Pages: [1]

Author Topic: How To Crawl A webSIte Using Web Robot  (Read 802 times)

0 Members and 1 Guest are viewing this topic.

Moiz

  • Guest
How To Crawl A webSIte Using Web Robot
« on: June 15, 2013, 05:02:12 pm »
How Can I Crawl Any Site With Help Of Robot or A Spider Or Crawler For

Example I want to crawl Evil Zone To Get All Links Listed in it So Suggest me A Good Spider And If there A tutorial Please Post It
Report to moderator   Logged

Offline vezzy

  • Royal Highness
  • ****
  • Posts: 771
  • Cookies: 172
    • View Profile
Re: How To Crawl A webSIte Using Web Robot
« Reply #1 on: June 15, 2013, 05:33:50 pm »
Use wget or write your own dedicated web scraper with some sort of HTML parsing library.
Report to moderator   Logged
Quote from: Dippy hippy
Just brushing though. I will be semi active mainly came to find a HQ botnet, like THOR or just any p2p botnet

Offline Kulverstukas

  • Administrator
  • Zeus
  • *
  • Posts: 6627
  • Cookies: 542
  • Fascist dictator
    • View Profile
    • My blog
Re: How To Crawl A webSIte Using Web Robot
« Reply #2 on: June 15, 2013, 07:25:14 pm »
Capitalizing Every Word In A Sentence Is Very Cool.

But what you really want is a web cloner, I can suggest HTTrack. Used it few times, worked fantastic.
Report to moderator   Logged

Offline kenjoe41

  • Symphorophiliac Programmer
  • Administrator
  • Baron
  • *
  • Posts: 990
  • Cookies: 224
    • View Profile
Re: How To Crawl A webSIte Using Web Robot
« Reply #3 on: June 15, 2013, 11:53:52 pm »
Bash or python would do the job just fine. But that's just me.
Report to moderator   Logged
If you can't explain it to a 6 year old, you don't understand it yourself.
http://upload.alpha.evilzone.org/index.php?page=img&img=GwkGGneGR7Pl222zVGmNTjerkhkYNGtBuiYXkpyNv4ScOAWQu0-Y8[<NgGw/hsq]>EvbQrOrousk[/img]
Pages: [1]