EvilZone

Programming and Scripting => Scripting Languages => : daxda November 17, 2013, 04:24:43 PM

: [Source] Crawler.py
: daxda November 17, 2013, 04:24:43 PM

Created a crawler today, it extracts links from the passed URL and stores the gathered list in a text file in the same folder as the script, called harvested.txt (it gets overwritten so be careful).

Usage is straight forward:
python Crawler.py -u <target url> [-d<0-20> optional level of depth, default 0] [-s optional flag that the crawler must stay in the same domain]

As always, free code!
Use it or loose it.
Thanks for reading my source, feedback, critique, motivation(any kind) is welcome.

clone link: https://gist.github.com/7514550.git
[gist]Daxda/7514550[/gist]

: Re: [Source] Crawler.py
: proxx November 17, 2013, 04:28:04 PM

Am I stupid or where is the link ?