Created a crawler today, it extracts links from the passed URL and stores the gathered list in a text file in the same folder as the script, called harvested.txt (it gets overwritten so be careful).
Usage is straight forward:
python Crawler.py -u <target url> [-d<0-20> optional level of depth, default 0] [-s optional flag that the crawler must stay in the same domain]
As always, free code!
Use it or loose it.
Thanks for reading my source, feedback, critique, motivation(any kind) is welcome.
clone link:
https://gist.github.com/7514550.git[gist]Daxda/7514550[/gist]
