Grep IP adresses
grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' <file> > output_fileGrab ip's and ports from a html proxy list
like: 127.0.0.1:80curl http://www.samair.ru/proxy/proxy-01.htm | grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}:[0-9]\{1,5\}'
this can be handy to get some proxy ip's, I have a few bash scripts
for collection all the proxy ip's from sites. (example samair.ru (http://dl.dropbox.com/u/4378489/Forums/evilzone/samair.ru_proxylist.sh))
there are a lot of sites who offered proxy ip's lists webbased.
Grab MD5 hashes from text/sql database, etc
HTML/TXT
grep -a -o -e "[0-9a-f]\{32\}" shop_users.html > userhash.txtSQL
grep -o -e "[0-9a-f]\{32\}" shop_users.sql > userhash.txt
lowercasing words
cat file | sed y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/ > <output file>
removing all digits in text file
sed 's/[0-9]*//g' input.txt > output.txt
Delete BOTH leading and trailing whitespace from each line
sed 's/^[ \t]*//;s/[ \t]*$//'
Delete all leading blank lines at top of file
sed '/./,$!d'
Removes ' from strings. (fixing up SQLi list)
cat sqli.txt | sed "s/[']*//g" > sqli_list.txt
display's all the url's in a file
curl http://<weblink>.html | grep -o -E "[a-zA-Z]{3,}://[a-zA-Z0-9\.]+/*[a-zA-Z0-9/\\%_.]*\?*[a-zA-Z0-9/\\%_.=&]*"
ahh well.. there are thousands more (if not.. millions :o )
but these I use often ..
for regex help:
info:
http://bashshell.net/category/regular-expressions/ (http://bashshell.net/category/regular-expressions/)
http://www.zytrax.com/tech/web/regex.htm (http://www.zytrax.com/tech/web/regex.htm)
software:
http://www.moskalyuk.com/blog/regular-expression-helpers (http://www.moskalyuk.com/blog/regular-expression-helpers)
online tools:
http://www.txt2re.com/ (http://www.txt2re.com/)
http://gskinner.com/RegExr/ (http://gskinner.com/RegExr/)
Libraries:
http://regexlib.com/ (http://regexlib.com/)
http://www.pement.org/sed/sed1line.txt (http://www.pement.org/sed/sed1line.txt)