some handy grep regex commands

[neusbeer]

Grep IP adresses 

Code: [Select]

grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' <file> > output_file

Grab ip's and ports from a html proxy list

like: 127.0.0.1:80

Code: [Select]

curl http://www.samair.ru/proxy/proxy-01.htm |  grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}:[0-9]\{1,5\}'

this can be handy to get some proxy ip's, I have a few bash scripts
for collection all the proxy ip's from sites. (example samair.ru)
there are a lot of sites who offered proxy ip's lists webbased. 

Grab MD5 hashes from text/sql database, etc
HTML/TXT

Code: [Select]

grep -a -o -e "[0-9a-f]\{32\}" shop_users.html > userhash.txtSQL

Code: [Select]

grep -o -e "[0-9a-f]\{32\}" shop_users.sql > userhash.txt
lowercasing words

Code: [Select]

cat file | sed y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/ > <output file>
removing all digits in text file

Code: [Select]

sed 's/[0-9]*//g' input.txt > output.txt
Delete BOTH leading and trailing whitespace from each line

Code: [Select]

sed 's/^[ \t]*//;s/[ \t]*$//'
Delete all leading blank lines at top of file

Code: [Select]

sed '/./,$!d'
Removes ' from strings. (fixing up SQLi list)

Code: [Select]

cat sqli.txt | sed "s/[']*//g" > sqli_list.txt
display's all the url's in a file

Code: [Select]

curl http://<weblink>.html | grep -o -E "[a-zA-Z]{3,}://[a-zA-Z0-9\.]+/*[a-zA-Z0-9/\\%_.]*\?*[a-zA-Z0-9/\\%_.=&amp;]*"
ahh well.. there are thousands more (if not.. millions  )
but these I use often ..

for regex help:
info:
http://bashshell.net/category/regular-expressions/
http://www.zytrax.com/tech/web/regex.htm

software:
http://www.moskalyuk.com/blog/regular-expression-helpers

online tools:
http://www.txt2re.com/
http://gskinner.com/RegExr/

Libraries:
http://regexlib.com/
http://www.pement.org/sed/sed1line.txt

[xzid]

Code: [Select]

grep -oP '(\d{1,3}\.){3}\d+'

Code: [Select]

awk '{print tolower($0)}'

Code: [Select]

grep -oP "\w{3,}://[\w\.-]{1,255}(/[^'\"<>\s]*){0,1}"
I helped!

edit: perl-style, neater. add URL regex.