[Python] Browser automation

[cr4zi8]

So a friend wanted me to give him some votes in a silly poll. The problem was there were checks to make sure that the requests were coming from a browser. I did not feel like working with posts requests to try to get around this so instead I came up with this:

Code: (Python) [Select]

import os
import time
prox=open("[Path to proxy list]")
for line in prox:
    f=open("C:\\Users\\[Your Username]\\Documents\\iMacros\\Macros\\#Current.iim","w")
    f.write("VERSION BUILD=8920312 RECORDER=FX\nTAB T=1\nTAB CLOSEALLOTHERS\nCLEAR\nURL GOTO=[REDACTED]\nPROXY ADDRESS="+line+"\nFRAME F=1\nTAG POS=1 TYPE=LABEL FORM=ID:form1 ATTR=TXT:Quentin<SP>Anderson\nTAG POS=1 TYPE=INPUT:RADIO FORM=ID:form1 ATTR=ID:PollX1_rblOPT_0\nTAG POS=1 TYPE=INPUT:SUBMIT FORM=ID:form1 ATTR=ID:PollX1_btnVote")
    f.close()
    os.system('start "" "C:\\Program Files (x86)\\Mozilla Firefox\\firefox.exe" imacros://run/?m=#Current.iim')
    time.sleep(20)


First you scrape a proxy list so that you can have several different users. Then you record the action you want the users to do using imacros(in this case vote on the poll). You then edit this script slightly adding CLEAR before the action (this clears cookies just in case it is storing that you have already been there in a cookie). You also add a line with PROXY ADDRESS=. You then loop through the proxy list appending each proxy to the line PROXY ADDRESS=. You then run the macros file with imacros then make sure to give it time to process 20 is probably excessive.

I know this is pretty basic but thought it might help simplify automation for people who did not know about this. Obviously this could be taken further to register multiple users since Google Captcha does not detect this.

[techb]

It should also be noted that this script is Windows only.

[proxx]

I am now an object Yay

So a friend wanted me to give him some votes in a silly poll. The problem was there were checks to make sure that the requests were coming from a browser. I did not feel like working with posts requests to try to get around this so instead I came up with this:

Code: (Python) [Select]

import os
import time
prox=open("[Path to proxy list]")
for line in prox:
    f=open("C:\\Users\\[Your Username]\\Documents\\iMacros\\Macros\\#Current.iim","w")
    f.write("VERSION BUILD=8920312 RECORDER=FX\nTAB T=1\nTAB CLOSEALLOTHERS\nCLEAR\nURL GOTO=[REDACTED]\nPROXY ADDRESS="+line+"\nFRAME F=1\nTAG POS=1 TYPE=LABEL FORM=ID:form1 ATTR=TXT:Quentin<SP>Anderson\nTAG POS=1 TYPE=INPUT:RADIO FORM=ID:form1 ATTR=ID:PollX1_rblOPT_0\nTAG POS=1 TYPE=INPUT:SUBMIT FORM=ID:form1 ATTR=ID:PollX1_btnVote")
    f.close()
    os.system('start "" "C:\\Program Files (x86)\\Mozilla Firefox\\firefox.exe" imacros://run/?m=#Current.iim')
    time.sleep(20)


First you scrape a proxy list so that you can have several different users. Then you record the action you want the users to do using imacros(in this case vote on the poll). You then edit this script slightly adding CLEAR before the action (this clears cookies just in case it is storing that you have already been there in a cookie). You also add a line with PROXY ADDRESS=. You then loop through the proxy list appending each proxy to the line PROXY ADDRESS=. You then run the macros file with imacros then make sure to give it time to process 20 is probably excessive.

I know this is pretty basic but thought it might help simplify automation for people who did not know about this. Obviously this could be taken further to register multiple users since Google Captcha does not detect this.

I like the way you did a complete workaround since I would probably end up using selenium or something like it to do the hard work.
Doing it the hard way will allow you to do it much faster etc.


Does it actually spawn a browser and does it keep the window open ?
Because if it does I would suggest killing the process first before doing the next interation.

Code: (Python) [Select]

import os
import time
prox=open("[Path to proxy list]")
for line in prox:
    f=open("C:\\Users\\[Your Username]\\Documents\\iMacros\\Macros\\#Current.iim","w")
    f.write("VERSION BUILD=8920312 RECORDER=FX\nTAB T=1\nTAB CLOSEALLOTHERS\nCLEAR\nURL GOTO=[REDACTED]\nPROXY ADDRESS="+line+"\nFRAME F=1\nTAG POS=1 TYPE=LABEL FORM=ID:form1 ATTR=TXT:Quentin<SP>Anderson\nTAG POS=1 TYPE=INPUT:RADIO FORM=ID:form1 ATTR=ID:PollX1_rblOPT_0\nTAG POS=1 TYPE=INPUT:SUBMIT FORM=ID:form1 ATTR=ID:PollX1_btnVote")
    f.close()
    os.system('start "" "C:\\Program Files (x86)\\Mozilla Firefox\\firefox.exe" imacros://run/?m=#Current.iim')
    time.sleep(20)
   !!!KILLFIREFOXHERE!!!!



Adjust as such ^

[cr4zi8]

After a timeout and/or completion of a Macro i macros automatically closes the tab; it is not necessary to kill Firefox.

[proxx]

After a timeout and/or completion of a Macro i macros automatically closes the tab; it is not necessary to kill Firefox.

Yeah its dirty but if it works right.

[aaanathan]

You can use casperjs for this.

Sent from my A0001 using Tapatalk

[NO_BOOT_DEVICE]

interesting way to do it and i'll definitely note that you can do this in the future.
but for further reference, most use-cases are covered by these 3, in ascending amount of browser-like-ness:
requests for python (urllib but not bad, if you're using ruby use httparty)
mechanize (i personally when using mechanize use the ruby version out of preference)
WebDriver (haven't used it before since i've always been able to use mechanize, but it's literally an api for controlling chrome or firefox)

[kenjoe41]

interesting way to do it and i'll definitely note that you can do this in the future.
but for further reference, most use-cases are covered by these 3, in ascending amount of browser-like-ness:
requests for python (urllib but not bad, if you're using ruby use httparty)
mechanize (i personally when using mechanize use the ruby version out of preference)
WebDriver (haven't used it before since i've always been able to use mechanize, but it's literally an api for controlling chrome or firefox)

Last time i used the first two, the make the socket connection through python and make the python call by themselves. There are instance you are going to need a browser or look as one as closely as possible, and that is where webDriver comes in or rather selenium 2.0.

But from the looks of that script, it actually calls a live version of the browser (firefox) with a macro as the argument, so that is different from the way requests will work.

[NO_BOOT_DEVICE]

Last time i used the first two, the make the socket connection through python and make the python call by themselves. There are instance you are going to need a browser or look as one as closely as possible,

i don't see what it making the socket connection instead of using urllib or such has to do with it looking like a browser? you can very easily dress up mechanize as a browser, and while it won't get all the little things (and javascript), it can go far enough that most servers will be okay with it. only real use case for webdriver (the python module is selenium, yes, but webdriver is it's own W3 spec! ) i can see is for when the javascript can't be simply parsed of it's info, like recaptcha.

[_Enigma]

I'm not sure how in-depth the application wanted you to appear as a browser  (running a java applet for example), and you have an interesting take on automation come out of it. But as a fun fact my first thought for a solution to your root issue was to alter the User-Agent header of the POST request.

I've never done it from python but it seems simple enough:

http://stackoverflow.com/questions/802134/changing-user-agent-on-urllib2-urlopen

[riptorrent]

i prefer selenium for this kind of thing but looks neat.  you gave me an idea though for a proxy checker.