DDOS cloudflare protection

[Kiuhnm]

I don't want to launch a DDOS attack. I just want to access a particular site through python (urllib). The problem is that the site is protected and when I try to open the url with Python I get a "503: Service Temporarily Unavailable".
If go to the same url with Firefox, I see this:

Checking your browser before accessing XXXXXX. This process is automatic. Your browser will redirect to your requested content shortly.
Please allow up to 5 seconds…
   DDoS protection by CloudFlare

Any solution?

[2d8]

In CloudFlare API is described, that browser check searches for uncommon HTTP headers and valid User-Agent.
So, you should set User-Agent in your script. E.g.:

Code: [Select]

urllib.URLopener.version = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461)'

[Kiuhnm]

In CloudFlare API is described, that browser check searches for uncommon HTTP headers and valid User-Agent.
So, you should set User-Agent in your script. E.g.:

Code: [Select]

urllib.URLopener.version = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461)'

It doesn't work. It seems that urrlib doesn't handle the redirection correctly. Maybe I should see what happens (with wireshark, for example) and use httplib directly...

[TheWormKill]

As far as I know, Requests supports Redirections and User-Agent setting. Downside: it's third-party
http://docs.python-requests.org/en/latest/
But if that's not an issue, I'd strongly suggest it.

[2d8]

It doesn't work. It seems that urrlib doesn't handle the redirection correctly. Maybe I should see what happens (with wireshark, for example) and use httplib directly...

To check if redirect persist there is geturl() option.
You could also look into mechanize as alternative to urllib, since it emulates full-fledged firefox browsing, supports redirects etc.


EDIT: my bad, geturl() won't follow redirects, nevermind

[Stackprotector]

You can also use curl or use the data from this action in chrome: F12 (dev menu) -> Network -> F5 -> right click on the main request -> Copy as cURL

[Kiuhnm]

OK. requests does a little better. As I suspected, one must overcome a Javascript challenge. Here's the script for your delight:

Code: [Select]

  (function(){
    var a = function() {try{return !!window.addEventListener} catch(e) {return !1} },
    b = function(b, c) {a() ? document.addEventListener("DOMContentLoaded", b, c) : document.attachEvent("onreadystatechange", b)};
    b(function(){
      var a = document.getElementById('cf-content');a.style.display = 'block';
      setTimeout(function(){
        var t,r,a,f, zxyHslq={"vRRShbXaOtY":!+[]+!![]+!![]+!![]+!![]+!![]};
        t = document.createElement('div');
        t.innerHTML="<a href='/'>x</a>";
        t = t.firstChild.href;r = t.match(/https?:\/\//)[0];
        t = t.substr(r.length); t = t.substr(0,t.length-1);
        a = document.getElementById('jschl-answer');
        f = document.getElementById('challenge-form');
        ;zxyHslq.vRRShbXaOtY*=+((+!![]+[])+(+!![]));zxyHslq.vRRShbXaOtY+=!+[]+!![]+!![]+!![]+!![]+!![];zxyHslq.vRRShbXaOtY+=!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![];zxyHslq.vRRShbXaOtY*=+((+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]));zxyHslq.vRRShbXaOtY+=+((!+[]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]));zxyHslq.vRRShbXaOtY+=!+[]+!![];zxyHslq.vRRShbXaOtY-=+((+!![]+[])+(!+[]+!![]));zxyHslq.vRRShbXaOtY+=+((!+[]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]));zxyHslq.vRRShbXaOtY*=+((!+[]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]));a.value = parseInt(zxyHslq.vRRShbXaOtY, 10) + t.length;
        f.submit();
      }, 5850);
    }, false);
  })();

That above is the challenge, of course.
I think I'll use an external javascript VM (maybe the one used for node.js?).

BTW, mechanize doesn't support Javascript! I think that, in general, no Python library does.
I'll need to handle Javascript myself. Either I build a simple specific interpreter for that kind of code, or I use an external Javascript VM. Even with a VM, I'll need to change the code a bit because a general VM doesn't understand browser related stuff.

Last edit:
I managed to do the entire procedure by hand in a Python REPL.
1) I get the javascript code and transform it so that it can run outside the browser.
2) I run the code in jsdb (a little interpreter) and read the result.
3) I send the answer to the site and receive two cookies which I must use for each future requests.
4) The rest is easy... It took me a while to realize that requests' automatic redirection messes things up.