The long versionI decided to add support for Bing in my search engine scraper which is written in Python (view the source
here)
Once implemented and tested, I found that you get a maximum of 10 results per query which is way too less for my taste.
So I tried searching what kind of parameters Bing offers, unlike Google there wasn't one for the amount of results.
Changing the amount of results in the
settings of Bing changed the request accordingly, turns out Bing uses a Cookie to decide how many results will get displayed.
The full request will look like this:
GET /search?q=query&go=&qs=ds&form=QBLH&filt=all HTTP/1.1
Host: www.bing.com
User-Agent: fuck you :)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: MUID=2AD1C087FaF66E6810F6CS11EED76E63; MUIDB=2AD1CF7EFF66E6810F6C5E1EED76E63; OrigMUID=2AD1C08sEFF66E6De0F6R501EED76E63%2ca7d25648b73544c8b8982d16c6452d1e; SRCHD=MS=30896x3&D=30894h2&AF=QBLH; SRCHUID=V=2&GUID=702445F423FDA4B0B82096DEA33499C79; SRCHUSR=AUTOREDIR=0&GEOVAR=&DOB=20131115; _UR=OMW=0&OMD=13028909743; SRCHHPGUSR=NEWWND=0&NRSLT=50&SRCHLANG=&AS=0&ADLT=OFF; _FS=NU=1; _SS=SID=2DB1B23g905CF4D4D82A02y1C3AAC297C7662&bIm=570781&nhIm=08-; _HOP=; SCRHDN=ASD=0&DURL=#; WLS=TS=63520119834
DNT: 1
Connection: keep-alive
After a long read on the Cookie I came by the parameter called "
NRSLT=50", which rang my bells. (
Number of
Re
Su
LTs)
I've tested out what arguments in the Cookie you have to pass to still get a valid query, this was the result:
Cookie: SRCHHPGUSR=&NEWWND=0&NRSLT=50&SRCHLANG=
The short versionAdd Cookie to request ^