python requests.get() returns an empty string

Question

When I run the below code it returns an empty string

url = 'http://www.allflicks.net/wp-content/themes/responsive/processing/processing_us.php?draw=5&columns[0][data]=box_art&columns[0][name]=&columns[0][searchable]=true&columns[0][orderable]=false&columns[0][search][value]=&columns[0][search][regex]=false&columns[1][data]=title&columns[1][name]=&columns[1][searchable]=true&columns[1][orderable]=true&columns[1][search][value]=&columns[1][search][regex]=false&columns[2][data]=year&columns[2][name]=&columns[2][searchable]=true&columns[2][orderable]=true&columns[2][search][value]=&columns[2][search][regex]=false&columns[3][data]=genre&columns[3][name]=&columns[3][searchable]=true&columns[3][orderable]=true&columns[3][search][value]=&columns[3][search][regex]=false&columns[4][data]=rating&columns[4][name]=&columns[4][searchable]=true&columns[4][orderable]=true&columns[4][search][value]=&columns[4][search][regex]=false&columns[5][data]=available&columns[5][name]=&columns[5][searchable]=true&columns[5][orderable]=true&columns[5][search][value]=&columns[5][search][regex]=false&columns[6][data]=director&columns[6][name]=&columns[6][searchable]=true&columns[6][orderable]=true&columns[6][search][value]=&columns[6][search][regex]=false&columns[7][data]=cast&columns[7][name]=&columns[7][searchable]=true&columns[7][orderable]=true&columns[7][search][value]=&columns[7][search][regex]=false&order[0][column]=5&order[0][dir]=desc&start=0&length=25&search[value]=sherlock&search[regex]=false&movies=true&shows=true&documentaries=true&rating=netflix&_=1451768717982'
print requests.get(url).text

but if I put the url in my browser it'll show my the json information. I did notice while debugging the browser must have the plugin tamper data installed to view the json. If a browser doesn't have the plug in a blank web page will appear. So, my theory is it has to do something with out the http request is being handed but I'm not where to go from here.

Any help would be great.

It seems I have no output in browser, are you sure it's the url you want? — pythad
– pythad, Commented Jan 3, 2016 at 5:48
@pythad if you would open the home allflicks page with browser developer tools and filter the XHR requests in the network tab, you will see the request the OP is asking about, thanks. — alecxe
– alecxe, Commented Jan 3, 2016 at 5:56

alecxe · Accepted Answer · 2016-01-03 05:32:52Z

You need to open up a session, visit the main page to get the cookies set and then make the XHR request to "processing_us.php":

url = "http://www.allflicks.net/wp-content/themes/responsive/processing/processing_us.php?draw=5&columns[0][data]=box_art&columns[0][name]=&columns[0][searchable]=true&columns[0][orderable]=false&columns[0][search][value]=&columns[0][search][regex]=false&columns[1][data]=title&columns[1][name]=&columns[1][searchable]=true&columns[1][orderable]=true&columns[1][search][value]=&columns[1][search][regex]=false&columns[2][data]=year&columns[2][name]=&columns[2][searchable]=true&columns[2][orderable]=true&columns[2][search][value]=&columns[2][search][regex]=false&columns[3][data]=genre&columns[3][name]=&columns[3][searchable]=true&columns[3][orderable]=true&columns[3][search][value]=&columns[3][search][regex]=false&columns[4][data]=rating&columns[4][name]=&columns[4][searchable]=true&columns[4][orderable]=true&columns[4][search][value]=&columns[4][search][regex]=false&columns[5][data]=available&columns[5][name]=&columns[5][searchable]=true&columns[5][orderable]=true&columns[5][search][value]=&columns[5][search][regex]=false&columns[6][data]=director&columns[6][name]=&columns[6][searchable]=true&columns[6][orderable]=true&columns[6][search][value]=&columns[6][search][regex]=false&columns[7][data]=cast&columns[7][name]=&columns[7][searchable]=true&columns[7][orderable]=true&columns[7][search][value]=&columns[7][search][regex]=false&order[0][column]=5&order[0][dir]=desc&start=0&length=25&search[value]=sherlock&search[regex]=false&movies=true&shows=true&documentaries=true&rating=netflix&_=1451768717982"
with requests.Session() as session:
    session.headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"}
    session.get("http://www.allflicks.net/")

    response = session.get(url, headers={"Accept" : "application/json, text/javascript, */*; q=0.01", 
                                         "X-Requested-With": "XMLHttpRequest", 
                                         "Referer": "http://www.allflicks.net/", 
                                         "Host": "www.allflicks.net"})
    print(response.json())

your're awesome that worked for me, thank you. I'll keep sessions in mind for the future.

Collectives™ on Stack Overflow

python requests.get() returns an empty string

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related