Python urllib2 - cannot open url

Question

I'm creating a script that reads information from search query on zhaopin.com using urllib2

When I try to open the url by copying it to my web browser (Chrome), I have no problem opening the site: http://sou.zhaopin.com/jobs/searchresult.ashx?p=1&isadv=0&bj=160000&in=160200

When I open the url using urllib2, I get the error moessage HTTPError: HTTP Error 502: Bad Gateway. From searching google, I could not figure out what I'm doing wrong.

import urllib
data = {}
data['in']='160200'
data['bj']='160000'
data['isadv']='0'
data['p']=1

url = 'http://sou.zhaopin.com/jobs/searchresult.ashx?'
url_values = urllib.urlencode(data)
full_url= url + url_values
print full_url
response = urllib2.urlopen(url)
html = response.read()
response.close()

Perhaps it is a problem with the URL: after opening the url in Chrome, the 'http://' disappears. I'd appreciate it if you could help me figuring this out.

Are you behind the Great Firewall of China? Try capturing the HTTP session using Wireshark and look at the raw data. The difference in the requests should be visible there. — John Zwinck
– John Zwinck, Commented Aug 6, 2017 at 7:34
That disappeared of http in address bar is nothing just a chrome feature nothing else. — Rajan Chauhan
– Rajan Chauhan, Commented Aug 6, 2017 at 7:42

gre_gor · Accepted Answer · 2019-10-31 16:25:50Z

1

Try urllib instead of urllib2:

response = urllib.urlopen(url)
html = response.read()
response.close()

edited Oct 31, 2019 at 16:25

gre_gor

6,65712 gold badges104 silver badges106 bronze badges

answered Aug 6, 2017 at 7:37

Surinder Batti

1091 silver badge4 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Pravitha V · Accepted Answer · 2017-08-06 12:42:42Z

HTTP Error 502: Bad Gateway

The above error occurs when there is a misconfiguration in the server you are trying. The misconfiguration can be due to the server is rebooting or not available at that moment.

This error can also be a result of poor IP communication between back-end computers, possibly including the server at the site you are trying to visit. It may be that the server is overloaded.

You can use urllib itself in your code to open the URL.

import urllib
data = {}
data['in']='160200'
data['bj']='160000'
data['isadv']='0'
data['p']=1

url = 'http://sou.zhaopin.com/jobs/searchresult.ashx?'
url_values = urllib.urlencode(data)
full_url= url + url_values
print full_url
response = urllib.urlopen(url)
html = response.read()
response.close()

Collectives™ on Stack Overflow

Python urllib2 - cannot open url

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related