python requests cannot get html

Question

I tried to get html code from a site name dcinside in Korea, i am using requests but cannot get html code

and this is my code

import requests
url = "http://gall.dcinside.com/board/lists/?id=bitcoins&page=1"
req = requests.get(url)
print (req)
print (req.content)

but the result was

Why I cannot get html codes even using requests??

hspandher · Accepted Answer · 2017-10-10 13:38:10Z

5

Most likely they are detecting that you are trying to crawl data dynamically, and not giving any content as a response. Try pretending to be a browser and passing some User-Agent headers.

headers = {
    'User-Agent': 'My User Agent 1.0', 
    'From': '[email protected]'
}

response = requests.get(url, headers=headers)

# use authentic mozilla or chrome user-agent strings if this doesn't work

answered Oct 10, 2017 at 13:38

hspandher

16.8k2 gold badges35 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Omer Hen · Accepted Answer · 2017-10-10 13:35:36Z

1

Take a look at this:

Python Web Crawlers and "getting" html source code

Like the guy said in the aforementioned post, you should use urllib2 which will allow you to easily obtain web resources.

answered Oct 10, 2017 at 13:35

Omer Hen

1721 silver badge12 bronze badges

1 Comment

ritiek Over a year ago

using urllib2 does not seem to be working with the website OP wants to fetch.

Collectives™ on Stack Overflow

python requests cannot get html

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related