Python how to get (decoded) html source code

Question

I am trying in python (2.7.13) to get the source code of a webpage (having the current foreign exchange rates). Normally that is no problem with requests.get(url, headers) etc. In this case I can download/get the webpage, but some parts seems to be (base64 ?) encoded.

However when I visit the page in a browser and I view the source code: the right (decoded) code will be shown in the browser. Question is: how can I get the (decoded) web page source. The url is: https://www.isbank.com.tr/en/foreign-exchange-rates

Part of the code I use is:

url = "https://www.isbank.com.tr/en/foreign-exchange-rates"
resp = requests.get(url)
out = resp.text

If possible, can you show some proof of concept of what you tried to do in a code block? — PythonKiddieScripterX
– PythonKiddieScripterX, Commented Sep 17, 2022 at 8:51
@PyhtonKiddieScripterX thanks. I visited the hyperlink you gave but it is not an utf-8 issue, so 'r.encoding = r.apparent_encoding' didn't help me. Som code added to 1st posting — ni_hao
– ni_hao, Commented Sep 17, 2022 at 9:09

bereal · Accepted Answer · 2022-09-17 09:34:10Z

1

The response contains the text in Turkish, saying that the request is rejected due to the "unusual traffic detected from your device". It seems that the site checks the User-Agent header to prevent simple scripts from crawling it. You can bypass it by adding some plausible header:

url = 'https://www.isbank.com.tr/en/foreign-exchange-rates'
ua = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'
resp = requests.get(url, headers={'User-Agent': ua})
out = resp.text

answered Sep 17, 2022 at 9:34

bereal

34.7k8 gold badges65 silver badges111 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python how to get (decoded) html source code

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related