Python BeautifulSoup returning incorrect html code

Question

I'm trying to get movie names from imdb list my problem is instead of returning the English name that appear in google chrome inspect element it return the Japanese name.

Note: I tried all BeautifulSoup parsers

Code:

import requests
from bs4 import BeautifulSoup
page=requests.get('https://www.imdb.com/list/ls040141830/')
soup = BeautifulSoup(page.text,'lxml')
name = soup.find('h3','lister-item-header')
print(name)

result:

<h3 class="lister-item-header">
<span class="lister-item-index unbold text-primary">1.</span>
<a href="/title/tt0245429/?ref_=ttls_li_tt">Sen to Chihiro no kamikakushi</a>
<span class="lister-item-year text-muted unbold">(2001)</span>
</h3>

expected result as show in page source:

as you can see the name in english but im getting it in japanese any help would be appreciated thanks.

Maybe, after initial page loading, some java script executes and translates local name to English, and in browser you already see it in English? — kosist
– kosist, Commented Oct 11, 2018 at 10:37
Are you trying to get imdb pages with an ip address from japan? — Mr Alihoseiny
– Mr Alihoseiny, Commented Oct 11, 2018 at 10:37

gherkin · Accepted Answer · 2018-10-11 10:45:06Z

3

Seems if you visit imdb from a non-browser client, imdb will translate the movie names into the original language. You should be able to fix it by add the Accept-Language header to requests

import requests
from bs4 import BeautifulSoup
headers = {"Accept-Language": "en-US, en;q=0.5"}
page=requests.get('https://www.imdb.com/list/ls040141830/', headers = headers)
soup = BeautifulSoup(page.text,'lxml')
name = soup.find('h3','lister-item-header')
print(name)

answered Oct 11, 2018 at 10:45

gherkin

4967 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python BeautifulSoup returning incorrect html code

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related