DataScraping in Python

Question

I am trying to scrape data from https://www.transfermarkt.co.uk/premier-league/startseite/wettbewerb/GB1

I have used this code to do so:

headers = {'User-Agent': 
           'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}

page = 'https://www.transfermarkt.co.uk/premier-league/startseite/wettbewerb/GB1'
pageTree = requests.get(page, headers=headers)
pageTree_text = pageTree.text

pageSoup = BeautifulSoup(pageTree_text, 'html.parser')

After, I want to find all the links that is connected to each team name, and use this code:

linkLocation = pageSoup.find_all("a", {"class": "vereinprofil_tooltip tooltipstered"})
linkLocation[0].text

output:

IndexError Traceback (most recent call last) in 1 linkLocation = pageSoup.find_all("a", {"class": "vereinprofil_tooltip tooltipstered"}) ----> 2 linkLocation[0].text

IndexError: list index out of range

Why doesn`t the list have any of the links within it?

Thnx in advcance!

vonschlager · Accepted Answer · 2020-03-01 18:10:25Z

0

"tooltipstered" class is added by javascript and is not available in the plain html document returned by the server. You can see that when you open the "source" of the page not using browser inspector.

As you can see "tooltipster" is some jquery plugin, you will need to use some other tool to scrape this page (eg.: selenium).

<script type="text/javascript" src="https://tmssl.akamaized.net//assets/e17e6900/js/jquery.tooltipster.js?lm=1574952016"></script>

answered Mar 1, 2020 at 18:10

vonschlager

3241 silver badge6 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Haroon Over a year ago

Hi So I cannot scrape the data from this page, using Python and BeautifulSoup?

vonschlager Over a year ago

You can using python, but not with BeautifulSoup alone. You can try with this SO answer: stackoverflow.com/questions/49939123/…

Collectives™ on Stack Overflow

DataScraping in Python

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related