1

I am trying to scrape the links that are stored in the href values from this website: https://www.fotmob.com/teams/8676/fixtures/wycombe-wanderers?page=1

Here is a photo of the links

I tried the following code:

url = "https://www.fotmob.com/teams/8676/fixtures/wycombe-wanderers?page=1"
r = requests.get(url)
html_doc = r.content
soup = BeautifulSoup(html_doc, 'html.parser')

[tag['href'] for tag in soup.find_all('a',{'class':'css-11smhdq-FtContainer e1ym2d3s2'})]

But this just returns an empty list. Does anyone know how to fix this?

2
  • Can you see those links in your soup object? Commented Oct 28, 2022 at 14:57
  • @JuanC I can yeah. Commented Oct 28, 2022 at 16:07

1 Answer 1

2

The required data is loaded dynamically by JavaScript that's why you are getting empty output because bs4 can't render JS but they are using API and you can extract data from API url instead.

import requests

api_url= 'https://www.fotmob.com/api/teams?id=8676&timezone=Asia%2FDhaka&ccode3=BGD'
headers={
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
    }
data=[]

res=requests.get(api_url,headers=headers)
#print(res)
for item in res.json()['fixtures']['allFixtures']['fixtures']:
    url = item['pageUrl']
    data.append({'url':'https://www.fotmob.com' + url})
print(data)

Output:

[{'url': 'https://www.fotmob.com/match/3915857/matchfacts/wycombe-wanderers-vs-burton-albion'}, {'url': 'https://www.fotmob.com/match/3915859/matchfacts/bolton-wanderers-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3917827/matchfacts/northampton-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915875/matchfacts/wycombe-wanderers-vs-shrewsbury-town'}, {'url': 'https://www.fotmob.com/match/3915880/matchfacts/exeter-city-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915866/matchfacts/barnsley-vs-wycombe-wanderers'}, {'url': 
'https://www.fotmob.com/match/3992753/matchfacts/wycombe-wanderers-vs-bristol-city'}, {'url': 'https://www.fotmob.com/match/3915922/matchfacts/wycombe-wanderers-vs-charlton-athletic'}, {'url': 'https://www.fotmob.com/match/3940648/matchfacts/wycombe-wanderers-vs-tottenham-hotspur-academy'}, {'url': 'https://www.fotmob.com/match/3915928/matchfacts/fleetwood-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915936/matchfacts/wycombe-wanderers-vs-accrington-stanley'}, {'url': 'https://www.fotmob.com/match/3915957/matchfacts/derby-county-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3940645/matchfacts/stevenage-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915945/matchfacts/sheffield-wednesday-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915987/matchfacts/wycombe-wanderers-vs-plymouth-argyle'}, {'url': 'https://www.fotmob.com/match/3915978/matchfacts/oxford-united-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3915997/matchfacts/wycombe-wanderers-vs-peterborough-united'}, {'url': 'https://www.fotmob.com/match/3940637/matchfacts/wycombe-wanderers-vs-peterborough-united'}, {'url': 'https://www.fotmob.com/match/3916030/matchfacts/milton-keynes-dons-vs-wycombe-wanderers'}, {'url': 
'https://www.fotmob.com/match/3916003/matchfacts/wycombe-wanderers-vs-cambridge-united'}, {'url': 'https://www.fotmob.com/match/3916015/matchfacts/wycombe-wanderers-vs-morecambe'}, {'url': 'https://www.fotmob.com/match/3915901/matchfacts/wycombe-wanderers-vs-port-vale'}, {'url': 'https://www.fotmob.com/match/4050422/matchfacts/wycombe-wanderers-vs-walsall'}, {'url': 'https://www.fotmob.com/match/3916041/matchfacts/forest-green-rovers-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916064/matchfacts/cheltenham-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916055/matchfacts/wycombe-wanderers-vs-portsmouth'}, {'url': 'https://www.fotmob.com/match/3916098/matchfacts/lincoln-city-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916093/matchfacts/wycombe-wanderers-vs-ipswich-town'}, {'url': 'https://www.fotmob.com/match/3916109/matchfacts/wycombe-wanderers-vs-bristol-rovers'}, {'url': 'https://www.fotmob.com/match/3916141/matchfacts/plymouth-argyle-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916154/matchfacts/peterborough-united-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916114/matchfacts/wycombe-wanderers-vs-oxford-united'}, {'url': 'https://www.fotmob.com/match/3916126/matchfacts/wycombe-wanderers-vs-sheffield-wednesday'}, {'url': 'https://www.fotmob.com/match/3916124/matchfacts/bristol-rovers-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916164/matchfacts/wycombe-wanderers-vs-fleetwood-town'}, {'url': 'https://www.fotmob.com/match/3916202/matchfacts/port-vale-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916186/matchfacts/wycombe-wanderers-vs-derby-county'}, {'url': 'https://www.fotmob.com/match/3916187/matchfacts/accrington-stanley-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916247/matchfacts/wycombe-wanderers-vs-bolton-wanderers'}, {'url': 'https://www.fotmob.com/match/3916223/matchfacts/shrewsbury-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916231/matchfacts/wycombe-wanderers-vs-exeter-city'}, {'url': 'https://www.fotmob.com/match/3916228/matchfacts/burton-albion-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916285/matchfacts/wycombe-wanderers-vs-barnsley'}, {'url': 'https://www.fotmob.com/match/3916279/matchfacts/charlton-athletic-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916314/matchfacts/wycombe-wanderers-vs-milton-keynes-dons'}, {'url': 'https://www.fotmob.com/match/3916321/matchfacts/ipswich-town-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916303/matchfacts/wycombe-wanderers-vs-forest-green-rovers'}, {'url': 'https://www.fotmob.com/match/3916358/matchfacts/morecambe-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916335/matchfacts/cambridge-united-vs-wycombe-wanderers'}, {'url': 'https://www.fotmob.com/match/3916349/matchfacts/wycombe-wanderers-vs-lincoln-city'}, {'url': 'https://www.fotmob.com/match/3916377/matchfacts/wycombe-wanderers-vs-cheltenham-town'}, {'url': 'https://www.fotmob.com/match/3916378/matchfacts/portsmouth-vs-wycombe-wanderers'}]

screenshot

Sign up to request clarification or add additional context in comments.

6 Comments

Thats brilliant, thanks so much. How did you know that it had an API and where did you find the url for it?
Thanks. You will find a couple of discussions about API from here: stackoverflow.com/questions/1820927/…
Thanks for sending that. I can't see on that link though how you found what headers to use?
Ok wait. I will show you screenshot
Open the website then click on F12 to go to network tab then refresh the url from far top left circular icon and see the above screenshot:
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.