How to extract hidden table from fbref.com website by id? [duplicate]

Question

I was trying to gather some data for a data analysis project concerning italian football. Unfortunately I can't extract one specific table from the web page, even though the id is correct.

The code is the following one:

import requests
import pandas as pd
from bs4 import BeautifulSoup

url = "https://fbref.com/it/comp/11/gca/Statistiche-di-Serie-A#all_stats_gca"

response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')

table = soup.find('table', {'id': 'stats_gca'})

df = pd.read_html(str(table))[0]

print(df)

With this id ('stats_squads_gca_for') I have no problem, but with ('stats_gca') in my example I get no table as output and also the compiler says that there is no such table.

What am I doing wrong?

I've also tried by not using pandas but the result is the same. I expect to have missed something.

page may use JavaScript to generate table - but BeautifulSoup can't run JavaScript. And server may also send different HTML for different browser (and different HTML when browser can't run JavaScript) so first check what you really have in response.content — furas
– furas, Commented Nov 25, 2023 at 18:08
@Walrus - I slightly adapted your example in the question by inserting the id with which it does not work, I think that makes the question easier to understand, after all we want to see the code that needs to be debugged. May take a minute or two taking the tour and read How to Ask. Thanks — HedgeHog
– HedgeHog, Commented Nov 25, 2023 at 18:26

HedgeHog · Accepted Answer · 2023-11-25 18:33:41Z

4

The table is hidden in comments , so simplest solution would be to remove the comments:

import requests
import pandas as pd

url= 'https://fbref.com/it/comp/11/gca/Statistiche-di-Serie-A#all_stats_gca'
pd.read_html(
    requests.get(url).text.replace('<!--','').replace('-->','')
    ,attrs={'id':'stats_gca'}
)[0]

In addition also check the attrs parameter of pandas.read_html() to select the table by its id.

edited Nov 25, 2023 at 18:33

answered Nov 25, 2023 at 17:51

HedgeHog

25.4k5 gold badges18 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to extract hidden table from fbref.com website by id? [duplicate]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related