Writing tabular data to a csv file from a webpage

Question

I've written a script in python to parse some data from a webpage and write it to a csv file via panda. So far what I've written can parse all the tables available in that page but in case of writing to a csv file it is writing the last table from that page to that csv file. Definitely, the data are being overwritten because of the loop. How can I fix this flaw so that my scraper will be able to write all the data from different tables instead of only the last table? Thanks in advance.

import csv
import requests 
from bs4 import BeautifulSoup
import pandas as pd


res = requests.get('http://www.espn.com/nba/schedule/_/date/20171001').text
soup = BeautifulSoup(res,"lxml")
for table in soup.find_all("table"):
    df = pd.read_html(str(table))[0]
    df.to_csv("table_item.csv")
    print(df)

Btw, I expect to write data to a csv file using panda only. Thanks again.

jezrael · Accepted Answer · 2017-09-20 15:02:37Z

2

You can use read_html what return list of DataFrames in webpage, so then need concat for one df:

dfs = pd.read_html('http://www.espn.com/nba/schedule/_/date/20171001')

df = pd.concat(dfs, ignore_index=True)
#if necessary rename columns
d = {'Unnamed: 1':'a', 'Unnamed: 7':'b'}
df = df.rename(columns=d)
print (df.head())
               matchup               a  time (ET)  nat tv  away tv  home tv  \
0          Atlanta ATL       Miami MIA        NaN     NaN      NaN      NaN   
1               LA LAC     Toronto TOR        NaN     NaN      NaN      NaN   
2  Guangzhou Guangzhou  Washington WSH        NaN     NaN      NaN      NaN   
3        Charlotte CHA      Boston BOS        NaN     NaN      NaN      NaN   
4          Orlando ORL     Memphis MEM        NaN     NaN      NaN      NaN   

                           tickets   b  
0  2,401 tickets available from $6 NaN  
1   284 tickets available from $29 NaN  
2  2,792 tickets available from $2 NaN  
3  2,908 tickets available from $6 NaN  
4  1,508 tickets available from $3 NaN

And last to_csv for write to file:

df.to_csv("table_item.csv", index=False)

EDIT:

For learning is possible append each DataFrame to list and then concat:

res = requests.get('http://www.espn.com/nba/schedule/_/date/20171001').text
soup = BeautifulSoup(res,"lxml")
dfs = []
for table in soup.find_all("table"):
    df = pd.read_html(str(table))[0]
    dfs.append(df)

df = pd.concat(dfs, ignore_index=True)
print(df)

df.to_csv("table_item.csv")

edited Sep 20, 2017 at 15:02

answered Sep 20, 2017 at 14:31

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

SIM Over a year ago

Thanks jezrael, for such an elaborative answer and the clarity. It serves the purpose. However, for the learning purpose: could you please suggest me how could i write those data to a csv file if i wanted to stick to the way I started in my post.

jezrael Over a year ago

Glad can help ;) Good luck!

Collectives™ on Stack Overflow

Writing tabular data to a csv file from a webpage

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related