Writing CSV from scraped HTML data

Question

I was able to extract data using below code from the Russian Statistics website and to create a CSV file. However, I have two issues, firstly, I don't know why there is always a blank row inserted between two-non-blank rows. Secondly, I am unaware how to write a nice table where data from the same month is spread across different columns. Right now, everything is just in one cell. Thanks.

from bs4 import BeautifulSoup
import lxml
import urllib2
import csv

f=csv.writer(open("Russia.csv","w"))
mainurl='http://www.gks.ru/bgd/free/B00_25/IssWWW.exe/Stg/d000/I000750R.HTM'
urlroot='http://www.gks.ru/bgd/free/B00_25/IssWWW.exe/Stg/d000/'

data = urllib2.urlopen(mainurl).read()
page = BeautifulSoup(data,'html.parser')

for link in page.findAll('a'):
    page = urllib2.urlopen(urlroot+link.get('href'))
    soup = BeautifulSoup(page, 'lxml')
    years=soup.findAll('title',text=True)

    table = soup.find('center').find('table')
    for row in table.find_all('tr')[3:]:
        cells = [cell.get_text(strip=True) for cell in row.find_all('td')]
        f.writerow([cells])

@alecxe; my desired output looks like this [example for row 1 in the csv]: January; 112,8; 28,2; 1,2 [then the next item on the next row, and so on for all items in my list], so a csv which lists each item's data in my list on each row. Right now, my problem is that the csv writer literally writes unicode " u' " before each item. — mr_bungles
– mr_bungles, Commented Jan 17, 2016 at 11:04

alecxe · Accepted Answer · 2016-01-17 14:38:42Z

1

You are unintentionally making a list of lists here:

cells = [cell.get_text(strip=True) for cell in row.find_all('td')]
f.writerow([cells])

Instead, write the cells list directly:

f.writerow(cells)

answered Jan 17, 2016 at 14:38

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Writing CSV from scraped HTML data

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related