Reading each line of HTML table into python list

Question

I am trying to web scrape a HTML table using python. I am using beautiful soup to do this web scraping. There are many tables in the HTML page and there are many rows in the table. I want each row to have a different name and if there are columns in the row, want them to be separate.

My code looks like this:

page = get("https://www.4dpredict.com/mysingaporetoto.p3.html")
html = BeautifulSoup(page.content, 'html.parser')
result = defaultdict(list)
tables = html.find_all('table')
for table in tables:
    for row in table.find_all('tr')[0:15]:
        try:
            #stuck here
        except ValueError:
            continue  # blank/empty row

Need some guidance on this.

SIM · Accepted Answer · 2018-09-15 07:07:52Z

1

If I correctly understood your requirement, the following script should do the trick:

import requests
from bs4 import BeautifulSoup

url = 'https://www.4dpredict.com/mysingaporetoto.p3.html'

res = requests.get(url).text
soup = BeautifulSoup(res, 'lxml')
num = 0
for tables in soup.select("table tr"):
    num+=1
    data = [f'{num}'] + [item.get_text(strip=True) for item in tables.select("td")]
    print(data)

Partial output:

['1', 'SINGAPORE TOTO2018-08-23 (Thu) 3399']
['2', 'WINNING NUMBERS']
['3', '02', '03', '23', '30', '39', '41']
['4', 'ADDITIONAL']
['5', '19']
['6', 'Prize:$2,499,788']
['7', 'WINNING SHARES']
['8', 'Group', 'Share Amt', 'Winners']
['9', 'Group 1', '$1,249,894', '2']
['10', 'Group 2', '$', '-']
['11', 'Group 3', '$1,614', '124']
['12', 'Group 4', '$344', '318']
['13', 'Group 5', '$50', '6,876']
['14', 'Group 6', '$25', '9,092']

edited Sep 15, 2018 at 7:07

answered Sep 15, 2018 at 6:15

SIM

22.5k6 gold badges45 silver badges116 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

SIM Over a year ago

If you are using python's latest version, I doubt there is any error. Check out the ouput the script produces.

lakshmen Over a year ago

how to fix that?

SIM Over a year ago

Try this ['%s' % num] replacing with [f'{num}'].

SIM Over a year ago

Then I failed to understand your requirement. Thanks.

Vijay Anand Pandian · Accepted Answer · 2018-09-15 07:46:31Z

Kindly check the below code, let me know if that doesn't works,

import requests
from bs4 import BeautifulSoup
import pprint
page = requests.get("https://www.4dpredict.com/mysingaporetoto.p3.html")
html = BeautifulSoup(page.content, 'html.parser')

tables = html.find_all('table')
table_data = dict()
for table_id, table in enumerate(tables):
    print('[!] Scraping Table -', table_id + 1)
    table_data['table_{}'.format(table_id+1)] = dict()
    table_info = table_data['table_{}'.format(table_id+1)]
    for row_id, row in enumerate(table.find_all('tr')):
        col = []
        for val in row.find_all('td'):
            val = val.text
            val = val.replace('\n', '').strip()
            if val:
                col.append(val)
        table_info['row_{}'.format(row_id+1)] = col
    pprint.pprint(table_info)
    print('+-+' * 20)

pprint.pprint(table_data)

Sample Output

[!] Scraping Table - 1
{'row_1': ['SINGAPORE TOTO2018-08-23 (Thu) 3399'],
 'row_10': ['Group 2', '$', '-'],
 'row_11': ['Group 3', '$1,614', '124'],
 'row_12': ['Group 4', '$344', '318'],
 'row_13': ['Group 5', '$50', '6,876'],
 'row_14': ['Group 6', '$25', '9,092'],
 'row_15': ['Group 7', '$10', '117,080'],
 'row_16': ['SHOW ANALYSISEVEN : ODD, 2 : 5SUM :138, AVERAGE :23 MIN :02, MAX '
            ':41, DIFF :39',
            'EVEN : ODD, 2 : 5',
            'SUM :138, AVERAGE :23',
            'MIN :02, MAX :41, DIFF :39'],
 'row_17': ['EVEN : ODD, 2 : 5'],
 'row_18': ['SUM :138, AVERAGE :23'],
 'row_19': ['MIN :02, MAX :41, DIFF :39'],
 'row_2': ['WINNING NUMBERS'],
 'row_3': ['02', '03', '23', '30', '39', '41'],
 'row_4': ['ADDITIONAL'],
 'row_5': ['19'],
 'row_6': ['Prize: $2,499,788'],
 'row_7': ['WINNING SHARES'],
 'row_8': ['Group', 'Share Amt', 'Winners'],
 'row_9': ['Group 1', '$1,249,894', '2']}
+-++-++-++-++-++-++-++-++-++-++-++-++-++-++-++-++-++-++-++-+

Igor Rivin · Accepted Answer · 2018-09-15 06:08:44Z

0

I would suggest abandoning BeautifulSoup (beautiful though it is) and using pandas (which uses BeautifulSoup or lxml on the back end). What you describe is bog standard with pandas, just read the docs.

answered Sep 15, 2018 at 6:08

Igor Rivin

4,8742 gold badges31 silver badges45 bronze badges

Comments

Suresh · Accepted Answer · 2018-09-15 06:47:53Z

0

I will suggest to use requests.get() instead of get() method

answered Sep 15, 2018 at 6:47

Suresh

4893 silver badges7 bronze badges

3 Comments

Giulio Caccin Over a year ago

Could you please use some of the OP code to enhance your answer? to make it clear what line of code will be the answer to OP question.

SIM Over a year ago

OP seems to have used requests library. However, he perhaps imported get from it like from requests import get. I still can't find any relevance between the answer to the question and your oneliner comment..

Suresh Over a year ago

Thanks SIM for suggestion.. I am new to python as well as stack over flow. try to learning and solving ..

Collectives™ on Stack Overflow

Reading each line of HTML table into python list

4 Answers 4

4 Comments

Comments

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

4 Comments

Comments

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related