Python Scraping Html Trouble

Question

I am currently trying to scrape my internet providers data usage. I tried looking for an api of sorts but they don't have one. I am resorting to scraping the html whch looks like this

</tr><tr class="top-border"><td>17&nbsp;&nbsp;Monday</td><td class='text-right'><span class='mb'>2,991.69&nbsp;MB</span><span class='gb'>2.92&nbsp;GB</span></td></td><td class='text-right'><span class='mb'>1,232.04&nbsp;MB</span><span class='gb'>1.20&nbsp;GB</span></td></td><td class='text-right'><span class='mb'>4,223.73&nbsp;MB</span><span class='gb'>4.12&nbsp;GB</span></td>         <td>
            <div class="progress"><div class="bar bar-success" style="width: 51%;"></div></div>         </td>

        </tr><tr><td>18&nbsp;&nbsp;Tuesday</td><td class='text-right'><span class='mb'>3,589.42&nbsp;MB</span><span class='gb'>3.51&nbsp;GB</span></td></td><td class='text-right'><span class='mb'>1,199.58&nbsp;MB</span><span class='gb'>1.17&nbsp;GB</span></td></td><td class='text-right'><span class='mb'>4,789.00&nbsp;MB</span><span class='gb'>4.68&nbsp;GB</span></td>           <td>
            <div class="progress"><div class="bar bar-success" style="width: 57%;"></div></div>         </td>

ect

I tried to use pythons re.search but I can only get a bit of info out of it. eg:

search = re.search("class='gb'>(.*)&nbsp;GB</span>",raw_info)
for i in range(0,100):
    try:
        print(search.group(i))
    except:
        break

output:

class='gb'>6.88&nbsp;GB</span></td></td><td class='text-right'><span class='mb'>
1,295.90&nbsp;MB</span><span class='gb'>1.27&nbsp;GB</span></td></td><td class='
text-right'><span class='mb'>8,340.12&nbsp;MB</span><span class='gb'>8.14&nbsp;G
B</span>
6.88&nbsp;GB</span></td></td><td class='text-right'><span class='mb'>1,295.90&nb
sp;MB</span><span class='gb'>1.27&nbsp;GB</span></td></td><td class='text-right'
><span class='mb'>8,340.12&nbsp;MB</span><span class='gb'>8.14

I learned I can't use groups like that to print out all of the numbers

tldr: I need to print all the numbers referring to gb and print them like this

2.92,1.20,4.12

3.51,1.17,4.68

Word of advice, never use regex on HTML. See this answer

Wondercricket
– Wondercricket

2016-10-28 20:46:17 +00:00
Commented Oct 28, 2016 at 20:46 — Wondercricket
– Wondercricket, Commented Oct 28, 2016 at 20:46

Ziyad Edher · Accepted Answer · 2016-10-28 20:45:15Z

3

You might want to try using BeautifulSoup, it's a very flexible library which can do exactly what you are looking for.

html = scraped
soup = BeautifulSoup(html)
spans = soup.findAll('span', attrs={'class': 'gb'})

You will then have a list of all the span tags that have the gb class. Producing the numbers and converting them to floats then applying whatever format you want to print them in is fairly simple.

edited Oct 28, 2016 at 20:45

answered Oct 28, 2016 at 20:39

Ziyad Edher

2,15020 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python Scraping Html Trouble

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related