Grabbing Data from Web Page using python 3

Question

I'm performing the same web scraping pattern that I just learned from post , however, I'm unable to scrap the using below script. I keep getting an empty return and I know the tags are there. I want to find_all "mubox" then pulls values for O/U and goalie information. This so weird, what am I missing?

from bs4 import BeautifulSoup
import requests
import pandas as pd

page_link = 'https://www.thespread.com/nhl-scores-matchups'

page_response = requests.get(page_link, timeout=10)

# here, we fetch the content from the url, using the requests library
page_content = BeautifulSoup(page_response.content, "html.parser")

# Take out the <div> of name and get its value
tables = page_content.find_all("div", class_="mubox")

print (tables)

# Iterate through rows
rows = []

Bertrand Martel · Accepted Answer · 2018-10-30 21:49:25Z

1

This site uses an internal API before rendering the data. This api is an xml file, you can get here which contains all the match information. You can parse it using beautiful soup :

from bs4 import BeautifulSoup
import requests

page_link = 'https://www.thespread.com/matchups/NHL/matchup-list_20181030.xml'
page_response = requests.get(page_link, timeout=10)
body = BeautifulSoup(page_response.content, "lxml")

data = [
    (
        t.find("road").text, 
        t.find("roadgoalie").text, 
        t.find("home").text,
        t.find("homegoalie").text,
        float(t.find("ot").text),
        float(t.find("otmoney").text),
        float(t.find("ft").text),
        float(t.find("ftmoney").text)
    )
    for t in body.find_all('event')
]

print(data)

answered Oct 30, 2018 at 21:49

Bertrand Martel

46k17 gold badges156 silver badges170 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user3066155 Over a year ago

This is a great technique. I have a couple questions related. Did you use developer tools to find the api? How do I move up and down the tags? For example, I want to start at <Matchup> then move down to <Road><TeamInfo><Team>.text.

Collectives™ on Stack Overflow

Grabbing Data from Web Page using python 3

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related