Rvest: using css selector pulls data from different tab in URL

Question

I am very new to scraping, and am trying to pull data from a section of this website - https://projects.fivethirtyeight.com/soccer-predictions/premier-league/. The data I'm trying to get is in the second tab, "Matches," and is the section titled "Upcoming Matches."

I have attempted to do this with SelectorGadget and using rvest, as follows -

library(rvest)
url <- ("https://projects.fivethirtyeight.com/soccer-predictions/premier-league/")
url %>%
   html_nodes(".prob, .name") %>%
   html_text()

this returns values, however corresponding to the first tab on the page, "Standings." How can I reference the correct section that I am trying to pull?

this page uses JavaScript to load data when you click Matches and probably rvest can't run JavaScript. You may need Rselenium to control real web browser which can run JavaScript. OR you can use DevTools in Firefox/Chrome to get url which JavaScript uses to download data - usually it get JSON data — furas
– furas, Commented Mar 17, 2022 at 17:24
it loads some values from projects.fivethirtyeight.com/soccer-predictions/forecasts/… — furas
– furas, Commented Mar 17, 2022 at 17:28

furas · Accepted Answer · 2022-03-17 21:34:18Z

First:I don't know R but Python.

When you click Matches then page uses JavaScript to generate matches and it loads JSON data from:

https://projects.fivethirtyeight.com/soccer-predictions/forecasts/2021_premier-league_forecast.json

https://projects.fivethirtyeight.com/soccer-predictions/forecasts/2021_premier-league_matches.json

https://projects.fivethirtyeight.com/soccer-predictions/forecasts/2021_premier-league_clinches.json

I checked only one of them - 2021_premier-league_matches.json - and I see it has data for Completed Matches

I made example in Python:

import requests

url = 'https://projects.fivethirtyeight.com/soccer-predictions/forecasts/2021_premier-league_matches.json'

response = requests.get(url)
data = response.json() 

for item in data:
    # search date
    if item['datetime'].startswith('2022-03-16'):

        print('team1:', item['team1_code'], '|', item['team1'])
        print('prob1:', item['prob1'])
        print('score1:', item['score1'])
        print('adj_score1:', item['adj_score1'])
        print('chances1:', item['chances1'])
        print('moves1:', item['moves1'])
        print('---')

        print('team2:', item['team2_code'], '|', item['team2'])
        print('prob2:', item['prob2'])
        print('score2:', item['score2'])
        print('adj_score2:', item['adj_score2'])
        print('chances2:', item['chances2'])
        print('moves2:', item['moves2'])

        print('----------------------------------------')

Result:

team1: BHA | Brighton and Hove Albion
prob1: 0.30435
score1: 0
adj_score1: 0.0
chances1: 1.244
moves1: 1.682
---
team2: TOT | Tottenham Hotspur
prob2: 0.43627
score2: 2
adj_score2: 2.1
chances2: 1.924
moves2: 1.056
----------------------------------------
team1: ARS | Arsenal
prob1: 0.22114
score1: 0
adj_score1: 0.0
chances1: 0.569
moves1: 0.514
---
team2: LIV | Liverpool
prob2: 0.55306
score2: 2
adj_score2: 2.1
chances2: 1.243
moves2: 0.813
----------------------------------------

Collectives™ on Stack Overflow

Rvest: using css selector pulls data from different tab in URL

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related