Problems extracting data using JSON in R (getting a lexical error)

Question

Related to the question asked here: R - Using SelectorGadget to grab a dataset

library(rvest)
library(jsonlite)
library(magrittr)
library(stringr)
library(purrr)
library(dplyr)

get_state_index <- function(states, state) {
  return(match(T, map(states, ~ {
    .x$name == state
  })))
}

s <- read_html("https://www.opentable.com/state-of-industry") %>% html_text()
all_data <- jsonlite::parse_json(stringr::str_match(s, "__INITIAL_STATE__ = (.*?\\});w\\.")[, 2])
fullbook <- all_data$covidDataCenter$fullbook

hawaii_dataset <- tibble(
  date = fullbook$headers %>% unlist() %>%  as.Date(),
  yoy = fullbook$states[get_state_index(fullbook$states, "Hawaii")][[1]]$yoy %>% unlist()
)

I am trying to grab the Hawaii dataset from the State tab. The code was working before but now it is throwing an error with this part of the code:

all_data <- jsonlite::parse_json(stringr::str_match(s, "__INITIAL_STATE__ = (.*?\\});w\\.")[, 2])

I am getting the error:

Error: lexical error: invalid char in json text.                                        NA                      (right here) ------^

Any proposed solutions? It seems that the website has remained the same for the year but what type of change is causing the code to break?

EDIT: The solution proposed by @QHarr:

all_data <- jsonlite::parse_json(stringr::str_match(s, "__INITIAL_STATE__ = ([\\s\\S]+\\});")[, 2])

This was working for a while but then it seems that their website again changed the underlying HTML codes.

QHarr · Accepted Answer · 2022-03-25 19:53:50Z

1

Change the regex pattern as shown below to ensure it correctly captures the desired string within the response text i.e. the JavaScript object to use for all_data

all_data <- jsonlite::parse_json(stringr::str_match(s, "__INITIAL_STATE__ = ([\\s\\S]+\\});")[, 2])

Note: in R the single escape is doubled e.g. \\s rather than shown \s above.

answered Mar 25, 2022 at 19:53

QHarr

84.5k14 gold badges58 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

OGC Over a year ago

How do I know what regex pattern to change though? They seem to be changing it every now and then. Right now the regex pattern you have posted in the solutions isn't working anymore.

OGC Over a year ago

Is it from looking at this: window.__INITIAL_STATE__={"authModal":{"isAuthModalOpen":f...} })(window) </script><script>_otbootstrap();</script><script> window.addEventListener('load', function() { var preloadPaths = JSON.parse("[]") ..........; }) });

QHarr Over a year ago

I'll have a look on weekend. Please remind me.

OGC Over a year ago

Did you get a chance to look at the problem yet? Thanks

Collectives™ on Stack Overflow

Problems extracting data using JSON in R (getting a lexical error)

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related