RSelenium will run when running each line individually but not when running in code chunks

Question

i am trying to learn the RSelenium package so that I can automate pulling data from webpages. When I am doing this I am encountering a strange error.

 "no such element: Unable to locate element: {\"method\":\"css selector\",\"selector\":\"[name='rankings_table_length']\"}
(Session info: chrome=87.0.4280.88)

What makes it strange is that I am able to run the script perfectly fine when I am running each line individually, but when I try to run the lines in a chunk or a loop rselenium is unable to find the elements. I know that I have the elements correct from inspecting them. Does anyone with more RSelenium experience know why code might work when running each line individually, but not in a chunk?

Full code below for reproducibility:

conn <- rsDriver(browser = "chrome",
                 port = 950L,
                 chromever = "87.0.4280.88")

chooseStat <- c("Total Offense", 
  "Total Defense", 
  "3rd Down Conversion Pct"
  )

year <- "2020"

conn$client$open()

conn$client$navigate("https://stats.ncaa.org/")

sportInput <- conn$client$findElement(using = "css", "[name = 'sport']")
sportInput$clickElement()
sportInput$sendKeysToElement(list("football", key = "enter"))

yearInput <- conn$client$findElement(using = "css", "[name = 'acadyr']")
yearInput$clickElement()
yearInput$sendKeysToElement(list(year, key = "enter"))

divInput <- conn$client$findElement(using = "css", "[name = 'u_div']")
divInput$clickElement()
divInput$sendKeysToElement(list("FBS", key = "enter"))

selectTeam <- conn$client$findElement(using = "css", "[id = 'stat_type_T_N']")
selectTeam$clickElement()

statInput <- conn$client$findElement(using = "css",
                                     "[id = 'Stats']")
statInput$clickElement()
statInput$sendKeysToElement(list(chooseStat[1], key = "enter"))

lenInput <- conn$client$findElement(using = "css selector",
                                "[name='rankings_table_length']")

lenInput$clickElement()
lenInput$sendKeysToElement(list("130", key = "enter"))



tbl <- conn$client$getPageSource()[[1]] %>% 
  readHTMLTable()

stat_tbl <- tbl$rankings_table

assign(str_replace_all(chooseStat[1], " ", "_"), stat_tbl)

hello_friend · Accepted Answer · 2020-12-26 00:20:09Z

1

This piece of code is where the error is arising:

lenInput <- conn$client$findElement(using = "css selector",
                                "[name='rankings_table_length']")

It is because this CSS Selector represents the path to an element that isn't loaded on the webpage (and therefore can't be operated on) prior to other selenium calls operating on that page. The real problem here is the time it takes for the webpage to reflect these operations (and load that particular element).

I would recommend inserting sys.sleep(2) function calls throughout your code in order to allow the elements to load prior to trying to operating on them. This should enable you to run your code as a chunk as opposed to line by line. Let me know how you go.

answered Dec 26, 2020 at 0:20

hello_friend

5,8381 gold badge13 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Jeff Gallini Over a year ago

That worked. I had played around with adding Sys.sleep() but didn’t put it throughout the code. Doing that seems to work. Thank you!

Collectives™ on Stack Overflow

RSelenium will run when running each line individually but not when running in code chunks

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related