RSelenium not recognising element

Question

I'm trying to scrape all the providers from this page: https://www.agedcareguide.com.au/nursing-homes/providers/vic

I'm using RSelenium on my Mac by running the following code in Terminal with Docker:

docker run -d -p 4445:4444 selenium/standalone-firefox

Then when I return to RStudio and run the following:

remDr <- remoteDriver(remoteServerAddr = "localhost", port = 4445L, 
browserName = "firefox")
remDr$open()
remDr$navigate("https://www.agedcareguide.com.au/nursing-homes/providers/vic")
remDr$getTitle()

All is good.

Then I try to get the element by using:

provs <- remDr$findElement()

and inside the brackets I have used the XPath, CSS Selector, everything I can think of but it always comes up saying:

Error in match.arg(using) : 'arg' should be one of “xpath”, “css selector”, “id”, “name”, “tag name”, “class name”, “link text”, “partial link text”

Anybody got any ideas where I'm going so terribly wrong?

This seems to work without an error provs <- remDr$findElement(using="class",value="c-result-list"). Note that this only finds the element, it does not get it without a bit of further processing. An alternative would be to use page <- remDr$getPageSource() after your navigate line, and then use rvest or similar to extract what you want from page. — Andrew Gustar
– Andrew Gustar, Commented Mar 27, 2018 at 10:00
Looking at the page source through right-clicking, the elements aren't there in text. Not sure how rvest would then be able to find them? And trying the findElement option returns an analysis on the browser. — Foothill_trudger
– Foothill_trudger, Commented Mar 27, 2018 at 22:11
Yes, although the 'selector' code seems to work - but you might need to also build in a delay. See answer below. Good luck! — Andrew Gustar
– Andrew Gustar, Commented Mar 28, 2018 at 9:21

Andrew Gustar · Accepted Answer · 2018-03-28 09:20:41Z

2

A partial solution...

with RSelenium...

remDr$navigate(...)
Sys.sleep(20) #the page keeps loading for some time
page <- remDr$getPageSource()

then, with rvest...

provs <- page[[1]] %>% read_html() %>% 
   html_node("#app > div > div.c-col-results > div:nth-child(3) > div > section") %>% 
   html_text()

after a bit of tidying (split by \\n, remove blanks)...

provs
 [1] "AdventCare"                                     "Providing nursing homes" 
 [3] "Alexandra Gardens SRS"                          "Providing nursing homes" 
 [5] "Allbright Manor"                                "Providing nursing homes"
 [7] "Alliance Care Services Group"                   "Providing nursing homes" 
 etc...

Hopefully this will help get you started, although it is a tricky one!

answered Mar 28, 2018 at 9:20

Andrew Gustar

18.6k1 gold badge26 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Foothill_trudger Over a year ago

Thanks for this - it works to get the data out which is great. Now I've got to master all the tidying which is a bit frustrating!

Collectives™ on Stack Overflow

RSelenium not recognising element

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related