Click Button via RSelenium

Question

I am attempting to scrape REI's Reviews (Hammocks) using Rselarium and Rvest. I want to hit the button at the bottom x amount of times so I can scrape all the reviews. I'm a little lost. Here's what I have so far. If you know too, how to preview in the finder what you're doing (not screen print) that would be cool. Thanks Stack Community.

    replicate(100,
          {
remDr$navigate("https://www.amazon.com/Eagles-Nest-Outfitters-DoubleNest-Portable/product-reviews/B00K30GXK8/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviewshttps://www.amazon.com/Eagles-Nest-Outfitters-DoubleNest-Portable/product-reviews/B00K30GXK8/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews")
webElem <- remDr$findElement("css", "body")
webElem$sendKeysToElement(list(key = "end"))
morereviews <- remDr$findElement(using = 'css selector', ".a-last a")
morereviews$clickElement
Sys.sleep(4)

reviews <- xml2::read_html(remDr$getPageSource()[[1]])%>%
  rvest::html_nodes(".review-text")%>%
  dplyr::data_frame(reviews = .)
})

Any helps/hints would be appreciated...

astronomerforfun
– astronomerforfun

2018-08-17 15:51:35 +00:00
Commented Aug 17, 2018 at 15:51 — astronomerforfun
– astronomerforfun, Commented Aug 17, 2018 at 15:51
Stack I'm stuck here.

astronomerforfun
– astronomerforfun

2018-08-20 02:15:38 +00:00
Commented Aug 20, 2018 at 2:15 — astronomerforfun
– astronomerforfun, Commented Aug 20, 2018 at 2:15

Pol Ferrando · Accepted Answer · 2018-08-29 13:02:01Z

4

Try this:

# Click the Load More button
replicate(100,
          {
            # scroll down
            webElem <- remDr$findElement("css", "body")
            webElem$sendKeysToElement(list(key = "end"))
            # find button
            morereviews <- remDr$findElement(using = 'css selector', "#BVRRContainer div.bv-content-pagination-container button")
            # click button
            morereviews$clickElement()
            # wait
            Sys.sleep(4)
          })

# Scrap the reviews
reviews <- xml2::read_html(remDr$getPageSource()[[1]])%>%
  rvest::html_nodes("#BVRRContainer div.bv-content-summary-body-text") %>%
  rvest::html_text() %>%
  dplyr::data_frame(reviews = .)
reviews

answered Aug 29, 2018 at 13:02

Pol Ferrando

6734 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

astronomerforfun Over a year ago

Still struggling with turning the page. Any ideas?

astronomerforfun Over a year ago

No. The page refreshes but doesn't actually change the page. I thought it was the selector (and may be) but I think I have tried every combination. What do you think Pol. BTW, I sent and email to you. Not sure if you got it.

astronomerforfun Over a year ago

Still can't figure this darn thing out... Are the page buttons Java or some other thing? Help stack.

Pol Ferrando Over a year ago

I noticed that you edited your question and changed the url you are scraping (rei vs amazon), so it's normal that the code of this answer won't work. Why did you change the url? You should go back to the last version of this question (since it has this solution which worked) and open a new one with this new url and explaining what you're trying to accomplish so we can help you with this new one.

astronomerforfun Over a year ago

Oh I didn't want to create a new one. I will though. That makes it cleaner. Thanks. Sorry for confusion.

Collectives™ on Stack Overflow

Click Button via RSelenium

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related