How to use scrapy to click on element and return JS

Question

I am trying to scrape names and contact details from this page https://www.realestate.com.au/find-agent/agents/sydney-cbd-nsw. I normally want to click into each of the list items and get the information from the resulting page, but there is no href to follow.

I'm presuming that the class type somehow points to some JS codes. When the list item is clicked the JS redirects you to the new url. Can I get at it somehow using Scrapy?

Note: I don't know much about JS

Michael Savchenko · Accepted Answer · 2019-07-23 09:59:00Z

3

This will give you all the links you need without JS rendering.

response.css('script::text').re('"url":"(.+?)"')

Don't use Chrome for scraping until there's no other way. It's really bad practice.

answered Jul 23, 2019 at 9:59

Michael Savchenko

1,4551 gold badge9 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

noahcoleman · Accepted Answer · 2019-07-23 09:16:09Z

0

I'd recommend using Selenium which will automate an instance of an actual browser. This means that sessions, cookies, javascript execution, etc. is all handled for you.

Example:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("http://example.com")
button = driver.find_element_by_id('buttonID')
button.click()

answered Jul 23, 2019 at 9:16

noahcoleman

896 bronze badges

2 Comments

Michael Savchenko Over a year ago

In fact, selenium is a testing tool not intended to be used for parsing. It's huge computational overhead and noticeably slowing crawler. Way better is to spend some time and reverse engineer some JS, ajax is quite simple

noahcoleman Over a year ago

Oh sorry. Thanks for the info though @MichaelSavchenko

Collectives™ on Stack Overflow

How to use scrapy to click on element and return JS

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related