Scrapy - Javascript rendering

Question

I would like to get some data from here:

https://www.drivy.com/location-voiture/liege/mitsubishi-colt-359699?address=Gare+de+Li%C3%A8ge-Guillemins&city_display_name=&country_scope=BE&distance=200&end_date=2019-05-27&end_time=06%3A00&latitude=50.6251&longitude=5.5659&start_date=2019-05-26&start_time=06%3A00

I'm searching for the ID of the owner of the car. This ID is within the aattribute of class car_owner_section. For the page above it is the numbers in the hrefattribute like this "/users/1228276". The issue is that this link is apparently rendered by javascript and I absolutely want to avoid scrapy-splash. Does anyone has an idea on how to find this ID ? It should be somewhere on a JSON I guess but I've searched for days now and found nothing.

Pasindu Gamarachchi · Accepted Answer · 2019-05-21 01:00:56Z

1

I tested it on scrapy shell, and the response returns the link you are looking for, without using splash. You might want to check your settings.

USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

answered May 21, 2019 at 1:00

Pasindu Gamarachchi

5666 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Scrapy - Javascript rendering

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related