Scrape dynamic webpage for data using scrapy

Question

I am trying to scrape some data from www.gbig.org. It was successful to scrape the Activity summary, Activity details and Why it's green section however, the response was empty when I crawl the LEED DASHBOARD section.

Specifically, I would like to get 6 scores, EA, MR, IEQ, SS, WE, and ID but these values are empty when I scrape with this xpath.

scrapy shell "http://www.gbig.org/activities/leed-1000020523"
response.xpath("//*[@id='overview']/div[1]/div[1]/div/div[2]/div[2]/div[1]/div[1]/div/div/p[1]/text()").extract()

I found it is because the values that want to scrape are dynamic values, but I have no idea how to get the values. Could you please guide me to obtain these?

There are a few solutions to this discussed in the docs here: docs.scrapy.org/en/latest/topics/dynamic-content.html — tomjn
– tomjn, Commented May 6, 2021 at 13:16
@tomjn Thanks for your comment first. I got a html file by using "scrapy fetch --nolog "gbig.org/activities/leed-1000020523" > response.html" however, there was no values I am looking for. Maybe I don't have enough web related knowledge. Can you give me some more information in detail? — liveinfootball
– liveinfootball, Commented May 6, 2021 at 13:34

Akshay Goyal · Accepted Answer · 2021-05-08 08:09:57Z

1

First of all this website is pretty slow and you need to increase wait time while crawling using scrapy.

There are few things you can experiment with to get this data you are looking for.

Experiment with increasing wait time
Crawl this website using splash docker for headless browser loading. This way you can load js files so, you will probably get the data you are looking for. Currently, you are crawling in scrapy shell which will give you basic HTML without any loading of js and CSS coming from your target website which may not contain all data.

Hopefully, this can solve your problem.

answered May 8, 2021 at 8:09

Akshay Goyal

713 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Scrape dynamic webpage for data using scrapy

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related