How to integrate javascript rendering module to scrapy?

Question

I am working on a web scraping program, but I have run into a problem using scrapy with javascript generated content. I know that scrapy is not built to do this type of scraping, but I have been trying to use scrapyjs or splash to accomplish what I need.

However, I cannot get either of these two modules to work correctly with scrapy. My question is if anyone has a minimal example they can show that uses scrapyjs or splash to render javascript pages?

Edit: My platform is ubuntu and I working with python. For scrapyjs I just put the source in the uppermost directory of the scrapy project and I have yet to find any real guides on how to use splash. The reason I am asking about splash is because it seems a more powerful module for javascript rendering and is mentioned a lot in the same conversation as scrapjs.

What's your platform, how did you install scrapyjs or splash? What errors if any are you getting? — paul trmbrth
– paul trmbrth, Commented Feb 5, 2014 at 9:56

Tony · Accepted Answer · 2014-02-05 16:38:39Z

1

I believe all you have to do is implement process_links in your Spider:

def proxy_url(url):
        return "http://localhost:8050/render.html?url=%s&timeout=15&wait=1" % url


def process_links(self,links):
        for link in links:
            link.url = proxy_url(link.url)
        return links

answered Feb 5, 2014 at 16:38

Tony

19.2k32 gold badges136 silver badges196 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Muttonchop Over a year ago

I don't fully understand what proxy_url() is doing. Could you explain?

Muttonchop Over a year ago

I got your solution to work in a slightly modified form.

Mahmoud M. Abdel-Fattah Over a year ago

@Adamkucera, Can you please share the modified form?

Muttonchop Over a year ago

@MahmoudM.Abdel-Fattah What I did was put the splash mechanism to generate js inside a parse function only if the response requires it. I didn't put it in front of all of the links if I don't need to. Does this help?

Hatem Jaber Over a year ago

Can you update your answer and show us a sample. I don't use Python on a daily basis and would like to see your sample code please.

Collectives™ on Stack Overflow

How to integrate javascript rendering module to scrapy?

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related