Running Scrapy from a script - Hangs

Question

I'm trying to run scrapy from a script as discussed here. It suggested using this snippet, but when I do it hangs indefinitely. This was written back in version .10; is it still compatible with the current stable?

This question and answer may be ready for update. Here is a recent snippet from Scrapy. It works, but the question, for me, becomes: how do you stop the Twisted reactor and move on when done? — scharfmn
– scharfmn, Commented May 31, 2013 at 17:28

warvariuc · Accepted Answer · 2011-06-29 05:26:01Z

7

from scrapy import signals, log
from scrapy.xlib.pydispatch import dispatcher
from scrapy.crawler import CrawlerProcess
from scrapy.conf import settings
from scrapy.http import Request

def handleSpiderIdle(spider):
    '''Handle spider idle event.''' # http://doc.scrapy.org/topics/signals.html#spider-idle
    print '\nSpider idle: %s. Restarting it... ' % spider.name
    for url in spider.start_urls: # reschedule start urls
        spider.crawler.engine.crawl(Request(url, dont_filter=True), spider)

mySettings = {'LOG_ENABLED': True, 'ITEM_PIPELINES': 'mybot.pipeline.validate.ValidateMyItem'} # global settings http://doc.scrapy.org/topics/settings.html

settings.overrides.update(mySettings)

crawlerProcess = CrawlerProcess(settings)
crawlerProcess.install()
crawlerProcess.configure()

class MySpider(BaseSpider):
    start_urls = ['http://site_to_scrape']
    def parse(self, response):
        yield item

spider = MySpider() # create a spider ourselves
crawlerProcess.queue.append_spider(spider) # add it to spiders pool

dispatcher.connect(handleSpiderIdle, signals.spider_idle) # use this if you need to handle idle event (restart spider?)

log.start() # depends on LOG_ENABLED
print "Starting crawler."
crawlerProcess.start()
print "Crawler stopped."

UPDATE:

If you need to have also settings per spider see this example:

for spiderConfig in spiderConfigs:
    spiderConfig = spiderConfig.copy() # a dictionary similar to the one with global settings above
    spiderName = spiderConfig.pop('name') # name of the spider is in the configs - i can use the same spider in several instances - giving them different names
    spiderModuleName = spiderConfig.pop('spiderClass') # module with the spider is in the settings
    spiderModule = __import__(spiderModuleName, {}, {}, ['']) # import that module
    SpiderClass = spiderModule.Spider # spider class is named 'Spider'
    spider = SpiderClass(name = spiderName, **spiderConfig) # create the spider with given particular settings
    crawlerProcess.queue.append_spider(spider) # add the spider to spider pool

Example of settings in the file for spiders:

name = punderhere_com    
allowed_domains = plunderhere.com
spiderClass = scraper.spiders.plunderhere_com
start_urls = http://www.plunderhere.com/categories.php?

edited Jun 29, 2011 at 5:26

answered Jun 28, 2011 at 7:09

warvariuc

60.1k45 gold badges183 silver badges234 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

ciferkey Over a year ago

I get this traceback. My scrapy project is named scraper. Could that be the problem?

warvariuc Over a year ago

I think that is the issue. This is from a real project. You can remove references to scraper. You just need some settings for spiders.

ciferkey Over a year ago

so after I remove the references to scraper how to I go about importing the settings for my project?

warvariuc Over a year ago

i made some comments. you need to make some changes to make it work - have a valid pipeline, fully implemented MySpider class, set all necessary settings.

Collectives™ on Stack Overflow

Running Scrapy from a script - Hangs

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest