Error Using Python's "Scrapy"

Question

I installed the newest version of Scrapy on Python 2.7 (Yes, I still use this version) and am getting an error while running Scrapy's tutorial script. The code I'm running is:

from scrapy.spider import BaseSpider

class DmozSpider(BaseSpider):
    name = "dmoz.org"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        filename = response.url.split("/")[-2]
        open(filename, 'wb').write(response.body)

The error is:

Traceback (most recent call last):
  File "C:/Users/Rainey/Desktop/dmoz/spiders/dmoz_spider.py", line 1, in <module>
    from scrapy.spider import BaseSpider
  File "C:\Python27\lib\site-packages\scrapy\spider.py", line 7, in <module>
    from scrapy import log
  File "C:\Python27\lib\site-packages\scrapy\log.py", line 10, in <module>
    from twisted.python import log
ImportError: No module named twisted.python

The tutorial script is on my desktop. Will this effect and should I have it in a different place?

Gustavo Breeze · Accepted Answer · 2012-02-16 16:58:07Z

5

The error message 'ImportError: No module named twisted.python' indicates that you are missing the twisted module.

Install twisted using a package manager like pip or download a Windows binary.

edited Feb 16, 2012 at 16:58

answered Mar 16, 2011 at 15:46

Gustavo Breeze

664 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Monster Z · Accepted Answer · 2012-05-29 03:00:48Z

2

You did not install scrapy successfully on windows. following steps should work for you to install scrapy 0.14(latest) on windows.

Install lxml-2.3 using windows installer here. I tried libxml but not work.
Install Twisted-12.0.0 using windows installer here
Install zope.interface with Mingw32 or not.
Install pyOpenSSL 0.13 using windows installer here
run "pip install scrapy", you are ready to go!

Hope it helps.

answered May 29, 2012 at 3:00

Monster Z

1476 bronze badges

Collectives™ on Stack Overflow

Error Using Python's "Scrapy"

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related