Unable to run scrapy for Python

Question

I am new to this platform as well as to Python scraping. I hope that my question will still be understandable and somebody can help me. Sorry, in case I make something unclear...

I have already checked other posts on a similar topic but could not manage to overcome my problem. I am currently getting into web-scraping and wanted to try Scrapy. Therefore, I followed the installation instructions on the website. http://doc.scrapy.org/en/0.16/intro/install.html#intro-install After I had figured out how it works I decided to run in a virtual environment.

I installed virtualenv and pip. Then I installed Scrapy.

When I now want to start with the tutorial

scrapy startproject tutorial

I get the following error message:

    File "/Users/XXX/environment_trial/bin/scrapy", line 3, in <module>
    from scrapy.cmdline import execute
  File "/Users/XXX/environment_trial/lib/python2.7/site-packages/scrapy/cmdline.py", line 7, in <module>
    from scrapy.crawler import CrawlerProcess
  File "/Users/XXX/environment_trial/lib/python2.7/site-packages/scrapy/crawler.py", line 3, in <module>
    from twisted.internet import reactor, defer
ImportError: No module named twisted.internet
(environment_trial)XXX-iMac:~ XXX$

I could not find a Twisted.py on my Mac as suggested by other posts.

Can somebody please tell me what to do?

Chris Clarke · Accepted Answer · 2013-05-20 14:56:24Z

1

Simply put, you need to install twisted. You can get it from the download page. It looks like you'll need to install from source on a newer Mac, but that's just a case of extracting the tarball and running python setup.py install in the extracted folder.

edit: Since you already have pip installed, you can also grab twisted with it.

pip -E twisted_env install -U twisted

answered May 20, 2013 at 14:56

Chris Clarke

2,2012 gold badges15 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

chunky · Accepted Answer · 2014-02-09 16:13:26Z

0

Please be sure that all the binaries that you are installing correspond to the exact same version of python that you have installed (For e.g python 2.7 ).

I did this mistake of installing pyopenSSL for python3.6 and it took me a lomng time to realize that versions did not match.

answered Feb 9, 2014 at 16:13

chunky

1

Collectives™ on Stack Overflow

Unable to run scrapy for Python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related