1

I have Windows 10, with Aptana Studio 3. - I installed python in it (v.3.5.0) with terminal - I installed scrapy 1.0

and now, if I try to run a crawler (scrapy) I get this message:

Traceback (most recent call last):
  File "c:\python3.5\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python3.5\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python3.5\Scripts\scrapy.exe\__main__.py", line 5, in <module>
  File "c:\python3.5\lib\site-packages\scrapy\__init__.py", line 48, in <module>
    from scrapy.spiders import Spider
  File "c:\python3.5\lib\site-packages\scrapy\spiders\__init__.py", line 10, in <module>
    from scrapy.http import Request
  File "c:\python3.5\lib\site-packages\scrapy\http\__init__.py", line 15, in <module>
    from scrapy.http.response.html import HtmlResponse
  File "c:\python3.5\lib\site-packages\scrapy\http\response\html.py", line 8, in <module>
    from scrapy.http.response.text import TextResponse
  File "c:\python3.5\lib\site-packages\scrapy\http\response\text.py", line 13, in <module>
    from scrapy.utils.response import get_base_url
  File "c:\python3.5\lib\site-packages\scrapy\utils\response.py", line 12, in <module>
    from twisted.web import http
  File "c:\python3.5\lib\site-packages\twisted\web\http.py", line 92, in <module>
    from twisted.internet import interfaces, reactor, protocol, address
  File "c:\python3.5\lib\site-packages\twisted\internet\reactor.py", line 38, in <module>
    from twisted.internet import default
  File "c:\python3.5\lib\site-packages\twisted\internet\default.py", line 56, in <module>
    install = _getInstallFunction(platform)
  File "c:\python3.5\lib\site-packages\twisted\internet\default.py", line 50, in _getInstallFunction
    from twisted.internet.selectreactor import install
  File "c:\python3.5\lib\site-packages\twisted\internet\selectreactor.py", line 18, in <module>
    from twisted.internet import posixbase
  File "c:\python3.5\lib\site-packages\twisted\internet\posixbase.py", line 18, in <module>
    from twisted.internet import error, udp, tcp
  File "c:\python3.5\lib\site-packages\twisted\internet\udp.py", line 53, in <module>
    from twisted.internet import base, defer, address
  File "c:\python3.5\lib\site-packages\twisted\internet\base.py", line 23, in <module>
    from twisted.internet import fdesc, main, error, abstract, defer, threads
  File "c:\python3.5\lib\site-packages\twisted\internet\defer.py", line 29, in <module>
    from twisted.python import lockfile, failure
  File "c:\python3.5\lib\site-packages\twisted\python\lockfile.py", line 52, in <module>
    _open = file
NameError: name 'file' is not defined

So far I get that file() is a python 2.x function and it does not work with python 3.x from this Q: python NameError: name 'file' is not defined But the problem is that I do not use file() anywhere in my script. And the message appears to reffer to scrapy default files. So what am I supposed to do? Edit the source files of scrapy? That sounds awfull and way above my knowledge level. So what can I do? I should install python 2.7 but for some reason (I don't remember) I could not install it properly when I tried

So anyone has a solution for this?

2 Answers 2

2

You likely need to update Twisted (to v15.4.0 or something similar), since your version is not Python 3 compatible.

As Matthias pointed out, scrapy is being ported to Python 3. However, the error you have is in lockfile.py from Twisted, not scrapy:

File "c:\python3.5\lib\site-packages\twisted\python\lockfile.py", line 52, in <module>
    _open = file

Twisted is a dependency of scrapy. If you look at the history of this file, you'll see that there was a merge that presumably brought in Python 3 compatibility on 9/22/2015. I can tell that you have an older version because only the previous versions have _open = file in that lockfile.py.

All that being said, you probably still need to use Python 2 until the rest of scrapy is updated to be Py3 compatible.

Sign up to request clarification or add additional context in comments.

Comments

1

It looks like scrapy is not yet fully ported to Python 3:

https://github.com/scrapy/scrapy/wiki/Python-3-Porting

So yes, use Python 2.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.