How to add third party python libraries in Google App Engine, which are not provided by Google? I am trying to use BeautifulSoup in Google App Engine and unable to do so. But my question is for any library I want to use in Google App Engine.
-
possible duplicate of How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)Wernight– Wernight2015-09-17 15:10:09 +00:00Commented Sep 17, 2015 at 15:10
-
I just posted an updated answer to the OP question here: stackoverflow.com/a/67863189/305689wescpy– wescpy2021-06-06 20:01:39 +00:00Commented Jun 6, 2021 at 20:01
Add a comment
|
7 Answers
Google has provided a documented way for included third-party libraries in your GAE project.
See the "Adding Third-party Packages to the Application" section of the Libraries in Python 2.7 docs.
If you want to include additional pure-python third-party packages, you can do so by setting up vendoring. Vendoring allows you to install packages to a subdirectory of your project and include them in your code. To use vendoring, create (or modify) appengine_config.py in the root of your project.
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
And then just put all your libs' source code in your lib dir
> pip install beautifulsoup4 -t lib
So your project directory structure looks like this:
project
- lib
- bs4
- your_code.py
This will allow your project's source files to import libs' packages/modules as if they were added to your PYTHON_PATH. For example:
# file: your_code.py
import bs4 # no need for 'from lib import bs4'
# do stuff with bs4...
You can also easily install everything from a requirements.txt file by doing the following command
> pip install -t lib -r requirements.txt
7 Comments
Alexander Trakhimenok
This answer seems to be better then the one that is accepted. Thanks Jesse!
bgenchel
this worked for me for a long time, but just began failing inexplicably. I am now getting 'virtualenv: cannot access lib: No such virtualenv or site directory'
Jesse Webb
@bgenchel @PrestonCrawford Were either of you able to resolve your problems? I have never seen that
virtualenv error and this approach is still currently working for me. If you still haven't solved it, please post a new question and link it here and I will try to help. Make sure to add many details.bgenchel
What happened for me, I think, was that I was trying to import from lib as a package, not understanding that vendor.add("lib") means you can access libraries inside as though they were on your python path. The error was thrown because after running the pip install on my requirements file, I had forgotten to put an init.py inside the folder. Thanks for offering!!
Preston Crawford
I was and I wasn't. I'm only getting the error through PyCharm right now. Trying to sort that out.
|
Actually I think this answer fits better here.
If you want to use 3rd party libraries that are not included in this list, then you'll have to add them manually.
In order to include manually any other library you have to have them inside the directory where the app.yaml lives. So for example if you have the following structure:
hello
├── libs
│ └── bs4
├── hello.py
└── app.yaml
then in your hello.py you have to put these two lines in the beginning of the file:
import sys
sys.path.insert(0, 'libs')
After doing that you'll be able to use any 3rd party library that you're going to put in that libs directory.
For example:
from bs4 import BeautifulSoup
2 Comments
Ann Kilzer
I am still having problems with this after following the directions. Has anything changed with GAE?
Alexander Trakhimenok
The answer from Jesse is much better IMHO.
You simply copy the folder containing the library you want to use into your app engine project.
Then when you deploy it's uploaded with your application and is available for use.
EDIT: Jesse's answer is how I now do this. So do that!
Comments
The way it worked here is:
import sys
# sys.path.insert(0, 'libs') #"Old" way, not working for me.
sys.path.append(os.path.join(os.path.dirname(__file__), "libs")) # This works!
Then import normally:
from bs4 import BeautifulSoup
Comments
Just put Beautifulsoup in the root of your project and upload it all
2 Comments
Moazzam Khan
Can i put it in some specific folder like - 'lib'?
Tim Hoffman
yes manipulate sys.path or use site module as in `site.addsitedir('lib')
pip install -t lib package_name
lib: the location for third party libraries
Then you are good to use this package like a normal library you use from ipython or terminal.
Comments
Update for 2024 for anyone trying to figure this out
I found the answer here: https://groups.google.com/g/google-appengine/c/e21mD63LCrs
He said
The vendored approach (installing into lib/ and adding an appengine_config.py file) is correct for Python 2.7 on the standard environment: https://cloud.google.com/appengine/docs/standard/python/tools/using-libraries-python-27 Python 3.7 on the standard environment and all Python versions on the flexible environment use virtualenv and will install libraries specified in your requirements.txt, so you don't need to create the lib folder, and appengine_config.py is ignored: https://cloud.google.com/appengine/docs/standard/python3/specifying-dependencies https://cloud.google.com/appengine/docs/flexible/python/runtime#dependencies Compute Engine is a whole different system that works in terms of VMs rather than apps, so none of this really applies there, I'm afraid.
