9

I want to use two Python libraries (Google's Cloud Library, and their Cloud SDK) in a single application, but they have conflicting names (they both use google in their base import names and do not use relative imports internally). How can I use them in a single app?

Changing the library's code to use proper relative imports is not practical. Also, I know I can use virtualenv to access these libraries from separate python applications, but how do I access them from within the same python app?

Details of the Naming Conflict

Here are some of the details on the import. When I import a module from the Cloud Library (I run import google.cloud.datastore), there is an exception about another import within that library:

>>> import libs.google.cloud.datastore
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "C:\[ProjectDIR]\libs\google\cloud\datastore\__init__.py", line 52, in <module>
    from google.cloud.datastore.batch import Batch
ImportError: No module named cloud.datastore.batch

The library is trying to do an absolute import, rather than a relative one. The reason that the Google Cloud Library cannot import google.cloud.datastore.batch is because google is already defined in the SDK, there is a naming conflict:

>>> print google.__path__
['C:\\Program Files (x86)\\Google\\Cloud SDK\\google-cloud-sdk\\platform\\google_appengine\\google']

Because the Cloud Library uses absolute imports, and the name google is already defined in the SDK, then the import fails.

Improve this question
8
  • The second link goes to the Google app engine SDK (providing code under the google.appengine package), the first is google.cloud. I don't see how the two have conflicting names. What issues are you experiencing? You can always import modules under a different name at any rate; from google.cloud import datastore as gc_datastore would bind the name gc_datastore in your current module; it doesn't matter what the original name was. Commented Dec 30, 2016 at 16:21
  • Hi @MartijnPieters, the problem has to do with the internals of the library. The library at the first link does not use relative imports, so when you try to import it, you get naming conflicts with the second. See line 57 here: github.com/GoogleCloudPlatform/google-cloud-python/blob/master/… Commented Dec 30, 2016 at 17:29
  • So does the other actually include a google.cloud.datastore package? You haven't clearly linked to anything there. And I suspect that the Google App Engine SDK already includes the Cloud SDK as part of the installation, so you wouldn't need to install the other one too. Commented Dec 30, 2016 at 17:31
  • @MartijnPieters, I added more detail. The problem is that both libraries define google and the cloud library uses absolute imports. And no, the SDK does not include the Cloud libraries. The Cloud Libraries are newer ways of accessing Cloud resources that use a REST interface. Commented Dec 30, 2016 at 17:38
  • Why are you using libs in this path: import libs.google.cloud.datastore? Why isn't that a top-level installation instead? That's your real problem. Commented Dec 30, 2016 at 17:58

1 Answer 1

Reset to default
9

The google packages take care to register themselves as a namespace package. With a properly set up sys.path there is no conflict here.

You need to set up your library environment correctly. Add a appengine_config.py file in the root of your project with:

from google.appengine.ext import vendor

# Add any libraries installed in the "lib" folder.
vendor.add('lib')

This adds the lib subdirectory in the right location of sys.path. See the Installing a third-party library section in the Developing Python Apps on App Engine How-To.

From here on out imports of google.cloud just work:

$ ls -1d lib *.py *.yaml
app.yaml
appengine_config.py
lib
main.py
$ pip install -t lib google-cloud
# installing into the lib subdirectory
$ cat main.py
import google
from google.cloud import datastore
from google.appengine.api import memcache
import os.path

here = os.path.dirname(os.path.abspath(__file__))

def app(*args, **kwargs):
    return '''
google: {}<br />
google.cloud.datastore: {}<br />
google.appengine.api.memcache: {}'''.format(
        os.path.relpath(google.__file__, here),
        os.path.relpath(datastore.__file__, here),
        os.path.relpath(memcache.__file__, here))

and in the browser I am served:

google: ../google-cloud-sdk/platform/google_appengine/google/__init__.py
google.cloud.datastore: lib/google/cloud/datastore/__init__.pyc
google.appengine.api.memcache: ../google-cloud-sdk/platform/google_appengine/google/appengine/api/memcache/__init__.pyc
Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Okay, this looks good, I'll give it a try. But can you also import the SDK libraries as well? In one file, could I do import google.cloud as gcloud and also import google.appengine as appengine?
@speedplane: updated to add an SDK sample too. Yes you can.
This is a bit of a nit, but the command pip install -t lib google-cloud adds a bunch of unnecessary stuff into the lib folder (e.g., egg-info, dist-info files). Any clean way to only install the libraries that will be used? Or should I just delete the unnecessary stuff after installing?
@speedplane: I'd not worry about those, and keep things as simple as possible. Don't complicate the bootstrap procedure for a new developer.
@speedplane: IIRC (it's been a year or 2 since I worked on GAE projects) you can add ignores to your yaml file instead so those egg-info directories are not uploaded, using the skip_files option.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.