0

I have a heroku project that is trying to install a Django reusable app using pip install -r requirements.txt (it's not the exact command from the official heroku/python buildpack, but it's basically doing that).

The problem is that my Django reusable app has a requirement itself, from a Github public repository.

A few answers in pip install dependency links point to:

  • The dependency-links syntax being not compatible with pip 19
  • The new syntax (i.e. package @ git+protocol://...) of course not being compatible with older versions of pip.

My Django project can be deployed in different ways: Heroku, local, Docker (dev and prod). Most of these deployment methods have the latest version of pip.

Heroku's python buildpack has Pip 9.0.2

My other deployment methods have Pip 19, so they have incompatibilities in the way they read requirements, and I can't have a method that works for both.

For older versions of pip, I do in my setup.py:

setup(
# [...]
install_requires=[
        "my_package == x.y.z"
    ],
    dependency_links=["git+https://github.com/account/package/tarball/master#egg=my_package-x.y.z"]
)

Whereas for Pip 19, I do:

setup(
# [...]
install_requires=[
        "my_package @ git+https://github.com/company/package.git"
    ],
)

I tried forking the repo and just increasing the version to pip's latest version. It does work, except for errors in pip-diff, which don't seem to have an effect on the build.

Interesting to note too is that since the buildpack has a cache system, you generally notice this problem only once until you require a newer version of a library. But it's a problem with review apps for example.

So what is the solution here? I could:

  1. Downgrade all other deployment methods to an older version of pip that works with dependency links, including all my local dev setups
  2. Use a forked version of heroku's python buildpack, with updated pip (pip-diff needs to be fixed though)
  3. Remove dependencies from github and add those dependencies at the project repo level instead of the reusable app level.
  4. Submit a PR to the official repo so that pip can get upgraded

That's all I can think of. Solutions 1. and 2. are really bad IMHO, 3. is even worse, and 4. can take time, which I can't really afford.

Any better ideas?

Improve this question

2 Answers 2

Reset to default
1

The pip notation is not the same as the the setuptools notation. There is no immediate reason to change the setup script. The install_requires and dependency_links seem fine in the first version, and probably actually are if it used to work.

Now, it is true that the newer version of pip do not take the dependency_links of setuptools into account at all. So these custom dependency links can be added to a requirements.txt file instead (with the pip notation of course).

Additionally you could choose to entirely remove the dependency_links argument in the setup script to avoid duplicates.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Do you mean adding it to the requirements.txt of the package being added, or to the requirements.txt of the project importing the package? because if it's the latter, it's not a good solution IMHO
From my (and the pip maintainers as well, I think) point of view, the reasoning is: if I use pip to install Python packages and their dependencies, I assume that they will all be downloaded from PyPI which is a hosting service that I trust, but I am certainly not OK if a dependency breaches this contract and requires code from a different host. That's why the dependency links should only be enforced by the one who actually triggers the installation, i.e. requirements.txt of the application. Libraries only need to clearly document if they need dependencies that are not on PyPI.
OK, I understand. I'll see if that works with my colleagues. Thanks for the clarification.
0

I don't know if it's the right answer, but I used this workaround

So basically in my setup.py I add a function and call it:

def install_github_package(github_url):
    try:
        import github_url
        # ... do some version checking ...
    except (ModuleNotFoundError, ImportError):
        if '--user' in sys.argv:
            subprocess.run([sys.executable, '-m', 'pip', 'install', '--upgrade',
                            '--user', github_url], check=False)
        else:
            subprocess.run([sys.executable, '-m', 'pip', 'install', '--upgrade',
                            github_url], check=False)

install_github_package("git+https://github.com/company/package.git")
Improve this answer

6 Comments

If it works for your use case, then go for it. But if you use pip and most importantly if you already have a requirements.txt file for your application anyway, then placing the custom links for the dependencies in this file is probably a much better solution.
I tried using the way you told me to, I forgot to mention I use pip-tools, and I'm having trouble doing a pip-compile if I follow what you said, i.e.: - keep install_requires and dependency_links - add my dependency in requirements.in - try to run pip-compile What should I do in this case? manually add a line to the file generated by pip-compile?
I don't know pip-tools, can't help much. But from quickly looking at its README, I'd say add the custom git dependency link to the requirements.in, yes. Or you may have to then edit requirements.txt manually afterwards.
yeah, does not work by putting it in the .in... I guess I have to add it afterwards... That's my problem with all this, it seems not totally compatible with the pip ecosystem, but it's my opinion only.
Surprised it doesn't work. If I were you, I would start with a fresh question specifically focused on pip-compile, or directly ask in their issue tracker.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.