2

Using AWS CDK v2, I want to deploy two lambda functions. As both depend on pandas, I want to deploy a layer first, which has pandas and its dependencies in requirements.txt located at entry.

One of the lambda functions also depends on pyarrow, which I put into the requirements.txt located at entry of that lambda function. As both pyarrow and pandas depend numpy, I currently end up with numpy and its dependencies being installed both in the layer and that lambda function, which obviously isn't what I want.

In the end, I'm looking for a way to pass --no-deps to the pip installation of requirements.txt of the lambda function that depends on pyarrow. Any ideas how to achieve this?

Improve this question
2
  • 1
    Have you tried using pyarrow --no-deps in your requirements.txt file? Commented Jul 30, 2022 at 3:34
  • I tried that after you suggested it: "ERROR: Invalid requirement: pyarrow==8.0.0 --no-deps" But it seems as if the feature might be available soon?! github.com/pypa/pip/pull/10837 Commented Jul 30, 2022 at 14:09

1 Answer 1

Reset to default
1

I don't think this is possible to specify no-deps when using a requirements.txt. CDK also can use pipenv's Pipfile and Poetry's poetry.lock. That being said, in my research none of those tools seem to support installing with --no-deps either unless you manually mess with the lock files (not advised or maintainable really).

Looking deeper at the specific problem, I don't see pyarrow listing pandas as a dependency but I do see it installing numpy which is a dependency of pandas

I think your best option presently is to just deal with having numpy in both places or put both pyarrow and pandas in the layer.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Oh, you're absolutely right, numpy is a dependency of pyarrow, not pandas, will change that in the question.
The following feature might be able to solve the problem -- do you agree? github.com/pypa/pip/pull/10837 I'd still prefer an additional argument without_dependencies or so by which --no-deps may be set on python3 -m pip install... where that command is executed on the lambda's requirements.txt.
Yes, it looks like that would do it

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.