I am working on a Google Dataflow project and here are the requirements for the project:
absl-py==0.7.1
apache-beam==2.5.0
astor==0.7.1
avro==1.8.2
backports.weakref==1.0.post1
cachetools==3.1.0
certifi==2019.3.9
chardet==3.0.4
crcmod==1.7
dill==0.2.6
docopt==0.6.2
enum34==1.1.6
fasteners==0.14.1
funcsigs==1.0.2
future==0.16.0
futures==3.2.0
gapic-google-cloud-pubsub-v1==0.15.4
gast==0.2.2
google-api-core==1.10.0
google-apitools==0.5.20
google-auth==1.6.3
google-auth-httplib2==0.0.3
google-cloud-bigquery==0.25.0
google-cloud-core==0.25.0
google-cloud-dataflow==2.5.0
google-cloud-datastore==1.7.3
google-cloud-pubsub==0.26.0
google-cloud-storage==1.15.0
google-gax==0.15.16
google-resumable-media==0.3.2
googleapis-common-protos==1.5.10
googledatastore==7.0.1
grpc-google-iam-v1==0.11.4
grpcio==1.20.1
h5py==2.9.0
hdfs==2.5.2
hdfs3==0.3.0
httplib2==0.9.2
idna==2.8
Keras-Applications==1.0.7
Keras-Preprocessing==1.0.9
Markdown==3.1
mock==2.0.0
monotonic==1.5
numpy==1.16.3
oauth2client==3.0.0
pbr==5.2.0
Pillow==6.0.0
ply==3.8
proto-google-cloud-datastore-v1==0.90.4
proto-google-cloud-pubsub-v1==0.15.4
protobuf==3.7.1
pyasn1==0.4.5
pyasn1-modules==0.2.5
pytz==2019.1
PyVCF==0.6.8
PyYAML==3.13
requests==2.21.0
rsa==4.0
six==1.10.0
tensorboard==1.13.1
tensorflow==1.13.1
tensorflow-estimator==1.13.0
termcolor==1.1.0
typing==3.6.6
urllib3==1.24.2
Werkzeug==0.15.2
The problem I am having is that one of the steps in the pipeline uploads a file to google storage (using blob.upload_from_file). This worked before but then I had to recreate the virtual environment as I was having other issues. I am working with Python2.7 and installed the necessary packages with pip.
I started with pip install google-cloud-dataflow, although according to the docs I should be doing pip install apache-beam[gcp] but that never worked for me (no matches found: apache-beam[gcp]). I know it's some packages versioning conflict but I am unable to find a solution.
Has anyone had a similar problem ?
I tried upgrading to latest google cloud packages but, as expected, it raised all sorts of issues.
