4

I installed gettyimages/spark docker image and jupyter/pyspark-notebook inside my machine.

However as the gettyimage/spark python version is 3.5.3 while jupyter/pyspark-notebook python version is 3.7, the following error come out:

Exception: Python in worker has different version 3.5 than that in driver 3.7, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

So, i have tried to upgrade the python version of gettyimage/spark image OR downgrade the python version of jupyter/pyspark-notebook docker image to fix it.

  • Lets talk about method 1, downgrade jupyter/pyspark-notebook python version first:

I use conda install python=3.5 to downgrade the python version of jupyter/pyspark-notebook docker image. However, after i do so , my jupyter notebook cannot connect to any single ipynb and the kernel seems dead. Also, when i type conda again, it shows me conda command not found, but python terminal work well

I have compare the sys.path before the downgrade and after it

['', '/usr/local/spark/python', '/usr/local/spark/python/lib/py4j-0.10.7-src.zip', '/opt/conda/lib/python35.zip', '/opt/conda/lib/python3.5', '/opt/conda/lib/python3.5/plat-linux', '/opt/conda/lib/python3.5/lib-dynload', '/opt/conda/lib/python3.5/site-packages']

['', '/usr/local/spark/python', '/usr/local/spark/python/lib/py4j-0.10.7-src.zip', '/opt/conda/lib/python37.zip', '/opt/conda/lib/python3.7', '/opt/conda/lib/python3.7/lib-dynload', '/opt/conda/lib/python3.7/site-packages']

I think more or less, it is correct. So why i cannot use my jupyter notebook to connect to the kennel?

  • So i use another method, i tried upgrade the gettyimage/spark image

sudo docker run -it gettyimages/spark:2.4.1-hadoop-3.0 apt-get install python3.7.3 ; python3 -v

However, I find that even i do so, i cannot run the spark well.

I am not quite sure what to do. May you share with me how to modify the docker images internal package version

Improve this question
4
  • 1
    Can you clarify what you mean by installing docker images? You pull a docker image so it is added to your docker system, and then spin up a container using the image. There is no installation that occurs. Hence, in this case, gettyimages/spark and jupyter/pyspark-notebook would be two different images that should not interact with each other unless you are setting up a connection between them through a docker network. Commented Aug 16, 2019 at 5:00
  • 1
    see this github.com/jupyter/docker-stacks/issues/… and this github.com/jupyter/docker-stacks/issues/603 Commented Aug 16, 2019 at 5:17
  • @Abhineet Gupta , i use port mapping method to enable the image container connect to the host. And these two image connect to each other through the host. What i mean of installing docker images is exactly what you said "pull a docker image so it is added to docker system, and then spin up a container using the image". And what i want is modify that image container and update the python version. By default, such notebook image has python 3.7 and python 2.7 only, while i wanna add python 3.5 version. Commented Aug 16, 2019 at 6:14
  • @LinPy, Thanks for your comment. However, by default, such python version is not inside the image. So, even i set the environment variable, is still fail. Commented Aug 16, 2019 at 6:18

1 Answer 1

Reset to default
6

If I look at the Dockerfile here, it installs python3 which by default is installing python 3.5 for debian:stretch. You can instead install python 3.7 by editing the Dockerfile and building it yourself. In your Dockerfile, remove lines 19-25 and replace line 1 with the following, and then build the image locally. 

FROM python:3.7-stretch

If you are not familiar with building your own image, download the Dockerfile and keep it in its own standalone directory. Then after cd into the directory, run the command below . You may want to first remove the already downloaded image. After this you should be able to run other docker commands the same way as if you had pulled the image from docker hub.

docker build -t gettyimages/spark .
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.