Run python script that imports local script in a docker

Question

I have a local script main.py that imports another local script submain.py. When I run it in a docker I get the following error:

qsub -cwd -soft -l docker,docker_images="*docker_imagename*" -S /usr/bin/python ./main.py --arg1 value1 --arg2 value2

from submain import func
ImportError: No module named submain

My Dockerfile looks like this:

FROM ubuntu:latest
RUN apt-get -y update && apt-get -y install build-essential libxml2-dev zlib1g-dev python-dev python-pip pkg-config libffi-dev libcairo-dev
RUN pip install --upgrade pip
RUN pip install python-igraph scikit-learn numpy scipy matplotlib

CMD /usr/local/bin/igraph

How can I run my script main.py with additional script stored locally? If this not possible, how I "attach" submain.py to docker image?

Can you add some more information? In particular, at least the docker command you're running. Also, this works when you run igraph locally (not in docker)? Also why is it running qsub? — dshockley
– dshockley, Commented Aug 24, 2017 at 18:32
Yes, the reason is that I'm working on a remote machine (for which I don't have sudo permissions, hence no igraph locally). I use qsub to submit a job to another server and I don't run any docker command except the one I specified. It worked for the scripts when there is no imports of local scripts happening, however, in this case it does not know where to look for submain.py. — Sergey Ivanov
– Sergey Ivanov, Commented Aug 24, 2017 at 19:39
If we forget about the fact that I'm launching via qsub, what would be the command/Dockerfile to run the script via docker command? — Sergey Ivanov
– Sergey Ivanov, Commented Aug 24, 2017 at 19:40
Are you sure you're running docker at all? I'm not familiar with qsub but I think maybe it's copying main.py to the remote server and running it directly (without docker, and also without copying submain.py). Is that possible? (To answer your question, something like docker run dockerimagename, also with the previous suggestion to add COPY directives to get the files in, and a different CMD.) — dshockley
– dshockley, Commented Aug 24, 2017 at 21:08

dshockley · Accepted Answer · 2017-08-25 07:23:37Z

Here's a minimal example of how this works, with docker + python (ignoring qsub).

First, main.py:

from submain import my_fn

if __name__ == '__main__':
    print('got val {} from submain.my_fn'.format(my_fn(12)))

And submain.py:

def my_fn(val):
    return val * 2

And the Dockerfile [1]:

FROM python:3

COPY main.py main.py
COPY submain.py submain.py

ENTRYPOINT python main.py

Then to test it:

$ docker build -t main-py-img .
# ... lots of output ...
Successfully built e79194e43094
Successfully tagged main-py-img:latest
$ docker run main-py-img

If possible, you should try to test this on a local environment, to work out the python + docker issues, before you move to submitting a job with qsub. Using qsub to submit a job that runs your docker image is a different issue -- probably you should create a new question about that, once you get this part resolved.

[1] For simplicity I've inherited from the python:3 docker image, to avoid having to install all the python dependencies. If for some reason you need Ubuntu, you can try with that instead. In case you don't need Ubuntu, you can just add RUN pip install igraph to the Dockerfile above (inherit from python:2 if you need python 2 instead of python 3), but I would recommend to start with the most minimal example possible, and build from there.

Roberto Salgado · Accepted Answer · 2017-08-28 09:25:00Z

In answer to

How can I run my script main.py with additional script stored locally?

Python will try to load a module using its libraries path and current dir. So, since you seem to be using /root/ as main folder, you could add this line to a docker run execution (I don't have experience with qsub):

-v /local/path/to/submain.py:/root/submain.py

It will map container's /root/submain.py to your local submain.py. So you don't need to copy your submain file to the docker image.

Mind that you are running your scripts as root inside a container. So any volume or data that the container can write/read could be exposed if you run any insecure service.

It's also a good idea to check WORKDIR and USER Dockerfile options.

In answer to

If this not possible, how I "attach" submain.py to docker image?

using this inside the Dockerfile will do the trick:

COPY submain.py submain.py

Mind that you must build a new image before every in-container testing or use a volume as in previous example before sending it to the grid.

Collectives™ on Stack Overflow

Run python script that imports local script in a docker

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related