4

Running jupyter in a docker container is a good solution for me, but I'm having trouble getting the notebook files to persist as advertised in the documentation here.

The docs say that after the session is closed and the server shutsdown, the .ipynb (notebook) files should be persisted in the ./work directory however, for me they are not. I have created notebooks both in the root directory and also in the /work directory which appears in the Jupyter Home page, but neither are to be found after shutdown, and if I restart the server, they are not in the directory list anymore either. I've tried launching the container in two ways -- first as suggested by the docs (substituting latest for the image tag):

docker run -p 8888:8888 jupyter/scipy-notebook:latest

and second by creating a docker-compose.yml file which allows me to capture the command text options and avoid the token security (which I don't need) as follows:

version: '3'

  services: # jupyter notebook
    jupyter_notebook:
    image: jupyter/scipy-notebook
    volumes:
      - ./work:/work
    ports:
      - "8888:8888"
    command: "start.sh jupyter notebook --NotebookApp.token=''"

I am running under Ubuntu 18.04.1 LTS with docker 18.06.1-ce I am expecting to find the notebook (at least the one I created in the /work folder) in the host system's ./work folder which is in the directory where I launched docker (or docker-compose), but nothing is there.

Here is a session transcript:

   s@VC66:ls -la
   -rw-r--r-- 1 steve steve  232 Nov  7 22:45 docker-compose.yml
   drwxr-xr-x 2 steve steve 4096 Nov  7 21:34 work

   s@VC66:~/sambashare/jupyter$ cat docker-compose.yml 

   version: '3'

     services:
       jupyter_notebook:
         image: jupyter/scipy-notebook
         volumes:
           - ./work:/work
         ports:
           - "8888:8888"
         command: "start.sh jupyter notebook --NotebookApp.token=''"

   s@VC66:~/sambashare/jupyter$ docker-compose up

   Creating network "jupyter_default" with the default driver
   Creating jupyter_jupyter_notebook_1 ... done
   Attaching to jupyter_jupyter_notebook_1
   jupyter_notebook_1  | Container must be run with group "root" to update passwd file
   jupyter_notebook_1  | Executing the command: jupyter notebook --NotebookApp.token=
   jupyter_notebook_1  | [I 16:08:40.454 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
   jupyter_notebook_1  | [W 16:08:40.597 NotebookApp] All authentication is disabled.  Anyone who can connect to this server will be able to run code.
   jupyter_notebook_1  | [I 16:08:40.625 NotebookApp] JupyterLab extension loaded from /opt/conda/lib/python3.6/site-packages/jupyterlab
   jupyter_notebook_1  | [I 16:08:40.625 NotebookApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
   jupyter_notebook_1  | [I 16:08:40.631 NotebookApp] Serving notebooks from local directory: /home/jovyan
   jupyter_notebook_1  | [I 16:08:40.631 NotebookApp] The Jupyter Notebook is running at:
   jupyter_notebook_1  | [I 16:08:40.631 NotebookApp] http://(62b087792f87 or 127.0.0.1):8888/
   jupyter_notebook_1  | [I 16:08:40.631 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
   jupyter_notebook_1  | [I 16:08:58.820 NotebookApp] 302 GET / (172.21.0.1) 0.48ms
   jupyter_notebook_1  | [I 16:09:07.941 NotebookApp] Creating new file in /work
   jupyter_notebook_1  | [I 16:09:17.360 NotebookApp] Saving file at /work/untitled.txt
   jupyter_notebook_1  | [I 16:09:24.725 NotebookApp] Shutting down on /api/shutdown request.
   jupyter_notebook_1  | [I 16:09:24.727 NotebookApp] Shutting down 0 kernels
   jupyter_jupyter_notebook_1 exited with code 0
   s@VC666:~/sambashare/jupyter$ ls work
   s@VC66:~/sambashare/jupyter$ ls
   docker-compose.yml  work

As you can see, it says it saved "untitled.txt" in the /work dir, but upon exiting there's nothing in there.

So to further refine the problem here, I altered the docker-compose.yml file to run a simple python script to create a file in the /work dir and see if it persists. It does!

command: "python3 /work/test.py"  # rather than start.sh...

here's the python test.py script:

   import os
   import pytz
   from datetime import datetime

   dir = "/work"
   if not os.path.isdir(dir):
      dir = "" # to test outside docker container...

   nyc_time = datetime.now( pytz.timezone("America/New_York"))
   fname = os.path.join(dir,"test.txt")
   f = open(fname, 'w')
   f.write(f"Test time is {nyc_time}\n")
   f.close()
   exit()

This time, after docker-compose up, the work folder contains "test.txt" which contains

Test time is 2018-11-09 11:55:28.472581-05:00

So the docker container mounting the /work dir seems fine -- problem might be something the jupyter image is doing on shutdown perhaps?

6
  • What do you mean by the host systems "~/work folder"? That is not the folder you're mounting from the host... Commented Nov 8, 2018 at 4:11
  • Sorry, I must have inadvertently added a few extra characters when typing the question. The edited version above is fixed now to read correctly. Commented Nov 9, 2018 at 16:06
  • Still, there is a difference in ~/work and ./work unless you put the Compose file in your home directory Commented Nov 9, 2018 at 16:15
  • so the compose file is in ~/sambashare/jupyter and there is a directory ~/sambashare/jupyter/work which is what I thought I was mounting as /work in the container. Just for kicks I also created ~/work in my home directory to see if we were mounting that instead. It too remained empty after running the same test. By the way, thanks for helping me out! Commented Nov 9, 2018 at 16:25
  • If you do ls ~/sambashare/jupyter and you see a docker-compose.yml, then you cd ~/sambashare/jupyter && docker-compose up, then ~/sambashare/jupyter/work should be mounted into /work of the container. A better way to test this would be (1) Make a file on your host machine in the work folder (2) Start the container, and see if data is indeed shown by Jupyter. That way, you are not trying to debug the opposite direction Commented Nov 9, 2018 at 18:17

1 Answer 1

3

I think your misconception is about the docker container using /work. AFAIKs it is /home/jovyan/work instead.

So you can solve your trouble by e.g. this volume mapping

mkdir -P /your-jupyter/work
docker run -p 8888:8888 -e JUPYTER_ENABLE_LAB=yes -v /your-jupyter/work:home/jovyan/work jupyter/scipy-notebook

HTH.

Sign up to request clarification or add additional context in comments.

3 Comments

thank you a lot ! I have been struggling with this for hours so far. for me I was using : (github.com/NVAITC/ai-lab) and I should've ran : docker run --rm -p 8888:8888 -v D:/quickstart-notebooks:/home/jovyan/work -e JUPYTER_ENABLE_LAB=no nvaitc/ai-lab:19.11
Thanks, for this answer, How should I mount a volume based on this? I run a pyspark with a volume that zllow me to access to my local files but once I shutdown. I lost all ly output file and work in /word repo.
The mount is controlled by the -v parameters you specify yourself. What are you using currently as full docker command line?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.