How to run a pytorch model server inside docker?

Question

I am trying to create a docker image to make it run as a server for serving a model in pytorch.

I converted the .pt model file to .MAR file in my local machine and i copied the .MAR file inside the docker image. I created a dockerfile:

FROM ubuntu:18.04
ENV TZ=Asia/Shanghai
ENV DEBIAN_FRONTEND noninteractive 
RUN apt-get update \
    && apt-get install --yes --no-install-recommends \
    tzdata

RUN ln -snf /usr/share/zoneinfo/${TZ} /etc/localtime\ 
    && echo ${TZ} > /etc/timezone \
    && dpkg-reconfigure -f noninteractive tzdata

RUN apt-get install python3 python3-dev python3-pip openjdk-11-jre-headless git wget curl -y
RUN python3 -m pip install torch torchvision torch-model-archiver torchserve==0.2.0
COPY densenet161.mar /model_store/
CMD torchserve --start --model-store model_store --models densenet161=densenet161.mar
EXPOSE 8080

I was able to create the image but I was not able to access that container when I tried to open the image and run the code it works

docker exec -it 4b126bd87f21 sh

# torchserve --start --ncs --model-store model_store --models densenet161.mar

The server is running. When I run the docker image it is not working. Docker container is running but I was not able to access the server.

I don't know what is the problem.

What kind of error do you get? If it is like not able to access host, then try adding --net host command when running the container using docker run — eroot163pi
– eroot163pi, Commented May 6, 2021 at 10:40

eroot163pi · Accepted Answer · 2021-06-22 04:26:46Z

2

I hope I understand the problem.

When you do docker run torchserve:local ...., by default it runs the CMD which is torchserve --start --model-store model_store --models densenet161=densenet161.mar but since the command runs in the background, your newly created docker container will immediately exit. Due to this same problem, i.e. to prevent docker exit it is possible to add tail -f /dev/null.

Look at the official docker entry point of torchserve https://github.com/pytorch/serve/blob/master/docker/dockerd-entrypoint.sh#L12 They tail it at the end to prevent docker exit

answered Jun 22, 2021 at 4:26

eroot163pi

1,8451 gold badge14 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

eroot163pi Over a year ago

@mohamed-fazil can you confirm?

eroot163pi Over a year ago

@mohamed-fazil if this solves the problem, please accept the solution

Irakli Salia Over a year ago

I came across this problem and also found out that using --foreground as a parameter with torchserve, runs it in foreground so the docker container doesn't exit. pytorch.org/serve/configuration.html

ganesh raj · Accepted Answer · 2021-06-16 12:56:57Z

0

My suggestion would be to rebuild the image.

Tip: Always give a name to the container you work with --name tag example: docker run --name container_name

Try to access with bash : docker exec -it 4b126bd87f21 bash

answered Jun 16, 2021 at 12:56

ganesh raj

411 bronze badge

Collectives™ on Stack Overflow

How to run a pytorch model server inside docker?

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related