2

I have a .Net 5 Core application which I am converting to a docker container(linux base). This application however calls python script to do some computation. The python script needs stix2 library, so I do an initial setup first to create a virtualenv and install stix2 library like this:

  pip install virtualenv
  virtualenv virtual_env
  virtual_env\Scripts\activate
  pip install stix2
  deactivate

When the python script is called, it reads some json, processes with stix2 library and stores the result in another json file in the same folder. The python script, input, output files reside in the ContentRootPath of the .net application. Currently I am using SlavaGu console launcher to run the python script from within the .net application (DoSomePythonProcess.cs):

using SlavaGu.ConsoleAppLauncher;
.
.
.
string cmdLine = @"/c <contentrootpath>\Python\virtual_env\Scripts\activate "+
                         @"&& python " + <AbsolutePathToPythonScriptFile> + " " + <JSONInputFilePath> + " " + <JSONOutputFilePath> + @"&& <contentrootpath>\Python\virtual_env\Scripts\Scripts\deactivate";
ConsoleApp.Result result = ConsoleApp.Run("cmd", cmdLine);

Now, I am trying to convert this application to docker. Currently, this is my dockerfile:

FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS base
RUN apt-get update -y && apt-get install python3-pip python3 -y && pip3 install stix2
WORKDIR /app
EXPOSE 5070

FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /src
COPY ["project.Web.csproj", "project.Web/"]
RUN dotnet restore "project.Web.csproj"
COPY . .
WORKDIR "/src/project.Web"
RUN dotnet build "project.Web.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "project.Web.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENV ASPNETCORE_URLS=http://*:5070
ENTRYPOINT ["dotnet", "project.Web.dll"]

Here are my questions:

How do I setup the virtualenv for the python script? Can it be done in dockerfile?

How can I call and execute the python script within this docker container?

Also it looks like the slavagu is not recognized when I am running the container. If that's the case, how can I call the python script within the .net application in a linux docker container?

I have searched online but I am not able to find anything relevant

Any help is much appreciated! Thanks in advance!

##UPDATE## Thanks to suggestions here, I was able to find a solution. Sharing it , in case someone needs it in future:

There is no change in dockerfile. The python will be installed in the container under /usr/bin/python3. You can try "which python3" in the container terminal to check this.

On how to call the python script within the .net file:

install cliwrap nuget package

using CliWrap;
using CliWrap.Buffered;

#eg, to check where python is installed
var pathToPy = await Cli.Wrap("which")
      .WithArguments(new[] { "python3" })
      .ExecuteBufferedAsync();
var resOutput = pathToPy.StandardOutput;
_logger.LogInformation(resOutput);
Improve this question
4
  • Tangentially, there is no reason to explicitly deactivate when you are done; the settings from the virtual environment will go away when the cmd instance exits. Commented Aug 1, 2021 at 5:39
  • 1
    Running a virtual environment inside a container seems superfluous anyway. Install the Python libraries you need system-wide within the container - which you already do; nobody else will be installing anything there anyway, so the isolation that the virtual environment provides from the rest of the container is just a needless complication. Commented Aug 1, 2021 at 5:42
  • Asking many questions in a single post is problematic; I can answer the parts about Python and Docker, but am grateful and unreasonably proud that I know nothing about the .NET parts. Your question is somewhat at risk of being closed as too broad. Can you split it up into multiple posts, each concentrating on one specific question? Obviously link between your questions for context. Commented Aug 1, 2021 at 5:46
  • @tripleee thank you for the suggestion! yes, you are right in pointing out that creating virtual environment is not needed in container, it is just for added security. For this usecase, as you said, it does not matter. As pointed out, I will try to split my questions over posts to keep the topic relevant. Thank you for the inputs! Commented Aug 2, 2021 at 9:39

2 Answers 2

Reset to default
6

Create a virtualenv in a python container, install dependencies inside this venv and copy it to your app container.

# prepare python env
FROM python:slim as python
# create a venv and install dependencies, use pip/poetry etc.
RUN python -m venv /venv
COPY requirements.txt .
# make sure you use the pip inside the venv
RUN /venv/bin/python -m pip install -r requirements.txt


# build your dotnet app
FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
# ...


# build app container
FROM mcr.microsoft.com/dotnet/aspnet:5.0

COPY --from=build /app/publish .

# copy the python environment we've prepared
COPY --from=python /venv /python

# other configurations
# ...  
ENTRYPOINT ["dotnet", "project.Web.dll"]

Then you can call /python/bin/python <args> from the app. Dependencies will work without problems, because they will be covered by python's default search paths.

Running python scripts from dotnet

Use Alexey Golub's CliWrap library when working with external processes.

var result = await Cli.Wrap("/python/bin/python")
    .WithArguments(new []{"my_python_script.py", "arg1", "arg2"})
    .WithWorkingDirectory("/app/python_scripts")
    .ExecuteBufferedAsync();

var output = result.StandardOutput;
// do something with the output
Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Hi! @abdusco, thank you for suggesting a very elegant solution! I was not aware of cliwrap! this helped a lot! Also want to highlight that: RUN /venv/bin/python -m pip install -r requirements.txt COPY --from=python /venv /python this does not create a python environment in my container.
I had to do it this way to create the python environment in my containers: FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS base RUN apt-get update -y && apt-get install python3-pip python3.7 -y && pip3 install stix2 Later, when i do #which python3, it points to /usr/bin/python3 in my container terminal
Hi @nv-2008, please let me know if your result image was large. If not, did you try any additional steps to reduce image size.
@everGreen final image size is determined by both python venv and dotnet app. make sure you're not installing unnecessary dependencies/extras. Use dive to inspect the layers and find out what's taking up a lot of space. github.com/wagoodman/dive
this isn't working for me with dotnet 7 as libPython is missing (when trying to use this for python2)
2

Though not a direct solution to your question, I would suggest that you reconsider the current design. A better approach may be deploy your python as a separate container with Flask or FastAPI to run as a API and then consume that API from your .Net Core container like an regular REST API. You wont need any custom application launcher or store intermediate result files.

For deployment, you can either chose to have it as a multi-container pod or deploy them as separate pods. So lets say, you add more computational functions into your Python API, you can scale it independently, if you are deploying them as separate pods.

Improve this answer

1 Comment

Hi! Thank you for the suggestion. I am bit hesitant for this approach as this process takes additional time and unnecessary python image overhead just for executing a single script. I was thinking of heading this way, if the original method fails.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.