Microsoft.ML C#: GPU not found in K8s/Docker container

I have created a .NET app that uses Microsoft.ML.OnnxRuntime.Gpu for interference. Now I'm trying to integrate it with Azure Kubernetes.

We have made the setup with Tesla T4 GPU and we confirmed it's visible:

So we know that T4 is visible under ID = 0.

This is basically my code that works locally on my Windows machine:

MLContext _mlContext = new();

var estimator = _mlContext.Transforms.ApplyOnnxModel(
    modelFile: _modelFile,
    inputColumnNames: _inputColumnNames,
    outputColumnNames: _outputColumnNames,   
    gpuDeviceId: gpuId
);

but when deploying to ACR, K8s etc., we are getting an exception:

System.InvalidOperationException: GPU with ID 0 is not found.

Bunch of information that I think may help.

• My local machine nvidia-smi log:

• libraries used:

      <PackageVersion Include="Microsoft.ML" Version="3.0.1" />
      <PackageVersion Include="Microsoft.ML.ImageAnalytics" Version="3.0.1" />
      <PackageVersion Include="Microsoft.ML.OnnxRuntime" Version="1.20.1" />
      <PackageVersion Include="Microsoft.ML.OnnxRuntime.Gpu" Version="1.20.1" />
      <PackageVersion Include="Microsoft.ML.OnnxTransformer" Version="3.0.1" />
  

• Image used in our Dockerfile:

  nvidia/cuda:12.3.2-runtime-ubuntu22.04
  

• My local nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver

  Copyright (c) 2005-2024 NVIDIA Corporation

  Built on Wed_Oct_30_01:18:48_Pacific_Daylight_Time_2024

  Cuda compilation tools, release 12.6, V12.6.85

  Build cuda_12.6.r12.6/compiler.35059454_0