Tensorflow-gpu issue (CUDA runtime error: device kernel image is invalid)

Question

I have a python virtual environment (conda) where I’ve installed CUDA toolkit 10.1.243 and tensorflow-gpu 2.3.0rc0. My CUDA driver is 11.0.

In order to test if tensorflow was installed to GPU correctly, I ran a series of commands from within the venv:

tf.test.is_built_with_cuda()

True

tf.config.list_physical_devices(‘GPU’)

Found device 0 with properties: pciBusID: 0000:01:00.0 name: Quadro M2000M computeCapability: 5.0 [PhysicalDevice(name=’/physical_device:GPU:0’, device_type=‘GPU’)]

python -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000,1000])))"

tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid

I am not sure how to troubleshoot this. I have a feeling that it is related to modifying the compilation such that tensorflow supports the compute capability of my device (5.0), but I am not sure how to proceed. Thank you!!

Ugurcan · Accepted Answer · 2020-08-04 08:41:40Z

5

I just had the same problem. I downgraded the Tensorflow2.3 version to 2.2 with following command.

pip install --upgrade tensorflow==2.2

It is working now but very slow.

answered Aug 4, 2020 at 8:41

Ugurcan

4162 gold badges7 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Clock ZHONG · Accepted Answer · 2020-08-30 04:48:04Z

4

According to this github issue's explanation, Google Tensorflow engineering team already discarded the support for some older version's GPUs: https://github.com/tensorflow/tensorflow/issues/41990

I believe your GPU is in those lower versions GPU family. So downgrading your TF from 2.3. to 2.2 is a solution. And TF engineers suggest us to build the TF2.3 by ourselves and and change its building configuration scripts to enable lower versions GPUs supoort, but Google TF team doesn't confirm it could work, and no resposbility to fix any problem we'll encounter.

edited Aug 30, 2020 at 4:48

answered Aug 30, 2020 at 4:21

Clock ZHONG

1,0501 gold badge11 silver badges28 bronze badges

2 Comments

Sidharth kaushik Over a year ago

I got rtx 2080, it's pretty weird they are not supporting this device.

Clock ZHONG Over a year ago

@SidKaushik Yes, it's weird, but I finally totally fixed this problem by switching my environment from TF to pytorch, TF is really a failure SW stack from google, at least it frustrated me many many times.

Prajot Kuvalekar · Accepted Answer · 2021-04-21 09:13:53Z

1

i recommend pip install tensorflow-gpu==2.2

answered Apr 21, 2021 at 9:13

Prajot Kuvalekar

6,7943 gold badges28 silver badges34 bronze badges

2 Comments

jochen Over a year ago

Maybe add more explanations to explain how this answers the question?

Prajot Kuvalekar Over a year ago

what i want to say is instead of just pip install tensorflow==2.2 use pip install tensorflow-gpu==2.2

Collectives™ on Stack Overflow

Tensorflow-gpu issue (CUDA runtime error: device kernel image is invalid)

3 Answers 3

Comments

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related