How to load models trained on GPU into CPU (system) memory?

Question

I trained a model on a GPU and now I am trying to evaluate it on CPU (GPU is being used for a different train run). However when I try to load it using:

with tf.device('/cpu:0'):
    model = tf.keras.models.load_model('models/resnet50_untrained.h5', compile=False)

I am getting a CUDA_ERROR_OUT_OF_MEMORY:

2019-10-17 09:25:23.088408: W tensorflow/compiler/xla/service/platform_util.cc:256] unable to create StreamExecutor for CUDA:0: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY: out of memory; total memory reported: 7981694976
2019-10-17 09:25:23.088505: F tensorflow/stream_executor/lib/statusor.cc:34] Attempting to fetch value instead of handling error Internal: no supported devices found for platform CUDA

(I tried also setting compile=True with the same result.)

It seems that the model is being loaded into GPU which is already used by another instance. How can I force keras/tensorflow to load it into system memory and execute it on CPU?

Rishabh Sahrawat · Accepted Answer · 2019-10-17 09:47:42Z

4

Can you define everything inside with tf.device('/cpu:0'): except the libraries import part and test.

If that doesn't work then create a virtual envoironment and install normal tensorflow not the gpu version and then try it. If it is still OOM error, then it is from the CPU being utilized and it has not enough memory to load this trained model.

answered Oct 17, 2019 at 9:47

Rishabh Sahrawat

2,5051 gold badge19 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

btel Over a year ago

Thanks for the suggestion. I tried to load the model into virtualenv without tensorflow-gpu and it worked! However, I find it suprising that the tf.device('/cpu:0') didn't do the trick. The code above is literally everything I tried to run.

Rishabh Sahrawat Over a year ago

Sometimes, because of some dependencies it just doesn't work how we expect it. But I am glad, it is working. Cheers!

Rishabh Sahrawat Over a year ago

Also please consider accepting it as the answer. ^^

btel Over a year ago

I would prefer to find a solution that does not require switching the virtualenvs. I installed everything in clean enviroment, so it's unlikely to be dependency problem.

Collectives™ on Stack Overflow

How to load models trained on GPU into CPU (system) memory?

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related