Using multiple CPU Cores with Tensorflow

Question

I am running a tensorflow training on a Linux machine with 4 cores. When checking the cpu utilization with htop, only one core is fully utilized, whereas the others are utilized only with ~15% (image below shows a screenshot of htop).

How can I make sure TF is using all CPUs to full capacity?
I am aware of this issue "Using multiple CPU cores in TensorFlow" - how to make it work for Tensoflow 2?

.

I am using the following code to generate the samples:

class WindowGenerator():

    def make_dataset(self, data, stride=1):
        data = np.array(data, dtype=np.float32)
        ds = tf.keras.preprocessing.timeseries_dataset_from_array(
            data=data,
            targets=None,
            sequence_length=self.total_window_size,
            sequence_stride=stride,
            shuffle=False,
            batch_size=self.batch_size,)
      
        ds = ds.map(self.split_window)
      
        return ds


    @property
    def train(self):
        return self.make_dataset(self.train_df)
    
    @property
    def val(self):
        return self.make_dataset(self.val_df)
    
    @property
    def test(self):
        return self.make_dataset(self.test_df, stride=24)

I'm using the following code to run the model training. sampleMgmt is of Class WindowGenerator. early_stopping defines the training termination criteria.

history = model.fit(sampleMgmt.train, epochs=self.nrEpochs,
                  validation_data=sampleMgmt.val,
                  callbacks=[early_stopping],
                  verbose=1)

0x6d64 · Accepted Answer · 2021-03-12 15:52:59Z

0

Have you tried this?

config = tf.ConfigProto(device_count={"CPU": 8})
    with tf.Session(config=config) as sess:

(source)

edited Mar 12, 2021 at 15:52

0x6d64

1532 silver badges9 bronze badges

answered Mar 12, 2021 at 9:32

Bhargav

4,9012 gold badges12 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

vfromate Over a year ago

I'm not sure how to use the sess object to start training. (Actually I thought that was a TF1.x syntax)

vfromate Over a year ago

In the link you referenced to, Yaroslav is mentioning that "By default cpu:0 represents all cores available to the process.". Does this mean, that it is perfectly normal, that htop only shows high load in one cpu, and not all of them?

vfromate · Accepted Answer · 2021-03-15 15:58:03Z

0

In the end, I did two things outside my code to make TF use all CPUs availible.

I used a SSD instead of a HDD
I used the Tensorflow Enterprise Image, provided in Google Cloud, instead of the Deep Learning Image

answered Mar 15, 2021 at 15:58

vfromate

418 bronze badges

Collectives™ on Stack Overflow

Using multiple CPU Cores with Tensorflow

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related