Does Tensorflow's tf.while_loop automatically capture dependencies when executing in parallel?

Question

I am interested in implementing a Recursive Neural Network in Tensorflow, like what has been done in How can I implement a recursive neural network in TensorFlow?.

However, in his implementation, the parallel_iterations of the tf.while_loop statement was fixed to be 1. I fear that this might be too slow. Since the tree I am going to feed into tensorflow have parts that are not dependent on each other, I would hope that I could set parallel_iterations to a higher value. However, it is inevitable that there are some dependencies required in the tree I feed in as input to tensorflow, and I am afraid that setting it to higher value may break the dependency property.

So my question is, had Tensorflow's tf.while_loop automatically captured dependencies already, in order to only use paralleism on placed that are not dependent on each other?

The tensorflow documentation says the following:

For correct programs, while_loop should return the same result for any parallel_iterations > 0.

But I am not sure what they mean by "correct programs".

guy · Accepted Answer · 2017-11-14 15:19:27Z

4

You can.

according to this issue, ops will run in parallel once all their inputs nodes are computed:

while_loop implements non-strict semantics. An iteration can start as soon as one of the ops for this iteration is ready (i.e., all its inputs are available.) for execution. So a while_loop can easily have multiple iterations running in parallel. For example, for scan, even if the accumulated value is not available in a step, the step can still start and execute any ops that don't depend on the accumulated value.

so you shouldn't encounter any issues.

edited Nov 14, 2017 at 15:19

answered Nov 14, 2017 at 7:29

guy

3522 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Does Tensorflow's tf.while_loop automatically capture dependencies when executing in parallel?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related