Tensorflow comes with eager execution enabled by default which makes debugging much more straightforward. What does eager execution mean, and why it’s not always so eager?
What is eager execution? The Tensorflow documentation has a great guide explaining this feature. To see the difference, try
>>> t = tf.constant(3) >>> print(t * t) # Executed eagerly tf.Tensor(9, shape=(), dtype=int32) >>> tf.compat.v1.disable_eager_execution() >>> t = tf.constant(3) >>> print(t * t) # Executed lazily, result 9 not available Tensor("mul_1:0", shape=(), dtype=int32)
Eager execution is the opposite of lazy evaluation, where operations such as
t * t are not actually executed. With lazy evaluation, the operations are recorded for later
execution. Lazy evaluation makes it possible, for example, to reuse the same
operations on a new batch of data. Lazy evaluation is good for performance but not for
debugging. You see in the above example that the intermediate result
9 is not
available with lazy evaluation. Without eager execution, you see the result of
t * t has just the symbolic value
With eager execution enabled, you can use your standard tools to debug your
code. Let it be a
breakpoint() or even just a simple
print(). You can
inspect your tensors at runtime to see where the code goes wrong.
Eager execution by default, so everything’s fine?
Well, not exactly. If you have worked with
keras and wrote your own
there’s a fair chance you had to debug the training step. However, this doesn’t
The training in keras, e.g., the method
train_step() is called from within a
This, in turn means, the whole training loop is evaluated lazily. The
benefits of eager execution for debugging are traded in favour of performance.
There is a way out. Pass
run_eagerly=True to your model’s
This will make the training step run in eager execution mode. This gives you back the ability to debug your code with standard tools. Once your code works properly with eager execution, you want to make sure to switch back to lazy evaluation for performance and portability reasons.