As part of learning about Neural Networks the topic of computation graphs comes up. What is a computation graph? Basically it’s a way of separating a computation into multiple steps. Instead of completing the whole computation at once you can follow the steps in the computation graph. Why is that so great? Because sometimes you have many computations that share the same first steps but behave differently in the second step. By using a computation graph you can avoid doing the same computation more than necessary.
Let’s do a simple example. We have the following formula:
J(a, b, c) = 3(a + bc)
Now, we can split this up into multiple steps by using the following definitions:
u = bc v = a + u
This gives us the formula:
J(a, b, c) = 3*v
The forward pass would calculate
J by first calculating
The backward pass is often used when doing derivation. Here you can use the chain rule of derivation stating that
dJ/da = dJ/dv * dv/da.
dJ/dv is equal to three always, for
c. This means that we only need to do this computation once.