In the previous segment, you learnt how gradient descent is used in learning neural networks. The training is done using backpropagation. We also discussed a detailed numerical example of backpropagation for a single input in a simple neural network.
We took the following steps when passing an input through the network:
- Forward propagation of the input through the network with random initial values for weights and biases
- Making a prediction and computing the overall loss
- Updating model parameters using backpropagation i.e., updating the weights and biases in the network, using gradient descent
- Forward propagation of the input through the network with updated parameters leading to a decrease in the overall loss
- Repeat the process until the optimum values of weights and biases are obtained such that the model makes acceptable predictions
The pseudocode/pseudo-algorithm is given as follows:
1: Initialise with the input
Forward Propagation
2: For each layer, compute the cumulative input and apply the non-linear activation function on the cumulative input of each neuron of each layer to get the output.
3: For classification, get the probabilities of the observation belonging to a class, and for regression, compute the numeric output.
4: Assess the performance of the neural network through a loss function, for example, a cross-entropy loss function for classification and RMSE for regression.
Backpropagation
5: From the last layer to the first layer, for each layer, compute the gradient of the loss function with respect to the weights at each layer and all the intermediate gradients.
6: Once all the gradients of the loss with respect to the weights (and biases) are obtained, use an optimisation technique like gradient descent to update the values of the weights and biases.
Repeat this process until the model gives acceptable predictions:
7: Repeat the process for a specified number of iterations or until the predictions made by the model are acceptable.
This is a consolidated algorithm for training a neural network.
You will now understand this process through code using TensorFlow. In the next few segments, you will understand the basics of TensorFlow required to implement the forward propagation and backward propagation steps.
Report an error