Common Training Issues
- Gradients: gradients of 0 indicate that the model is not training and gradients that grow infinitely mean that the model is changing too much. Note that it's usually sufficient to focus on the gradient in the last layer.
- Loss: if loss increases during validation, this means the model is overfitting.
- Predictions per class: there should be one color per class, otherwise the model is mixing classes together. The model may need to be made more complex to rectify this.
- Gradients: if gradients die (i.e., decrease to nothing) quickly, try normalizing the input using batch normalization and/or try a different output activation function.
- Prediction: if the predictions seem to contain a lot of noise then the model has not been trained enough yet or needs to be made more complex. If you are only predicting one's (i.e., a white image) follow the tips in the previous point.