What to Look for During Training and Testing
Last updated
Was this helpful?
Last updated
Was this helpful?
In addition to with your models, there are a number of statistics to watch during training and testing. These can be viewed in real-time via PerceptiLabs' and views respectively.
Below are common things to look for while training and testing different types of models.
Tip: Also be sure to check out our where we provide brief overviews of modeling and debugging.
Loss: if loss increases during validation, this means the model is overfitting.
Predictions per class: there should be one color per class, otherwise the model is mixing classes together. The model may need to be made more complex to rectify this.
Gradients: if gradients die (decrease to nothing) quick, try normalizing the input using batch normalization and/or try a different output activation function.
Prediction: if the predictions seem to contain a lot of noise then the model has not been trained enough yet or needs to be made more complex. If you are only predicting one's (i.e., a white image) follow the tips in the previous point.
Table: if test Dice is a lot worse than it is training and validation, the model is likely overfitting.
Gradients: of 0 indicate that the model is not training and gradients that grow infinitely mean that the model is changing too much. Note that usually the gradient in the last layer is sufficient to watch.
CPU/GPU usage: if resources aren't being utilized used as much as they should be, try larger batch sizes (this can be adjusted in the ).
Confusion matrix: if the shows that classes are mixed up, try building a more complex model.
Table: look closely at . Also if the test accuracy is a lot worse than it is for training and validation, the model is likely overfitting.
Image: see above.