number of parameters in neural networks used in practice can be pretty large. Particularly, in images.
Resolution
- Early Stopping
- Add data
- fake data if getting data is difficult
- eg Images can be translated slightly, rotated slightly, change of brightness, etc
- Adversarial Training
- fake data if getting data is difficult
- Regularization
- L1 L2 Regularization can be used in neural networks too.
- Dropout
- Gradient Clipping
- Hard constraints on weights
- inject noise into the system
- Unsupervised Pre-Training
- enforce sparsity in the network