Regularization is important technique for preventing overfitting problem while training a learning model.

Regularization

Regularization is a way to prevent overfitting and get a model generalizes the data. Overfitting problem usually caused by large weight value , so common way of regularization is to simply adds the bigger penalty as model complexity increases. Regularization parameter penalizes all the parameters except intercept so that it will decrease the importance given to higher terms and will bring the model towards less complex equation.

Regularization Figure 1: Example of cost function with regularization term.

L1/L2 Regularization

L1 Regularization

L1 Regularization also called Lasso Regression adds absolute value of magnitude of coefficient as penalty term to the loss function. If is zero, it will be the same with original loss function. If is very large, it will add too much weight and it will lead to under-fitting. So, it is important how is chosen.

L1 Figure 2: L1 regularization.

L2 Regularization (weight decay)

L2 Regularization also called Ridge Regression is one of the most commonly used regularization technique. It adds squared magnitude of coefficient as penalty term to the loss function. If is zero, it will be the same with original loss function. If is very large, it will add too much weight and it will lead to under-fitting. So, it is important how is chosen as well.

L2 Figure 3: L2 regularization.

Simply thinking that we add on the loss function and after computing gradient descent, will be updated by,

So, will work as penalty according to the amount of .

Dropout

Dropout [N. Srivastava et al., 2014] is one of the simplest and the most powerful regularization techniques. It prevents units from complex co-adapting by randomly dropping units from the network.

Dropout

Figure 4: Comparison between network with dropout and without dropout.

  • In the training stage
    • It randomly drops units using dropout rate
    • It has the effect of sampling a large number of different thinned networks.
    • Dropout rate: probability that a unit will be retained.
      • Optimal dropout rate is suggested around 0.5 in hidden layers and close to 1 for in input layer.
      • As it is a hyperparameter, it can be decided empirically.
  • In the inference stage
    • A single unthinned network is used (no dropout applied)
    • Weights of the unthinned network has to be multiplied by the dropout rate to keep the output the same as training stage.
    • It approximates the effect of averaging the predictions of all thinned networks from the training process. (as ensemble learning)
  • Drawbacks of dropout
    • It increases training time because of noisy parameter updates caused when each training step tries to train a different architecture.

Label Smoothing

Most datasets have some amount of mistakes in the labels and it makes minimizing cost function be harsh. One way to handle this is to explicitly model the noise on labels. This is done through setting a probability for which it thinks the labels are correct. This probability is easily incorporated into the cross entropy cost function anlytically. Label smoothing is an example of this.

  • Output vectors provided
  • Softmax output form
  • Label smoothing replaces the label vector with
    • It prevents the pursuit of hard probabilities without discouraging correct classification.
  • Label smoothing estimates the marginalized effect of label noise during training.
    • When the prior label distribution is uniform, label smoothing is equivalent to adding the KL divergence between the uniform distribution and the network’s predicted distribution to the negative log-likelihood.

Examples

References