Regularization in TensorFlow using Keras API
Regularization is a technique for preventing over-fitting by penalizing a model for having large weights. There are two popular regularization parameters: L1 and L2. L1 is called Lasso, and L2 is called Ridge. Both of these are usually introduced when you learn Linear Regression.
In the past, when working with TensorFlow Estimators, regularization had to be implemented by the optimizer. This is not the case with Keras. Let’s look at a simple Keras model.
The code above is a simple model. We can proceed to add regularization to the layers. In the code below, we will add L2 regularization.
We add regularization by assigning to the kernel_regularizer parameter. To add L2 regularization, we pass keras.regularizers.l2(). This takes one parameter, which is the regularization strength l. We can pass an L1 regularizer by simply replacing l2() with l1().
There are situations in which we would like to use both L1 and L2 regularization. This is frequently called Elastic Net. We would use l1_l2() and this time around, pass two parameters, like so.
The Dense layer takes three regularizers, which all default to None. These are kernel_regularizer(applied to weights), bias_regularizer(applied to bias unit), and activity_regularizer(applied to layer activation).
Good luck tuning that model.
