How to change Learning rate in Tensorflow after a batch end?
I need to create a class for searching the optimal learning rate for the model, incrementing the value of the learning rate by %5 in each batch. I have see the on_train_batch_end() callback but i am not able to set it.
Reduce Learning rate on plateau only adjust the learning rate at the end of an epoch. It does not due so at the end of a batch. To do that you need to create a custom callback. If I understand you correctly you want to reduce the learning rate by 5% at the end of each batch. The code below will do that for you. In the callback model is the name of your compiled model. freq is an integer that determine how often the learning rate is adjusted. If freq=1 it will be adjusted at the end of every batch. If freq=2 it will be adjust on every other batch etc. reduction_pct is a float. It is the percent the learning rate will be reduced by. Verbose is a boolean. If verbose=True, a print out will occur each time the learning rate is adjusted showing the LR used for the just completed batch and the LR that will be used for the next batch. If verbose=False no printout is generated.
class ADJLR_ON_BATCH(keras.callbacks.Callback): def __init__ (self, model, freq, reduction_pct, verbose): super(ADJLR_ON_BATCH, self).__init__() self.model=model self.freq=freq self.reduction_pct =reduction_pct self.verbose=verbose self.adj_batch=freq self.factor= 1.0-reduction_pct * .01 def on_train_batch_end(self, batch, logs=None): lr=float(tf.keras.backend.get_value(self.model.optimizer.lr)) # get the current learning rate if batch + 1 == self.adj_batch: new_lr=lr * self.factor tf.keras.backend.set_value(self.model.optimizer.lr, new_lr) # set the learning rate in the optimizer self.adj_batch +=self.freq if verbose: print('\nat the end of batch ',batch + 1, ' lr was adjusted from ', lr, ' to ', new_lr)
Below is an example of use of the callback with the values I believe you wish to use
model=your_model_name # variable name of your model reduction_pct=5.0 # reduce lr by 5% verbose=True # print out each time the LR is adjusted frequency=1 # adjust LR at the end of every batch callbacks=[ADJLR_ON_BATCH(model, frequency, reduction_pct, verbose)]
Remember to include callbacks=callbacks in model.fit. Below is a sample of the resulting printout starting with an LR of .001
at the end of batch 1 lr was adjusted from 0.0010000000474974513 to 0.0009500000451225787 1/374 [..............................] - ETA: 1:14:55 - loss: 9.3936 - accuracy: 0.3333 at the end of batch 2 lr was adjusted from 0.0009500000160187483 to 0.0009025000152178108 at the end of batch 3 lr was adjusted from 0.0009025000035762787 to 0.0008573750033974647 3/374 [..............................] - ETA: 25:04 - loss: 9.1338 - accuracy: 0.4611 at the end of batch 4 lr was adjusted from 0.0008573749801144004 to 0.0008145062311086804
Answered By – Gerry P