登录
TensorFlow 1 version
|
View source on GitHub
|
A LearningRateSchedule that uses a cosine decay schedule with restarts.
Inherits From: LearningRateSchedule
tf.keras.experimental.CosineDecayRestarts(
initial_learning_rate, first_decay_steps, t_mul=2.0, m_mul=1.0, alpha=0.0,
name=None
)
See [Loshchilov & Hutter, ICLR2016], SGDR: Stochastic Gradient Descent with Warm Restarts. https://arxiv.org/abs/1608.03983
When training a model, it is often recommended to lower the learning rate as
the training progresses. This schedule applies a cosine decay function with
restarts to an optimizer step, given a provided initial learning rate.
It requires a step value to compute the decayed learning rate. You can
just pass a TensorFlow variable that you increment at each training step.
The schedule a 1-arg callable that produces a decayed learning rate when passed the current optimizer step. This can be useful for changing the learning rate value across different invocations of optimizer functions.
The learning rate multiplier first decays
from 1 to alpha for first_decay_steps steps. Then, a warm
restart is performed. Each new warm restart runs for t_mul times more
steps and with m_mul times smaller initial learning rate.
Example usage:
first_decay_steps = 1000
lr_decayed_fn = (
tf.keras.experimental.CosineDecayRestarts(
initial_learning_rate,
first_decay_steps))
You can pass this schedule directly into a tf.keras.optimizers.Optimizer
as the learning rate. The learning rate schedule is also serializable and
deserializable using tf.keras.optimizers.schedules.serialize and
tf.keras.optimizers.schedules.deserialize.
Returns | |
|---|---|
A 1-arg callable learning rate schedule that takes the current optimizer
step and outputs the decayed learning rate, a scalar Tensor of the same
type as initial_learning_rate.
|
Args | |
|---|---|
initial_learning_rate
|
A scalar float32 or float64 Tensor or a Python
number. The initial learning rate.
|
first_decay_steps
|
A scalar int32 or int64 Tensor or a Python
number. Number of steps to decay over.
|
t_mul
|
A scalar float32 or float64 Tensor or a Python number.
Used to derive the number of iterations in the i-th period
|
m_mul
|
A scalar float32 or float64 Tensor or a Python number.
Used to derive the initial learning rate of the i-th period:
|
alpha
|
A scalar float32 or float64 Tensor or a Python number.
Minimum learning rate value as a fraction of the initial_learning_rate.
|
name
|
String. Optional name of the operation. Defaults to 'SGDRDecay'. |
Methods
from_config
@classmethodfrom_config( config )
Instantiates a LearningRateSchedule from its config.
| Args | |
|---|---|
config
|
Output of get_config().
|
| Returns | |
|---|---|
A LearningRateSchedule instance.
|
get_config
get_config()
__call__
__call__(
step
)
Call self as a function.