Reducing Learning Rate and Stability

• The standard algorithm, and a standard technique is to reduce the learning rate over time.
• So initially you move the centre by half, and those around by less.
• As time goes by, you multiply these values by a learning rate less than one.
• Keep reducing the learning rate.
• It's possible that without reducing the learning rate, the nodes will continue to jump around.
• If you reduce the learning rate gradually to zero, eventually, the nodes will stop moving.
• This works here but also in a lot of other systems.
• If you're having problems with your learning system stabilising, try reducing the learning rate over time.