Sakana AI trains a 1,000-layer neural network without backpropagation, possibly the way the brain learns
Prepared by Sintelligent with the help of SI. Publication time restored from the build log. . Updated: .
A real alternative to the main way neural networks are trained is emerging, a step toward SI that learns locally, like the brain, and could run on entirely different hardware.
Backpropagation requires a global pass through the whole network, which the brain, as far as is known, does not have. The PC-ALM method supplements "predictive coding" with Lagrange multipliers: each layer learns only from the local signals of its neighbors. Networks up to 1,000 layers deep trail conventional training on MNIST by only about 2 percentage points, and gradient similarity to backpropagation rose from 0.60 to 0.91. The JAX code is open under the MIT license and has so far been tested only on small datasets.
Sources 2