All news

Nature study identifies a fractal cause of unavoidable irreproducibility in neural-network training

Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .

Even deterministic training can have a fundamental reproducibility limit, which matters when evaluating and comparing future models.

Andrew Ly and Pulin Gong linked unpredictable training outcomes to riddled basins of attraction: initialisations leading to different solutions are intertwined so that an arbitrarily close starting point can produce another outcome. Increasing the precision of initial parameters therefore does not eliminate differences even when random factors are controlled. The authors confirmed the mechanism in a minimal model and then trained approximately 500,000 small networks, including VGGNet, ResNet and BERT configurations on MNIST, CIFAR-10 and IMDB. In one illustrative experiment, halving the uncertainty error would require reducing the perturbation by almost 24 orders of magnitude. At the functional level, prediction differences plateaued at around 2.9% regardless of further precision increases. The work proposes measuring this limit rather than treating every irreproducible outcome as random noise or a computing error.

Sources 1