All news

AWS proposes three checks for detecting silent GPU errors during model training

Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .

Long model training runs will have a way to stop a rare error on one GPU before it silently corrupts the entire model and wastes weeks of computation.

Silent data corruption occurs when a GPU returns a plausible but incorrect result without a hardware error; in distributed training, it enters the gradients and weights of every replica. AWS added three checks to a PyTorch DistributedDataParallel loop: detection of non-finite gradients, cross-replay of a microbatch on a GPU on another node with comparison of losses and gradient fingerprints, and comparison of weight fingerprints after the optimizer step. When a check fires, the system identifies the suspect node, does not apply the corrupted step, rolls back to a checkpoint, and hands the node to SageMaker HyperPod for replacement. The method was tested with software-injected faults on a cluster of 32 Nvidia H100 GPUs and repeated on the minimum three-instance configuration. The code is available on GitHub.