All news

AWS adds managed NVIDIA Dynamo to SageMaker HyperPod Inference

Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .

Developers can roughly double the output per H200 GPU in this test by splitting prefill and decode across nodes without configuring the network by hand.

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo: a managed add-on installs the Dynamo CRDs and its components, while HyperPod handles EFA networking, GPU scheduling and node replacement. Multi-node deployment, separate prefill (input request processing) and decode (response generation) stages, and KV-aware routing with the vLLM, SGLang and TensorRT-LLM engines are available. In AWS's benchmark on Qwen3-235B-A22B-FP8 (two p5en.48xlarge instances, 16 H200 GPUs) the disaggregated mode delivered 207 output tokens per second per GPU versus 101 for the combined mode, with P90 time to first token under 1 s. For Qwen3.5-397B-A17B a two-node deployment retained 99.1% of a single node's throughput; the GLM-5.3 model in FP8, according to the publication, does not fit into the 640 GB of memory on a single ml.p5.48xlarge.