Amazon unveils a single inference gateway across SageMaker HyperPod clusters and regions
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
A single cluster failure no longer requires manual switching: the gateway itself sends traffic to healthy clusters, and prompts are not stored.
On November 20 Amazon introduced SageMaker HyperPod Global Inference Gateway, a managed gateway with a single entry point. It routes inference requests across all registered HyperPod clusters in one or more AWS regions. A cluster is selected by four signals with configurable weights: free KV cache and GPU memory (0.20), queue depth (0.25), region proximity (0.40) and error rate (0.15). If a cluster fails, traffic is automatically moved to healthy ones with no changes on the client side. Authentication uses SigV4 and IAM, traffic goes only through AWS PrivateLink, and the gateway does not log or store prompts and responses; OpenAI-compatible requests are supported. The service is available in 13 AWS regions, including Virginia, Ohio, Oregon, São Paulo, Frankfurt, Ireland, London, Stockholm, Tokyo, Mumbai, Singapore and Sydney. In beta tests, Bagelcode's gateway served two regions through a single entry point.
Sources 1