AI / ML

Amazon SageMaker HyperPod now tracks AMI versions across clusters to detect drift and applies security patches automatically without interrupting long-running training jobs. This update eliminates the previous manual, reactive patching process that risked altering bundled software like NVIDIA drivers or CUDA. The feature allows administrators to maintain secure, consistent infrastructure while reducing operational overhead.

• Gain visibility into AMI versions running across SageMaker HyperPod clusters to detect configuration drift.
• Automate security patching to avoid disrupting multi-day foundation model training workloads.
• Prevent accidental changes to critical bundled software such as NVIDIA drivers or CUDA during updates.
• Reduce manual operational burden by shifting from reactive to automated patch management.

Amazon SageMaker Inference now supports automatic container image caching to significantly reduce cold-start latency during scale-out events. By pre-pulling large generative AI container images (often 10 GB+) from ECR, new instances can start serving traffic immediately without waiting for full downloads. This feature specifically targets the bottleneck where every new instance previously had to pull the entire image, adding several minutes of delay.

• Scale-out latency drops by up to 50% as new instances bypass ECR pull times
• Large GenAI containers (10 GB+) are pre-cached locally before scaling occurs
• Eliminates multi-minute cold-start delays for deep learning framework images
• No code or configuration changes required to enable automatic caching