SageMaker AI adds GPU partitioning for real-time endpoints

AWS announced GPU partitioning for Amazon SageMaker AI real-time inference endpoints on October 30, 2026. The feature allows teams to place several models on one GPU, with partitions sized for individual workloads. AWS identifies small and mid-size models—including fine-tuned variants, embedding models, rerankers and classifiers—as target workloads in its release announcement.

How endpoint partitioning works

The feature uses NVIDIA Multi-Instance GPU (MIG) technology. According to AWS, it can divide a GPU into as many as seven isolated partitions, each with its own memory and compute. AWS describes the isolation between partitions as hardware-enforced.

Teams configure GPU partitioning when creating an endpoint that hosts inference components, then place models on appropriately sized partitions. Models continue to be invoked by name, so AWS says application code does not need to change. The deployment configuration changes while the model-invocation interface remains the same, according to the announcement.

Announced availability

AWS lists GPU partitioning as available on P4d, P5, G7 and G7e instances in seven regions:

• US East (N. Virginia)
• US East (Ohio)
• US West (Oregon)
• Asia Pacific (Singapore)
• Asia Pacific (Tokyo)
• Europe (Frankfurt)
• Europe (Ireland)

These are the instance families and regions specified in the October 30 announcement.

What engineers should evaluate

Treat partitioning as a consolidation option, not a guarantee of lower costs. Before choosing a layout, measure each model’s peak GPU memory demand and latency under realistic concurrency. Validate memory fit and service-level targets separately, then compare the capacity and cost required by candidate configurations.

The useful sizing question is no longer just “Which GPU can host this model?” but “Which partition allocation can sustain this workload?” Evaluate a layout by whether it meets performance targets with less provisioned capacity—not simply by how many models it packs onto an accelerator.