OpenAI details the inference and harness optimizations behind GPT-5.6

OpenAI published an engineering account of GPT-5.6 efficiency on July 29. The article describes work across inference routing, GPU kernels, speculative decoding, caching, and the agent harness used by Codex and ChatGPT Work. It also says GPT-5.6 Sol in Codex helped identify, implement, and validate several of the production optimizations.

Efficiency comes from the complete serving path

At global routing, requests can be assigned using geography, capacity, and accelerator type. Within a cluster and model instance, context length, cache availability, batching, model sub-networks, and compute cores create further placement choices. OpenAI says analyzing production traffic exposed imbalance that broad heuristics had missed.

The team also used the model to rewrite Triton and Gluon kernels, with floating-point validation tooling checking correctness. For speculative decoding, a smaller draft model proposes tokens that the main model verifies in parallel. OpenAI reports that Sol designed experiments for its own draft model and monitored training. Workload-specific tuning then adjusted batching, sharding, and key-value cache management for different request shapes.

Agent harnesses have their own efficiency budget

A capable model can still waste work if an agent repeatedly expands context or breaks cacheable prompt prefixes. Builders should measure tokens by task phase, tool-result growth, cache-hit rate, repeated file reads, and compaction frequency rather than relying only on model-level tokens per second. Stable tool schemas and deliberate context replacement can matter as much as another inference optimization.

The performance numbers in OpenAI’s post describe its own models, workloads, and production system. Teams should reproduce the method on their traffic: define success first, compare end-to-end latency and cost, and validate numerical equivalence whenever an AI-generated kernel or scheduling change reaches production.