OpenAI turns tax-agent corrections into evals and Codex tasks

OpenAI and Thrive Holdings described a production improvement loop for tax agents on May 27. Their Tax AI system captures practitioner corrections and the trace that produced each result, converts recurring failures into targeted evaluations, and gives Codex scoped engineering work to improve the product. The announcement frames self-improvement as an instrumented software process, rather than a model silently rewriting itself.

Corrections become structured evidence

A changed tax field is ambiguous on its own: it may reveal extraction failure, mapping error, unsupported workflow, or an acceptable practitioner override. The system therefore preserves source documents, extracted values, provenance, downstream submissions, and expert corrections. Engineers and domain specialists classify a failure, turn it into a reproducible eval case, and define a measurable target. Codex can then investigate, propose a change, and run targeted plus regression evaluations.

This creates three distinct control points. Practitioners decide which errors matter. Product traces make the failure inspectable. Evals determine whether a candidate change fixes the case without harming established behavior. Human review remains part of preparing the final return.

The pattern transfers, with conditions

Teams can apply the loop outside tax when experts can correct outputs and the system can retain enough provenance to reproduce them. Start by defining the unit of correction, the allowed use of production data, redaction and retention rules, and who may promote a correction into an eval. Separate product noise from model failure before automating task creation.

The hardest engineering work is the evidence pipeline. Without stable traces, versioned prompts and tools, held-out regression cases, and an approval boundary, an automated loop can optimize toward mislabeled feedback. The practical contribution here is the closed-loop design: production supplies candidate failures, experts supply meaning, and repeatable evaluations govern changes.