OpenAI traces rare Rockset crashes to hardware and an 18-year-old libunwind race

OpenAI reported the resolution of rare crashes in its Rockset data infrastructure on June 30. What initially looked like one impossible C++ stack-corruption bug turned out to be two unrelated failures: silent CPU corruption on one host and a race in GNU libunwind that had existed for 18 years. The investigation succeeded after the team stopped treating a few core dumps as isolated specimens and built a dataset across the full crash population.

Similar symptoms concealed different causes

The affected processes sometimes returned to invalid addresses or showed a stack pointer displaced by eight bytes. Conventional log queries could not classify the crashes reliably because the stack traces themselves were damaged. Manual inspection also reinforced an early assumption that every case shared one root cause.

The team extracted comparable facts from core files, cleaned and grouped them, and looked for differences across hosts, regions, code paths, and stack state. That population view isolated a cluster associated with one machine, leading to the hardware finding. Removing those cases made a second pattern visible: an exception-unwinding race whose tiny instruction window could corrupt the return path. OpenAI says it fixed the upstream libunwind issue as part of the work.

Preserve evidence before deciding the hypothesis

For low-frequency native crashes, logs alone may erase the distinguishing evidence. Engineering teams should retain symbolized cores under strict access controls, record build and host identities, normalize comparable stack and register features, and make grouping criteria reproducible. Hardware should remain in the hypothesis set even when failures appear across a fleet.

The case also shows why debugging datasets need provenance. Sampling, upload failures, symbol mismatches, and duplicate crash loops can bias the apparent population. A good crash pipeline lets investigators revise classifications without losing the raw artifact that produced them.