# What should CI upload when a failed job cannot be reproduced locally? > A failed CI job should upload a small, predictable diagnostic bundle, not a giant copy of the workspace. Canonical URL: https://www.devobs.io/articles/qa-ci-failure-artifact-contract/ By: Jonah Reed Published: 2023-11-06T16:02:08.273Z Updated: 2026-09-06T08:31:04.426Z Section: Architecture A failed CI job should upload a small, predictable diagnostic bundle, not a giant copy of the workspace. In practice, that means failure-only artifacts containing structured test results, the exact command that ran, toolchain and dependency versions, runner details, bounded logs, and framework-specific evidence such as Playwright traces. GitHub Actions artifacts are intended for build and test output used in debugging, and they support named uploads with retention controls through [GitHub Actions artifact documentation](https://docs.github.com/en/actions/tutorials/store-and-share-data). [Playwright Trace Viewer](https://playwright.dev/docs/trace-viewer) is a good example of why targeted evidence matters. ## What should the artifact contract include? Use one contract for every failing job so engineers know where to look. The minimum set is: - a machine-readable test report such as JUnit, NUnit, or pytest XML - a short human-readable failure summary - the exact invoked command, including flags and selected test target - runner metadata: OS, image, architecture, container image, and whether the runner was hosted or self-hosted - toolchain metadata: language runtime, package manager, compiler, browser version, and lockfile digest - bounded logs from the failing step, not unbounded full-job output - framework evidence when available: screenshots, crash dumps, coverage fragments, or traces GitHub explicitly documents artifacts as a way to store output for “debugging failed tests or crashes” and lets you upload multiple named files or directories, exclude paths, and set `retention-days` in [the artifacts guide](https://docs.github.com/en/actions/tutorials/store-and-share-data). ## Why upload metadata as well as logs? Because many “cannot reproduce locally” failures are environment mismatches, not code mysteries. For Python, GitHub recommends using `setup-python` because it “ensures consistent behavior across different runners and different versions of Python” in [the Python workflow guide](https://docs.github.com/en/actions/tutorials/build-and-test-code/python). That same idea applies to diagnostics: if the artifact does not record the resolved runtime and tools, you are missing the first thing investigators need. Include the enforced ordering boundary too. If one step generated inputs that a later step consumed, record that sequence in the summary. A preflight check alone is not enough; investigators need to know which command completed before another began. ## How large should failure artifacts be? Keep them deliberately bounded. Upload artifacts only on failure, compress text outputs, and cap noisy logs to the relevant tail plus a small context window. Set short retention for routine failures and longer retention for flaky or release-blocking runs. GitHub supports custom artifact retention periods in [the same artifact documentation](https://docs.github.com/en/actions/tutorials/store-and-share-data). Do not upload secrets, full environment dumps, or whole home directories. Redact tokens, cookies, authorization headers, and private keys before writing files. Prefer allowlists over blocklists: explicitly write the fields you need instead of dumping everything and hoping redaction catches it. ## What does a concrete failed-run bundle look like? For a [browser test](https://www.devobs.io/articles/flaky-browser-test-waits/) job: 1. Run tests normally. 2. On failure, write `failure-summary.md` with the failed suite, exact command, commit SHA, and runner image. 3. Export `junit.xml`. 4. Save `versions.txt` with Node, package manager, browser, and OS details. 5. Save the last 2,000 lines of the relevant test log. 6. Upload Playwright `trace.zip` and screenshots. Playwright documents traces as “a great way for debugging your tests when they fail on CI” in [Trace Viewer docs](https://playwright.dev/docs/trace-viewer). 7. Upload all of the above as one named artifact such as `failed-e2e-linux-chrome`. This does not apply unchanged to jobs handling regulated data or proprietary datasets. In those cases, keep the same contract shape but replace raw evidence with sanitized summaries, hashes, and reproducible references to protected storage. ## Which artifacts make failed CI jobs debuggable? The ones that answer four questions quickly: what failed, under which exact environment, with which inputs, and with what nearby evidence. If an uploaded bundle cannot answer those, it is too thin. ## What should the team verify before adopting this workflow? Verify that failures trigger artifact upload reliably, retention matches investigation timelines, redaction is tested, and artifact names are stable enough for automation and incident links. Next step: define a repository-wide `ci-failure-artifact-contract` document and enforce it in shared workflow templates so every failed job leaves the same minimum evidence. Reviewed: 2026-09-05 ## Source references - - -