# How do we verify software artifact provenance at deployment without turning every environment exception into an unsigned bypass? > A practical deployment pattern for enforcing provenance with digest-bound attestation checks and auditable break-glass handling. Canonical URL: https://www.devobs.io/articles/qa-ge50-provenance-deployment-verification-exceptions/ By: Owen Park Published: 2024-01-13T05:26:35.110Z Updated: 2026-09-06T08:31:04.426Z Section: Architecture Verify provenance at deployment admission against immutable digests, and treat exceptions as a separate emergency path rather than a skipped check. The safe pattern is: resolve tags to digests, fetch attestations for that digest, verify signer identity and issuer, require the right predicate type, evaluate policy over the attestation payload, and deny on missing or malformed proofs. If an environment can bypass that path with a normal deploy flag, you do not have provenance enforcement. ## What should the deployment decision verify? The deployment question is simple: does this exact artifact digest have the attestations required by policy from identities you trust? That is why digest binding matters more than tag naming. The [Sigstore Cosign verification docs](https://github.com/sigstore/docs/blob/main/content/en/cosign/verifying/verify.md) state that Cosign signature payloads include the container image digest and that Cosign validates that digest during verification by default. For provenance, require the SLSA provenance predicate type, `https://slsa.dev/provenance/v1`, as defined in the [SLSA provenance v1 spec](https://slsa.dev/spec/v1.0/provenance). Then inspect the attestation fields that actually affect trust: the top-level `subject` digest and the builder identity in `predicate.runDetails.builder.id`, along with any source or dependency references your release process expects. The same [SLSA provenance v1 spec](https://slsa.dev/spec/v1.0/provenance) shows `subject` as a top-level statement field and `runDetails.builder.id` inside `predicate`. Deployment policy should accept only the exact combinations you intended to trust. ## Where should enforcement happen? Put the final allow-or-deny decision in deployment admission or equivalent enforcement, because that is the point that can block acceptance of the submitted workload before it runs. CI checks are still valuable, but they are preflight checks. They cannot stop a later retag, a manual deploy, or an environment-specific override. A good layered design is: - CI builds, signs, and publishes attestations. - The registry stores images and attached attestations. - Admission resolves the submitted image to a digest, verifies attestations for that digest, and then admits or rejects. That ordering is the important control. You are not trying to make every upstream system perfectly coordinated; you are making admission repeat the trust decision at the enforcement point. ## What does a concrete verification flow look like? Prerequisite: your deployment path can resolve OCI image tags to digests and reach the registry metadata needed for attestation lookup. Worked example for `registry.example/payments:prod`: 1. Resolve `:prod` to `sha256:abc...`. 2. Fetch attestations attached to that digest. 3. Run [Cosign verify-attestation](https://github.com/sigstore/cosign/blob/main/doc/cosign_verify-attestation.md) with explicit signer constraints such as `--certificate-identity` and `--certificate-oidc-issuer` for keyless verification. 4. Require the provenance type with `--type slsaprovenance1` or the full SLSA URI, as supported by [Cosign verify-attestation](https://github.com/sigstore/cosign/blob/main/doc/cosign_verify-attestation.md). 5. Validate policy over the predicate payload. The [Cosign attestation docs](https://github.com/sigstore/docs/blob/main/content/en/cosign/verifying/attestation.md) document CUE and Rego policy validation and note that validation is written against the `predicate` portion of the attestation. 6. Admit only if every required attestation is present, parses correctly, and matches policy. The practical tradeoff is latency and dependency sensitivity at deploy time. In return, you stop environment exceptions from becoming silent unsigned lanes. ## How do we handle exceptions without creating a bypass? Never add `skip_provenance=true` to the normal deployment path. Use a distinct break-glass route with a short TTL, named approver, ticket reference, and narrow scope such as one digest or one namespace. The workload should still be marked as emergency-admitted rather than provenance-verified, and that state should expire automatically. Also decide recovery behavior up front. If attestations are missing, malformed, or temporarily unreachable, deny by default. The [Cosign attestation guidance](https://github.com/sigstore/docs/blob/main/content/en/cosign/verifying/attestation.md) says systems should be designed to fail closed because an attacker may delete or hide specific attestations. **Checklist:** verify by digest, pin signer identity and OIDC issuer, require the SLSA predicate type, validate builder and source expectations, deny on absent or invalid attestations, and isolate emergency overrides. **Q: Should we verify a multi-arch tag once?** No. Verify the manifest digest actually selected for deployment, or verify each platform-specific digest your environments may run. **Q: Should transparency-log trouble automatically allow deploys?** No. Treat that as an outage decision and use the temporary break-glass path, not a hidden fallback. Next step: implement one admission policy for a single critical workload that denies images lacking a valid SLSA provenance attestation for an approved builder identity, then test break-glass expiry before broader rollout. Reviewed: 2026-09-06 ## Source references - - - -