# Put every production change on the incident timeline

> Use one change-event schema for deploys, configuration, flags, and infrastructure so responders can correlate regressions quickly.

Canonical URL: https://www.devobs.io/articles/deployment-markers-in-telemetry/
By: Claire Dubois
Published: 2023-01-26T17:55:03.540Z
Updated: 2026-09-06T08:31:04.426Z
Section: Architecture

Every production change should emit a structured event into the same timeline responders use for metrics and traces. Code deploys alone are insufficient: configuration edits, feature-flag changes, dependency upgrades, certificate rotations, and infrastructure replacements can all alter behavior. A common schema lets an incident responder test a change hypothesis in minutes.

## Define one change event

Use a stable event name and include change ID, kind, service or resource, environment, region, previous and new version, rollout phase, actor or automation identity, source link, start and completion times, and outcome. Add the commit, image digest, [configuration revision](https://www.devobs.io/articles/configuration-change-delivery-pipeline/), flag key, or infrastructure plan identifier appropriate to the kind. Avoid secrets and raw configuration values; link to an access-controlled diff instead.

OpenTelemetry [resource semantic conventions](https://opentelemetry.io/docs/specs/semconv/resource/) define attributes for entities that produce telemetry, including service and deployment context. Propagate stable resource attributes such as service name, service version, deployment environment, cloud region, and orchestrator identifiers into traces, logs, and metrics. The change event then uses the same values, allowing dashboards to filter affected telemetry without fuzzy name matching.

Version every change independently. A code version without a configuration revision cannot distinguish two pods running the same binary under different behavior. Include feature-flag evaluation metadata in traces only where privacy and volume allow; aggregate change markers should still record when a flag rollout percentage or rule changed.

## Mark phases, not only completion

Emit events for rollout start, each material traffic step, pause, rollback, and completion. A regression can begin at 10 percent exposure and disappear during rollback before a final deployment record is written. Phase events also show whether a metric changed before or after the canary received traffic.

GitHub's guide to [controlling deployments with Actions](https://docs.github.com/en/actions/deployment/about-deployments/deploying-with-github-actions) documents environments, deployment protection, concurrency, and deployment status. If GitHub Actions is the delivery system, carry its deployment or run identifier into the runtime change event. The same principle applies to another delivery system: preserve a durable link from observed runtime state back to the change record.

Clock alignment matters. Emit timestamps from a trusted collector or record both source and ingestion time. Keep a unique event ID so retries do not draw duplicate dashboard annotations. If a rollout controller and CI both emit, designate which is authoritative for actual traffic exposure and correlate the two IDs.

## Put the data where responders already look

Add markers to golden-signal dashboards and make them queryable alongside traces. During an incident, start with changes affecting the impacted service, region, or dependency during the relevant window. A marker is a hypothesis, not proof; responders should compare affected and unaffected cohorts and examine whether the regression follows version or configuration boundaries.

Alert when runtime versions appear without a corresponding change event, when a rollout remains in progress beyond its expected window, or when regions disagree on configuration revision. These are observability failures even before customer impact appears.

## Instrument one path end to end

Choose the most frequent production deployment. Emit start, exposure, completion, and rollback events; attach the resulting version attributes to application telemetry; and add dashboard markers filtered by service and region. Run a controlled staging rollback and confirm the timeline reconstructs what happened without opening the CI system. Next, extend the schema to configuration and feature flags, keeping the same identifiers and query experience.

## Include non-deployment changes

Test a feature-flag rollback, emergency configuration edit, expired certificate replacement, database failover, and autoscaling-policy change. Each should produce the same searchable identity and affected-resource fields as a code release. Confirm the marker appears before the behavior changes and that a failed or reversed action gets a terminal outcome. This prevents responders from blaming the nearest commit when another control system caused the regression.

Reviewed September 2026.

## Source references

- <https://opentelemetry.io/docs/specs/semconv/resource/>
- <https://docs.github.com/en/actions/deployment/about-deployments/deploying-with-github-actions>