Delayed Distribution Metrics

Incident Report for Datadog Govcloud

Resolved

Monitors and live data are no longer affected and all live data is shown without delays. Unfortunately we are not able to recover data for distribution metrics with percentiles disabled from 19:00-19:40 UTC and those metrics will display gaps. You can see your affected metrics here: https://app.ddog-gov.com/metric/summary?facet.percentiles=-enabled&filter=dist.
Posted Mar 08, 2024 - 18:24 EST

Update

Remediation for this issue is still in progress. Monitors are no longer affected, and live data are no longer delayed, but some users might continue to see gaps in historical data on metrics and dashboards until remediation is complete.
Posted Mar 08, 2024 - 16:46 EST

Identified

The issue has been identified and a fix is being implemented.
Posted Mar 08, 2024 - 15:44 EST

Investigating

We are investigating increased latency processing Distribution Metrics.
As a result of this issue, some users may see delays or gaps for metrics on graphs.
To prevent spurious alerts, we have temporarily disabled monitors based on this data.
Posted Mar 08, 2024 - 14:50 EST
This incident affected: Metrics and Infra Monitoring.