Operations metrics that do not drift

The eleven-minute MTTR that wasn't

A platform vendor once told a room of buyers their mean time to resolve had dropped to eleven minutes. It had. What the slide didn't say was that the clock started at ticket creation, not at incident start, and that a fifth of their incidents were auto-created and auto-closed by a synthetic monitor before a human ever saw them. Take those out, and the real MTTR for human-handled incidents was closer to ninety minutes. Nobody lied. The metric was defined honestly, once, by someone who needed a good number, and it kept that definition because nobody asked to see it again.

That's the pattern behind almost every one of the nine metrics in this piece. The formula is rarely secret. The start and end points, the population being averaged, and the exclusions are where the real editorial choices happen, and those choices almost never make it onto the dashboard tile.

Time to acknowledge (MTTA)

The formula, as the platform's own scorecard model computes it, is the mean of acknowledgedAt minus startedAt across every incident that carries an acknowledgement timestamp, with the median reported alongside so a single outlier can't quietly drag the mean. The thresholds in that same model treat a mean under five minutes as healthy, under fifteen minutes as attention, and anything slower as critical.

The edge case that breaks a naive MTTA is silent exclusion: incidents that were never acknowledged simply don't enter the average, so a team that lets its worst incidents sit unacknowledged for hours can show a fast MTTA built entirely from the easy ones it did pick up. The platform's model separately counts unacknowledged active incidents for exactly this reason, alongside the acknowledged count the average is drawn from. When a vendor quotes MTTA, ask what share of incidents were excluded from the average because nobody ever acknowledged them, and ask to see that number, not just the average of the rest.

Time to resolve (MTTR)

The formula is the mean of resolvedAt minus startedAt across incidents that have actually been marked resolved. AI Ops reports the median beside it for the same reason as MTTA: a single nine-hour outlier can move a mean by half an hour across a modest population, and a reader who only sees the mean has no way to know that happened.

MTTR gets redefined in three common ways. The clock can start at detection instead of at the underlying fault, which flatters slow detection more than fast recovery. The population can quietly exclude incidents still open at reporting time, which is survivorship bias wearing a service-level badge: the slowest, still-unresolved incidents are excluded from the average that's supposed to describe how fast a team resolves things. And "resolved" itself can mean "ticket closed" rather than "customer impact ended," which are frequently different moments. Ask which of these three is happening before trusting any MTTR figure, including this platform's own.

Time to detect (MTTD): the metric most platforms don't actually have

Time to detect is supposed to measure the gap between when a fault began and when it was noticed, and it's arguably the most operationally important of the nine, because everything downstream of detection is response time, while detection lag is pure blind time. It's also the metric most platforms, including this one today, don't actually compute, because it requires a timestamp for when the underlying condition started, independent of and earlier than the moment a signal crossed a threshold and a situation got named.

The AI Ops incident record currently uses the moment an incident is opened as its "detected" event, which is really a detection-and-naming timestamp, not a true fault-onset timestamp. The honest label for that field is "time from incident open to acknowledgement," not MTTD. If a vendor quotes MTTD, the single most useful question is what timestamp they use for "the fault began," because most operations telemetry doesn't reliably carry one, and a number computed without it is measuring something else wearing MTTD's name.

Noise reduction and the signal funnel

Noise reduction is usually presented as a single ratio: raw events in, incidents out, with the gap between them relabeled as noise the platform "eliminated." The honest version of this number needs four stages, not two: records admitted, incidents opened, situations named, and decisions made ready, each expressed as a share of the stage before it, because a big reduction from records to incidents can mean excellent filtering, or it can mean under-instrumented sources that never generated many records to begin with.

The platform's own funnel keeps a second honesty flag on the first stage specifically: whether the record count is a measured platform counter or a fallback sum from a reference fixture, because those are not the same claim and shouldn't carry the same confidence. A noise-reduction percentage without that funnel underneath it is a marketing number dressed as an engineering one.

Error budget and burn rate

Error budget remaining is the minimum, across every registered service objective, of budget-remaining-percent, not an average, because an average hides the one service that's actually about to breach behind nine that are fine. Burn rate is the maximum burn rate across the same set, for the same reason. The AI Ops alert policy states the bands plainly: watch at 1x burn, attention at 4x, critical at 8x, the same bands SRE teams have used since the multi-window burn-rate alert became standard practice.

The edge case is scope creep in what counts as an "objective." A team under pressure to show a healthy error budget can narrow the objective's definition, shorten its measurement window, or simply decline to register an objective for the service that's struggling, and a minimum-across-objectives calculation is only as honest as the set of objectives someone chose to register. Ask for the full objective registry, not the summary number, before trusting an error-budget figure from anyone.

Change-failure rate: correlation wearing causation's clothes

Change-failure rate is supposed to be the share of changes that cause an incident. The quiet redefinition here isn't in the formula, it's in the word "cause." Most platforms, this one included, can only reliably compute a correlation: which change happened on a shared service shortly before an incident started. The platform's own preceding-change logic looks for a change on a shared service within a twenty-four-hour window before an incident and explicitly offers it as the first hypothesis, never as the cause, because two things happening in sequence on the same service is not proof one produced the other.

A change-failure rate computed from correlation and reported as causation will systematically overstate how often changes actually cause incidents, and it will understate it for the incidents where the real causal change fell outside whatever window the correlation logic uses. When a vendor states a change-failure rate, ask whether it comes from a post-incident causal determination made by a person, or from a time-window correlation the tool ran automatically. Those are different metrics that happen to share a name.

Automation rate and autonomous-action success rate

These two are the newest arrivals on operations dashboards, and they're also the easiest to inflate, because "automation" and "autonomous" don't have settled definitions across the category yet. An automation rate can mean the share of remediation steps a script executes with a human clicking approve first, or it can mean the share where nothing human touched the decision at all. Those are very different claims compressed into one percentage.

Autonomous-action success rate has a sharper problem: it presumes autonomous actions are actually happening. In a recommend-and-propose posture, where every consequential effect is a proposal until a named human approves it, which is where the AI Ops reference deployment sits today and where PagerDuty's fully autonomous responder still sits as a stated 2026 target rather than a shipped capability (PagerDuty: Spring 2026 release), there is no autonomous-action population to compute a success rate over. A vendor who quotes one anyway is usually measuring "proposals a human accepted," a real and useful number, just not the number its label claims to be. Ask directly: of the actions in this rate, how many executed without a human approving each one first? If the answer is zero, the metric needs a different name.

What to ask for instead

None of this means the nine metrics are useless. They're the right vocabulary for the category. It means the vocabulary is cheap to fake and expensive to verify, so the questions that matter are about a metric's construction, not its value. Five questions cover most of the ways these numbers get quietly redefined:

• The population the average is computed over, and who or what is excluded from it, by name.
• Whether the number is measured from a platform counter or derived from a fixture, sample, or model.
• The start and end timestamp definitions in plain language, not just the metric's initials.
• Whether a causal claim in the metric is backed by a determination or only by a time-window correlation.
• The full objective or service registry the number was drawn from, not the single summary figure.