How do you evaluate an AIOps or agentic-operations vendor?

The frame buyers are actually using in 2026

Buyers evaluating an operations control plane in 2026 are typically comparing against three groups at once, and expecting one product to hold up against all three: workflow and incident platforms adding agents to an existing ticket and on-call ownership (ServiceNow, PagerDuty, BMC), observability platforms adding agentic reasoning to their own telemetry (Datadog, Dynatrace, Splunk), and agentic-SRE startups built agent-first (Resolve AI, Cleric, Rootly, Traversal). A large services incumbent's own platform, bundled into its managed-services contract, is increasingly a fourth comparison buyers raise unprompted.

A checklist you can run in a proof of concept

Seven questions separate a vendor's marketing claims from what a proof of concept can actually verify.

• Permission model: ask for the actual role and permission table, not persona names, and test whether one identity can both propose and approve the same action.
• Evidence discipline: does every headline number on a dashboard name the field or formula it was computed from, or is it an unlabeled trend line?
• Autonomy posture: is there a visible, per-workflow dial (observe, propose, act with approval, act) with an audit trail of changes to it, or is "autonomous" only a slide adjective?
• Reasoning trace: can you click into the agent's chronology, ranked hypotheses, and counterevidence, or can you only read a narrated summary of what it did?
• Deployment dependency: does the product require the vendor, or an affiliated services arm, to operate your infrastructure before you can use it?
• Self-reported versus independent: ask directly which outcome numbers are self-reported and which are independently measured, and by whom.
• What they admit they don't have: analyst placement, independent customer measurement, and any feature that is on the roadmap but explicitly not shipped yet.

What tends to win a demo, and why that is a useful filter

Across the field, the demonstrations that land in 2026 share a pattern: showing the reasoning rather than only the answer, treating graduated autonomy as a first-class visible control, putting the agent on the roster as a rostered responder the way PagerDuty now does, surfacing a first finding in minutes, carrying memory across incidents, naming the preceding change as the first hypothesis rather than an afterthought, opening a collaborative room already pre-loaded with context the way Splunk's AI Canvas does, and making agent governance and interoperability (MCP, agent-to-agent) inspectable rather than asserted.

That pattern is a useful filter precisely because it rewards inspectability over narration. A vendor whose demo consists of a person describing what the agent did, rather than a screen you can click through yourself, is asking you to trust the narration.

The trap to watch for

A scripted co-pilot demo can look identical to a fully autonomous one if nothing goes wrong on stage. The useful question is not "what does it do when it works," it's "what happens, live and unscripted, when the agent is wrong." Does it stop, does a named human get pulled in, and is that routing enforced by the system rather than assumed by the presenter.

What HELM by LockedIn Labs would show if you asked these questions of us

The permission model on /access, the basis line behind every indicator on /command#scorecards, the autonomy dial and its ceiling on /control-plane, and the replayable investigation trace on /command#investigations. And plainly: HELM by LockedIn Labs has no analyst placement from Forrester or Gartner as of this writing, and no independent customer measurement of outcomes yet. We would rather name that gap in the same conversation than let a demo imply it is already closed.

A note on scoring vendors against each other

It is tempting to build a single weighted scorecard across every vendor in a shortlist and pick the highest total. That approach tends to reward vendors who answer every question with a confident number, whether or not the number is independently verifiable, over vendors who are precise about what they do and do not have evidence for. A more reliable approach treats "admits what it hasn't proven" as a scored category in its own right, not a tiebreaker, since a vendor's willingness to name its own gaps is one of the few signals in an evaluation that a scripted demo cannot fake.

It is also worth weighting the evaluation toward what you can operate in a sandbox over a fixed evaluation window, rather than what a sales engineer clicks through on a call. A vendor confident in its own governance will generally welcome an evaluator poking at the edges of the permission model; hesitation to grant that kind of hands-on access is itself a data point.