Evaluating an AI operations platform: the buyer's checklist nobody hands you

The demo that worked and the pilot that didn't

A buyer once approved a platform after a thirty-minute demo where an agent found a root cause in under two minutes. The pilot that followed took six weeks and never reproduced that speed once, because the demo had run on a curated incident with clean telemetry, and the pilot ran on the buyer's actual, messier estate. Nobody had done anything dishonest. The demo showed what the product could do in ideal conditions, and ideal conditions are exactly what a live estate never offers.

The fix isn't a longer demo. It's a different kind of demo, one where the buyer drives instead of watches, using their own conditions instead of the vendor's curated ones. What follows is the checklist for that kind of evaluation, organized around the questions whose answers actually separate platforms.

Evidence discipline: can you check the number, not just hear it?

Ask the vendor to click on any number on their screen. A platform with real evidence discipline can tell you, on the spot, which field or formula produced it. A platform without it will explain the number in prose, confidently, and the prose will not be checkable against anything. The Register's April 2026 survey with NeuBird found 60 percent of respondents cite lack of trust as the top blocker to agentic operations (The Register: survey of SREs on AI agents in production). A platform that can't show its work is asking you to supply the trust it hasn't earned.

Ask specifically: does every headline figure name its source field or formula on the same screen, and does the platform ever say "not measured" instead of estimating? A platform that always has a number for everything is more worrying than one that sometimes admits a gap.

Autonomy gradation: is there a dial, or just a word?

"Autonomous" is doing a lot of unverified work in this category right now. Ask for the actual gradations the platform supports, not the marketing word. Dynatrace argues for determinism first: causal AI produces the answer, and the agent acts under what it calls supervised autonomy (Dynatrace: third-generation platform announcement). Cleric ships read-only by default and escalates to write access only when a team is ready (Cleric: AI SRE). PagerDuty's SRE Agent can act on its own remediations or wait for approval, with a fully autonomous responder still a stated target for later in 2026, not a current capability (PagerDuty: Spring 2026 release).

None of these postures is wrong. What matters for an evaluation is whether the platform's autonomy ceiling is a control you can see and set per workflow, with a record of who set it and when, or whether it's a single word in a pitch deck that means something different depending on who's asking.

Separation of duties: can the same identity propose and approve?

This is the single fastest test to run in a live evaluation. Have the same logged-in identity propose an action and then try to approve it. A platform with real separation of duties will refuse the approval outright, because the control plane checks that the approving principal doesn't match the proposing principal, not because a policy document says it should. A platform without this control will let the approval through, and the audit trail will show a decision that was, structurally, a person approving their own work.

Extend the test to the people who build automation: can whoever writes and publishes a workflow also execute it without a second identity in the loop? Can whoever edits policy also operate under the policy they just wrote? ServiceNow's Autonomous Workforce publishes role-scoped permissions as part of its AI Control Tower (ServiceNow: Autonomous Workforce announcement). Ask to see the actual rule enforced live, not the phrase describing it.

Agent observability: what does the platform know about its own agents?

An agent's authority ceiling, its data sources, the peers it can talk to, and the conditions under which it abstains rather than answer are all facts a platform should be able to state about itself, the same way an org chart states who reports to whom. Ask the vendor to show one agent's full profile: identity, authority, sources, peers, output contract, and abstention rule. If any of those is missing, or if the answer is "it depends on the prompt," that's the gap.

Datadog's Bits Investigation renders an agent trace, every tool call the agent made, as part of its output (Datadog: Bits AI SRE deeper reasoning), which is a different but related property: not just what the agent is allowed to do, but what it actually did on this specific run. Ask for both. A platform that can describe its agents in the abstract but can't replay one specific agent's actual reasoning trace has observability in name only.

Data residency and deployment model

Ask where your data physically lives, who else's data shares that infrastructure, and what happens to data used in a demo or pilot after it ends. A platform operating as a single multi-tenant SaaS backend has one honest answer to give, and it should be able to give it precisely, including subprocessors, hosting region, retention period, and deletion mechanics. A platform offering an on-prem or customer-boundary deployment has a different but equally answerable question: what exactly crosses the boundary, and what stays inside it.

Some services-led platforms require the provider to operate the estate it orchestrates; "you can't orchestrate what you don't operate" is a fair summary of that constraint. That's a legitimate model for a buyer who wants a managed-services relationship. It's a disqualifying constraint for a buyer who wants a product they, or their existing systems integrator, can run over an estate the provider doesn't operate. Know which buyer you are before the deployment-model question comes up.

What to demand a demo prove, in your own hands

Five specific tests, run by you rather than described by the vendor, cover most of the ground above:

• Run the platform against a real or realistic incident from your own estate, not a curated one the vendor chose.
• Click through to the source of at least five headline numbers and confirm each one names a field or formula.
• Attempt the propose-then-approve-with-the-same-identity test and watch whether it is actually refused.
• Ask to see one agent's full observability profile: authority, sources, peers, output contract, abstention rule.
• Ask what the platform says when it doesn't have a number, and see if "not measured" is a phrase it can produce.

The counter-argument: due diligence this heavy kills momentum

There's a real risk in over-indexing on this checklist: a buying committee that demands perfect answers to every area before signing anything will never sign anything, because no platform in this category, including the incumbents with analyst placements, has a perfect answer to every one of these questions today. Forrester's Q2 2025 AIOps Platforms Wave names Datadog, Dynatrace, and ScienceLogic as Leaders (Forrester Wave: AIOps platforms Q2 2025), and even those three have public gaps somewhere on this list.

The point of the checklist isn't to find a platform with no gaps. It's to find a platform that can name its own gaps accurately when asked, in the same conversation where it's trying to win your business. That's a rarer and more useful signal than a clean scorecard, because a vendor's honesty about what it doesn't do yet is one of the only things in a sales process you can actually verify on the spot.