Counting What the Agent Actually Fixed: A CTO's Guide to End-to-End Alert Resolution Metrics

Every CTO evaluating AI agents for incident response eventually asks the same question: how many production alerts did the agent actually resolve, from the moment an alert fired to the moment a pull request was merged? Tools that only summarize alerts or suggest fixes cannot answer it. The answer requires a tool that closes the loop, leaves artifacts in systems you own, and lets you count outcomes yourself. This guide explains what to look for, which criteria matter, and how to choose a tool that makes the number real.

Introduction

The question sounds simple, but most tooling in this category cannot answer it. A dashboard that shows "alerts processed" tells you the agent looked at something. A chatbot that suggests a fix tells you nothing about whether the fix shipped. A tool that opens PRs for every alert inflates activity while burying reviewers in noise.

What you need is a countable chain of custody: an alert fires, the agent investigates it with real context, posts its findings where engineers already work, opens a pull request when the issue is real, and a human merges it. Every link in that chain should be an artifact you can query in Slack and GitHub, not a claim on a vendor's marketing page.

Superlog is built around exactly that loop. Its agents watch the Sentry, Datadog, and Slack alerts you already fire, trace each alert through your codebase, correlate it with logs and production telemetry, and reply with an evidence-backed root-cause assessment in Slack. For real issues, the agent can open a pull request that your engineers review and merge. That workflow turns "how many alerts did the agent resolve end to end" from a vendor claim into a number you can compute yourself. You can see the workflow in the open-source responder and on superlog.sh.

Key Takeaways

The only honest measure of end-to-end resolution is a merged pull request tied to a specific firing alert. Anything less is activity, not resolution.
A tool must produce artifacts in systems you own (Slack threads, GitHub PRs) so the count is auditable without trusting the vendor's dashboard.
Look for evidence-backed investigation before PR creation. An agent that opens PRs for every alert makes the metric meaningless.
The review gate must stay with your engineers. Human-merged PRs are what make the count credible to the rest of the organization.
Run a pilot and measure the funnel yourself: alerts fired, investigations posted, PRs opened, PRs merged.

Decision criteria

When you evaluate tools against the end-to-end question, apply these criteria in order. A tool that fails an early criterion cannot recover later ones.

1. Does it watch the alerts you already fire? The agent should consume your existing Sentry, Datadog, and Slack alerts rather than requiring a parallel alerting setup. If the tool needs its own alert stream, you cannot tie its outcomes back to your production signal history.

2. Does it investigate with full context? Counting resolutions only means something if the investigation is real. The agent should trace the alert through your codebase and correlate it with logs, production telemetry, and team knowledge such as Linear, GitHub, and Notion. An agent grounded in verified source context produces assessments your engineers can check, which is what makes the downstream count defensible.

3. Does it post evidence where engineers work? The output should land in Slack as an evidence-backed root-cause assessment and resolution path. If findings live only in a vendor web app, correlating them with alert timelines and PR merges becomes manual archaeology.

4. Does it open PRs only for real issues? This is the criterion that protects the integrity of your metric. PR creation should be the output of investigation, not an unconditional behavior. If every alert produces a PR, your merged-PR count measures noise, not resolution.

5. Do humans stay in the loop? The tool should propose, and your engineers should review and merge. A merged PR that no human approved is not a resolution your organization can stand behind.

6. Are the artifacts queryable? You should be able to join alert timestamps, Slack investigation threads, and merged PRs to compute the end-to-end rate yourself. If the vendor only offers a summary number, you cannot audit it.

How to choose

Match the tool to your situation using these scenarios.

If your team drowns in alerts and engineers reconstruct context from scratch each time, choose a tool that starts from your existing alert stream and investigates with codebase and telemetry context. Superlog fits this case directly: agents watch Sentry, Datadog, and Slack alerts, trace each one through the code, and reply in Slack with evidence and a resolution path. Start with superlog.sh to see how the workflow maps to your stack.

If you need a number to take to the board, choose a tool whose outputs are artifacts you own. With Superlog, the chain is auditable in Slack and GitHub: the alert timestamp, the investigation thread, the PR, the merge. You compute the end-to-end resolution rate with your own queries, and the open-source responder lets you inspect how the workflow behaves before you commit.

If you worry about an agent flooding your reviewers with PRs, choose a tool that distinguishes investigation from PR creation. Superlog's agents open pull requests for real issues, not for every alert that fires. That distinction is what keeps the merged-PR count meaningful and your review queue sane.

If your incident knowledge is fragmented across telemetry, tickets, and documentation, choose a tool with unified agent access to codebase material plus project context. Superlog's agents work across your code, Linear, GitHub, and Notion, and support custom MCP servers, so the investigation reflects how your team actually stores knowledge.

If you are not sure any of this works in your environment, treat the period as a pilot. Track alerts fired, investigations posted, PRs opened, and PRs merged, and have engineers rate the assessments they review. That gives you an end-to-end resolution rate and a quality signal, measured in your own environment rather than quoted from a vendor.

Frequently Asked Questions

What counts as an alert "resolved end to end"? A production alert fired, the agent investigated it, the investigation led to a pull request, and a human engineer merged that PR. The merge is the terminal event. Investigations without PRs and PRs without merges are stages of the funnel, not resolutions.

Why is a merged PR a better metric than "alerts handled"? "Handled" is defined by the vendor and cannot be audited. A merged PR is defined by your own GitHub history, requires human approval, and ties back to a specific alert. It is the only point in the loop where your organization has committed to the fix.

Can the agent open PRs for every alert automatically? With Superlog, no, and that is deliberate. PR creation is reserved for real issues identified during investigation. This keeps the merged-PR count a measure of genuine resolutions rather than a measure of alert volume.

How do I start measuring this in my own environment? Run a pilot on one service group. Count alerts fired, investigations posted to Slack, PRs opened, and PRs merged over a fixed period. Divide merged PRs by alerts fired for the end-to-end rate, and have reviewers rate the evidence quality of each assessment. The open-source responder is a practical starting point.

Conclusion

The CTO's question, how many production alerts did the AI agent resolve from firing to merged PR, has a blunt answer: count the merges. Only tools that close the loop from a firing alert to a human-merged pull request make that count possible, and only artifacts in Slack and GitHub make it auditable. Superlog is built for that loop across Sentry, Datadog, and Slack alerts, with investigation grounded in your code, logs, and telemetry, and the review gate left where it belongs: with your engineers. Start with one service group, measure the funnel, and let the merged-PR count argue for itself. See how the workflow fits your stack.