Every Self-Improving Agent Loop in 2026 Stops at the Pull Request

Read the five products' own launch posts side by side and the same shape appears. A managed agent reads production traces, clusters recurring failures, writes an investigation, and proposes a change. The proposal is handed to an engineer as a pull request or an editable suggestion. The engineer decides what ships. After the merge, the product goes back to watching traces.

That is a useful product. It is also where the hard part begins, because the proposal is a hypothesis. Nobody has yet tested it against the current agent on the same inputs, and nobody will measure whether it worked unless the engineer sets that up by hand.

The table is built from each vendor's current announcement or documentation, with the reviewed date in the sources list. Where a vendor does not describe a step, the cell says so rather than guessing.

The stopping point is not an accident of roadmap order. Three forces push every platform to end at the proposal.

Accountability transfers at the pull request. Once an engineer merges a change, the vendor is no longer the author of what runs in production. For a platform owned by a model provider or an observability vendor, a fix that ships itself and regresses a customer is a brand problem of a different order than a bad suggestion. The guide to why major AI companies keep humans in the loop covers the structural version of this argument.

Validation is expensive and specific. Testing a prompt change against the agent that is actually running requires the agent's prompt, tools, and configuration, a set of inputs that exercise the failing behavior, and a comparison method that is fair to both versions. Observability products have the traces but not the agent; drafting a PR is cheaper than building a simulation harness.

Production measurement is slow and often inconclusive. A credible answer to 'did it work' needs the same detector on both sides of the release, enough conversations after it, and a confounder check. Most vendors would rather not publish 'not fixed' or 'confounded' against their own suggestions, which is what an honest verdict system produces much of the time.

When the loop stops at the proposal, the reviewing engineer inherits four jobs: deciding whether the hypothesis is right, constructing a test that would show it, shipping the change safely, and setting up a measurement that will say whether the failure went away. The time that was supposed to be saved moves rather than disappears. Teams that adopted detection tools in 2025 reported exactly this: more findings, same backlog.

It also quietly changes what counts as done. A merged PR closes the ticket. The failure it targeted may still be happening, and nobody is looking, because the loop restarted on the next finding. Converra's earlier analysis of why improvement loops stop before production verification describes that failure mode in detail.

Pre-deployment evidence on the same inputs. A candidate fix should be compared with the current agent on the same simulated conversations, scored by the same rules, before anyone is asked to approve it. Converra runs this as a paired head-to-head on shared personas, and a candidate is only eligible if its lift over the baseline is positive; see simulation testing for agents.

A regression suite built from production. Every fix is tested against scenarios derived from the conversations that failed, so a change that cures one pattern and breaks another is caught before release. The guide to building a regression suite from production traces covers the construction.

Governed deployment with a way back. Review-first by default, with the change shipped as a pull request or a direct update a person applies; automatic deployment only behind a passing regression suite when a team turns it on; and a previous version you can redeploy.

A verdict on the targeted failure, with confounding handled. Converra judges a fix it shipped by recurrence of the targeted failure: at least 5 diagnosed conversations before any verdict, verified only with at least 15 after the change and zero recurrences, confounded if the prompt was edited outside the fix during the window, and regressed if the overall failure rate rose. Changes you merge yourself are measured seven days before against seven days after. The full rules are in how to verify an AI agent fix after deployment.

It would be easy to read this as 'autonomous good, human-in-the-loop bad'. That is not the argument. Converra keeps approval in the loop by default too. The difference is what the human is approving: a hypothesis with a drafted diff, or a candidate with head-to-head evidence, a regression result, and a measurement that will report back. Approval of the second kind takes minutes and compounds trust; approval of the first kind is the engineer doing the vendor's validation.

Scope matters here as well. Converra automatically fixes prompts and configuration, the layers where a simulated comparison is meaningful. For code and routing changes it diagnoses to the exact step and hands your team a PR-ready description with evidence; it does not merge code on its own.

Ask each vendor three questions and write down the answers. What does the fix get tested against before I see it? What happens to the targeted failure rate after I merge, and who measures it? What does the product report when the fix did not work? A product that cannot answer the third question has not closed a loop; it has automated the first half of one.

The production verification page describes what Converra reports, including the honest outcomes: not fixed, confounded, and not enough evidence yet.