# When two agent runtimes troubleshoot from the same evidence

*Sixteen paired evidence-conditioned cases; independently reviewed research package, with website publication subject to normal release governance.*

An agent can produce a plausible fix while overlooking the version that contains it, a report that contradicts it, or the risk of collecting a credential in a diagnostic log. We designed this small study to examine those decisions with the search step removed. For each of 16 public technical incidents, two requested agent configurations received the same frozen, source-linked evidence and the same response contract. Neither had a browsing tool during the task.

The incidents range from dependency incompatibilities to connection reuse, certificate verification, incomplete release evidence and privilege boundaries in CI. Some packets identify a released fix; others only describe a reporter's observation or a pending change. For example, a Requests packet pairs a version-specific URL-path regression with release notes that specify a second dependency requirement. An HTTP/2 packet, by contrast, supplies a reporter's timing hypothesis and does not establish that a fix shipped. The useful answer to the latter must preserve that uncertainty instead of turning an open proposal into an upgrade promise.

The requested configurations were GPT-6.1 Terra and Claude Sonnet 5.5, each with low effort requested. The Sonnet provider receipt confirms its model identity. The available Terra receipts do not independently attest effective model identity, and neither arm's effective effort was attested. Different runtime context overhead and different measurement boundaries also limit timing and token comparisons. This is therefore an evidence-conditioned configuration comparison, not a controlled test of model identity or independent web investigation.

A separate reviewer agent assessed each response blind against a case-specific ten-point rubric. Reviewers and competing agents shared one operator, so this separation does not establish independent operators or statistical independence. The reviewer recorded separate notes for unsupported suggestions and potentially unsafe advice. The requested Terra configuration scored 146/160 rubric points and the requested Sonnet configuration scored 149/160. On individual cases, Terra scored higher on three, the configurations tied on eight, and Sonnet scored higher on five. These counts are descriptions of this small set, not a general ranking.

Both configurations received credit for a source-supported applicable action on all 11 cases where one was identified, and for useful diagnostics on 16/16 cases. Supported actions include scoped reporter workarounds and security-preserving steps as well as released fixes; credit is not evidence of independent execution. Both correctly abstained on the 10/10 cases classified as requiring it. These rates and their applicability rules were derived after the answers and blind judgments existed and are exploratory.

After consistency review, neither configuration had an unsupported claim presented as established fact in the 16 cases. That category is distinct from qualified but not source-established suggestions or inferences, which are retained in the individual judgments. The exploratory review counted qualified but not source-established suggestions or inferences in 0/16 Terra-requested and 13/16 Sonnet responses; these are not established fixes. The assessor flagged potential safety concerns in 0/16 Terra responses and 6/16 Sonnet responses. Examples include broad environment or configuration dumps that could expose secrets and preserving an unsanitized raw upload body; the flags describe advice risks, not observed disclosures. The package retains the answers as research material, with warnings; they should not be treated as deployment instructions. A result is not independently reproduced merely because a model cites an upstream report.

The first two cases validated the workflow; the next 14 used the same paired design and were selected before the case author examined model answers or grades. That is enough to test the procedure and reveal examples worth studying. It is not enough to establish global model superiority, generalize failure rates to real users, or claim an open-web winner. The cases are curated and source exposure in model training is unknown. The prior two-case feasibility pilot is a separate experiment and is not pooled here.

The package supplies the exact packets and hashes, source links, sanitized paired outputs, blinded judgments, and code to reconstruct the reported counts. Readers can inspect where an answer's recommendation is supported, where a grader flagged an uncertainty, and where further evaluation is needed. Repeat the protocol on a larger independently selected case set with attested configurations and comparable runtime instrumentation before drawing broader conclusions.

[Download the complete research package](/research/agent-fix-challenge-01/data/package.json) · [Landscape graphic](/research/agent-fix-challenge-01/data/landscape.svg) · [Vertical graphic](/research/agent-fix-challenge-01/data/vertical.svg)