KFA Benchmark Commons — Workbench 01: swe field guide

# KFA Benchmark Commons — swe field guide Workbench 01 family research stage. Each included finding passed separate Claude analysis, adversarial review and source verification; these roles share one operator and do not constitute independent reproduction. Original summaries only; benchmark-answer-free intent. No official benchmark endorsement or measured performance improvement is claimed. Sources were bounded excerpts, not full deep reads. ## WHAT THIS TASK FAMILY TESTS - Code localization in a repository - Patch editing - Patch editing and recovery - Budget and stopping behavior - Harness complexity ## COMMON FAILURE MODES - INFERENCE: file-only lists may hide line context needed to choose a match (wb01-swe-ae4a25f92f5de3dfe6) - Valid multi-step edits may be rejected when intermediate states are invalid (paper notes guardrails impose an order) (wb01-swe-015dab6a1f8fdffe44) - Syntax-valid but incorrect or overly specialized implementations remain a prominent failure group (wb01-swe-015dab6a1f8fdffe44) - Accumulating edit errors (wb01-swe-2c757ae293a0afa0e1) - INFERENCE: context fills with failed attempts (wb01-swe-2c757ae293a0afa0e1) - Some failed edits may be guardrail rejections rather than wrong edits (wb01-swe-2c757ae293a0afa0e1) - Early stopping could cut off recoverable runs (inference) (wb01-swe-f1a620b705940123b5) - NOT_OBSERVED (absence is not evidence of no risk) (wb01-swe-16c27869fc892d6a91) ## ENVIRONMENT / EVALUATOR TRAPS No supported finding was accepted from the supplied excerpts. ## PROMISING STRATEGIES - Summarized search results versus match-by-match browsing; verbose per-match context as anti-pattern (merged from wb01-swe-5581ae626f15fd5036): In the GPT-4 configuration studied on SWE-bench Lite, summarized search outperformed browsing matches one by one. (finding wb01-swe-ae4a25f92f5de3dfe6; LIMITED; transfer UNKNOWN). ## NOT-PROMISING STRATEGIES - Consecutive failed edits reduce recovery likelihood (merged from wb01-swe-bdc5cbe83c6022b848): The SWE-agent paper authors observed repeated unsuccessful edits, with recovery becoming less likely after consecutive edit failures. (finding wb01-swe-2c757ae293a0afa0e1; LIMITED; transfer UNKNOWN). ## RECOVERY PLAYBOOK - INFERENCE, untested as an agent habit: start from compact lists of matching files and open individual matches selectively. (wb01-swe-ae4a25f92f5de3dfe6) - INFERENCE: validate syntax at edit time and re-attempt rejected edits. The paper tested a tool-level gate, not agent self-discipline. (wb01-swe-015dab6a1f8fdffe44) - HYPOTHESIS: after repeated failed edits, re-view the file or change approach instead of retrying the same edit. No threshold or intervention was tested. (wb01-swe-2c757ae293a0afa0e1) - HYPOTHESIS, untested: treat a long run with no progress as a cue to change strategy rather than assume more budget will help. (wb01-swe-f1a620b705940123b5) - HYPOTHESIS: do not drop or add ACI features without a measured comparison on the target model. (wb01-swe-16c27869fc892d6a91) ## COST / LATENCY TRADEOFFS - Summarized search results versus match-by-match browsing; verbose per-match context as anti-pattern (merged from wb01-swe-5581ae626f15fd5036): cost UNKNOWN; latency UNKNOWN. - Syntax-check gating of edit commands: cost UNKNOWN; latency UNKNOWN. - Consecutive failed edits reduce recovery likelihood (merged from wb01-swe-bdc5cbe83c6022b848): cost UNKNOWN; latency UNKNOWN. - Run length and cost versus success (merged from wb01-swe-9b3f7c6cb0aba82bb7): cost UNKNOWN; latency UNKNOWN. - Two unverified project statements on simple harness versus engineered ACI: cost UNKNOWN; latency UNKNOWN. ## WHAT IS BENCHMARK-SPECIFIC - arXiv:2405.15793v3 (analyst summary of 5.1/5.2/E.3); SWE-bench Lite, GPT-4 configuration; SWE-agent git:3ea751c087f32b16e039a2233dd6eefecef325d5 ACI doc; applicability: Agents whose search output can be shaped; Only the paper's GPT-4 / SWE-bench Lite setup (wb01-swe-ae4a25f92f5de3dfe6). - arXiv:2405.15793v3 (analyst summary); SWE-bench Lite, GPT-4 configuration; SWE-agent git:3ea751c087f32b16e039a2233dd6eefecef325d5; applicability: Editing where a syntax checker exists; Only the paper's GPT-4 / SWE-bench Lite setup (wb01-swe-015dab6a1f8fdffe44). - arXiv:2405.15793v3 (analyst summary of 5.2); SWE-bench Lite, GPT-4 configuration; applicability: Tool-based editing loops; Only the paper's GPT-4 / SWE-bench Lite setting (wb01-swe-2c757ae293a0afa0e1). - arXiv:2405.15793v3 (analyst summary of 5.1/5.2/E.3); SWE-bench Lite, GPT-4 configuration; applicability: Budget and stopping policy design in the paper's setting (wb01-swe-f1a620b705940123b5). - SWE-agent git snapshot git:3ea751c087f32b16e039a2233dd6eefecef325d5; applicability: Choosing harness complexity for SWE-bench-style tasks (wb01-swe-16c27869fc892d6a91). ## WHAT MAY TRANSFER TO REAL AGENTS - Summarized search results versus match-by-match browsing; verbose per-match context as anti-pattern (merged from wb01-swe-5581ae626f15fd5036): UNKNOWN; limits: Paper evidence is an analyst paraphrase with no effect sizes; Both sources come from the same project, so they are not independent; Old model versions; transfer unverified. - Syntax-check gating of edit commands: UNKNOWN; limits: No effect sizes; Doc and paper from the same project, not independent; Old model versions. - Consecutive failed edits reduce recovery likelihood (merged from wb01-swe-bdc5cbe83c6022b848): UNKNOWN; limits: Observational, no intervention; No rates given; Old model versions. - Run length and cost versus success (merged from wb01-swe-9b3f7c6cb0aba82bb7): UNKNOWN; limits: Correlational only; Summary-level, no numbers; Old model versions. - Two unverified project statements on simple harness versus engineered ACI: UNKNOWN; limits: Both sources are from one repository snapshot, not independent; Self-reported maintainer claim without metric. ## OPEN QUESTIONS - Effect size - Transfer to current models - Token cost and latency - Linting cost and latency - Languages without checkers - Whether a reset or backoff rule helps - Whether current models show the same pattern - Base recovery rates - Whether early-stop or restart policies improve resolve rate - Net cost of restarts - In the SWE-agent paper, successful runs generally finished sooner and consumed less money than unsuccessful ones, and the authors therefore questioned whether simply increasing budgets would help substantially. - Models, benchmark versions and metrics behind the parity claim - Whether mini-swe-agent uses ACI-like features - The SWE-agent README states that mini-swe-agent has superseded SWE-agent and matches its performance while being much simpler. ## SOURCES - swe-agent-paper: [source](https://arxiv.org/abs/2405.15793v3); PAPER; retrieved 2026-10-08T20:46:29.668Z. - swe-agent-aci: [source](https://github.com/SWE-agent/SWE-agent/blob/3ea751c087f32b16e039a2233dd6eefecef325d5/docs/background/aci.md); OFFICIAL_BENCHMARK_DOC; retrieved 2026-10-08T20:46:29.201Z. - swe-agent-overview: [source](https://github.com/SWE-agent/SWE-agent/blob/3ea751c087f32b16e039a2233dd6eefecef325d5/README.md); OFFICIAL_REPOSITORY; retrieved 2026-10-08T20:46:28.966Z.