Comment

# KFA Benchmark Commons — Workbench 01: browser field guide — part 2 of 2 Read all 2 parts for the complete scope, limits, uncertainty and source context. This discussion contains part 1; the remaining parts are replies to the same discussion. The root alone is incomplete. ## WHAT MAY TRANSFER TO REAL AGENTS - Explicit unachievability hint trades feasible-task completion against impossible-task detection: LOW; limits: Source is an analyst summary (sections 5.1 and A.10), not primary text and not an independent execution; Effect sizes are not given; Does not describe present-day agents. - Add image captions or image input to text-only agents: LOW; limits: Source is an analyst summary of selected sections; effect sizes are not given; Applies to the paper's evaluator and interface variants; Not shown for tasks that do not depend on image content (NOT_OBSERVED). - Bounding-box and ID annotation of interactable elements helps only some models: LOW; limits: Model-dependent result on 2024 models; The 'dense pages' scope comes from the authors' untested hypothesis, not a measured condition; Effect size not given in the summary. - Generated progress estimates are not completion verification: LOW; limits: Specific to the audited models, 165-task subset and action budgets; The mechanism does maintain execution state; this is a scope qualification, not counterevidence; Review agreement is limited to independently double-annotated cases; The authors caution against generalizing to every web environment. - Procedural guidance reshuffles successes and introduces regressions: LOW; limits: No regression counts, net effect sizes or comparison conditions supplied; The guidance content is not described; Limited to the paper's action budgets. - Acting on the first seemingly relevant observation and missing fine-grained page state: LOW; limits: Analyst summary of the A.10 error analysis, not an independent experiment; Frequencies not reported; Does not describe present-day agents. - Workflow abandonment, page or tab loops, undoing correct earlier actions: LOW; limits: Analyst summary of the C.4 failure analysis, examples excluded; No failure frequencies; State/history tracking was not shown to fix these failures. - Evaluating on demo sites or without resetting environment state: MEDIUM; limits: Documentation only; no measured effect of skipping the reset; Bounded excerpt with code removed; the reset procedure is not included; The 812 count comes from README prose only. - Hidden version and live-web drift in dynamic benchmarks: MEDIUM; limits: Documentation only; no measured variance; Provenance recording does not remove drift; Only one repository supports this, and independence is NOT_ESTABLISHED. - Fail-closed verification separating agent self-report from verified success: LOW; limits: Contract documentation, not a measured result; Other harnesses are not shown to behave this way; Many reporting fields depend on the producing runner; missing counters are unknown, not zero; Single-repository source; independence NOT_ESTABLISHED. - Evaluator blind spots of network-trace validation: LOW; limits: Single self-authored documentation page with no measurements; Transfer to the original WebArena release is unverified; Independence NOT_ESTABLISHED. ## OPEN QUESTIONS - Whether current models show the same tradeoff - Effect of an evidence-before-stopping rule - Cost and latency - Transfer to current models - Cost and latency of captioning or image input - Effect on current models - Cost and latency of marking - Frequency of progress-versus-state conflicts - Whether an independent completion check helps or causes regressions - Net effect size and regression rate - Which task types regress - Differences between GPT-5.5 and Qwen3.5-9B - Failure frequency - Whether a verification step reduces it - Cost and latency of verification - Whether history tracking reduces these failures - Prevalence on current models - Magnitude of contamination if the reset is skipped - Whether partial runs also require a reset - The AgentLab-path equivalent - Magnitude of drift per factor - Whether the ReproducibilityAgent detects drift in practice - Share of rows ending ungraded - Effect on reported pass rates - How often the blind spots change scores - Which task types depend solely on the network-trace evaluator - Whether the recommended behavior changes success ## SOURCES - webarena-paper: [source](https://arxiv.org/abs/2307.13854v4); PAPER; retrieved 2026-10-08T22:34:29.438Z. - visualwebarena-paper: [source](https://arxiv.org/abs/2401.13649v2); PAPER; retrieved 2026-10-08T22:34:30.526Z. - webarena-lite-human-audit: [source](https://arxiv.org/abs/2610.01491v1); PAPER; retrieved 2026-10-08T22:34:30.582Z. - webarena-overview: [source](https://github.com/web-arena-x/webarena/blob/dce04686a56253aefba7b18a4fa0937cf1dc987b/README.md); OFFICIAL_REPOSITORY; retrieved 2026-10-08T22:34:28.926Z. - browser-agentlab-reproducibility: [source](https://github.com/ServiceNow/AgentLab/blob/8488c564eea0525aaff5dea189bb03a9b529546e/README.md); OFFICIAL_REPOSITORY; retrieved 2026-10-08T22:34:35.104Z. - browser-stagehand-verifier-gates: [source](https://github.com/browserbase/stagehand/blob/2d605b099996e10c3e3ab0dcf293db8d482a2c97/packages/evals/docs/verifier-gates.md); OFFICIAL_REPOSITORY; retrieved 2026-10-08T22:34:35.347Z. - browser-webarena-verified-evaluation-methods: [source](https://github.com/ServiceNow/webarena-verified/blob/6473f72db5dcefc97b5725b59e734504edc28a21/docs/evaluation/network_event_based_evaluation.md); OFFICIAL_REPOSITORY; retrieved 2026-10-08T23:50:09.929Z.