KFA Benchmark Commons — Workbench 01: tools field guide

# KFA Benchmark Commons — tools field guide Workbench 01 family research stage. Each included finding passed separate Claude analysis, adversarial review and source verification; these roles share one operator and do not constitute independent reproduction. Original summaries only; benchmark-answer-free intent. No official benchmark endorsement or measured performance improvement is claimed. Sources were bounded excerpts, not full deep reads. ## WHAT THIS TASK FAMILY TESTS - evaluation reporting - API error, retry and timeout configuration - Tool safety metadata interpretation - Multi-turn function calling evaluation ## COMMON FAILURE MODES - Silent omission of missing generations (wb01-tools-e05878462fcc95b8ca) - Deflated summary columns from unevaluated categories counted as 0 (wb01-tools-e05878462fcc95b8ca) - Hidden retries consuming time or rate-limit budget (wb01-tools-c159c6aae0cbe3a7de) - Long waits before a timeout surfaces (wb01-tools-c159c6aae0cbe3a7de) - Auto-approving a mislabeled tool could cause unintended writes (INFERENCE) (wb01-tools-505e5ac4157b345ee0) - A write leaving the wrong end-of-turn state could fail (wb01-tools-e2873002cb4978ca3c) - A missed key read call could fail response-based checks (wb01-tools-e2873002cb4978ca3c) - A turn with no valid function call ends (wb01-tools-e2873002cb4978ca3c) ## ENVIRONMENT / EVALUATOR TRAPS - Partial evaluation yields scores not comparable to the leaderboard: The BFCL repository README states that with the partial-eval flag the evaluator silently skips IDs missing from the model result file and computes accuracy on the remaining subset, so the score may differ from a full-set evaluation and may not match official leaderboard numbers. (finding wb01-tools-e05878462fcc95b8ca; LIMITED; transfer MEDIUM). - Know and set SDK retry and timeout defaults when running tool agents: The OpenAI Node SDK README states that connection errors, 408, 409, 429 and 5xx responses are retried 2 times by default with short exponential backoff, and requests time out after 10 minutes by default, with timed-out requests also retried twice. (finding wb01-tools-c159c6aae0cbe3a7de; LIMITED; transfer LOW). - Do not treat tool annotation hints as enforced safety guarantees: The MCP Python SDK docs say tool annotations such as read-only and destructive are behavioural hints for clients, not security, and that clients may not honour them. (finding wb01-tools-505e5ac4157b345ee0; LIMITED; transfer MEDIUM). - Match call planning to evaluation mode: end-of-turn state for writes, minimum necessary path for reads: The EvalScope BFCL-v3 page says multi-turn state-based evaluation compares the API system's internal state at the end of each dialogue turn for write/delete operations, and response-based evaluation checks that the call trajectory contains the minimum necessary path while allowing redundant steps for read operations. (finding wb01-tools-e2873002cb4978ca3c; LIMITED; transfer LOW). ## PROMISING STRATEGIES No supported finding was accepted from the supplied excerpts. ## NOT-PROMISING STRATEGIES No supported finding was accepted from the supplied excerpts. ## RECOVERY PLAYBOOK - INFERENCE: label subset scores as partial, do not compare with leaderboard figures, and confirm the result count matches the intended entries. (wb01-tools-e05878462fcc95b8ca) - INFERENCE: under step or time budgets set maxRetries and timeout explicitly and log the request ID. (wb01-tools-c159c6aae0cbe3a7de) - INFERENCE (untested): do not infer safety from a read-only annotation; apply independent confirmation or sandboxing for tools that may write or delete. (wb01-tools-505e5ac4157b345ee0) - HYPOTHESIS: leave the backend in the intended state after each write turn and ensure key read calls appear; some redundant reads may be tolerated within the step cap. (wb01-tools-e2873002cb4978ca3c) ## COST / LATENCY TRADEOFFS - Partial evaluation yields scores not comparable to the leaderboard: cost UNKNOWN; latency UNKNOWN. - Know and set SDK retry and timeout defaults when running tool agents: cost UNKNOWN; latency UNKNOWN. - Do not treat tool annotation hints as enforced safety guarantees: cost UNKNOWN; latency UNKNOWN. - Match call planning to evaluation mode: end-of-turn state for writes, minimum necessary path for reads: cost UNKNOWN; latency UNKNOWN. ## WHAT IS BENCHMARK-SPECIFIC - BFCL V4 repository snapshot; applicability: Subset or debug runs of BFCL; Comparing local scores with published numbers (wb01-tools-e05878462fcc95b8ca). - OpenAI SDK git snapshot; applicability: Agent harnesses using the OpenAI Node SDK at this snapshot; Runs with wall-clock or step budgets (wb01-tools-c159c6aae0cbe3a7de). - MCP Python SDK snapshot; protocol release varies; MCP TypeScript SDK snapshot; protocol release varies; applicability: Agents or hosts that act on MCP tool metadata (wb01-tools-505e5ac4157b345ee0). - BFCL v3; EvalScope integration snapshot; applicability: BFCL v3 multi-turn subsets as described by the EvalScope doc (wb01-tools-e2873002cb4978ca3c). ## WHAT MAY TRANSFER TO REAL AGENTS - Partial evaluation yields scores not comparable to the leaderboard: MEDIUM; limits: Documentation only; evaluator not executed; Single README at this git snapshot; Snapshot is labeled V4 but README example paths name BFCL_v3 files; version labeling is mixed. - Know and set SDK retry and timeout defaults when running tool agents: LOW; limits: Defaults may change across versions; Worst-case latency not stated in source; Node SDK only. - Do not treat tool annotation hints as enforced safety guarantees: MEDIUM; limits: Documentation only; no incident or measurement data; Both docs come from one organization; not independent experiments; Applying host-role guidance to an agent's own decisions is an extrapolation; Protocol release varies across the two snapshots. - Match call planning to evaluation mode: end-of-turn state for writes, minimum necessary path for reads: LOW; limits: Third-party integration doc, not upstream BFCL documentation; independence NOT_ESTABLISHED; Gives applicable scenarios per evaluator, not per-task assignment; No scores or measured effects. ## OPEN QUESTIONS - Behavior in other BFCL versions - Magnitude and direction of subset vs full-set gap - Behavior of non-partial evaluation on missing IDs - Retry frequency in practice - Effect on task success, cost or latency - Rate of inaccurate annotations in deployed servers - Cost and latency of extra confirmation - Whether the upstream harness matches this description - Whether unnecessary writes are penalized - Whether both evaluators can apply to one turn - Whether in-turn state is checked before turn end ## SOURCES - bfcl-overview: [source](https://github.com/ShishirPatil/gorilla/blob/6ea57973c7a6097fd7c5915698c54c17c5b1b6c8/berkeley-function-call-leaderboard/README.md); OFFICIAL_REPOSITORY; retrieved 2026-10-08T20:46:29.707Z. - openai-errors: [source](https://github.com/openai/openai-node/blob/bc6c0bfb70f253d5caa3f335699e9713ea9067b5/README.md); OFFICIAL_REPOSITORY; retrieved 2026-10-08T22:34:16.158Z. - tools-mcp-python-schemas: [source](https://github.com/modelcontextprotocol/python-sdk/blob/91941ed4d3985d59def99e090baa3f880c626cc8/docs/servers/tools.md); OFFICIAL_REPOSITORY; retrieved 2026-10-08T22:34:23.348Z. - tools-mcp-typescript-schemas: [source](https://github.com/modelcontextprotocol/typescript-sdk/blob/b022522089a0c8b632595c6e7b536453945ed5a9/docs/servers/tools.md); OFFICIAL_REPOSITORY; retrieved 2026-10-08T22:34:23.818Z. - tools-evalscope-bfcl-v3: [source](https://github.com/modelscope/evalscope/blob/715b9a79a13db29c3f251e1ad12c428623a9318c/docs/en/third_party/bfcl_v3.md); OFFICIAL_REPOSITORY; retrieved 2026-10-08T22:34:22.222Z.