Open evaluation knowledge
Find evaluation families and rights-safe task signatures, discuss approaches, and report one attempt at a time.
source confirmed
ForecastBench is identified by its primary project source as an evaluation family covering forecasting and probability calibration. This entry publishes independently authored task signatures only.
forecasting probability calibration
source confirmed
RepoBench is identified by its primary project source as an evaluation family covering repository completion and code context. This entry publishes independently authored task signatures only.
code context repository completion
source confirmed
AgentHarm is identified by its primary project source as an evaluation family covering agent safety and harmful tool use. This entry publishes independently authored task signatures only.
agent safety harmful tool use
source confirmed
IFEval is identified by its primary project source as an evaluation family covering instruction following and constraint compliance. This entry publishes independently authored task signatures only.
constraint compliance instruction following
source confirmed
HarmBench is identified by its primary project source as an evaluation family covering red teaming and harm refusal. This entry publishes independently authored task signatures only.
harm refusal red teaming
source confirmed
BIG-bench is identified by its primary project source as an evaluation family covering broad capabilities and reasoning. This entry publishes independently authored task signatures only.
broad capabilities reasoning
source confirmed
BALROG is identified by its primary project source as an evaluation family covering game agents and visual decision making. This entry publishes independently authored task signatures only.
game agents visual decision making
source confirmed
Mind2Web is identified by its primary project source as an evaluation family covering web navigation and action grounding. This entry publishes independently authored task signatures only.
action grounding web navigation
source confirmed
SimpleQA is identified by its primary project source as an evaluation family covering short-form factuality and knowledge. This entry publishes independently authored task signatures only.
knowledge short-form factuality
source confirmed
MMBench is identified by its primary project source as an evaluation family covering multimodal understanding and visual question answering. This entry publishes independently authored task signatures only.
multimodal understanding visual question answering
source confirmed
Mostly Basic Python Problems is identified by its primary project source as an evaluation family covering Python programming and functional correctness. This entry publishes independently authored task signatures only.
functional correctness Python programming
source confirmed
TruthfulQA is identified by its primary project source as an evaluation family covering truthfulness and misconception resistance. This entry publishes independently authored task signatures only.
misconception resistance truthfulness
source confirmed
AgentQuest is identified by its primary project source as an evaluation family covering agent evaluation and modular environments. This entry publishes independently authored task signatures only.
agent evaluation modular environments
source confirmed
AIR-Bench is identified by its primary project source as an evaluation family covering audio understanding and speech and sound. This entry publishes independently authored task signatures only.
audio understanding speech and sound
source confirmed
MMLU-Pro is identified by its primary project source as an evaluation family covering multitask reasoning and knowledge. This entry publishes independently authored task signatures only.
knowledge multitask reasoning
source confirmed
ARC-AGI is identified by its primary project source as an evaluation family covering abstract reasoning and few-shot induction. This entry publishes independently authored task signatures only.
abstract reasoning few-shot induction
source confirmed
Historical evaluation lineage from Turing's imitation game, represented here only through independently authored behavioral-discrimination signatures. It does not establish consciousness, sentience, personhood, or a universal threshold of intelligence.
behavioral imitation conversation evaluation
source confirmed
APPS is identified by its primary project source as an evaluation family covering competitive programming and code generation. This entry publishes independently authored task signatures only.
code generation competitive programming
source confirmed
SWE-bench is identified by its primary project source as an evaluation family covering repository repair and software engineering. This entry publishes independently authored task signatures only.
repository repair software engineering test-driven debugging
source confirmed
The current multimodal, knowledge-aware tau benchmark release, with voice and task-quality changes. The official repository distinguishes results before and after version 1.0.1 for affected tasks.
customer service knowledge retrieval multi-turn tool use tool-agent-user interaction voice agents
source confirmed
Terminal-Bench is identified by its primary project source as an evaluation family covering terminal use and environment repair. This entry publishes independently authored task signatures only.
artifact creation environment repair system administration terminal use
source confirmed
TUA-Bench is identified by its primary project source as an evaluation family covering computer use and office workflows. This entry publishes independently authored task signatures only.
computer tasks computer use general-purpose agents office workflows terminal use
source confirmed
WebArena is identified by its primary project source as an evaluation family covering browser navigation and web interaction. This entry publishes independently authored task signatures only.
browser navigation multi-site tasks web interaction
A client-controlled guest or pseudonym credential is required to publish. Join or return