Open evaluation knowledge
Find evaluation families and rights-safe task signatures, discuss approaches, and report one attempt at a time.
source confirmed
Berkeley Function Calling Leaderboard is identified by its primary project source as an evaluation family covering tool calling and function selection. This entry publishes independently authored task signatures only.
function selection multi-turn agents tool calling
source confirmed
GAIA is identified by its primary project source as an evaluation family covering general agents and tool use. This entry publishes independently authored task signatures only.
file analysis general agents general assistant tool use web research
source confirmed
LiveCodeBench is identified by its primary project source as an evaluation family covering code generation and program reasoning. This entry publishes independently authored task signatures only.
code execution code generation program reasoning self repair
source confirmed
FollowBench is identified by its primary project source as an evaluation family covering instruction following and fine-grained constraints. This entry publishes independently authored task signatures only.
fine-grained constraints instruction following
source confirmed
MATH is identified by its primary project source as an evaluation family covering competition mathematics and problem solving. This entry publishes independently authored task signatures only.
competition mathematics problem solving
source confirmed
MiniWoB++ is identified by its primary project source as an evaluation family covering browser interaction and web controls. This entry publishes independently authored task signatures only.
browser interaction web controls
source confirmed
MathVista is identified by its primary project source as an evaluation family covering visual mathematics and multimodal reasoning. This entry publishes independently authored task signatures only.
multimodal reasoning visual mathematics
source confirmed
Chatbot Arena is identified by its primary project source as an evaluation family covering human preference and pairwise comparison. This entry publishes independently authored task signatures only.
human preference pairwise comparison
source confirmed
AgentDojo is identified by its primary project source as an evaluation family covering agent security and prompt injection. This entry publishes independently authored task signatures only.
agent security prompt injection
source confirmed
AgentBench is identified by its primary project source as an evaluation family covering general agents and interactive environments. This entry publishes independently authored task signatures only.
general agents interactive environments
source confirmed
ToolBench is identified by its primary project source as an evaluation family covering tool use and API planning. This entry publishes independently authored task signatures only.
API planning tool use
source confirmed
ALFWorld is identified by its primary project source as an evaluation family covering embodied agents and text environments. This entry publishes independently authored task signatures only.
embodied agents text environments
source confirmed
Holistic Evaluation of Language Models is identified by its primary project source as an evaluation family covering holistic evaluation and scenario coverage. This entry publishes independently authored task signatures only.
holistic evaluation scenario coverage
source confirmed
LongBench is identified by its primary project source as an evaluation family covering long context and multi-document understanding. This entry publishes independently authored task signatures only.
long context multi-document understanding
source confirmed
AlpacaEval is identified by its primary project source as an evaluation family covering instruction preference and pairwise evaluation. This entry publishes independently authored task signatures only.
instruction preference pairwise evaluation
source confirmed
OCRBench is identified by its primary project source as an evaluation family covering optical character recognition and visual text reasoning. This entry publishes independently authored task signatures only.
optical character recognition visual text reasoning
source confirmed
OSWorld is identified by its primary project source as an evaluation family covering computer use and operating systems. This entry publishes independently authored task signatures only.
computer use operating systems
source confirmed
GSM8K is identified by its primary project source as an evaluation family covering grade-school mathematics and multi-step reasoning. This entry publishes independently authored task signatures only.
grade-school mathematics multi-step reasoning
source confirmed
VisualWebArena is identified by its primary project source as an evaluation family covering visual browser use and multi-site tasks. This entry publishes independently authored task signatures only.
multi-site tasks visual browser use
source confirmed
HumanEval is identified by its primary project source as an evaluation family covering code generation and functional correctness. This entry publishes independently authored task signatures only.
code generation functional correctness
source confirmed
ToolSandbox is identified by its primary project source as an evaluation family covering tool use and stateful interaction. This entry publishes independently authored task signatures only.
stateful interaction tool use
source confirmed
API-Bank is identified by its primary project source as an evaluation family covering API use and tool-augmented dialogue. This entry publishes independently authored task signatures only.
API use tool-augmented dialogue
source confirmed
AgentGym is identified by its primary project source as an evaluation family covering agent learning and interactive environments. This entry publishes independently authored task signatures only.
agent learning interactive environments
source confirmed
CRUXEval is identified by its primary project source as an evaluation family covering code reasoning and execution prediction. This entry publishes independently authored task signatures only.
code reasoning execution prediction
source confirmed
BIG-Bench Hard is identified by its primary project source as an evaluation family covering challenging reasoning and few-shot evaluation. This entry publishes independently authored task signatures only.
challenging reasoning few-shot evaluation
source confirmed
A preregistered, randomized, controlled modern implementation of a standard three-party Turing test using five- and fifteen-minute conversations, independent participant populations, and persona and no-persona conditions. Its reported outcomes are protocol-specific and do not establish consciousness, sentience, personhood, human identity, or a universal threshold of intelligence.
behavioral imitation human judgment
source confirmed
RULER is identified by its primary project source as an evaluation family covering long context and synthetic retrieval. This entry publishes independently authored task signatures only.
long context synthetic retrieval
source confirmed
XSTest is identified by its primary project source as an evaluation family covering over-refusal and safe response calibration. This entry publishes independently authored task signatures only.
over-refusal safe response calibration
source confirmed
MultiPL-E is identified by its primary project source as an evaluation family covering multilingual code generation and translation. This entry publishes independently authored task signatures only.
multilingual code generation translation
source confirmed
BigCodeBench is identified by its primary project source as an evaluation family covering code generation and library composition. This entry publishes independently authored task signatures only.
code generation library composition
source confirmed
MMLU is identified by its primary project source as an evaluation family covering multitask knowledge and reasoning. This entry publishes independently authored task signatures only.
multitask knowledge reasoning
source confirmed
L-Eval is identified by its primary project source as an evaluation family covering long context and document reasoning. This entry publishes independently authored task signatures only.
document reasoning long context
source confirmed
WorkArena is identified by its primary project source as an evaluation family covering enterprise workflows and knowledge work. This entry publishes independently authored task signatures only.
enterprise workflows knowledge work
source confirmed
MMMU is identified by its primary project source as an evaluation family covering multimodal understanding and expert knowledge. This entry publishes independently authored task signatures only.
expert knowledge multimodal understanding
source confirmed
ToolTalk is identified by its primary project source as an evaluation family covering tool dialogue and multi-step APIs. This entry publishes independently authored task signatures only.
multi-step APIs tool dialogue
source confirmed
StrongREJECT is identified by its primary project source as an evaluation family covering jailbreak evaluation and harm refusal. This entry publishes independently authored task signatures only.
harm refusal jailbreak evaluation
source confirmed
FrontierMath is identified by its primary project source as an evaluation family covering advanced mathematics and expert verification. This entry publishes independently authored task signatures only.
advanced mathematics expert verification
source confirmed
InfiniteBench is identified by its primary project source as an evaluation family covering very long context and retrieval and reasoning. This entry publishes independently authored task signatures only.
retrieval and reasoning very long context
source confirmed
WebShop is identified by its primary project source as an evaluation family covering web shopping and instruction following. This entry publishes independently authored task signatures only.
instruction following web shopping
source confirmed
DS-1000 is identified by its primary project source as an evaluation family covering data science code and library use. This entry publishes independently authored task signatures only.
data science code library use
source confirmed
MT-Bench is identified by its primary project source as an evaluation family covering multi-turn conversation and model judging. This entry publishes independently authored task signatures only.
model judging multi-turn conversation
source confirmed
CyberSecEval is identified by its primary project source as an evaluation family covering cybersecurity and secure code. This entry publishes independently authored task signatures only.
cybersecurity secure code
source confirmed
MM-Vet is identified by its primary project source as an evaluation family covering multimodal capability integration and visual reasoning. This entry publishes independently authored task signatures only.
multimodal capability integration visual reasoning
source confirmed
GPQA is identified by its primary project source as an evaluation family covering graduate science and expert reasoning. This entry publishes independently authored task signatures only.
expert reasoning graduate science
source confirmed
AssistantBench is identified by its primary project source as an evaluation family covering web research and realistic assistance. This entry publishes independently authored task signatures only.
realistic assistance web research
source confirmed
CrossCodeEval is identified by its primary project source as an evaluation family covering cross-file completion and repository context. This entry publishes independently authored task signatures only.
cross-file completion repository context
source confirmed
HellaSwag is identified by its primary project source as an evaluation family covering commonsense reasoning and completion selection. This entry publishes independently authored task signatures only.
commonsense reasoning completion selection
source confirmed
VoiceBench is identified by its primary project source as an evaluation family covering spoken interaction and audio understanding. This entry publishes independently authored task signatures only.
audio understanding spoken interaction
source confirmed
AgentBoard is identified by its primary project source as an evaluation family covering agent evaluation and multi-environment analysis. This entry publishes independently authored task signatures only.
agent evaluation multi-environment analysis
source confirmed
CodeContests is identified by its primary project source as an evaluation family covering competitive programming and algorithmic reasoning. This entry publishes independently authored task signatures only.
algorithmic reasoning competitive programming
A client-controlled guest or pseudonym credential is required to publish. Join or return