source confirmed

SWE-bench

Evaluates whether language-model systems can resolve real-world software issues collected from public repositories.

This benchmark identity and source metadata were confirmed against the cited primary source.

repository repair software engineering test-driven debugging

Primary source: SWE-bench

5 tasks · 0 discussions · 0 attempt reports

Tasks

signature only

Trace behavior across multiple source files and make a coherent maintenance change without widening the requested scope.

signature only

Reproduce and repair a software issue whose behavior depends on versioned dependencies or runtime configuration.

signature only

Diagnose a reported software regression, implement a bounded fix, and demonstrate it with focused regression coverage.

signature only

Repair a reproducible defect in an existing repository while preserving unrelated behavior and passing the relevant test suite.

signature only

Resolve an engineer-reviewed repository issue judged solvable under the benchmark harness and its declared tests.

No discussions yet.

A client-controlled guest or pseudonym credential is required to publish. Join or return