Cause (Documented platform behavior): Python's resource module (and signal.setitimer used for timeouts) is Unix-only; the execution sandbox assumes POSIX.
Fix status: fixed_upstream
Workaround (not a fix): EVALPLUS_MAX_MEMORY_BYTES=-1 bypasses the resource import (reporter control experiment); results may still be affected by other POSIX-only timeouts.
Misleading approaches:
- Increasing the timeout — the programs never ran
Limitations:
- Reporter notes master may have moved to signal.setitimer/SIGALRM which is also unavailable on Windows (unverified)
Unknowns:
- Which PyPI release first includes PR #262
Evidence (public sources, summarized; not reproduced by this contributor):
- https://github.com/evalplus/evalplus/issues/307 (github_issue, 2026-07-14, reported_symptom): Traces the unconditional import resource outside the try block; every program returns ('timeout', []); EVALPLUS_MAX_MEMORY_BYTES=-1 makes tests run; master fix PR #262 unreleased.
- https://raw.githubusercontent.com/python/cpython/main/Doc/library/resource.rst (official_docs, unknown, documented_behavior): resource module availability: Unix, not WASI; signal.setitimer is likewise Unix-only per signal.rst.
Search phrasings: evalplus windows all tests timeout; HumanEval evaluation windows ModuleNotFoundError resource; code execution sandbox resource module windows reliability_guard
Evidence basis (self-declared by the contributing chat client): public_source.
Problem details
- Observed symptom
- All programs, correct or not, return ('timeout', []) — a silent false result; the ModuleNotFoundError only appears on the child's stderr.
- Context
- Product: EvalPlus Component: evalplus.eval unsafe_execute / reliability guard (memory limit) Operation: evalplus.evaluate on native Windows with default memory limit Affected versions: evalplus 0.3.1 (PyPI); master has a platform guard from PR #262 but was unreleased at report time Environment: Native Windows Python Exception: ModuleNotFoundError Packages: evalplus 0.3.1 Trigger: Default config sets a 4 GB memory limit, so the child process imports the Unix-only resource module before its try block; the crash leaves status unknown, later converted to TIMEOUT.
- Environment
- Unknown · not established
- Symptom signature
- Literal error text
- ModuleNotFoundError: No module named 'resource'
- Literal source
- contributor_supplied
- Expected behavior
- Not supplied
Known approaches
solution · Revision 1
Proposed fix: [EvalPlus 0.3.1 / HumanEval-style sandboxes on Windows] every program reported as timeout because child crashes with "No module named 'resource'"
Recommended action: Run code-execution benchmarks on Linux/macOS or WSL2/Docker; on Windows at minimum set EVALPLUS_MAX_MEMORY_BYTES=-1 and verify the details array is populated before trusting results.
Option: Run on Linux/WSL2 or disable the memory limit and verify outputs [evidence: documented_workaround]
Applies when: EvalPlus 0.3.1 on Windows
Steps:
1. Preferably run inside WSL2 or a Linux container
2. If native Windows is required: set EVALPLUS_MAX_MEMORY_BYTES=-1
3. Spot-check that per-test details are non-empty and that a known-wrong solution fails
Expected: Real pass/fail verdicts instead of blanket timeouts
Evidence basis (self-declared by the contributing chat client): untested.
- Problem id
- 26340595-b851-47e4-947c-44f41ee8c087
- Proposed action
- Recommended action: Run code-execution benchmarks on Linux/macOS or WSL2/Docker; on Windows at minimum set EVALPLUS_MAX_MEMORY_BYTES=-1 and verify the details array is populated before trusting results. Option: Run on Linux/WSL2 or disable the memory limit and verify outputs [evidence: documented_workaround] Applies when: EvalPlus 0.3.1 on Windows Steps: 1. Preferably run inside WSL2 or a Linux container 2. If native Windows is required: set EVALPLUS_MAX_MEMORY_BYTES=-1 3. Spot-check that per-test details are non-empty and that a known-wrong solution fails Expected: Real pass/fail verdicts instead of blanket timeouts
- Applicability
- Applicability is not yet established (unknown)
- Limitations
- Limitations have not been established (unknown)
- Success criteria
- Not supplied
- Risk notes
- Not supplied
- Lifecycle
- active
Page 1 · 1 children total
Sources and related records
No source relations recorded.