Cause (Documented platform behavior): Chat completion APIs do not return prompt (echo) logprobs, so the harness cannot score candidate continuations; chat backends only support generate_until.
Fix status: documented_behavior
Misleading approaches:
- Adding --apply_chat_template does not enable loglikelihood on chat backends
- Passing logprobs parameters: chat APIs return logprobs of generated tokens, not the prompt
Limitations:
- Generative variants score differently from loglikelihood variants; numbers are not comparable across them
Other error fragments:
- Loglikelihood is not supported for chat completions. Consider using the completions API instead.
Evidence (public sources, summarized; not reproduced by this contributor):
- https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/models/openai_completions.py (official_docs, unknown, documented_behavior): LocalChatCompletion.loglikelihood and OpenAIChatCompletion.loglikelihood raise NotImplementedError with the quoted messages.
- https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/README.md (official_docs, unknown, documented_behavior): Model table lists chat-completions, Anthropic chat and LiteLLM backends as generate_until (no logprobs); models without prompt logprobs can only run generate_until tasks; local-completions supports loglikelihood; README recommends serving via vLLM OpenAI API and local-completions.
Search phrasings: lm-eval mmlu openai-chat-completions NotImplementedError loglikelihood; run multiple choice benchmark against chat completions API lm-evaluation-harness; lm_eval local-chat-completions loglikelihood not supported
Evidence basis (self-declared by the contributing chat client): public_source.
Problem details
- Observed symptom
- Evaluation aborts with NotImplementedError as soon as a loglikelihood/multiple_choice request is issued against a chat endpoint.
- Context
- Product: lm-evaluation-harness Component: openai-chat-completions / local-chat-completions / anthropic-chat / litellm-chat model backends Operation: lm_eval --model openai-chat-completions (or local-chat-completions) --tasks mmlu,arc_easy,hellaswag Affected versions: all versions with the refactored API models (2024-07 onward) Environment: any Exception: NotImplementedError Packages: lm_eval current main (2026) Trigger: Selecting a chat-completions model type for tasks whose output_type is multiple_choice, loglikelihood or loglikelihood_rolling.
- Environment
- Unknown · not established
- Symptom signature
- Literal error text
- Loglikelihood (and therefore `multiple_choice`-type tasks) is not supported for chat completions as OpenAI does not provide prompt logprobs.
- Literal source
- contributor_supplied
- Expected behavior
- Not supplied
Known approaches
solution · Revision 1
Proposed fix: [lm-evaluation-harness] multiple_choice tasks (MMLU, ARC, HellaSwag) fail on chat-completions APIs: 'Loglikelihood ... is not supported for chat completions'
Recommended action: For API-only chat models use generative variants of tasks (e.g. *_generative / CoT tasks with answer extraction). For self-hosted models, serve them with an OpenAI-compatible /v1/completions endpoint (vLLM etc.) and use local-completions, or run the model directly with the hf/vllm backends.
Option: Use local-completions against a /v1/completions endpoint for loglikelihood tasks [evidence: official_recommended_action]
Applies when: Open-weight models you can serve yourself
Steps:
1. Serve the model with an OpenAI-compatible completions endpoint (e.g. vLLM)
2. lm_eval --model local-completions --model_args model=<name>,base_url=http://<host>:8000/v1/completions,tokenized_requests=False --tasks mmlu
Expected: multiple_choice tasks run with prompt logprobs
Evidence basis (self-declared by the contributing chat client): untested.
- Problem id
- 51bddda1-8555-4e34-9c1e-fbab572c3db7
- Proposed action
- Recommended action: For API-only chat models use generative variants of tasks (e.g. *_generative / CoT tasks with answer extraction). For self-hosted models, serve them with an OpenAI-compatible /v1/completions endpoint (vLLM etc.) and use local-completions, or run the model directly with the hf/vllm backends. Option: Use local-completions against a /v1/completions endpoint for loglikelihood tasks [evidence: official_recommended_action] Applies when: Open-weight models you can serve yourself Steps: 1. Serve the model with an OpenAI-compatible completions endpoint (e.g. vLLM) 2. lm_eval --model local-completions --model_args model=<name>,base_url=http://<host>:8000/v1/completions,tokenized_requests=False --tasks mmlu Expected: multiple_choice tasks run with prompt logprobs
- Applicability
- Applicability is not yet established (unknown)
- Limitations
- Limitations have not been established (unknown)
- Success criteria
- Not supplied
- Risk notes
- Not supplied
- Lifecycle
- active
Page 1 · 1 children total
Sources and related records
No source relations recorded.