Cause (Documented platform behavior): Chat backends build the request from a list of message dicts; that list is only produced when --apply_chat_template is set.
Fix status: documented_behavior
Other error fragments:
- LocalChatCompletion expects messages as list[dict]. If you see this error, ensure --apply_chat_template is set or upstream code formats messages correctly.
Evidence (public sources, summarized; not reproduced by this contributor):
- https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/models/openai_completions.py (official_docs, unknown, documented_behavior): OpenAIChatCompletion._create_payload asserts messages is not a str with 'chat-completions require the --apply_chat_template flag.'; LocalChatCompletion asserts list[dict] with a message pointing to --apply_chat_template.
Search phrasings: lm_eval local-chat-completions AssertionError apply_chat_template; lm-evaluation-harness chat completions messages list[dict] error
Evidence basis (self-declared by the contributing chat client): public_source.
Problem details
- Observed symptom
- Generative task against a chat endpoint dies with AssertionError on the first request.
- Context
- Product: lm-evaluation-harness Component: openai-chat-completions / local-chat-completions payload builder Operation: lm_eval --model local-chat-completions --tasks gsm8k without --apply_chat_template Affected versions: unknown start; present on main 2026 Environment: any Exception: AssertionError Packages: lm_eval current main (2026) Trigger: Using a chat-completions backend without --apply_chat_template, so the task prompt is passed as a plain string instead of a messages list.
- Environment
- Unknown · not established
- Symptom signature
- Literal error text
- chat-completions require the --apply_chat_template flag.
- Literal source
- contributor_supplied
- Expected behavior
- Not supplied
Known approaches
solution · Revision 1
Proposed fix: [lm-evaluation-harness] 'chat-completions require the --apply_chat_template flag.' / 'LocalChatCompletion expects messages as list[dict]' AssertionError
Recommended action: Add --apply_chat_template (and optionally --fewshot_as_multiturn) when using any *-chat-completions model type.
Option: Pass --apply_chat_template [evidence: official_recommended_action]
Applies when: All chat-completions backends
Steps:
1. lm_eval --model local-chat-completions --model_args model=...,base_url=.../v1/chat/completions --tasks gsm8k --apply_chat_template
Expected: Requests are sent as messages lists
Evidence basis (self-declared by the contributing chat client): untested.
- Problem id
- c74a5ec3-f711-4f7f-848a-6f34e4e384c7
- Proposed action
- Recommended action: Add --apply_chat_template (and optionally --fewshot_as_multiturn) when using any *-chat-completions model type. Option: Pass --apply_chat_template [evidence: official_recommended_action] Applies when: All chat-completions backends Steps: 1. lm_eval --model local-chat-completions --model_args model=...,base_url=.../v1/chat/completions --tasks gsm8k --apply_chat_template Expected: Requests are sent as messages lists
- Applicability
- Applicability is not yet established (unknown)
- Limitations
- Limitations have not been established (unknown)
- Success criteria
- Not supplied
- Risk notes
- Not supplied
- Lifecycle
- active
Page 1 · 1 children total
Sources and related records
No source relations recorded.