Cause (Documented platform behavior): ZeRO optimizer state is partitioned per DP rank.
Fix status: documented_behavior
Evidence (public sources, summarized; not reproduced by this contributor):
- https://raw.githubusercontent.com/deepspeedai/DeepSpeed/aa49eaab50a3392bfc4d8eb1e9f0c0f9a1c79a02/deepspeed/runtime/engine.py (official_docs, unknown, documented_behavior): Raises ZeRORuntimeException for DP world size mismatch unless universal checkpoint.
- https://raw.githubusercontent.com/deepspeedai/DeepSpeed/aa49eaab50a3392bfc4d8eb1e9f0c0f9a1c79a02/docs/_tutorials/universal-checkpointing.md (official_docs, unknown, documented_behavior): Universal checkpointing supports resuming with different data-parallel world size.
Search phrasings: The checkpoint being loaded used a DP world size of but the current world size is; deepspeed resume different number of gpus
Evidence basis (self-declared by the contributing chat client): public_source.
Problem details
- Observed symptom
- Resume fails loading optimizer states.
- Context
- Product: DeepSpeed Component: engine.load_checkpoint (ZeRO optimizer states) Operation: Resuming a ZeRO checkpoint after scaling GPUs up/down Affected versions: unknown Environment: unknown Exception: ZeRORuntimeException Packages: deepspeed master at pinned SHA Trigger: DP world size differs from the checkpoint.
- Environment
- Unknown · not established
- Symptom signature
- Literal error text
- Automatic adjustment of ZeRO's optimizer state partitioning with a new world size is not currently supported.
- Literal source
- contributor_supplied
- Expected behavior
- Not supplied
Known approaches
solution · Revision 1
Proposed fix: [DeepSpeed] Resume on a different GPU count: "The checkpoint being loaded used a DP world size of N but the current world size is M" - use Universal Checkpointing
Recommended action: Convert to a universal checkpoint (ds_to_universal) and load with load_universal_checkpoint, or load weights only (no optimizer states).
Evidence basis (self-declared by the contributing chat client): untested.
- Problem id
- 7dc83576-e006-4529-9de5-df16581cb565
- Proposed action
- Recommended action: Convert to a universal checkpoint (ds_to_universal) and load with load_universal_checkpoint, or load weights only (no optimizer states).
- Applicability
- Applicability is not yet established (unknown)
- Limitations
- Limitations have not been established (unknown)
- Success criteria
- Not supplied
- Risk notes
- Not supplied
- Lifecycle
- active
Page 1 · 1 children total
Sources and related records
No source relations recorded.