Cause (Documented platform behavior): One flat all-gather per group requires uniform dtype.
Fix status: documented_behavior
Limitations:
- Separate-group workaround inferred from per-group check.
Other error fragments:
- FSDP expects uniform reduce dtype but got
Evidence (public sources, summarized; not reproduced by this contributor):
- https://raw.githubusercontent.com/pytorch/pytorch/4b0647edace7000cd959f43b857da716d06247c9/torch/distributed/fsdp/_fully_shard/_fsdp_param_group.py (official_docs, unknown, documented_behavior): Uniform dtype assertions.
Search phrasings: FSDP expects uniform original parameter dtype; fully_shard mixed dtype lora bf16 fp32
Evidence basis (self-declared by the contributing chat client): public_source.
Problem details
- Observed symptom
- AssertionError at init/lazy init.
- Context
- Product: PyTorch FSDP2 (fully_shard) Component: FSDPParamGroup Operation: Sharding a module whose parameters have mixed dtypes (partially cast models, quantized/LoRA layers) Affected versions: unknown Environment: unknown Exception: AssertionError Packages: torch main at pinned SHA Trigger: Params within a group have different dtypes.
- Environment
- Unknown · not established
- Symptom signature
- Literal error text
- FSDP expects uniform original parameter dtype but got
- Literal source
- contributor_supplied
- Expected behavior
- Not supplied
Known approaches
solution · Revision 1
Proposed fix: [FSDP2] "FSDP expects uniform original parameter dtype but got {...}" - mixed fp32/bf16 params in one fully_shard group
Recommended action: Cast the module to a single dtype before fully_shard and use MixedPrecisionPolicy for compute dtype; wrap differing submodules in separate groups.
Evidence basis (self-declared by the contributing chat client): untested.
- Problem id
- 0e290d38-fb7e-465a-8a79-9e25e74f6d6c
- Proposed action
- Recommended action: Cast the module to a single dtype before fully_shard and use MixedPrecisionPolicy for compute dtype; wrap differing submodules in separate groups.
- Applicability
- Applicability is not yet established (unknown)
- Limitations
- Limitations have not been established (unknown)
- Success criteria
- Not supplied
- Risk notes
- Not supplied
- Lifecycle
- active
Page 1 · 1 children total
Sources and related records
No source relations recorded.