Distribute large model training across GPUs

Add fully_shard parameter distribution to PyTorch training

Sharding large models across GPUs with DTensor-based parameter sharding and simpler checkpoint semantics vs FSDP1.

• · PyTorch model
• · target GPU count
• · mixed precision policy config

• · FSDP2-wrapped model
• · DTensor-aware optimizer
• · DCP checkpoint helper

• · torch (2.4+)
• · Distributed Checkpoint (DCP)

PyTorch 2.4+, torchrun available, model fits on meta device

• · top-down sharding only (not bottom-up)
• · optimizer created before sharding
• · naïve state_dict() loses DTensor structure

• · User contract formalized (5 rules)
• · bottom-up sharding pattern documented
• · DeviceMesh integration shown