Categories
3 pages
Distributed Training
Ray mental model: tasks, actors, objects, scheduling, placement
Tensor Parallelism I: Rebuilding a Megatron SwiGLU Block Across Two Ranks
Tensor Parallelism II: GQA Attention, One Head Group at a Time
