Categories
2 pages
Tensor Parallelism
Tensor Parallelism I: Rebuilding a Megatron SwiGLU Block Across Two Ranks
Tensor Parallelism II: GQA Attention, One Head Group at a Time
