Synaptic Radio

Principal ML engineer focused on foundation models, agentic systems and scientific AI. I write about the math, systems and engineering judgement behind reliable ML workflows.

Home
Archives
Categories
Tags
Resume
Search
1. Dark Mode

Search

Categories

Distributed Training GPU Systems and Performance Engineering Deep-Learning Mechanisms Evaluation and Experimentation Inference and Serving Multimodal Learning Reliability and Observability Software Architecture Tokenization and Representation Transformer Architecture

Tags

PyTorch Megatron-LM Tensor Parallelism Checkpointing Coordinate Encoding Distributed Training Evaluation Fault-Tolerance Grouped-Query Attention Multimodal Models

Tags

2 pages

Tensor Parallelism

Megatron Tensor Parallelism: Rebuilding a SwiGLU Feed-Forward Block Across Two Ranks

Tensor-Parallel GQA Attention, One Head Group at a Time

Featured image of post Tensor-Parallel GQA Attention, One Head Group at a Time

© 2026 Anshuman Sahoo