Synaptic Radio

Principal ML engineer focused on foundation models, agentic systems and scientific AI. I write about the math, systems and engineering judgement behind reliable ML workflows.

Home
Archives
Categories
Tags
Resume
Search
1. Dark Mode

Search

Categories

Distributed Training GPU Systems and Performance Engineering Inference and Serving Tensor Parallelism Deep-Learning Mechanisms Research Methodology Software Architecture Transformer Architecture

Tags

Megatron-LM PyTorch Tensor Parallelism Actors Book-Review Circe Distributed Systems Distributed Training Greek-Mythology Grouped-Query Attention

Categories

2 pages

GPU Systems and Performance Engineering

Tensor Parallelism I: Rebuilding a Megatron SwiGLU Block Across Two Ranks

Featured image of post Tensor Parallelism I: Rebuilding a Megatron SwiGLU Block Across Two Ranks

Tensor Parallelism II: GQA Attention, One Head Group at a Time

Featured image of post Tensor Parallelism II: GQA Attention, One Head Group at a Time

© 2026 Anshuman Sahoo