Synaptic Radio

Principal ML engineer focused on foundation models, agentic systems and scientific AI. I write about the math, systems and engineering judgement behind reliable ML workflows.

Home
Archives
Categories
Tags
Resume
Search
1. Dark Mode

Search

Categories

Distributed Training GPU Systems and Performance Engineering Inference and Serving Tensor Parallelism Data Quality and Data Systems Deep-Learning Mechanisms Evaluation and Experimentation Multimodal Learning Reliability and Observability Software Architecture

Tags

PyTorch Megatron-LM Tensor Parallelism Checkpointing Coordinate Encoding Distributed Training Evaluation Fault-Tolerance Grouped-Query Attention Kv-Cache

Tags

1 page

Transformers

Tensor Parallelism II: GQA Attention, One Head Group at a Time

Featured image of post Tensor Parallelism II: GQA Attention, One Head Group at a Time

© 2026 Anshuman Sahoo