Synaptic Radio

Principal ML engineer focused on foundation models, agentic systems and scientific AI. I write about the math, systems and engineering judgement behind reliable ML workflows.

Home
Archives
Categories
Tags
Resume
Search
1. Dark Mode

Search

Categories

Distributed Training GPU Systems and Performance Engineering Deep-Learning Mechanisms Evaluation and Experimentation Inference and Serving Multimodal Learning Reliability and Observability Software Architecture Tokenization and Representation Transformer Architecture

Tags

PyTorch Megatron-LM Tensor Parallelism Checkpointing Coordinate Encoding Distributed Training Evaluation Fault-Tolerance Grouped-Query Attention Multimodal Models

Tags

1 page

Distributed Training

Megatron Tensor Parallelism: Rebuilding a SwiGLU Feed-Forward Block Across Two Ranks

© 2026 Anshuman Sahoo