At the Ray Summit 2026, I noticed a distinct engineering design pattern of “disaggregation” or as I like to think about it - phase separation emerging.
Data pipeline, RL, multimodal models, serving etc. all run optimally on different hardware/resource profiles. Disaggregation means allowing those phases to run, scale, or use hardware independently - akin to microservices.
Each of these phases then creates a complex system boundary/contract. Data must cross it, the sender and receiver can run at different speeds leading to queues and suboptimal performance, and either side can fail without the other due to version conflicts or transient failures, breaking down the entire system.
There is a clear trade off - Put every phase in one process and one accelerator pool, and unlike work competes for the same resources. Split every phase into its own service, and each handoff requires a queue, network transfer, compatibility rules, and failure handling.
Some examples of these phases:
| Disaggregation | Why phases differ | State crossing the boundary |
|---|---|---|
| Rollout inference → RL learner | Autoregressive generation vs forward/backward + optimizer state | Trajectories, logprobs, rewards, policy versions; weights return |
| Encoder → LM → generative decoder | Vision/audio encoders, language model, diffusion/decoder stages need different hardware/batch shapes | Embeddings, hidden states, conditioning tensors |
| Draft training → target serving | Speculator learning and target inference scale independently | Hidden states/logits/training samples, draft weights |
The main point is - Disaggregation is not free. It replaces idle time with queues, transport, consistency, and scheduling problems. The break-even question is therefore: does independent scaling plus better specialization save more time/cost than the boundary adds in transfer, orchestration, and staleness?
From a platform design perspective, variables like the scale and resource costs also play a part in these kind of decisions. Two big things to ponder for me - do we keep referring to model-, data-, task- specific recipes to build these pipelines? can we automate this at a system level such that we can meet the demands of each specific workload without affecting the ergonomics of training and inference?