> For the complete documentation index, see [llms.txt](https://handbook.modular.com/llms.txt). > Markdown versions of all pages are available by appending .md to any URL. # Multi-model inference pipelines A multi-model inference pipeline is a system where several models work together to produce one result. Instead of asking a single model to do everything, you split the work into stages. Each stage focuses on a specific task, like retrieval, OCR, classification, generation, or post-processing. This is different from running one model behind a single endpoint. It’s also different from [pipeline parallelism](https://handbook.modular.com/inference-optimization/data-tensor-pipeline-expert-hybrid-parallelism.md), which splits one model across multiple devices. Here, the main question is not how to distribute one model. It is how to design, deploy, and operate a system where several models cooperate in one request path. ## What a multi-model pipeline looks like The simplest way to picture this is a pipeline. Many real systems, however, look more like an inference graph than a straight line. Some stages run in parallel. Some requests branch into different downstream paths. Some stages are optional. Different patterns mean different trade-offs. ### Sequential pipelines This is the most straightforward setup. Each stage feeds into the next. ```text Input → Stage A → Stage B → Stage C → Output ``` Sequential pipelines are conceptually simple, but their latencies add up. If each stage takes 50 ms, four stages can easily turn into a few hundred milliseconds of end-to-end latency before the final generation step even begins. ### Parallel fan-out / fan-in In this pattern, one request is sent to several models at once. Their outputs are then merged, voted on, or scored. Examples include: - Ensemble predictions - Running several candidate generators and selecting the best result - Combining object detection and segmentation on the same image Parallel fan-out can improve quality or coverage, but it increases total compute use. Even if latency stays reasonable, cost per request can rise fast. ### Conditional routing In this pattern, an early stage decides what happens next. Examples include: - A small classifier sends only difficult requests to a larger model - A language detector picks the right downstream model - A safety filter blocks or redirects unsafe inputs This can save cost and protect latency, but it only works if the router is reliable. A bad early decision can send requests down the wrong path and hurt quality. ### Multimodal pipelines These systems mix different data types and therefore different model types. Examples include: - Image encoder → language model - Speech model → language model → moderation model - Document parser → table extractor → language model The core challenge here is usually not just model quality. It is how to move and normalize intermediate data between stages without creating bottlenecks or fragile interfaces. ## Why multi-model pipelines matter Many production AI applications aren't really one-model problems. One large model can often perform several tasks reasonably well, but that does not mean it is the best choice for each stage. ### Better capability fit Different models are optimized for different jobs. - OCR models are tuned for extracting text from noisy images or PDFs. - Embedding models are tuned for semantic retrieval. - Rerankers are tuned for relevance scoring. - Smaller classifiers or guard models are often good enough for routing, filtering, or moderation. - Larger generative models are best reserved for final reasoning or response synthesis. That division of labor is often more effective than forcing one model to stretch across every stage. ### Better hardware fit Not every stage deserves the same hardware. A lightweight preprocessing or validation stage may run well on CPU. A vision encoder, reranker, or large generator may need [high-performance GPUs](https://handbook.modular.com/getting-started/choosing-the-right-gpu.md). Some stages batch nicely, while others are highly latency-sensitive and should stay small and fast. This lets teams place each stage on the hardware that matches its workload instead of overprovisioning the whole pipeline around the most expensive stage. ### Independent scaling Different stages usually have different traffic profiles. A retriever may be cheap to run but hit every request, while a large generator is expensive and may only run on a subset of traffic after filtering. When each stage is an independent deployable unit, it can scale on its own signal (queue depth, GPU utilization, concurrency) rather than being coupled to the slowest component. ### Lower cost per request Multi-stage pipelines create room for cost savings that a single model cannot easily replicate: - A cheap classifier or router can send only hard requests to a larger, more expensive model. - Lightweight stages can run on CPU or smaller GPUs. - Smaller specialist models can replace general-purpose LLMs for narrow tasks (extraction, classification, moderation). These savings only show up if the pipeline is well tuned. A poorly designed pipeline can easily cost more than a single monolithic model. ### Better iteration speed Multi-model systems are more modular. If the retrieval stage is underperforming, you can replace or retune it without changing the generation stage. If the final model is too expensive, you can test a smaller alternative without redesigning the rest of the system. That kind of local iteration is one reason teams adopt inference graphs rather than monolithic services. ## When not to use a multi-model pipeline It’s easy to over-engineer this. If a single model already meets your needs, keep it simple. Extra stages only make sense when they add clear value. Before splitting a workload into multiple stages, ask: - Does each stage solve a distinct problem that one model does not solve well enough? - Does the pipeline improve quality, cost, or control by enough to justify the added complexity? - Can the latency budget absorb the extra hops and queueing points? - Can the interfaces between stages stay stable as models evolve? - Will independent scaling actually save money, or just create more operational overhead? Model composition comes with a real tax: - More services to deploy - More contracts between stages - More [observability](https://handbook.modular.com/infrastructure-and-operations/comprehensive-observability.md) work - More failure modes - More tuning at the end-to-end level Start with the smallest architecture that meets the requirement, then add stages only when they clearly earn their place. ## Example architectures Here are a few concrete patterns where multi-model inference pipelines are a natural fit. ### RAG pipeline A common RAG path looks like this: ```cpp Query → Embed → Retrieve → Rerank → Generate → (Optional) Verify ``` Each stage has a clear role: - The embedding model finds similar content - The retriever narrows the search space - The reranker improves relevance - The generator turns that evidence into a response - A final verifier or citation checker reduces hallucination risk ### Document AI pipeline A document workflow might look like this: ```text Document image → OCR → Layout extraction → Classify → Summarize → Structured Output ``` This is hard to replace with one model if accuracy, formatting, or traceability matters. OCR and layout extraction are very different tasks from summarization. The trade-off is that large intermediate artifacts can move across several stages, so payload design matters. If the output needs to feed another system directly, [structured outputs](https://handbook.modular.com/model-interaction/structured-outputs.md) can make that handoff much easier to maintain. ### Multimodal assistant A multimodal application may route image, audio, and text through separate encoders before a downstream language model uses the combined signal. These systems are often strong examples of hardware specialization. The speech stage, image stage, and language stage may have very different runtime profiles and scaling needs. ## Single model vs. multi-model pipeline There is no universal winner. The right choice depends on what constraint matters most. | Dimension | Single model | Multi-model pipeline | |------------------------------|---------------|----------------------| | Simplicity | Simpler | More moving parts | | Latency | Usually lower | Often higher | | Hardware flexibility | Limited | Higher | | Independent scaling | Limited | Stronger | | Specialization | Limited | Stronger | | Ops burden | Lower | Higher | | Experimentation at one stage | Harder | Easier | As a rule of thumb: - Start with one model if it already meets your product requirement. Learn how to [choose the right model](https://handbook.modular.com/getting-started/choosing-the-right-model.md) before you decide to compose several models. - Add stages when they clearly help - Keep the number of stages as small as possible ## FAQs ### Should every RAG system be treated as a multi-model pipeline? Conceptually, yes, because retrieval, reranking, and generation are separate stages. Operationally, not always. Some teams package those stages behind one service boundary and treat them as one deployable unit. The important part is to understand the stage-level bottlenecks even if the abstraction looks simple. ### Should all stages live in one service? Not always. One service can reduce hop latency and simplify local coordination. Separate services are better when you need different hardware, scaling policies, release cadence, or failure isolation for different stages. ### Can multi-model pipelines lower inference cost? They can reduce cost when small specialist models filter or route requests before a large model runs, or when different stages use cheaper hardware more efficiently. However, poor pipeline design can easily do the opposite. ### How is this different from an agentic workflow? Both chain multiple model calls, but a multi-model pipeline is a mostly fixed graph that you design up front. An agent decides dynamically which tools or models to call, how many times, and in what order. Agents are a superset of the pipeline idea, with more flexibility and more variance in latency and cost. If you want the stage interfaces in either setup to stay predictable, [function calling](https://handbook.modular.com/model-interaction/function-calling.md) and [structured outputs](https://handbook.modular.com/model-interaction/structured-outputs.md) are often part of the solution.