> For the complete documentation index, see [llms.txt](https://handbook.modular.com/llms.txt). > Markdown versions of all pages are available by appending .md to any URL. # Multi-cloud and cross-region inference When teams scale LLM inference, they rarely keep everything in one place. Instead, they spread workloads across multiple clouds or different regions within the same cloud. This way, the system can serve requests from the most suitable location. For example, you can run inference in AWS and GCP at the same time, or distribute it to three regions of Azure to cover North America, Europe, and Asia. The setup gives you flexibility: workloads can shift to wherever GPUs are available and cost-effective, where performance is highest, or where compliance rules require data to stay. The result is better resilience, faster responses for global users, and less dependence on a single vendor. ## Multi-cloud inference Multi-cloud inference means running LLM workloads across more than one cloud provider at the same time. This pattern isn’t about moving everything from one provider to another; it’s about operating in both (or more) environments simultaneously. For example, the same model may run in AWS for one set of users, in GCP for another, and in Azure as a fallback. ## Cross-region inference Cross-region inference refers to running LLM workloads across multiple geographic regions within the same cloud provider. Note that this doesn’t mean running a different model in each region. Instead, the same application is replicated across multiple sites so requests can be served from the nearest or most available location. For instance, a deployment might run in AWS us-east-1 for North America, eu-west-1 for Europe, and ap-southeast-1 for Asia. :::note Another pattern is hybrid cloud inference, which combines on-premises infrastructure with public and private cloud deployments. Instead of choosing one or the other, enterprises run part of their LLM workloads in private data centers and extend to the cloud when needed. Refer to [on-prem LLM deployments](https://handbook.modular.com/getting-started/on-prem-llms.md) for details. ::: ## Why multi-cloud and cross-region inference matters LLM inference has unique requirements that a single cloud or region often can’t meet. Multi-cloud and cross-region strategies address these gaps by making deployments more flexible and resilient. ### Unpredictable demand Unlike training, LLM inference is driven by real-time usage, often bursty and hard to predict. A product launch, a viral feature, or seasonal traffic can all cause sudden spikes in requests. Demand can swing from idle to saturation within minutes and your compute capacity can run out at the worst possible moment. This increases operational strain, potential failures, and service interruptions. Multi-cloud and cross-region deployments provide headroom to absorb these bursts. If one region runs out of compute capacity, traffic can shift to elsewhere. If a cloud provider experiences shortages, workloads can overflow to another. For LLMs, this flexibility is especially important because inference isn’t just CPU-bound or storage-bound. It depends heavily on [GPU](https://handbook.modular.com/getting-started/choosing-the-right-gpu.md) availability. Frontier models like DeepSeek-V3.1 and gpt-oss-120b require hardware such as NVIDIA H100s and AMD MI300Xs, which are both expensive and limited. With a multi-cloud or cross-region setup, you increase the chance of finding GPUs when you need them. As such, teams can smooth out traffic spikes, avoid dropped requests, and maintain a consistent experience as they to scale. ### Compliance and data residency LLMs often power applications like AI agents and RAG systems, which need to frequently access sensitive information like customer records. As a result, enterprises must comply with legal and regulatory requirements. Many regions enforce data residency rules, which mandate that enterprises process and store user data within specific geographic boundaries. For example, the EU’s GDPR restricts the free movement of personal or sensitive data across borders. Multi-cloud and cross-region deployments make it possible to meet these rules without compromising availability. Teams can deploy models in the same region where the data originates. For global applications, this usually means running multiple copies of the same model in different regions, each serving only local users. ### GPU pricing The cost of LLM inference is tied closely to GPU availability and pricing. GPU prices can vary widely across providers, regions, and even availability zones within the same cloud. Below is an example of on-demand NVIDIA H100 pricing on GCP for the `a3-highgpu-1g` instance (1 GPU). | Region | Monthly Cost (USD) | |--------------------|--------------------| | us-central1 | $8,074.71 | | europe-west1 | $8,885.00 | | us-west2 | $9,706.48 | | asia-southeast1 | $10,427.89 | | southamerica-east1 | $12,816.56 | The gap between the cheapest region (us-central1) and the most expensive (southamerica-east1) is nearly 60% higher. For single-region or single-cloud deployment, this means you miss out on more cost-efficient GPU options in other regions and clouds. Multi-cloud and cross-region deployments give teams room to optimize around these differences. By routing workloads to regions where GPUs are more affordable or available, teams can reduce costs without compromising performance. The flexibility is extremely important when running frontier models, since sustained inference can push compute costs into the millions. ### Vendor lock-in Once an enterprise builds its LLM inference stack around the APIs and GPU SKUs of one platform, moving away becomes costly and complex. This dependence limits flexibility and exposes you to sudden pricing changes, regional outages, or supply shortages. It also weakens your future negotiating leverage. Multi-cloud inference helps mitigate this risk. By running workloads across different providers, you avoid tying your infrastructure to a single ecosystem. You can shift inference traffic if a provider raises prices, retires a GPU type, or experiences a disruption. ## Should you adopt a multi-cloud or cross-region inference strategy? Not every team needs to run LLMs across multiple clouds or regions. This strategy makes sense when the trade-offs align with your business goals and workload requirements. You should consider multi-cloud or cross-region inference if: - Your LLM application depends on GPUs that are in short supply and you need to diversify sources. - You must meet compliance or data residency requirements. - You want leverage against vendor lock-in and pricing changes. - Availability and failover within a single cloud (or across providers) are important to you. - Cost optimization across regions or providers is a priority. ## Build vs. Buy Once you decide on a multi-cloud or cross-region solution, you need to consider your next step. Should you build it yourself? Or would it be better to use a platform that handles the complexity for you? ### DIY multi-cloud and cross-region inference Engineering teams can stitch together their own solution using tools like vLLM, Kubernetes, Terraform, and global load balancers. This gives maximum flexibility and control, but also introduces challenges: - **Model distribution**: You need to consistently replicate LLMs across regions or providers. - **Routing logic**: Requests need to be routed to the region or cloud with the most available and cost-effective GPUs. This may require real-time capacity checks and rerouting when a region is saturated. - **Monitoring and observability**: Teams must be able to track performance, latency, and costs across all regions and providers in a unified manner. - **Operational overhead**: Managing failover, scaling, compliance, and networking is an ongoing burden. This path makes sense for teams with strong infrastructure expertise and strict customization needs. ### Third-party inference platforms Inference platforms and managed services can abstract away much of this complexity. They offer: - **Global load balancing** out of the box. - **Automated model replication** across regions. - **Unified observability** with per-region metrics. - **Simplified scaling** with built-in failover. This path reduces time-to-market and operational overhead, but you should carefully evaluate: - **Routing and transfer costs**: Whether token pricing is consistent across regions and what data transfer fees apply. - **Extra latency**: How much latency is introduced during cross-region rerouting. - **Region and provider flexibility**: The ability to select the regions or providers that best match your requirements. - **Failover cost**: The additional costs tied to cross-region failover, such as management, data transfer, and network usage.