> For the complete documentation index, see [llms.txt](https://handbook.modular.com/llms.txt).
> Markdown versions of all pages are available by appending .md to any URL.

# Build and maintenance cost

Building self-hosted LLM inference infrastructure isn’t just a technical task;
it’s a costly, time-consuming commitment.

## Complexity

LLM inference requires much more than standard cloud-native stacks can provide.
Building the right setup involves:

- Provisioning high-performance GPUs (often scarce and regionally limited)
- Managing CUDA version compatibility and driver dependencies
- Configuring autoscaling, concurrency control, and scale-to-zero behavior
- Applying advanced inference optimization techniques such as
  [prefix caching](https://handbook.modular.com/inference-optimization/prefix-caching.md) and
  [prefill-decode disaggregation](https://handbook.modular.com/inference-optimization/prefill-decode-disaggregation.md)
- Setting up observability tools for GPU monitoring, request tracing, and
  failure detection
- Handling model-specific behaviors like streaming, caching, and routing

None of these steps are trivial. Most teams try to force-fit these needs onto
general-purpose infrastructure, but it only results in reduced performance and
longer lead time.

Even if a team pulls it off, every week spent setting up infrastructure is a
week not spent improving models or delivering product value. For high-performing
AI teams, this opportunity cost is just as real as the infrastructure bill.

## Limited flexibility for ML tools and frameworks

Many AI stacks lock model runtimes, such as PyTorch, vLLM, or specific
transformers, to fixed versions. The primary reason is to cache container images
and ensure compatibility with infrastructure-related components. While this
simplifies deployment in clusters, it also restricts flexibility when you need
to test or deploy newer models or frameworks that fall outside the supported
list.

But this rigidity creates real limitations:

- You can’t easily test or deploy newer models or framework versions.
- You inherit more tech debt as your stack diverges from community or vendor
  updates.
- LLM deployment speed slows down, putting your team at a competitive
  disadvantage.

Scaling LLMs should mean exploring faster, better models, without being stuck
waiting for infra to catch up.

## Support for complex AI systems

An LLM alone doesn’t deliver value. It has to be part of an integrated system,
often including:

- Pre-processing to clean or transform user inputs
- Post-processing to format model outputs for front-end use
- Inference code that wraps the model in logic, pipelines, or control flow
- Business logic to handle validation, rules, and internal data calls
- Data fetchers to connect with databases or feature stores
- Multi-model composition for retrieval-augmented generation or ensemble
  pipelines
- Custom APIs to expose the service in the right shape for downstream teams

Here’s the catch: most LLM deployment tools aren’t built for this kind of
extensibility. They’re designed to load weights and expose a basic API. Anything
more complex requires glue code, workarounds, or splitting logic across multiple
services.

That leads to:

- More engineering effort just to deliver usable features
- Poor developer experience for teams trying to consume these AI services
- Blocked innovation when tools don’t support use-case-specific customization

## The hidden cost: talent

LLM infrastructure requires deep specialization. Companies need engineers who
understand GPUs, Kubernetes, ML frameworks, and distributed systems — all in one
role. These professionals are rare and expensive, with salaries often 30–50%
higher than traditional DevOps engineers.

Even for teams that have the right people, hiring and training to maintain
in-house capabilities is a major investment.
[In this survey](https://www.salesforce.com/news/stories/public-sector-ai-statistics/),
over 60% of public sector IT professionals cited AI talent shortages as the
biggest barrier to adoption. It’s no different in the private sector.