Your team's API bill last month: $9,000. This month: $14,000. The product is scaling, traffic is steadily increasing, and the CFO is starting to ask the familiar question: "Why don't we run the model ourselves?" It's a valid question. But if you answer "buy GPUs" without calculating the break-even point, six months later you'll be explaining why your H100 cluster is running at 15% utilization.
This is a decision-making problem, not a religious one. If there are numbers, it can be calculated.
Break-even formula: don't guess, divide.
Monthly self-hosting costs consist of three items:
- GPU: buy outright (3-year depreciation) or rent cloud. Renting an 8×H100 node costs tens of thousands of USD/month depending on the provider; buying outright means dividing the cost over 36 months plus infrastructure.
- Electricity + datacenter: an H100 node at full load consumes ~10kW. Multiply by industrial electricity rates, plus cooling.
- People: at least a fraction of the time of 1–2 engineers who know CUDA, can read vLLM logs, and are willing to handle on-call.
The other side of the equation: your actual throughput (tokens/second with good batching) multiplied to get tokens/month, compared to the $/1M token price of the API you are currently using.
break_even_tokens/tháng = chi_phí_self_host_tháng / giá_API_per_token
Ví dụ: node tốn $25.000/tháng (GPU + điện + 0.5 FTE),
API blended $2/1M token
→ hòa vốn tại 12.5 tỷ token/tháng ≈ 420M token/ngày
It sounds like a lot, but a B2B product with tens of thousands of active users running a RAG pipeline in the background can easily hit that level. The key point: the 8–18x cheaper per token figure is only true when volume is stable and utilization is high. Idle GPU is burning money. If your traffic spikes during business hours and dies at night, the margin shrinks quickly. Organizations that execute this correctly often reduce cost per request by 60–80% — but "doing it correctly" is a conditional proposition.
And in 2026, Vietnam will have an additional variable not found in spreadsheets: Law 134/2025 takes effect from March 2026, with high-risk industries having a roadmap through 2027. For banking, healthcare, and citizen data — the question is no longer "how much cheaper" but "is the data leaving the territory." On-prem or local GPU cloud (FPT AI Factory is running thousands of H100s from a $200M project with NVIDIA) shifts from a cost-saving option to a compliance requirement.
vLLM is the default, and migration is almost free
By mid-2026, choosing a serving engine will no longer be a debate. vLLM wins because of three things:
- PagedAttention: manages KV cache like an OS manages virtual memory, eliminating VRAM waste due to fragmentation.
- Continuous batching: new requests jump into the running batch instead of queuing for the old batch to finish — throughput is 2–3x higher than static batching.
- Prefix caching: long system prompts, repetitive few-shot examples — compute once, use forever. For RAG pipelines with fixed prompt templates, this is real money.
Most important for migration: vLLM exposes an OpenAI-compatible API. Client code only needs to change one line:
client = OpenAI(
base_url="http://llm.internal:8000/v1", # đổi mỗi dòng này
api_key="dummy",
)
resp = client.chat.completions.create(
model="Qwen/Qwen3-32B", messages=messages
)
This means you can run in parallel: route 10% of traffic to the self-hosted cluster, measure quality and latency, then decide. No big-bang migration.
Models worth running by 2026: Qwen 3.x (MoE, surprisingly strong in Vietnamese), DeepSeek V4, Llama 4, Gemma for lightweight tasks. Open-weight is no longer a "inferior version of frontier" — for narrow tasks, small models fine-tuned via QLoRA can be more accurate than frontier APIs while being dozens of times cheaper.
Three levers to squeeze costs after self-hosting
Break-even is just the starting point. These three levers decide whether you save 2x or 10x:
| Lever | Gain | Trade-off |
|---|---|---|
| FP8 on H100 | 1.3–2x throughput | <2% quality loss, almost unmeasurable in common tasks |
| KV cache INT8/INT4 | Reclaim 30–50% VRAM → larger batch size, longer context | Need thorough eval with very long context |
| Small model + fine-tune | 7B–32B instead of 70B+, costs decrease exponentially | Must have an eval set and fine-tune pipeline (Unsloth performs QLoRA in almost a single command) |
Execution order: FP8 first (almost a free lunch), KV cache quantization when VRAM is the bottleneck, fine-tune small models once you have proper data and an eval set.
Another exit is opening: ROCm. As of 1/2026, AMD passes ~93% of the vLLM test suite — meaning MI300X is no longer a gamble. When H100 is scarce or overpriced, having a Plan B provides bargaining power. NVIDIA's monopoly is seeing its first real crack at the serving layer.
The cost is not on the invoice
This is the part people often overlook when excited about 8–18x numbers. Leaving APIs means you inherit:
- On-call. A model server crashing at 2 AM is your team's problem, not Anthropic's or OpenAI's.
- Security patching. vLLM, drivers, CUDA/ROCm, model weights — all are attack surfaces you must monitor. Prompt injection is still OWASP LLM01, and now the infra layer is also your responsibility.
- Capacity planning. APIs scale infinitely; your GPU cluster does not. A 3x traffic spike after a marketing campaign is an incident, not good news.
- Chasing new models. APIs provide you with the best model every quarter without effort. Self-hosting means every model change requires a new cycle of benchmarking, quantizing, and canarying from scratch.
Stick with APIs when: volume is below the break-even point several times; traffic is erratic; the team has fewer than 5 backend engineers and no one has operated GPUs; tasks require true frontier reasoning; and there are no data legal constraints.
Move to self-hosting when: token volume is consistently above break-even, there is at least one person comfortable with nvidia-smi and vLLM logs, and — for many industries in VN from 3/2026 — when the law leaves you no other choice.
Checklist before signing a GPU purchase order
- Measure actual token volume for the last 3 months, separating prefill/decode. Is there steady growth, or seasonal spikes?
- Calculate break-even against GPU rental prices First — do not commit to a full purchase before running in production for 6 months.
- Benchmark open-weight models on your own eval set; do not trust leaderboards.
- Run 10–20% shadow traffic through vLLM for at least one month, comparing p95 latency and quality.
- Be able to answer: who is on-call, who patches CVEs, who does capacity planning — with specific names.
- Check compliance with Law 134/2025 regarding the type of data you process; if on-prem is not feasible, evaluate local GPU cloud.
- Have a fallback plan to return to APIs if it fails — since APIs are OpenAI-compatible, the exit is cheap, but it must be written down beforehand.
Item number 5 disqualifies more teams than all other items combined.
Self-hosting in 2026 is no longer technically difficult — vLLM has handled that. The difficulty lies in the fact that it is an operational commitment disguised as a procurement decision. Teams that account for both sides will truly save money; teams that only look at the 8–18x figure are buying a second production system to maintain.
