Once You've Framed the Metrics, the Stack Is the Next Gate
"We've mapped out the KPI table. But we've been stuck in meetings for three weeks trying to decide which model to actually run it on." That was a DX executive at a manufacturing subsidiary in a meeting last month. I couldn't give an immediate answer that day. In truth, there's no 30-minute right answer to this question.
I covered the effect-measurement framework earlier (the ROI and KPI design from episode 3), but which stack you use to hit those KPIs can double or triple both difficulty and cost. Stack selection isn't a technical decision — it's an approval decision. It's a document the CIO, CFO, legal, and information security all have to align on around the same page.
Commercial APIs — Claude and OpenAI Families
This is the fastest path to get started. One account, one API key, and you have a prototype up in a few hours. Claude 3.5 Sonnet holds benchmark results around 92% on HumanEval and 98% on tool-call accuracy, and shows particular strength in long-document summarization and complex reasoning. The GPT-4o/5 family plays well with multilingual workloads, latency, and the Microsoft stack, so it slots naturally into Office and Teams-based workflows.
From an approver's perspective, the real value of commercial APIs isn't performance — it's offloading operational burden. Model updates, GPU issues, scaling — the internal team barely has to think about any of it. In return, two things trip you up: the fact that data leaves your perimeter (which drags out legal and infosec review), and the fact that costs climb in step functions as traffic grows.
Honestly, if your monthly token usage is still small, this choice is close to the right answer in nearly every case. If you can secure zero-data-retention agreements and private endpoints for the top frontier models from Claude or OpenAI, production deployments are becoming increasingly common even in finance and legal.
Open Source — Llama and Mistral Families
Open-weight models like Llama 3 70B and the Mistral family sit around 81% on HumanEval and 82% on tool calling — roughly a 5-10% gap versus frontier commercial models. Two years ago that gap was 20%, so it narrows every year. For most practical tasks — document summarization, internal FAQ, RAG-based knowledge retrieval — the perceived difference is small.
The appeal is twofold. First, you can design the system so data never takes a single step outside the company. Second, you handle high-volume traffic at 5-50x lower per-token cost versus commercial models (on hosted providers). If you run your own GPUs, the effective cost collapses to operations only.
That said, open source isn't "free." Fine-tuning (LoRA-based, 7-14B models) runs from KRW 500,000 to 3,000,000 per run, and full fine-tuning on a 70B model exceeds KRW 30 million. Add model-serving infrastructure, monitoring, prompt management, and version-rollback pipelines, and you need at least one MLOps person in-house. That headcount is what tends to get left off the quote.
On-Premise Self-Hosting — Where Break-Even Actually Sits
For fully air-gapped networks, regulated industries, defense, and public sector, on-premise is often the only option. Initial GPU server build-out runs into the hundreds of thousands of dollars depending on scale. Monthly operating cost for the model-serving infrastructure itself starts around KRW 1-5 million, but on top of that you carry 24x7 operations and DevOps headcount. Even US research puts infrastructure ops alone in the $5,000-$30,000+ per month range as a common baseline.
The break-even point from an approver's perspective looks roughly like this: once monthly token usage crosses the 5M-2B token range, self-hosting starts to beat commercial APIs on cost. Below that, self-hosting is usually a net loss on total cost. In our consulting practice, we repeatedly see cases that start with "our data is sensitive, so it has to be on-prem, period," only for the actual traffic simulation to reveal that a private endpoint on a commercial API was the more sensible choice.
Recommended Stack by Data Sensitivity
Here's the matrix we frequently drop into approval documents in practice.
| Sensitivity / Traffic | Recommended Stack | Core Rationale |
|---|---|---|
| Low / Low | Commercial API only | Time to Market first, minimize initial investment |
| Mid / Mid | Commercial API + VPC private endpoint, self-hosted vector DB | Control the data path while offloading operations |
| High / Low-Mid | ZDR-contract commercial API + PII masking proxy, or Llama serving on Bedrock/Vertex | Reduce legal risk while minimizing infra team burden |
| High / High, air-gapped | On-prem open source + hybrid routing | Cost control at high traffic, with approved commercial API paths reserved for frontier requests |
In 2026, the Answer Is Hybrid
The current market consensus is hybrid routing. Low-cost, high-frequency traffic (FAQ, classification, summarization) flows through open source or low-cost commercial models, and only the 5-25% of traffic that genuinely needs frontier reasoning gets routed to Claude or top-tier GPT models. Total cost drops by more than half, while the quality end-users experience stays nearly identical.
What made this structure viable is that open-source performance has caught up to production standards. "30% cheaper" sounds small, but when you route 75% of traffic through a model that's 5x cheaper, more than 60% of your total cost disappears. In an approval document, that number carries weight.
The catch is that hybrid adds another stack, along with routing logic, monitoring, and cost dashboards. Do you have enough traffic to justify carrying that management overhead — that's the practical baseline for adopting hybrid.
Wrapping Up — And the Next Gate
Stack selection is coupled with the RAG versus fine-tuning decision from the previous episode. If you're RAG-centric, commercial APIs will take you far enough; if you need domain-specific fine-tuning, the case for open-source self-hosting gets stronger. You don't need to pick the perfect stack up front. On the assumption that you'll rebalance six months in based on traffic and quality data, starting with the stack you can validate fastest has consistently produced the lowest total cost in our engagements.
In episode 5, we'll cover how these stacks translate into actual quotes by scale — Budgeting for Your In-House LLM Build — A Scale-by-Scale Quote Guide for 2026. You'll see exactly how today's stack matrix maps to real numbers.
At 5years+, we've been supporting in-house LLM adoption for mid-sized and small enterprises across Korea and Japan, and we work through this stack decision alongside client teams. If you're currently evaluating a stack internally, our Free consultation puts together a recommendation grounded in your current data sensitivity and traffic forecasts.