RAG vs Fine-Tuning vs Prompting: 3 Approaches to Building Your Own LLM, Compared

Three Paths on the Boardroom Table

In the previous post, we brushed past the three main approaches to building a company-specific data LLM. This time, we're laying each of them out on the boardroom table and taking them apart, one by one.

Not long ago, I spent about two hours in conversation with a DX lead at a manufacturing firm. His parting comment stuck with me: "Every vendor tells us their option is great, but I honestly can't figure out where our company should start." Frankly, I hesitate every time I hear that question — there isn't one right answer. Still, for those of you standing on the edge of a sign-off decision, let me walk through — plainly, without embellishment — what we've actually chosen and why on our recent projects.

What Each of the Three Approaches Actually Changes

Start with the principles. The three approaches often get listed side by side, but each one touches a completely different layer.

Prompt engineering leaves the model itself untouched and instead refines the question (the prompt) going in, coaxing out the desired answer. You use off-the-shelf models like GPT or Claude as they are, and design purely around "how to ask." It's usually the first thing teams reach for during early experiments or PoCs.

RAG (Retrieval-Augmented Generation) pre-indexes your internal documents, manuals, and product databases as vectors. When a question comes in, the system retrieves the most relevant passages and hands them to the LLM as reference material. In plain terms, it's like "letting the model bring your company's reference binder into the exam." From an executive-approval standpoint, the ability to cite the exact source document beneath every answer is particularly powerful.

Fine-tuning takes a pre-trained model and further trains its weights on your company's own data. The model absorbs your tone, domain vocabulary, and decision patterns. The trade-off: you need GPU capacity, and the training data has to be carefully curated.

A Feel for Cost and Maintenance

Two questions come up most often at approval meetings: how much does it cost, and how much upkeep does it demand later. Here's how the three approaches look on a single page.

The numbers swing widely by project scale. Directionally, though, the entry barrier climbs in the order Prompting → RAG → Fine-tuning. Conversely, the amount of "proprietary asset" you're left with grows in the reverse order.

Why Most SMBs Start with RAG

To cut to the chase: a majority of the Korean and Japanese SMB projects we've delivered over the past year started with RAG. Not out of preference — but because, at each decision point, the alternatives simply came out worse.

Several reasons stack up. First, initial cost is low. Organizing internal documents, loading them into a vector DB, and wiring up a retrieval pipeline doesn't demand GPUs the way fine-tuning does. Second, updates are trivial — swap the documents and the answers refresh instantly. For companies whose regulations or product catalogs change frequently, this maintenance simplicity is decisive for the operations team. Third, you can attach source documents to answers. It doesn't eliminate hallucination risk entirely, but being able to display "this answer came from page 3 of manual X" noticeably lowers the internal approval barrier.

This ties back to a point we brushed on in the previous post's concept and roadmap overview. On that roadmap diagram, the slot labeled "Stage 1: PoC" is, in real projects, most often filled by RAG.

So When Does Fine-Tuning Make Sense?

There are absolutely moments when fine-tuning is the right call. The situations where we actually recommend it usually look like this.

Cases where more than 5,000 labeled training examples have accumulated, and where an industry-specific tone or stylistic consistency directly drives service quality. Think legal document summarization, a brand-specific customer-service script, or domains like medical and bio where terminological precision is the foundation of the business. The other case is high-volume repeat traffic. Running hundreds of thousands of simple, repetitive tasks per day through an API racks up heavy token bills. In situations like that, fine-tuning a smaller model and self-hosting it can be far cheaper over the long run. It's not uncommon to hear reports of 80–90% operating cost savings compared to straight API usage.

The pattern we see most often in practice is the hybrid. Launch quickly with RAG, then — after 6 to 12 months of operational data and clearer tonal requirements — layer on partial fine-tuning like LoRA. Don't try to draw out the hybrid architecture from day one; letting the need surface organically through operations dramatically lowers the failure rate.

Where Does Prompt Engineering Fit In?

A common misconception worth clearing up: prompt engineering isn't a "lower tier" beneath RAG or fine-tuning. All three operate as parallel layers. Whether you're running RAG or fine-tuning, you still have to design how the model gets asked, every single time. Some teams run just fine for six months on well-crafted prompts alone. As a sequence, Prompting → RAG → (if needed) Fine-tuning is the natural order of evaluation. Companies that start with fine-tuning, by contrast, almost always regret it a few months in.

Questions Worth Asking at the Approval Meeting

When your team is debating the three approaches internally, working through these questions first will make the decision markedly easier. Comparing vendor quotes without these answers in hand tends to leave the direction wobbling.

• How frequently does the knowledge you want the LLM to reference actually change?
• Do answers need to display "which document this came from" as evidence?
• How much labeled training data has actually accumulated — measured reality, not aspiration?
• What's the daily usage profile — low-volume precision, or high-volume repetition?

Once you can answer these four questions, which of the three approaches to prioritize usually narrows itself down naturally.

Wrapping Up — Once You've Chosen an Approach, What Comes Next

Choosing an approach doesn't mean the project has begun. The genuinely hard problem starts after. What language will you use, three months from now, to show your leadership team "how much this has improved, and by what measure." The next installment will cover ROI and KPI design — how to measure the impact of deploying your own LLM. We'll share several metric frameworks we've actually built alongside clients.

If your team is currently in the middle of this debate, just tell us your document types, the nature of the work, and a rough sense of traffic — we can sketch out a rough direction on which approach fits best. Feel free to reach out through a Free consultation, and if you're curious about related cases we've delivered, the previous post's roadmap summary is worth a look too.