AI for CIOs
RAG or fine-tuning: which problem are you actually trying to solve?
SimplSolutions editorial team · AI architecture · 4 min read
Published
AI-assisted original editorial guidance. Calculations and scenarios are illustrative, not customer results.
Separate knowledge from behavior
Retrieval-augmented generation, commonly called RAG, supplies selected source material to a model when it answers. Fine-tuning changes model behavior through additional training. These are different interventions. If employees need the current approved expense procedure with evidence they can inspect, begin by testing retrieval and source governance. If an evaluated model repeatedly fails a narrow output format despite good instructions and examples, a behavior intervention may deserve investigation.
Neither method excuses poor source ownership. Training on an obsolete procedure does not make it current. Retrieving an obsolete procedure does not make it authoritative. Before comparing technologies, name the employee task, required source, update frequency and permission boundary. Also test whether ordinary search and a clearer guide already solve the problem at lower complexity.
Use a decision table, not a vendor category
| Observed problem | First thing to test | Evidence to inspect |
|---|---|---|
| Current policy changes often | Retrieval from maintained sources | Correct version and supporting passage |
| Employee cannot find the right document | Search and information architecture | Successful task completion |
| Output violates a stable format | Instructions, validation and examples | Format failures on held-out cases |
| Correct source is retrieved but answer is wrong | Generation and answer evaluation | Claim-by-claim support |
| Wrong audience receives information | Authorization design | Denied-user test results |
The table is a troubleshooting aid, not a universal architecture prescription. Several interventions may be needed. Fixing authorization by adjusting model prose is the wrong category of response: authorization needs enforcement independent of whether the model agrees with the instruction.
Illustrative editorial photograph, not a customer result.
Diagnose where the answer fails
For a synthetic remote-access question, save the expected source, retrieved passages, generated answer and reviewer decision. If the relevant guide never appears, inspect ingestion, indexing, metadata and retrieval. If the guide appears but the model invents a step, inspect the answer instructions and validation. If the guide is correct for another employee population, inspect applicability and user context.
Without that separation, teams change the model when the real problem is source selection. They may then get a more articulate wrong answer. Keep a small held-out test set that was not used to adjust the system. Reusing the same examples for every improvement can conceal failure on ordinary questions you did not anticipate.
Ask how updates and removals propagate
For retrieval, ask what happens when the original source changes, is deleted or loses an audience permission. Inspect the actual refresh and cache behavior. For training-based approaches, ask how an obsolete learned behavior is corrected and what evidence demonstrates the correction. Do not assume deleting a training file makes the model forget its contents.
Treat retrieval as another governed data path. Source copies, indexes, logs and generated responses have their own access and retention implications. Review the real deployment and contractual terms with the responsible teams. A claim that knowledge is grounded does not establish privacy, security or compliance.
Compare alternatives on accepted tasks
Run the same permitted questions, ambiguous questions and no-evidence questions through your candidates. Record accepted results, review minutes, maintenance effort and total cost. Include the simpler search-only option if it is viable. A conversational interface may help users, but that benefit should appear in task completion, not just a demonstration's polish.
Put this to work this week
Take five failed answers and label the first point where each went wrong: source absent, retrieval wrong, applicability missing, unsupported generation or permission failure. Assign the repair to that stage instead of buying a different model immediately. Preserve the original cases for comparison, and add fresh held-out questions. Use the claim-support ledger to distinguish a fluent rewrite from a genuinely supported improvement.
Use the AI Knowledge Access Test Sheet to define the comparison. OWASP's RAG guidance explains that retrieval introduces a governed pipeline, not a security shortcut. Request an architecture-focused demo and bring an approved source plus three representative questions. SimplSolutions can help scope the useful intelligence layer without assuming your existing systems need replacing.
