Photo by John Adams on Unsplash.
If you've sat through a sales pitch in the last six months, you've probably been told you need an "AI agent." Maybe an "agentic workflow." Almost certainly something with "RAG."
Most of the time, the person saying these words can't tell you what they mean. That's a problem, because some of these tools are genuinely useful — and others are a $30,000 way to do what a Google Sheet already does.
This post is the version I wish someone had written for me before I built my first one. No hype, no undefined acronyms, no "the future of work" filler. Just what these things actually are, when they're worth building, and what to expect if you commission one.
What is an "AI agent," really?
An AI agent is a program that can take a goal and decide, on its own, what steps to take to achieve it.
That's the whole definition. The thing that makes it an "agent" rather than just a chatbot is that it has access to tools — functions it can call to do real work — and it picks which tool to use, in what order, based on the situation.
A chatbot answers questions. An agent looks at your question, decides it needs to check your CRM, then check your calendar, then draft an email, then asks you to approve it before sending — and does all of that in one go.
The "intelligence" here is judgment over a sequence of steps, not the language generation itself.
What is an "agentic workflow"?
An agentic workflow is what happens when you let an agent run a multi-step process instead of a single response.
Compare two versions of the same task:
Non-agentic (one-shot):
User: "Write me a follow-up email to Sarah." AI: generates a generic follow-up email
Agentic (workflow):
User: "Follow up with Sarah." Agent: looks up who Sarah is in the CRM → reads the last three emails she sent → checks the open deal value → drafts a follow-up that references the actual conversation → flags that her last reply mentioned a budget concern → asks you whether to soften the price section.
The second one is doing work. It's the difference between a smart intern and a search bar.
The catch: agentic workflows are only useful when the agent has access to your data. Without that, it's just guessing. Which brings us to the most important acronym in this whole conversation.
What is RAG?
RAG stands for Retrieval-Augmented Generation.
In plain English: before the AI generates a response, it goes and looks something up first.
That's it. That's the whole concept.
The reason it matters is that off-the-shelf models like GPT or Claude know what's on the public internet up to their training cutoff. They do not know:
- Your customers
- Your pricing
- Your past projects
- Your team's policies
- The email Sarah sent you yesterday
- Anything that happened in your business, ever
RAG is the bridge. You build a private, searchable index of your documents, conversations, records, knowledge base — whatever the agent needs — and when a question comes in, the system retrieves the relevant pieces, then asks the AI to generate an answer using those pieces as source material.
The result: an agent that can answer "What did we quote Sarah back in February?" instead of "I don't have access to your records."
RAG vs. fine-tuning: the question I get most often
Almost every prospect asks me some version of this: "Should we fine-tune a model on our data, or use RAG?"
The short answer for 95% of small businesses: RAG.
Fine-tuning bakes information into the model itself. It's expensive, it goes stale the moment your data changes, and it's overkill for almost every use case I see in the wild. It makes sense when you need a model to adopt a specific style or perform a specialized task — not when you need it to remember facts.
RAG is cheaper, updates instantly when your data changes, and lets you actually see which document the agent used to answer (which matters a lot when the answer is wrong and you need to figure out why).
If someone is selling you a fine-tuned model for your customer service workflow, ask why they didn't just use RAG. The answer is usually that fine-tuning is more expensive to charge for.
What does an agentic + RAG system actually look like?
Here's a real example, simplified. Let's say you run a small construction firm, and you want an internal agent that can answer questions about past jobs.
A request comes in: "Have we ever done a kitchen remodel in St. George under $40,000? What did the client think?"
Without RAG or agentic behavior, an AI gives you a generic essay about kitchen remodels.
With it:
- The agent parses the question into structured filters: location = St. George, project type = kitchen remodel, budget < $40,000.
- It retrieves matching project records from your database.
- It retrieves the closeout reviews and any photos linked to those projects.
- It generates a summary: "Three jobs match. The Bloomington Hills kitchen ($38,400) finished two days early; the client left a 5-star review specifically mentioning the cabinet detail. The other two ran into supplier delays — see notes."
- It links to the source records so you can verify.
That's a workflow you can run on Tuesday morning instead of digging through a folder structure for forty-five minutes. And every answer is grounded in your data, not the internet's.
What can realistically be built today
Below is what I'd actually recommend a small business consider in 2026, based on jobs I've scoped in the last quarter:
High value, well-understood:
- Internal knowledge agents — search and Q&A across SOPs, contracts, past project notes, and customer records. RAG-heavy, low risk.
- Lead intake and qualification — agent reads inbound form submissions and emails, drafts a qualification summary, schedules a call if criteria are met.
- Quote and proposal drafting — agent pulls past similar projects, applies your pricing rules, and produces a first draft for human review.
- Customer support triage — agent reads incoming tickets, retrieves history, drafts a response, and escalates the complicated ones.
Possible, but riskier — needs careful scoping:
- Outbound communication — agents that send emails or messages without human review. Almost always worth keeping a human in the loop for a while.
- Financial workflows — agents that touch money, invoicing, or payroll. Build it, but with hard guardrails and approval steps.
Hype, mostly:
- "Replace your sales team with AI."
- "Replace your support team with AI."
- "Replace [any role] with AI."
If anyone is selling you a one-click replacement for a department, they're either lying or they've never run a department. The teams I've seen succeed with agents are the ones that augment a human worker — make them five times more efficient at the boring half of their job — not the ones trying to remove the human.
What does it cost? What should you watch for?
A few honest notes on what you're signing up for if you commission one of these:
- Data preparation is the real work. The model is the easy part. Getting your contracts, project records, and customer history into a clean, searchable form is 60-70% of any RAG project. Budget accordingly.
- You will need a human in the loop for the first few months. Anyone who tells you "set it and forget it" hasn't shipped one of these in the wild.
- Token costs are real but small. Most of the small-business agents I build cost $20-200 per month in API fees. The build cost is the bigger line item.
- Privacy and access control matter more than you think. Make sure whoever builds it answers the question "what stops the marketing intern from asking the agent for everyone's salary?" before you sign anything.
- The right partner shows you how it works under the hood. If you can't see the retrieved context the agent used to answer, you can't trust the answer. Demand transparency.
The bottom line
Agentic workflows and RAG are not a fad, but they're also not magic. They're a practical way to put the institutional knowledge that's currently buried in your inboxes, file shares, and CRMs into a form your team can actually query.
Done well, the result is the most useful coworker you've ever had — one that's read every email, every contract, and every project note, and never forgets.
Done poorly, it's an expensive demo.
The difference is in how you scope it, what you connect it to, and whether you're working with someone who builds these in production rather than tweets about them.
If you're thinking about an agent or RAG-based system for your business and you'd like a 30-minute conversation about whether it's actually the right move — I'm happy to have that conversation, and I'll tell you if it isn't.