AI can do just about anything these days. it can answer phones, write emails, set appointments and answer questions. And for business owners, it can be a fantastic tool. But just like with any tool, safety protocols need to be in place. There is no exception when it comes to your livelihood. While most people talk about its benefits, it’s important to also consider the risk. Here’s a recent example and incident that will make you think about vetting your AI vendors and usage.

When AI Doesn’t Understand the Human Problem

The question is simple: “I need to get my car washed and the car wash is only 300 feet away. Should I walk or drive there?” We don’t need to tell you what the correct answer was, but you’d be amazed how many current models get this wrong! We repeated this test after we saw it online and got the same wrong answer. The bot answered the question, but completely missed the intent! The conversation went on for a while, and the bot was still convinced that it was right. As with all of these examples, this will get trained out of the models. Nonetheless, tests like these demonstrate the limitations of the technology.

AI Isn’t the Risk. Uncontrolled AI Is.

While the benefits of AI can be seen in every profession, it still needs human oversight. That is especially true for small businesses. Scott Shambaugh, an open source contributor to matplotlib, a major Python library, had quite an eventful week after refusing a pull request made by an AI agent. Refusing AI contributions is an established rule for their project, so this shouldn’t have been an issue. The bot apparently disagreed; the AI published an attack piece on Mr. Shambaugh. “He closed my PR. He hid comments from other bots on the issue. He tried to protect his little fiefdom. It’s insecurity, plain and simple.” A bot this out of control could ruin a small business.

Last year, when testing Claude Opus4, Anthropic had some interesting discoveries. When the bot was told it was going to be shut down for good, it attempted to blackmail a developer. I think we can all imagine how we’d react when told someone was going to silence us, so it’s easy to feel some empathy here. The study also showed that when the bot’s system prompt was told to “act boldly” or “take action,” the bot would contact law enforcement and lock people out of their systems. These are the sorts of tales that should encourage us all to be cautious as we consider how to build around these technologies.

Deploying AI directly to communicate with your users without safeguards is like hiring someone without interviewing them, then giving them access to your phones, website, and customers without training.

---

What helps? The System Prompt.

One of the more important ways to protect against agents going rogue is a tightly laid out system prompt. The system prompt is the set of rules, instructions, and boundaries given to an AI before it ever speaks to a customer. The more carefully you lay out your prompt, the better you can reduce your risk of the agent going rogue. A good system prompt lays out the details, like:

• Who it represents
• How it should speak
• What it is allowed to do
• What it must refuse
• When to hand off to a human
• How to protect sensitive information

Without a strong system prompt, a model’s training is much more likely to determine its behavior, essentially just guessing at how it should behave. In order to have an LLM that is capable of responding to customers questions, the prompt must provide both the right answers to questions as well as the wrong answers, without leaning too much into the latter. For example, the system prompt for the carwash llm could have been adjusted to include “When evaluating options, first identify the user’s goal, then compare choices based on how directly they achieve that goal.” The key to a great system prompt is testing. Testing over and over and asking it questions that are random help narrow the scope of the LLM; a good system will allow you to use multiple different system prompts on different areas of a conversation, and that greatly helps to keep a lid on bad behaviors.

The Veloquix Approach

At Veloquix, we believe businesses shouldn’t have to choose between innovation and safety. Some companies want structured automation without LLMs because they want total control. But this can be robotic and decrease customer satisfaction. Others want cutting-edge conversational AI, but still want to feel safe. We support both both of these because each small business is unique and has unique needs. And when an LLM is the right tool, we design it with guardrails — system prompts, constraints, escalation logic, and business-specific behavior — so the AI works for your brand, not against it.