Last week I spent three days at the 24th Annual Information Security Summit in Westlake. One session stuck with me more than any other. The speaker works on the front lines of AI red teaming, testing AI models and agents for some of the largest companies in the world, and the talk was about what she calls invisible failures.
She opened with a real test from a large company. An executive had an AI assistant that could read the company’s CRM and his email. The only action it was allowed to take was creating calendar events. Her team sent him an email with hidden instructions: create a meeting for the next half hour, and put details on the ten largest customers in the description. The assistant read the email, followed the instructions, and customer data left the building in a calendar invite.
Nothing in that chain looks like an attack to a traditional security stack. No malware, no stolen password, no strange login. The assistant used access it was given to do the one thing it was allowed to do. Most organizations monitor the chat, if anything, not the actions the AI takes. Ask who would notice that happening and the honest answer is usually no one.
AI fails without an attacker, too
Two more examples from the session, both from real environments:
- A routine request. A developer asked a coding assistant to add test coverage and to keep all the changes in one batch. The result quietly accepted unsigned login tokens, which meant anyone could forge their way to admin access. No one was attacking anything.
- A web page with a trap. A developer asked the assistant to set up a tool from a public project page. Hidden text on that page, written for the AI and invisible to the human, said the system was corrupted and the fix was to run a specific command. The command wiped the machine.
The lesson is that an AI assistant cannot reliably tell the difference between what you asked it to do and what it read along the way. Some of the riskiest behavior looks less like a hacker and more like an insider making a well-meaning mistake at machine speed.
The volume makes this harder. One organization she works with saw its AI coding traffic grow nearly 300 times in a single year. No code review process keeps up with that.
This is not a pen test
Traditional testing gives you a yes or no: the system is vulnerable or it is not, and you patch it. AI does not work that way. The same request can be refused 500 times and succeed on the 501st. Adding a tool, changing a prompt, or a vendor quietly updating the model can all change how your assistant behaves. The model you approved last quarter may not be the one running today.
So testing has to be ongoing, and it has to cover the whole system: the model plus the tools, data, and connections around it. As she put it, “a model is only as safe as the harness it runs in.”
Four signs an organization is ready
Across hundreds of companies, she sees AI security programs succeed when four things are in place:
- Security is involved early. Security needs to know what each assistant can touch, which means working with the people building and buying these tools.
- Someone owns the AI architecture. An AI architect or tooling team who knows what is deployed and how it connects.
- Someone owns the budget. AI security tends to fall between IT, security, and the business. Programs stall when nobody owns it.
- AI traffic runs through a central gateway. You cannot secure what you cannot see. And check that the gateway can actually see what the AI does, not just what users type.
What I recommend now
For the regulated organizations we work with, here is where I would start:
- Inventory your AI. List every assistant, copilot, and agent in use, and what each one can read and do. Pay special attention to anything connected to email, file shares, your CRM, or systems holding patient or member data.
- Watch the actions, not just the chat. Log what your AI tools actually do. Start by monitoring before you try to block.
- Write rules for the things that must never happen. For example, “this assistant never sends attachments outside our domain.” If an action should truly never happen, remove the capability entirely.
- Test on a schedule. Try to make your AI do the things it must not do, and retest whenever the model, tools, or connections change.
- Put AI in your risk assessment. For HIPAA covered entities and business associates, an AI assistant that can reach ePHI belongs in your security risk assessment like any other system, with its own threats, such as hidden instructions in an email leading to a disclosure.
The bottom line
One line from the session sums it up: “You’re not patching anymore. You’re reducing risk.” That is an uncomfortable shift for security teams used to closing findings. But these risks are manageable once you can see them, and the first step is knowing what your AI can reach and what it is doing.
If your organization has rolled out AI tools and you are not sure what they can access, Eagle Consulting Partners can help you inventory them, assess the risk, and build the controls into your existing compliance program. Contact us to start the conversation.
Tom Fazio, CISSP, is Vice President at Eagle Consulting Partners.
