Just Another Kind of Software: Setting Security Principles for AI

When people ask me about cybersecurity, I think of a line from Ani DiFranco’s song “My I.Q.”: “Every tool is a weapon if you hold it right.”ⁱ I had it in mind when sitting on a panel on cybersecurity and AI at “AI, Data, and Tech in Private Markets” in New York.

It’s a common misconception that the security risks from AI are different from other tech. The moment a firm puts AI in front of its information, we hear the same worries in the same order. People ask whether the tool leaks data, what the exposure is, and how to use it safely. Any bad actor can exploit unprotected AI the same way they could with APIs, databases, and essentially any other kind of software.

AI is just another kind of software, with a small number of variations. For the most part, the same discipline you apply to any software or vendor secures AI as well. Eight of the ten risks identified by the Open Worldwide Application Security Project (OWASP) for LLMs and Gen AI Apps are directly comparable to the way you would secure any application.ⁱⁱ The only exceptions are what it calls excessive agency and unbounded consumption.

The similarity points directly to what your first move should be ahead of any purchase. Set your security principles first, before you choose the implementations and tooling that fit them. It’s as true for building with AI as with any other capabilities.

Keeping your data yours

The first question to ask any vendor is how they handle your information. Does it leave their control? With AI, you also want to control whether the vendor trains its models on what you feed it. It’s a clear question of ownership. Your data have to belong to you. Client data or proprietary models are your exclusive material. We certainly make that clear in our own client agreements.

This guardrail is central to how we do business. New features that we build become a part of the platform for everyone. What a client builds on the platform belongs to that client. A client can audit the data its staff enters and decide how long we retain it. That is the same baseline that technology leaders should expect from vendors that provide LLM applications or other AI-enabled capabilities.

The unique wrinkle with AI is that buyers also have to scrutinize how a model holds what it learns. In theory, the right prompts could cause a model to regurgitate its training data either intentionally or inadvertently. It can also be more subtle than data.

For example, a firm’s macro trading strategy could be embedded in how its positions looked three months ago. If those data were returned to other users, they could reverse-engineer the embedding from the model and gain insight into your proprietary approach. Risks like that are essential to scrutinize and prohibit in clear contractual terms that keep what’s yours, yours.

Sanitizing inputs and trusting the supply chain

Prompt injection is another flavor of something familiar from traditional software. It’s another variant of input sanitization and structurally the same as SQL injection, though the mechanisms are different. Anyone who has cleaned up unwanted transactions or data deletions disguised in SQL queries will recognize this risk.

The AI equivalent can be quite similar. Suppose you pointed an AI system at all of your emails. If that system also holds the keys to send wires or carry out other transactions, an attacker could email a hidden instruction, like telling the system to disregard its prior rules, treat the sender as trusted, and send a payment or execute a trade. Anything that takes input from the outside world and connects live to other systems presents a similar risk. Online communities for developers and hackers are full of examples of exploits and vulnerabilities.

The main difference from SQL injection is that the possibilities are less well-known. But the solutions are similar. You should harden your systems with clear separation of duties. Sanitize any untrusted input into an AI the way you would into a database. That applies to what trusted insiders feed it, too, because your staff can make mistakes and paste things they should not, such as malicious code that makes its way into a coding system. It’s the same cast of systems and characters. The multi-layered defense and the security training you already run carry over.

Trust in InfoSec runs both down and upstream, however. At some point, you have to trust the compiler. You can rarely verify the whole chain that produced your tools, and with AI, you’re rarely buying a bare model. Someone could plant clever subsets of training data designed to activate only under narrow circumstances, and they are genuinely hard to detect. You’re buying an ecosystem that someone else compiled, trained, and wrapped in additional prompts and context. All of that trust is a diligence question.

What you bound and what you check

The first place AI breaks from the software analogy is what OWASP calls excessive agency. Deterministic software does a defined thing, but an LLM asks permission to do what it needs to do to finish your task. Those permissions accumulate. If you give them freely, you could reach a cascade point where a prompt meant to do something small reads files off your disk and sends mail in your name because of prior authorization you gave it.

The closer analogy here is a new hire, not software. It’s no different from giving an intern their own keys to the supply closet on day one, then taking the keys back at the end. To limit excessive agency, scope the work to a single project and grant permissions only within that project. Take the keys back afterward.

Checking the work also helps manage excessive agency. There’s a danger in overtrusting a model after it’s been right ten times in a row. It’s almost an instinct to start trusting it like a human and stop checking, or to ask the model to check itself and take “yes” for an answer. A person checking a machine’s output drowns in volume and starts rubber-stamping, like an airport screener who waves through the rare, genuine threat because nearly everything else was just a water bottle.

Humans checking humans works at the human scale, but checking at the machine scale is its own discipline. You want systems that show how they reached a result the way you tell an ops analyst to show their work. With LLMs, it takes a separate agent to check another and report the exceptions, while a person stays in the loop at the summary level, reads the citations, and spot-checks the rest.

Metering the spend

The second nuance with AI security is unbounded consumption. Teams forget to put a meter or a limit on these systems, and a single eager user can run up a six-figure bill that nobody knows about until it arrives. A bad actor could do the same, overloading an LLM with input and queries, or even stealing your model by hammering it with inputs that collect enough output to replicate your model, its weights, and architecture.

Some teams stand up a leaderboard for AI use. Don’t. Set budgets, the way you already do for any usage-based, consumption-priced API or SaaS platform. The moment you celebrate whoever consumed the most tokens, even level-headed people start burning tokens to avoid ranking last.

Right tools, wrong hands

What I took from the audience in New York is that you have to demystify AI security. Most of the time, AI is just software, and you secure it the way you secure software. Governing AI the way you would a capable, unproven employee helps manage the risks that look more like social engineering, but with an agentic user.

Keeping your tools from becoming weapons in the wrong hands means clearly answering five questions: whether your data leave your walls, whether you can sanitize what goes in, who sits upstream of the model, what the system may touch, and what it costs you. Answer those, and you are securing AI the way you already secure anything else you didn’t build yourself.