AI Model Governance: Tackling the Challenge of Explainability and Risk in Hedge Fund AI Adoption

AI governance meets data governance

While everybody is familiar with data governance, AI governance might be a novel concept to some fund managers. Naturally, the concepts are intricately intertwined. Aside from guardrails that police identity and access management to prevent prompt hijacking, shadow AI, and other breaches, AI governance shares other DNA strands with data governance. They share common responsibilities in guiding data as a product that AI systems create and consume, both overseeing data integration, quality, security, privacy, and accessibility.ⁱⁱⁱ The most practical overlap is providing everybody in the firm with good information to advance business needs. AI agents differ in that they are asked to generate insights and make decisions.

Refereeing the army of AI agents

It won’t be long before each fund has scores and, subsequently, hundreds of AI agents to conduct operations, all communicating with each other as relevant. Model context protocol (MCP) will accelerate AI adoption for asset and fund managers by enabling data interoperability, context, and workflow standardization to connect LLMs to financial data, tools, and systems. A report manager agent can schedule a report, amend its frequency, or report on its generation status. Other agents might handle transactions, trade booking, corporate action booking, or provide information about data quality breaks and exceptions.

Hallucination is a well-documented risk, especially if multiple agents are involved, as one hallucinatory step could lead to a devastating result. Because hedge fund systems deal with real-world breaks, cash movements, and P&Ls, platforms cannot afford hallucinated answers from AI agents. Additionally, if two agents have similar-sounding capabilities or titles, LLMs might make mistakes, necessitating clear definitions of each agent's capability.

AI explainability + observability = integrity

Just as in data quality management, AI agents' decision-making must preserve integrity, lineage, and auditability. The human in the loop needs to be able to track back to the agent’s sources. How did the agent arrive at that decision? How did it go about it? There must always be an audit trail. Whenever it generates the output, the users can then backtrack it to the data on the platform. It's all about AI explainability, transparency, drill-through capabilities, and ensuring humans can cite its thought processes, so to speak.

AI governance should be considered part of the overall AI rollout and adoption structure. In the initial user acceptance testing phase, designated beta testers should check and review the results before general rollout. Ideally, there is a permanent governing body that determines what any open specifications should be and keeps decisions on AI standardized. This is an AI foundational pillar of sorts, which provides us governance in a centralized manner, determining which models will be used and for what purposes. Firms should not be duplicating the same work in different products or modules.

AI is not always the answer

Hedge funds are engaged in another technology arms race, to be sure. However, they shouldn’t rush into rolling agents out across departments merely because they know that eventually all departments will be using agents. When there is a right moment to use AI, we certainly want to use it, but not everything needs to be solved with AI. When certain functions are probabilistic (non-deterministic), then firms should bring in the help of AI agents or LLMs. However, if there are some deterministic steps and deterministic rules that can be taken, then I would recommend firms move forward with their existing processes. LLMs take time in terms of response, they cost us money and expend power, so we want to be very thoughtful. It's also a performance issue. AI implementation brings a degree of uncertainty into operations, at least initially. Firms want to avoid it wherever they can, and use it to maximize impact, wherever it is truly needed. Critically, firms should avoid “AI washing” — false or exaggerated claims about their AI usage enhancing operations — to dazzle potential allocators and other stakeholders.^iv

Closing the governance gap in hedge fund AI models

Hedge funds possessing the most substantive, highest quality proprietary data stand to hold a competitive edge in the AI era. Funds that establish robust AI governance will lead the drive toward integrating AI models in hedge fund operations. The US Department of Homeland Security wrote, “AI use in the financial services sector presents unique concerns, including with respect to explainability, bias, and accountability”; and that humans should remain in the loop so investment managers can “ensure a continuous chain of human responsibility across the entire AI project lifecycle.”^v

Meticulous governance is imperative to prevent model drift, hallucinations, and opaque decision-making — risks that can impair investment processes, compliance, and performance as AI adoption accelerates.