For years, the AI governance conversation centered on models. What data trains them, how outputs get reviewed, who owns accountability for results. Those are still the right questions. But they're no longer the only ones.
A new category of risk is taking shape inside enterprise organizations, and it's moving fast: agentic AI. AI agents aren't just generating content or surfacing recommendations. They're executing workflows, making decisions, accessing systems, and chaining actions across cloud environments, often with minimal human review at each individual step.
When an AI agent recommends a credit decision, drafts and sends a customer communication, or autonomously executes a procurement process, it isn't behaving like a tool. It's behaving more like a co-worker. That changes how governance needs to work.
The Gap Between Deployment and Governance Is Already a Problem
Enterprise adoption of agentic AI is accelerating quickly. Gartner projects that 40% of enterprise applications will embed task-specific AI agents by the end of 2026, up from less than 5% in 2025. A 2026 State of Agentic AI survey by CrewAI of 500 senior executives found that 100% of enterprises plan to expand their use of agentic AI this year, and yet only 34% cite security and governance as a top evaluation factor when selecting agentic platforms.
That gap, between deployment velocity and governance maturity, is precisely where agentic AI risk takes root. And the risks aren't hypothetical. They're structural, predictable, and already showing up in production environments.
Why Traditional Governance Frameworks Aren't Enough
Here's what makes agentic AI different from everything that came before it. Data governance was built to manage passive assets: records, schemas, definitions. You control what goes in; you set rules for access. The framework is relatively linear: define, catalog, steward, measure. AI agents are fundamentally different. They don't just store and retrieve, they reason, decide, and act. And they do so across boundaries that traditional governance was never designed to span.
At First San Francisco Partners (FSFP), we've long held that AI governance is an evolution, not a revolution, from data governance. The two disciplines share structural DNA (decision rights, risk oversight, accountability), but they diverge significantly in scope and execution. AI governance isn't simply data governance applied to model management. It covers the full lifecycle of AI solutions, including the increasingly autonomous systems now operating inside enterprise environments.
What agentic AI demands is a governance model that keeps pace with autonomous action. That means:
- Visibility into what agents exist, what they can access, and what permissions they hold. Most organizations currently lack this. What you can't see, you cannot govern.
- Clear decision rights over who can approve, deploy, monitor, or retire AI agents and agentic workflows. This is the agentic equivalent of data stewardship, and it requires a named human on the other end.
- Audit trails that capture not just what an agent did, but why, on whose behalf, and under what policy conditions. Intent matters when decisions have downstream business, regulatory, or ethical consequences.
- Human-in-the-loop checkpoints that reflect actual risk tiers. Not every agent action requires a human review. But some do and governance must define which.
Agentic AI needs governance, the same as any other data model would.
The Decision Steward: A New Role for a New Reality
As agentic AI scales, a new organizational role is emerging at the intersection of AI governance and accountability: the Decision Steward. This isn't a job title in the traditional sense: it's a capability. Decision Stewards critically examine AI-generated decisions for alignment to organizational values, societal ethics, and regulatory compliance. They verify that human-in-the-loop checks are functioning. They assess for bias. And critically, they treat AI agents with the same behavioral accountability expected of any co-worker.
This concept matters because accountability in agentic environments can't be diffuse. When an autonomous system makes a decision that causes harm, "the model did it" is not an acceptable answer for regulators, customers, or boards of directors. Someone has to own the outcome.
The Semantic Layer Is the Foundation
One of the most underappreciated governance requirements for agentic AI is the semantic layer — the shared business definitions, data meanings, and contextual intelligence that give AI systems the grounding they need to produce trustworthy, explainable outputs. Consider a practical example: What does "customer" mean in your organization? Is it the purchaser, the account owner, or the individual end user? The difference matters enormously when an AI agent is autonomously making decisions based on that concept, and without a governed definition, the agent is essentially reasoning in a vacuum.
Without semantic intelligence, even well-governed AI can produce results that are technically fluent but contextually wrong. When agentic systems are acting on data, the quality and governance of that data becomes a direct governance obligation, not just a data quality concern. This is why we say trusted data is both the output of data governance and the input to AI governance. For agentic systems, that relationship becomes even more critical: bad data doesn't just produce bad reports. It produces bad autonomous decisions.
What Agentic Governance Looks Like in Practice
Agentic AI governance isn't a separate framework bolted onto existing data governance. It's an extension of it — structured around the unique risks that autonomous action creates. Based on our work with enterprise organizations, here's where to start:
1. Take Inventory
You cannot govern what you haven't cataloged. That includes every agent in use, every system it can access, and every action it's authorized to take. Treat agents as digital identities with defined permissions and audit trails.
2. Define Risk Tiers
Not all agentic actions carry equal consequence. An agent that drafts a summary for human review poses different risk than one that autonomously sends a communication or executes a transaction. Risk tier definitions drive human-in-the-loop requirements.
3. Build in Observability
Governance without visibility is unenforceable. Every agent action, tool invocation, and policy decision needs to be captured in a format that supports monitoring and compliance review.
4. Extend Your Semantic Layer
The governed definitions, data products, and metadata that support your existing AI governance program must extend to the data agentic systems reason over. Semantic gaps become decision gaps.
5. Assign Accountability Explicitly
Every agentic workflow needs a named owner — a human accountable for what the agent does, not just what it was designed to do.
Agentic AI, when well governed, is a powerful coworker.
The Competitive Case for Getting This Right
There's a business case here beyond risk mitigation. Forrester Research notes that enterprises are entering AI's "hard hat" phase, where cost control, governance, and operational reliability matter more than impressive demos. CFOs are pushing harder for ROI. Boards are asking sharper questions. Gartner's 2026 Hype Cycle for Agentic AI signals the same tension: strong momentum without corresponding maturity across supporting governance, security, and cost-management capabilities. The need for oversight is becoming evident early in the adoption curve, not after large-scale deployment failures.
AI governance has never been purely a compliance exercise. Done well, it's a capability that makes AI more useful, not less. Governance-first organizations are positioned to scale faster, with fewer failures, and with the stakeholder confidence required to do so sustainably.
The Time to Build Is Now
The window to get ahead of agentic AI governance is narrow. Organizations deploying agents without the governance infrastructure to manage them are taking on structural risk that compounds with every new deployment. The good news: you don't have to start from scratch. If your organization has invested in data governance, you have a foundation.
The work is to evolve that foundation, extending decision rights, stewardship, semantic intelligence, and accountability structures into the agentic layer. That's exactly the kind of work FSFP was built to do. Ready to build governance that keeps pace with agentic AI? Talk to our team or explore our AI Governance Playbook for a practical roadmap.
Array
