Artificial intelligence has moved from the innovation lab into the fabric of everyday business operations. As organizations race to integrate AI into workflows, a parallel and deeply troubling trend is accelerating, the rise of adversarial AI: the deliberate exploitation of AI systems by malicious actors to scale, sharpen, and disguise attacks in ways that were not possible just a few years ago.
This is no longer a theoretical risk. According to the 2025 OneTrust and ISMG Securing the GenAI Era Report, adversarial use of AI is already a measurable reality, and most organizations are not adequately prepared to meet it.
Not long ago, the primary concerns around AI centered on model behavior: hallucinations, bias, and inaccurate outputs. Those concerns remain valid, but the threat profile has matured considerably. The OneTrust/ISMG report surveyed 180 cybersecurity leaders, technology executives, and business decision-makers across industries. The findings paint a clear picture of an adversarial AI landscape that has moved firmly into enterprise reality.
Nearly half of respondents (46%) reported experiencing AI-generated social engineering at scale. Attacks that use AI to craft convincing, personalized messages in far greater volume than human attackers ever could. Another 36% reported enhanced phishing attacks powered by generative AI, and an equal 36% flagged automated vulnerability discovery and exploitation as threats they've observed firsthand.
Deepfakes (synthetic video or audio designed to impersonate trusted individuals) were reported by 33% of respondents. Malicious code generation was flagged by 28%, and voice cloning used in business email compromise scenarios was identified by 24%. Perhaps most striking: only 14% of respondents reported observing no adversarial AI activity at all.
The direction of travel is clear. In earlier surveys from the same research series, a meaningful share of respondents said they hadn't seen adversarial AI in practice. That number has steadily collapsed. Adversarial AI has moved from a future concern to a present-day operational challenge.
With deepfakes on the rise, organizations need to be prepared to spot the tells.
Why Traditional Defenses Fall Short
What makes adversarial AI particularly dangerous isn't just its technical sophistication; it's how it undermines the basic signals organizations have historically used to validate trust. Writing style, voice, and even video have long served as informal verification mechanisms. Adversarial AI can replicate all of them convincingly.
This creates a verification problem that outdated security frameworks simply weren't built to solve. When an executive's voice can be cloned, a phishing email can be indistinguishable from genuine communication, and social engineering attacks can be deployed at scale with minimal human effort, the traditional "spot the anomaly" approach to security loses much of its effectiveness.
The same OneTrust/ISMG report highlights a parallel governance gap that compounds the threat:
- Only 15% of organizations have AI governance frameworks that are both centrally defined and operationally enforced across the enterprise.
- A full 40% have defined frameworks that are not systematically applied. Without enforcement, governance is theater, and adversarial actors know it.
Even the risk exposure organizations are most focused on has shifted. Employee-driven sensitive data leakage through AI tools was cited as the top implementation concern by 48% of respondents, reflecting a broader pivot from model risk to user risk.
Organizations that focus only on controlling AI models while leaving user behavior and data handling ungoverned are managing the wrong end of the risk spectrum.
Controls Exist, Readiness Does Not
To be fair, organizations are not standing still. The report identifies a range of controls that have been deployed: staff training (49%), data loss prevention tools (45%), tool whitelisting and blacklisting (33%), monitoring and logging (31%), and prompt filtering (28%). These are meaningful starting points.
But having controls in place is not the same as being prepared to use them under pressure. Fewer than one-third of respondents (just 29%) have a tested, AI-specific incident response plan. Most validation still relies on manual human-in-the-loop review (51%), and 16% report no formal validation process at all.
Workforce training coverage is similarly uneven, with 8% of organizations providing no GenAI training to employees.
The gap between defined controls and operational readiness is where adversarial AI finds its opening. Attackers don't test organizations in controlled conditions: they probe in real time, at scale, when defenses are least prepared.
Governance Is the Foundation for Defense
Defending against adversarial AI is not primarily a technology problem — it's a governance problem. The organizations best positioned to withstand AI-enabled attacks are the ones that have built structured accountability, clear ownership, and enforceable policies around how AI is used, monitored, and managed.
This is the core of what First San Francisco Partners (FSFP) is built to help organizations achieve. FSFP's approach addresses the specific conditions that adversarial AI exploits:
- Poor data governance creates blind spots; organizations can't detect data poisoning or unauthorized model access if they don't have complete visibility into their data ecosystem.
- Weak accountability structures mean no one owns the response when an AI-enabled incident occurs.
- Undefined roles leave gaps between the people who use AI tools and the people responsible for securing them.
Through its AI governance consulting services, FSFP helps organizations build frameworks that go beyond policy definition: establishing the monitoring, enforcement mechanisms, and clear ownership structures that allow governance to function as an active control system rather than a static document.
This approach includes designing Data Steward roles with explicit AI accountability, implementing Semantic Intelligence practices that make data lineage and context traceable, and applying the Data Decision Framework (DDF) to ensure decisions about AI systems are made with appropriate rigor and transparency.
The FSFP AI Governance Playbook (a free resource available on the FSFP website) outlines seven building blocks for responsible enterprise AI. Each building block directly addresses a category of adversarial risk: from aligning AI use cases to business strategy, to building transparency and ethical oversight across models and workflows.
The FSFP team tailors each engagement to the needs of the individual client, vs taking a one-size-fits-all approach.
What Organizations Need to Do Now
Adversarial AI is not a future problem that organizations can defer until governance matures. It is a present-tense operational challenge, and the gap between deployment and governance is exactly the opening attackers are using.
The most effective responses involve three interconnected priorities:
- First, operationalize governance, not just define it. Governance must be enforced, monitored, and tested against actual conditions.
- Second, assign clear accountability. The 29% of organizations reporting no single accountable owner for AI-related risk represent a critical structural vulnerability. Every AI system in production needs an owner responsible for its behavior, security, and governance compliance.
- Third, shift from model-centric to usage-centric risk management. Adversarial attacks increasingly target the human layer: how employees use AI tools, how data moves through AI workflows, and where sensitive information is exposed. Organizations that govern only the model while leaving usage ungoverned are managing less than half the risk.
The Path Forward
The pace at which AI has entered enterprise operations is extraordinary. The pace at which governance has followed is not. That gap is where adversarial AI operates — and it will continue to do so until organizations treat AI governance as mission-critical infrastructure rather than a compliance afterthought.
At FSFP, we believe that responsible AI doesn't begin with the model. It begins with the governance foundation beneath it. Trusted data, accountable people, enforceable processes, and the right technology controls working together create the conditions in which AI can be both transformative and secure.
If your organization is navigating the intersection of AI adoption and adversarial risk, we'd welcome the conversation. Connect with the FSFP team to learn how governance can become your strongest defense.
