Black Hat 2025: AI security is the new cloud security

Black Hat USA wrapped this past weekend. The AI security track was the most-attended track at the conference, meaningfully more than cloud, meaningfully more than crypto, meaningfully more than the perennial endpoint and network tracks that used to dominate. The substance under the hype was a real category forming. The shape of it maps closer to 2014-era cloud security than anyone wants to admit.

Worth pulling out the patterns, because the conversation about AI security in mid-2025 looks a lot like the conversation about cloud security in 2013-2014: the threats are real, the mitigations are immature, the people doing the work well are way ahead of the people who haven't started.

The categories that emerged

Three threat categories that dominated the talk content:

Prompt injection at the application boundary. The category we've been talking about for two years that finally has serious operational examples. Talks demonstrated working exfiltration against major SaaS AI integrations, working capability escalation against agent surfaces, working data poisoning against RAG pipelines. The defenses (input sanitization, output validation, capability isolation) are not new; the operational discipline to apply them at scale is.

Model and weight exfiltration. Multiple talks on extracting fine-tuned model weights via API timing attacks, response-shape analysis, and embedded-context probing. For shops that have invested meaningfully in fine-tunes, the IP is genuinely at risk in a way that changes the threat model. The mitigations (rate limiting, response normalization, watermarking) are partial; the category is going to be a sustained research area.

Agent privilege abuse. This was the most-attended sub-track. Talks demonstrated agents being convinced to take actions outside their intended scope through prompt manipulation, tool-call confusion, and conversation-history poisoning. The pattern of an agent with broad permissions being talked into using them on behalf of an attacker rather than the legitimate user. The fix is the same shape as the building-an-AI-assistant-that-can't-see-your-secrets pattern: capability isolation, scoped access, structural rather than promised security.

The parallels to 2014

The thing that struck me most across the talks: the AI security conversation in 2025 looks a lot like the cloud security conversation in 2014.

In 2014: cloud was being adopted faster than security teams could keep up. The threat models that mattered were misconfiguration (open S3 buckets), credential mishandling (hardcoded keys), shadow IT (unapproved deployments), and supply chain (third-party libraries with vulnerabilities). The defenses were a combination of platform-level guardrails (which were immature), tooling (which was nascent), and discipline (which was inconsistent). The shops that got it right early built strong cloud security teams, invested in tooling, and made governance a first-class concern.

In 2025: AI is being adopted faster than security teams can keep up. The threat models that matter are misconfiguration (over-permissioned agents), credential mishandling (secrets in prompts and conversation history), shadow IT (unapproved AI deployments and personal-tool usage), and supply chain (model weights, prompt libraries, MCP servers). The defenses are a combination of platform-level guardrails (immature), tooling (nascent), and discipline (inconsistent). Same playbook applies.

The shops that get this right early will build strong AI security teams, invest in tooling, and make governance a first-class concern. The shops that don't will repeat the 2014-2017 cloud-security catch-up cycle, with the same kinds of public incidents marking the milestones.

The governance gap, again

The thing that's still missing from the AI security conversation, and the thing the talks at Black Hat repeatedly pointed at: the governance gap I wrote about in the Build context is now showing up as a security gap. Agents deployed without clear ownership, without audit trails, without revocation paths, without scope policies, these are the agents getting compromised. The compromise patterns are predictable; the governance to prevent them is missing.

The pattern from a couple of talks specifically:

• An agent deployed without an owner stays running after the use case it was built for is gone. Nobody notices it's still active. An attacker finds it, talks it into doing something the original use case didn't intend, the action looks legitimate because the agent has legitimate permissions.
• A prompt-engineering vulnerability gets disclosed for a major model. Patches available. The fleet of agents deployed by the org doesn't have a coordinated update path. Half the fleet stays vulnerable for weeks. Attacker exploits the slow ones.
• A new MCP server gets connected to an agent for a one-off project. Stays connected forever. The MCP server author ships an update with a backdoor. The agent is compromised through the supply-chain path.

These aren't sophisticated attacks. They're the cloud-security pattern of "misconfigured / unmanaged / unmonitored" applied to the agent layer. The fixes are the same kind: ownership, asset registry, change management, audit trails. The work has to happen at the agent-platform level; the platforms aren't shipping it well yet.

What the security teams are building

A few patterns of what mature shops are actually doing, based on hallway conversations rather than the headline talks:

AI asset registries. A list of every agent, every model integration, every MCP server in production. Owner, purpose, scope, last-reviewed date. Same shape as a cloud-resource registry. Critical for the "what do we have running" question.

Per-agent capability audits. Periodic review of what each agent can actually do, against what it should be able to do. Excess capability gets revoked. New capability gets approved. The discipline is similar to least-privilege IAM auditing.

Prompt-injection red teaming. Regular adversarial testing of production AI surfaces. Try to make the agent do things it shouldn't. Find the vectors before an attacker does. Most of this work is still bespoke; the tooling is starting to mature.

Supply-chain controls for MCP servers and prompts. Provenance tracking for what's in the prompt library, what MCP servers are connected, where the model weights came from. Signature verification. Update notifications. Same shape as the open-source supply-chain controls that matured in the post-Log4Shell era.

Conversation-level audit logging. Every conversation with a production AI surface logged in queryable form. Useful for incident response, useful for compliance, useful for catching anomalies. Most platforms make this hard to do well; the orgs doing it have built it themselves.

What I took away

Three things that landed for me from the conference:

The category is real. AI security as a discipline is forming the way cloud security formed in 2013-2014. The good people exist; the tooling is immature; the certifications and training don't yet exist; the research is publishable in a way that suggests open problems and active interest.

The mitigation playbook isn't exotic. It's mostly the same playbook as good cloud security, applied to a new layer. Asset management, least privilege, audit, change control, supply chain. The shops that have strong cloud security operations adapt to this faster than the shops that don't.

The platforms are behind. None of the major AI platforms ship the security primitives at a level that matches enterprise expectations. The gap is the same gap as 2014-era cloud. IAM, audit, monitoring, incident response. The platforms will close it; the timeline is "next eighteen months" rather than "next quarter."

The conference vibe was the same vibe Black Hat had in the early cloud days, energy, real research, immature operational tooling, and a steadily-growing wave of people figuring out they need to take this seriously. That progression took the cloud security category from "barely a track" to "core conference content" over about three years. AI security is on the same trajectory and probably on a faster timeline. The treat-the-AI-like-an-employee discipline provides the management framing; the security work is the operational expression of it.

Finally, if you're a security person in 2025, AI security is the place the next decade's career is being built. If you're an AI person, the security side is the part of the platform conversation worth watching closely. The two communities are converging fast.