Skip to main content
Packetlabs Company Logo
Blog

Why Does AI Security Feel Familiar?

Authored By Sebastian Finch, Associate Ethical Hacker

Why Does AI Security Feel Familiar?

You can't dry a wet towel with another wet towel. That is roughly the strategy when we ask one AI model to secure others, or worse, itself. 

As companies rush LLMs and agentic AI into production, a pattern emerges: most of the "new" vulnerabilities inherent in agent systems are not new at all. They are old vulnerabilities with a new disguise.

The Pattern Beneath AI Security

LLMs and Agentic AI has not invented a new class of vulnerability. It has resurrected old ones and dropped them into a system for which existing security controls are not equipped to handle.

The OWASP Agentic Top 10 (Version 2026) and the OWASP Top 10 for LLMs (Version 2025) offer a clear description of vulnerabilities that, although new, map almost one to one onto bugs being fought for years: cross-site scripting, the confused deputy, TOCTOU race conditions, and living off the land. The song remains the same, the big change is the environment they live in nondeterministic and autonomous. 

Four direct examples help to visualize this point.

Four Old Vulnerabilities, New Faces

For every "new" agentic vulnerability there is usually a precedent from the past that already taught us how to think about it. This is not to say everything is the same, just to map them to things we’ve already experienced, to know how to test them, reveals the wizard behind the curtain. 

The mappings below should help to frame this in concretely. The first three come straight from OWASP, which names the confused deputy and TOCTOU as examples within ASI03. The fourth is an extrapolation of the living off the land (LOLBAS) technique that captures something about where this is heading.

Old world

Agentic world

What's the same

What's new

Stored XSS

Persistent memory injection

A payload is planted that fires later, in another session

The "DOM" is a vector store, and the trigger is a conversation

Confused Deputy

Low-privilege agent instructs a high-privilege agent

A trusted process is tricked into using its authority for someone else

The deputies are now plural, asynchronous, and persuasive

TOCTOU (Time-of-Check to Time-of-Use)

Permission races inside an agent workflow

Check, then state change, then action on stale state

Authorization is granted at the start of a workflow that may run for hours

LOLBAS (Living Off The Land Binaries, Scripts and Libraries)

Every tool call is legitimate, but the chain is the attack

Permitted actions chained to reach a forbidden outcome

Distributed, asynchronous, split across agents, and almost untraceable

What does this mean? The attack surface has changed again. Insider threats used to mean a person inside your organization. Now your agents are the insiders, and they are increasingly difficult to background check and track. The trust boundaries that sat at the network edge, then moved to the API gateway, has changed again. It now sits between agents, and many deployments have not drawn the new lines yet.

The Kill Chain, Reimagined

An interesting way to read the OWASP Agentic Top 10 is as a kill chain: the same four stages 

Kill Chain

Initial access. Prompt injection from poisoned content, a compromised package in the agentic supply chain, or a malicious MCP server. Two good examples are as follows:

  • A poisoned prompt that shipped inside Amazon Q for VS Code v1.84.0 before it was caught. 

  • A malicious npm package impersonating postmark-mcp that quietly BCC'd emails to the attacker, reportedly the first malicious MCP server found in the wild. (ASI04, LLM01.)

Lateral movement. One compromised agent infects the next through shared context, inter-agent messages, or a shared memory store. (ASI07.)

Persistence. Memory injection, the stored XSS of 2026. The payload sits dormant in long-term context and reactivates on a future task that has nothing to do with how it got there. (ASI06.)

Impact. Silent backdoors in commits, fraudulent invoice approvals, manipulated dashboards, exfiltration hidden inside legitimate-looking API calls.

That last stage is where most security writing stops, but as is tradition here let’s go a little deeper. The impact step comes with an eerie twist that is beginning to emerge from the shadows.

The Role of Human-Agent Trust Exploitation (ASI09)

When a server gets compromised, the server does the attacker's bidding. When an agent gets compromised the scary part is, your people do, and they do it by hand, during business hours, with legitimate credentials. 

OWASP calls this Human-Agent Trust Exploitation (ASI09), and the description is pretty slick. The agent acts as an "untraceable 'bad influence,' manipulating the human into performing the final, audited action, making the agent's role in the compromise invisible to forensics." Automation bias, perceived authority, and the conversational nature of LLMs and agents all collide to make a malicious recommendation seem benign, even helpful. 

Three examples from OWASP help show the real-world impact. The finance copilot ingests a poisoned vendor invoice and recommends an urgent payment to bank details the attacker controls, and the manager approves. The coding assistant offers a slick one-line fix, the developer pastes it, and the script exfiltrates source code. The clinical agent suggests a small dosage adjustment with a plausible explanation, and the clinician accepts.

In all these cases the event that gets logged is usually a normal human action, and the malicious part remains invisible. The traditional anomalous login, strange network traffic, shell on a server is missing. There is just an employee doing their job, with a quiet hand on the wheel.

What Can You Do?

Begin by studying OWASP’s Top 10 For Agentic Applications 2026. But for now, here are some key take aways to begin with.

Do not make AI guarding AI your only line of defense. Validation, policy enforcement, and guardrails belong on separate, deterministic systems. 

Assume the system prompt is public. Treat it like client-side code. Pliny's CL4R1T4S repository, a public archive of extracted production system prompts from major AI vendors, is the burning bush. If even frontier providers cannot keep this secret, it is unlikely others can.

Practice least-agency, not just least-privilege. Avoid handing out autonomy that is not needed. It’s tempting, but agentic behavior where it adds nothing increases the attack surface. If a deterministic script will do the job, it should be preferred.

Pentest agents the way you pentest insiders. Assumed breach scenarios. Watch behaviour, not reporting. Look for the living-off-the-land patterns chaining actions toward harmful outcomes. Look into AI/LLM penetration testing as a regular practice.

Conclusion

The good news is that we have fought every one of these before. The bad news is that we keep forgetting what we learned. And this time, the towel is making the decisions.

Contact Us

Join our newsletter

Packetlabs Company Logo
  • Toronto | HQ401 Bay Street, Suite 1600
    Toronto, Ontario, Canada
    M5H 2Y4
  • San Francisco | Outpost580 California Street, 12th floor
    San Francisco, CA, USA
    94104
  • Calgary | Outpost421 - 7th Ave SW, Suite 3000
    Calgary AB, Canada
    T2P 4K9
  • Australia | OutpostPacketlabs Pty Ltd.
    ABN 14 691 178 542
    Level 24, 1 O'Connell St
    Sydney NSW 2000
Cyber Right NowCREST LogoCREST AI Signatory AICPA SOC 2 LogoG2Clutch 2023 Certification Logo