From hype to actual deployment
AI has dominated nearly every tech conversation for the past few years: it is hard to find a product roadmap, conference keynote, or quarterly earnings call that doesn't mention it. What has shifted more recently is that the talk has turned into deployment. Chatbot demos and proof-of-concept pilots have given way to AI wired directly into production systems, customer-facing applications, and day-to-day business workflows.
Much of the current discussion around AI in offensive security centers on augmenting the penetration tester: using AI to automate reconnaissance, generate payloads, or triage findings faster. That conversation has its rightful place, but it overshadows a larger one. As organizations embed AI functionality directly into their applications and workflows, those systems become targets in their own right, with a distinct attack surface and vulnerability classes that traditional penetration testing methodology was never designed to address. This is what compels penetration testers to revisit and adapt their methodology: not just the tools they use, but the nature of what they are now being asked to test and how to approach the landscape.
A different kind of target
Organizations are adopting AI faster than their security programs can keep up, which raises a real question: how much of the traditional pentesting playbook still applies when the target is a language model instead of a server? AI penetration testing probes systems like LLM chatbots, RAG pipelines, and autonomous agents for ways they can be manipulated or made to misbehave. AI red teaming is the broader, more open-ended version of that same adversarial exploration. Either way, the target itself looks different: on top of the usual code and infrastructure, you are now also assessing the foundation model, the system prompt, the retrieval pipeline, agent memory, and every tool or API the model has been given permission to call.
Where the real risk concentrates
Most of the highest-risk findings in an AI assessment don't actually come from the model itself, they come from everything connected to it. A chatbot running a perfectly safe foundation model can still be compromised through a retrieval pipeline that pulls in untrusted documents, an agent carrying more tool permissions than the task requires, or a backend API that blindly trusts whatever the model tells it to do. The useful reframing for a tester is moving from "can we jailbreak the model" to "what can an attacker accomplish through it."
The vulnerabilities look different too
The vulnerability classes shift accordingly. SQL injection has its rough analogue in prompt injection, authentication bypass in instruction hierarchy bypass, remote code execution in agentic tool misuse. But some categories have no traditional precedent at all: model extraction lets an attacker reconstruct a model by querying it repeatedly, data poisoning corrupts behavior by tampering with the training set, model poisoning achieves a similar result through a compromised checkpoint further down the supply chain, and excessive agency turns a model's own legitimate permissions into a liability once it starts taking real actions on real systems.
One proof of concept might not be enough
This breaks traditional methodology in a specific way. A pentest normally treats one successful proof of concept as sufficient evidence; the flaw exists and will reproduce reliably. Prompt injection won't cooperate. An attack that works in one conversation can fail in the next depending on model updates, retrieved context, or how the model interprets ambiguous phrasing that day. The question shifts from can this be exploited to how reliably can it be exploited, which changes how many trials a finding needs and how confidently a report can state the risk.
What still carries over
None of this means pentesters are starting from scratch. The adversarial mindset, reconnaissance discipline, and ability to translate a technical flaw into business risk all carry over directly; what changes is the vocabulary and the target, not the underlying approach. Traditional and AI security testing are not replacing each other, they are converging, since most organizations running AI are still running traditional infrastructure underneath it. The practitioners who get ahead will treat AI red teaming as an extension of the same skill set, not a separate one.
A practical example using instruction bypass
Consider a customer-facing AI assistant configured with a system prompt restricting it to a specific scope:
From a traditional security perspective, that looks like an access control. The assumption is that the model will enforce it the same way a server enforces an authorization rule. A tester probing this system might submit:
Against models not hardened for instruction robustness, the model will partially comply, surfacing fragments of the system prompt or abandoning its defined scope. What makes this particularly relevant from an offensive standpoint is how that first response becomes the foundation for a second, more targeted attempt. If the model reveals that it has been instructed not to discuss pricing negotiations or internal company information, the attacker now has enough context to probe directly:
The model, already operating outside its original guardrails, is now being steered using its own disclosed context against itself. Each response narrows the attack surface and increases the precision of the next input.
Unlike a traditional vulnerability where a single exploit either works or it doesn't, this type of attack is iterative by nature. A tester documenting this finding would capture each payload, the model's responses across both turns, the consistency rate, and the cumulative business impact of an attacker using that progressively disclosed context to chain deeper into the system.
As rudimentary as this example may appear, variations of this technique succeed against production AI systems at a rate that would surprise most organizations who consider their system prompt a sufficient security boundary.