Separating AI Hype from Reality for Defenders and Adversaries

While headlines may have you believe that AI is supercharging cyberattacks, that narrative misses the larger point.

Yes, threat actors can wield language models to bypass defenses, write perfect phishing emails, or generate undetectable malware — but defenders have access to the same arsenal of tools, the same public models, the same APIs, etc. The difference isn’t about what they’re using. It’s about how they apply it.

This idea builds on the premise I first explored in What Generative AI Means for Cybersecurity: Risk & Reward, which examined both the optimism and skepticism surrounding early generative AI adoption, and why AI should augment human defenders, not replace them.

AI levels the playing field, but not the ethics

There’s no secret “dark LLM” optimized for offense. Instead, “malicious” LLMs are often repurposed foundational open-source models, stripped of their safety filters and fine-tuned. They don’t introduce new offensive capabilities or insights but rather exploit readily accessible technology by removing ethical constants.

Both attackers and defenders have access to the same foundational models. But context is the biggest differentiator between the two sides. Defenders can pair AI models with real, privileged, structured, data to give them purpose and perspective. Attackers can’t, largely because they operate in fragmented, disconnected environments with partial visibility and limited context. What sometimes gets overlooked is that adversaries are also experimenting with poisoned or tainted open-weight models, inserting malicious data, instructions, or hidden triggers during fine-tuning. But this isn’t new. For years, we’ve seen the same pattern in compromised software packages, rogue Python libraries, system tools compromised by a Trojan Horse, and back-doored open-source utilities.

For adversaries, the attack mechanism has evolved, but the intent hasn’t. Their aim is to weaponize trust and exploit distribution channels.

For cyber defenders, the takeaway is simple — these aren’t “evil AIs.” They’re just the latest generation of compromised software. The path forward remains the same: secure the supply chain, verify provenance, and monitor system behavior.

LLMs don’t discover zero-days or invent exploits; they simply predict text that sounds plausible based on what they’ve seen before. Without access to proprietary data or environmental context, LLMs can’t identify or make decisions around unseen systems or vulnerabilities. An attacker might use an LLM to generate boilerplate code, rewrite an email to nail the tone, or summarize reconnaissance notes — but none of that is truly new. It mainly helps them move faster, speeding up routine attack prep rather than creating entirely novel threats.

Offensive operations thrive on predictability and repeatability. LLMs are, by nature, nondeterministic — that is, they don’t give the same output twice. That’s fine for a defender generating summaries or hypotheses, but it’s problematic for attackers executing malicious payloads or exploit chains that must behave identically every time to be successful. This innate non-predictability is a real headache for attackers, as it undermines the precision required for successful, repeatable attacks.

Attackers want predictable scripts, not creative prose.

AI is a force multiplier for all, but defenders hold the high ground

AI changes the tempo of the game, not the rules. Both sides are using LLMs as force multipliers to speed up operations. For attackers, AI means faster, more prolific phishing campaigns and malware attacks. For defenders, AI offers accelerated threat detection, rapid incident digestion, and smarter insights from logs, user behavior, and other complex data. What they both gain is efficiency.

As Ryan Fetterman, Splunk Senior Security Strategist, highlighted in Defending at Machine Speed, by operationalizing these models, and integrating them directly into workflows, we accelerate reasoning and triage while keeping analysts accountable for making the final call on whether something is actually malicious.

However, defenders hold the high ground when AI is integrated responsibly, which includes privileged access to ground truth data identity, process lineage, session context), seamless integration with core security infrastructure, and the ability to audit and validate LLM reasoning. The power lies in our ability to fuse AI's probabilistic analytical insights with the deterministic, verifiable truth from our systems of record — a combination that provides more trustworthy and actionable intelligence and drives better decision-making.

LLMs amplify our understanding when paired with evidence, not when they're filling in holes with guesswork.

The real threat isn’t that attackers are developing polymorphic malware with AI, it’s that defenders might misuse AI themselves.

Treating LLM outputs as ground truth can lead to flawed decisions, such as misguided budget allocations or automation policies. Placing too much trust in LLMs as a single source of truth can also create operational blind spots, resulting in missed threats and disruptions. Ultimately, misuse of AI can weaken the very security you’re trying to build.

The danger isn’t that attackers are using AI better, it’s that defenders might start using it poorly.

When AI becomes the junior analyst

The promise of AI-driven SOCs often drifts toward letting models make the call. This can be problematic because we still need humans in the loop. But keeping humans in the loop can also be risky when the people relying on those outputs are still learning AI applications and protocols.

Junior analysts, by definition, are developing their intuition. They haven’t yet built the internal library of “I’ve seen this before” moments that help a seasoned analyst recognize when something doesn’t add up. A language model, on the other hand, sounds confident every time, even when it’s wrong.

That’s a subtle but serious risk: fluency feels like truth. When a system presents well-written reasoning, a junior analyst is far less likely to question it, even if the logic or conclusion is flawed. Over time, this erodes critical thinking; analysts stop reasoning through problems and start reasoning around the model.

If AI becomes a crutch rather than a coach, we risk building faster analysts who learn less. The right role for LLMs in the SOC is to empower human judgment, not bypass it. AI should serve as a mentor, accelerating understanding by explaining why an alert might matter and suggesting next steps. This flow enhances human expertise rather than replacing critical decision-making

 Why novel AI attacks are unlikely

Some research has explored whether LLMs could generate polymorphic or self-rewriting malware. Oesch et al. (2025) describes a proof-of-concept where ChatGPT produced key-logger functions at runtime, injected in memory rather than written to disk, a hypothetical “Living Off the LLM” scenario. Yet even the authors note the limitation: their example depended on remote APIs and was confined to a controlled environment.

Crucially, these systems remain nondeterministic. The same prompt rarely produces identical code twice. That lack of reproducibility and reliability makes them ill-suited to the precision that real-world attacks demand. As with most AI malware demonstrations, the output may look plausible but often fails to execute consistently.

The distinction between appearing plausible and the practical reality of execution matters. We need to separate a theoretical possibility from an operational reality. The current wave of super-powered malicious AI headlines risk creating a Chicken Little moment for cybersecurity — that is, loud warnings of falling skies that distract from real, measurable progress by using AI responsibly.

AI’s value is amplifying context

The evolution from our earlier blogs to today’s machine-speed workflows shows a continuous trajectory: progress in AI security isn’t about removing humans from the loop, but empowering them to make better, faster decisions. AI makes context scalable, not infallible. The organizations that win won’t be those that use AI first, but those that integrate it best — responsibly, contextually, and with human oversight.

To learn more about how AI is empowering both defenders and adversaries, subscribe to the Perspectives by Splunk monthly newsletter.