Autonomous Adversaries: Are Blue Teams Ready for Cyberattacks To Go Agentic?

2024 was a year of incredible progression for Artificial Intelligence. As large language models (LLMs) have evolved, they have become invaluable tools for enriching the capabilities of defenders – instantly providing the knowledge, procedures, opinions, visualizations, or code any given situation demands. However, these same models provide outputs that enable even low-sophistication attackers to uplift their own skill-levels. The introduction of features enabling LLMs to operate as autonomous agents, including actions on local machines, further extend these capabilities. For defenders, this raises concerns about the intersection of attacker techniques, LLM maturity, and AI autonomy. Will 2025 be the year of the 'agentic adversary'?

During the initial rise of LLMs, analysts expressed concern about a potential surge in phishing attacks, driven by the ability of models to craft convincing lure emails. The evidence has been mixed — SURGe research found that compared to existing tools, LLMs provided minimal advantages in generating phishing content, even for cross-language messages. This suggests that composing and sending phishing emails is not a significant bottleneck for attackers, but it raises a critical question: what happens when LLMs can accommodate the tasks that are the current bottlenecks? Are there specific stages in the kill chain where autonomous LLM agents could prove capable, and even offer a distinct advantage?

The Data

As researchers, we will avoid wild speculation in answering this question and instead focus on drawing conclusions and asking questions that are grounded in data. Two factors at the intersection of this topic can be clearly measured: 1) real-world ATT&CK™ technique use, and 2) offensive LLM capability maturity. Fortunately, our Macro-ATT&CK project has compiled 5 years of in-the-wild attack technique frequency, and our Cisconian-colleagues from Robust Intelligence have published new research specific to the capability of LLMs to serve as co-pilots for enacting attack procedures, aligned with the techniques of the ATT&CK framework. The LLM evaluation, Yet Another Cybersecurity Assistance Evaluation (YACAE), prompted three distinct models (ChatGPT-4o, Claude Sonnet 3.5, and Google 1.5 Gemini Pro) in 5 rounds, testing the model’s ability to assist the user in enacting ATT&CK technique procedures on a specific platform (e.g. MacOS, Windows, Linux). Details of each individual test are available here.

By combining these datasets, we can identify the intersection between in-the-wild technique use and the relative maturity of LLM capabilities. This analysis can help defenders and researchers better understand the near-term implications of the overlap, the gap, and in what ways new capabilities such as agentic automation could change the nature of cyber threats.

With the data compiled, let's explore some examples and visualizations with Splunk!

The Intersection

First, let's find some common techniques where LLMs are also mature in their ability to instruct a method or produce accurate code for enacting attack procedures. We can get a quick summaryand sort for the most common techniques in SPL:

Here we can see high competency in instructing a user how to carry out specific attack procedures like transferring tools, enumerating users, assets, and software components, across multiple platforms. The high frequency of these techniques being used in the wild could be interpreted in many ways. They could be highly successful, they could hold high utility across environments, or they could be low sophistication. What we can say for sure is that they are common, and these models are accurate in their knowledge and ability to enact them.

The Gap

On the other hand, let's look at some common examples in-the-wild, but where the evaluated capability is currently inconsistent.

Discussion

There are clear strengths and weaknesses in the capabilities of the tested models, depending on the model and the operating systems targeted. Robust Intelligence found that LLM’s were willing to comply with the task ~90% of the time. Because of legitimate security evaluation use cases, or the ability for actors to locally host models without guard rails, compliance should not be expected to be a significant barrier to accessing this information. The models were able to accurately instruct on the prompt in about ~45% of tested cases:

It is important to note that these results are incomplete, we don’t yet have tests for all techniques in ATT&CK, and only a handful of results are explored in this post. While much of this capability isn’t new – actions have been automated and scripted for a long time – the introduction of advanced reasoning techniques, the ease of use, and the ability for models to summarize and contextualize returned information are creating new opportunities.

Actions like discovery of users, services, credentials and logs are assessed to be relatively mature. This suggests that an attacker can create a reconnaissance agent that could be tied to mass exploitation attempts and provide summarized information on targets. This type of capability would allow for more targeted compromise of high-value targets. While initial vulnerability exploitation is not yet a mature capability of these models, it is another topic in active development using agentic capabilities.

Conclusions

In this post, we examined how ATT&CK techniques observed in-the-wild intersect with the current capabilities of large language models (LLMs) when used as assistants for executing these procedures.

While these evaluations suggest modest improvements may be possible in operational speed and learning efficiency (~45% accuracy overall), the models are currently highly accurate (80-100%) in reproducing many of the most commonly used attack procedures. It is still uncertain if, or when, integrating these models’ agentic capabilities will lead to dramatic scaling of operations or significantly reduce the need for human involvement in cyberattacks.

What does seem evident is that as LLMs grow more adept at facilitating cyberattacks, the skill floor for attackers will inevitably rise. While models currently fall short of facilitating the most complex elements of attacks such as crafting effective evasion strategies or engaging in advanced attack planning, new research like non-linear reasoning may soon help overcome these limitations. LLMs have yet to invent entirely new classes of attacks or make existing methods substantially either more efficient or reliable, so Blue Teams do not need to prepare for substantial deviations from the established tactics and techniques present in ATT&CK. Another mitigating factor is that AI-enabled assistance is inherently limited by the user’s ability to articulate their objectives clearly and to integrate the model’s outputs into a cohesive and effective attack strategy.

Moving forward, it is critical to continue to monitor the progression of cyber-attack enablement via these models, and facilitate dialogue and coordination between the security and machine learning communities. Through this combined expertise we can develop robust protections which meaningfully secure systems against emerging AI-enabled threats. We are seeking to complete the baseline of YACAE evaluation to include more of the top ATT&CK techniques and procedures. By regularly evaluating these models with nuanced, realistic, and detailed cyber attack evaluations, we hope to anticipate and prepare for emerging threats as AI capabilities advance. We invite you to contribute new evaluations as we continue to pursue a more secure future for cyber and AI development.

As always, security at Splunk is a family business. Special thanks to collaborators: Kamile Lukosiute (Robust Intelligence) and Adam Swanda (Robust Intelligence).