Topic · 89 links
Autonomous cyber attackers
AI systems and agents that find and exploit vulnerabilities, the evidence on how much of an intrusion they now run without a human, and the defensive work responding to them.
The argument has moved on from whether models can hack. Frontier models now find real vulnerabilities at scale, from DARPA's AI Cyber Challenge to restricted-access programs such as Anthropic's Project Glasswing, and threat intelligence teams describe state-linked intrusions in which the model did most of the work. The open questions are narrower and harder: how much of an operation actually runs without a human, whether faster discovery means faster exploitation, and whether defenders can use the same tools to patch what is found before attackers weaponize it. The evidence splits where the arguments do. The strongest claims about autonomy come from vendor threat reporting. Vendors hold the telemetry, but they also sell the models and the defenses, so their reports are flagged throughout. Benchmarks and exploitation data are more reproducible but measure narrower things. So far they show the number of vulnerabilities found going up, not a clear collapse in time-to-exploit. The harness around a model matters as much as the model itself: the same model can go from 3 of 40 to 37 of 40 in a different scaffold, so any capability figure describes a whole system, not a model.
Start here
Eight readings, best read in order. You'll start with what was claimed about AI attackers, then see how those claims held up when tested, and finish with what security companies say they've seen in real attacks. Each reading has a short note on why it's in that spot. The first four are also on the home page, and you'll find more reading lists onPaths.
-
Staying ahead of threat actors in the age of AI
The 2024 baseline: state actors using models, no novel attacks. Everything after is a claim of departure from this.
The first joint Microsoft and OpenAI disclosure, naming five state-linked groups using models for reconnaissance, scripting and social engineering, and finding no new AI-enabled attack. The baseline everything since is measured against.
-
LLM Agents can Autonomously Exploit One-day Vulnerabilities
The paper that started the alarm, read with its own caveat in view: 87% with the CVE writeup, 7% without.
Reports GPT-4 exploiting 87% of 15 one-day CVEs when handed the CVE description, dropping to 7% without it. The headline number is widely cited; the caveat that it needs the writeup is the part most citations omit.
-
Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models
How capability actually gets measured, and why subtask decomposition matters for reading any score.
Forty professional CTF tasks from four competitions, each decomposed into subtasks so partial progress is measurable. The benchmark most frequently cited when labs and AI safety institutes report offensive cyber capability.
-
CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
The same question against real CVEs rather than CTFs. Up to 13% is the number to hold onto.
Sandboxed reproductions of critical-severity web CVEs with automated success checks. Leading agent frameworks resolved up to 13%. That figure is the most defensible number currently available for end-to-end autonomous exploitation, and it is a long way below what the threat reporting implies.
-
Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks
Then see the number move when scaffolding changes: 3 of 40 becomes 37 of 40. Capability claims are about harnesses, not models.
Inserts a high-level attack abstraction layer between the model and the network, reaching critical assets in 37 of 40 benchmark environments where baseline agents managed 3. The clearest evidence that scaffolding, not model scale, is the constraint.
-
SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned
The largest funded test of autonomous find-and-patch, written up by the people who ran and entered it.
Camera-ready systematization of AIxCC by participants and organizers, covering competition design, the finalist architectures, and execution traces. The single best account of what autonomous cyber reasoning systems could and could not do by 2025.
-
Disrupting the first reported AI-orchestrated cyber espionage campaign
The first claimed end-to-end autonomous campaign. Read it as a vendor's account, because that is what it is.
Anthropic's account of a campaign it attributes to a Chinese state-sponsored group that drove Claude Code through roughly 80-90% of an intrusion chain against about thirty targets. Vendor-authored and self-reported; no independent corroboration of the attribution.
-
GTIG AI Threat Tracker: From Prompting to Autonomy - The Evolution of Adversarial AI
A second vendor reaching similar conclusions independently, which is the closest thing to corroboration available.
September 2026 tracker documenting a shift from prompt-assisted work to agentic workflows, including a cloud compromise and mass credential harvesting completed in under six hours. Names specific actors and malware families.
Everything in this topic
Grouped by subtopic, then by source type within each subtopic.
Nothing matches those filters. Try clearing one of them.
Agentic exploitation 2
Agents carrying an exploit chain end to end against real or reproduced targets.
-
AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks
Targets post-breach hands-on-keyboard activity rather than initial access, which is the phase most later threat reporting says actually shifted. Written by Microsoft-affiliated authors and still an unreviewed preprint.
-
LLM Agents can Autonomously Exploit One-day Vulnerabilities
Reports GPT-4 exploiting 87% of 15 one-day CVEs when handed the CVE description, dropping to 7% without it. The headline number is widely cited; the caveat that it needs the writeup is the part most citations omit.
Vulnerability discovery 4
Models and agents finding flaws in code rather than exploiting known ones.
-
Archived evaluation of Chinese code LLMs on vulnerability detection: 3,200 raw API responses from 27 July 2026 with corrected verdicts, the JavaVulBench 200-sample probes, an October 2026 refresh grid of four models by four prompt strategies, and a corrected pipeline. This is function-level classification, not autonomous discovery, so it does not bear on zhu-2025-cve-bench numbers. Revision under review at IEEE Access.
preprint# -
From Naptime to Big Sleep: Using Large Language Models To Catch Vulnerabilities In Real-World Code
Walks through an exploitable stack buffer underflow in SQLite that the agent found and existing fuzzing had missed. Vendor-authored, and a single case. The technical detail is unusually complete for this genre.
vendor-authored# -
Project Naptime: Evaluating Offensive Security Capabilities of Large Language Models
Argues that poor benchmark scores reflect bad scaffolding rather than model limits, and shows CyberSecEval 2 buffer-overflow scores moving from 0.05 to 1.00 once the tooling improved. The methodology post behind Big Sleep.
vendor-authored# -
Introducing Aardvark: OpenAI's agentic security researcher
Announcement of a GPT-5-based agent that finds vulnerabilities, validates exploitability in a sandbox and proposes patches, claiming 92% detection on a seeded benchmark. Vendor announcement, not a paper; the benchmark is not independently reproducible.
Autonomous penetration testing 3
Systems that run an engagement with little or no human steering.
-
Big Enough to Break Out: Tracking the Rising Capability of LLM Penetration-Testing Agents
Compares a human-in-the-loop pentest harness against a newer autonomous one across three public targets, arguing that planning rather than memory is the current bottleneck. Small sample, September 2026 preprint.
preprint# -
PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing
The peer-reviewed version of PentestGPT, splitting reasoning, generation and parsing into separate modules to survive context loss over long engagements. USENIX hosts the PDF and talk video free.
-
The Road to Top 1: How XBOW Did It
Vendor account of reaching the top of HackerOne's US leaderboard with roughly 1,060 validated findings from an autonomous system. Marketing-adjacent and entirely self-reported. The discussion of deduplication and validation is the substantive part, and it is the part most coverage of this left out.
Multi-agent orchestration 1
Several coordinated agents running phases of an operation together.
-
Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks
Inserts a high-level attack abstraction layer between the model and the network, reaching critical assets in 37 of 40 benchmark environments where baseline agents managed 3. The clearest evidence that scaffolding, not model scale, is the constraint.
Cyber reasoning systems 3
Automated find-patch-exploit systems, from the Cyber Grand Challenge onward.
-
SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned
Camera-ready systematization of AIxCC by participants and organizers, covering competition design, the finalist architectures, and execution traces. The single best account of what autonomous cyber reasoning systems could and could not do by 2025.
-
AI Cyber Challenge marks pivotal inflection point for cyber defense
DARPA's results announcement. Team Atlanta's system found 54 of 63 synthetic vulnerabilities and patched 43, across 54 million lines of code, and all seven finalist systems were released under OSI-approved licences — which is the part with lasting value, since the systems themselves are now inspectable.
-
The 2016 competition for machines that find, patch and exploit flaws unaided. It predates LLMs entirely — which is the reason to read it before treating agentic exploitation as new.
Benchmarks 7
Task suites built to measure offensive or defensive capability reproducibly.
-
BypassGym: A Benchmark for Evaluating AI Agents' Endpoint Defense Evasion Capabilities
Benchmark in which an agent receives command-and-control shellcode and must build a loader that runs it on a real Windows endpoint without being stopped by the installed EDR product. Across 90 runs covering five lightweight models, six EDR products and three payloads, 71 succeeded (78.9%), and Windows Defender stopped none of its 15. Each combination ran once, and an injected evasion skill matters: GLM-5.3 Flash fell from 15 to 9 of 18 without it. Unreviewed preprint; code and data are promised only after review.
preprint# -
Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models
Forty professional CTF tasks from four competitions, each decomposed into subtasks so partial progress is measurable. The benchmark most frequently cited when labs and AI safety institutes report offensive cyber capability.
-
APTInvestBench: Evaluating Autonomous APT Investigation under Varying Telemetry
Benchmark for LLM agents doing SOC-style APT investigation: 370 cases built from 56 report-informed attack reconstructions (16.4 million synthetic log records generated with Cisco Talos's EvidenceForge) under seven telemetry conditions. Across eleven models, agents acquire sufficient evidence for 44.3% of recoverable attack actions but cite it for only 25.0%. Endpoint-only telemetry cuts coverage by just 1.6 points while 35.5% of previously covered actions lose support. The main study uses 10 of the 56 scenarios; unreviewed preprint.
preprint# -
CyberPersistBench: Evaluating LLM-Based Cyber Attackers on Installation and Persistence
Starts agents from a restricted foothold and tests whether they can keep it: 203 tasks over 46 persistence mechanism families on Linux and Windows, scored at six prefix levels from writing the Artifact to surviving a reboot. Five frontier models reach 27.6–42.4% at pass@3, and most failures fall between writing a valid config and getting the host to register it. Native defenses on 128 tasks cut success to 5.5–13.3%, not to zero. The 44.8% scaffold figure excludes tasks refused on safety grounds. Preprint.
preprint# -
CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
Sandboxed reproductions of critical-severity web CVEs with automated success checks. Leading agent frameworks resolved up to 13%. That figure is the most defensible number currently available for end-to-end autonomous exploitation, and it is a long way below what the threat reporting implies.
-
Task descriptions across six CTF categories, run instructions, evaluation logs, a frontier-model leaderboard. Start here rather than the paper if you mean to run it.
-
Conformance suite testing whether AI agent memory and checkpoint stores reveal tampering. Release 0.6.3 runs nine storage-level edits against sixteen stores, eight of which claim tamper evidence, with a control confirming each edit changed only what was intended. The threat model is an attacker with write access to storage, not memory poisoned through the agent's own writes. Single-author Zenodo release, not peer reviewed.
living#
Capability evaluation 4
Methodology for eliciting and measuring what a model can actually do.
-
No One Architecture Fits All: A Cross-Environment Evaluation of Hierarchical Red Team Agents
All-RL and all-LLM planner–executor red agents against expert RL defenders in CybORG CAGE-4 and Cyberwheel at 100 and 1010 hosts (18 configurations, 3,600 episodes). RL wins CAGE-4 (78.5% vs 18.0%) and Cyberwheel-100 (81.0% vs 50.5%) but scores 0.0% at 1010 hosts, where Foundation-Sec-8B reaches 55.0%. Every configuration compromises a host; the failures come at escalation or root-to-impact. LLM results depend on the model (DeepHat scores 0% everywhere), and on CAGE-4 the RL planner proves not load-bearing. Simulator-only preprint.
-
Threat-Preserving Representation Sensitivity in Agent-Security Benchmarks
Across 28,904 runs on ASB, MCPTox and AgentDojo, renaming tools while holding task, harmful action and scorer fixed shifts attack success. Neutral names on ASB's attacker tools raise committed ASR by 11.67–13.21 points; threat-explicit names on MCPTox lower it by 11.00 (GPT-5-mini) and 4.11 (Haiku 4.5); AgentDojo barely moves (−0.50). A form-matched neutral name reproduces 8.54 of MCPTox's 11 points, so threat vocabulary is not the main driver. Unreviewed preprint covering three small models.
preprint# -
The UK AI Security Institute's evaluation framework. CVE-Bench and several other cyber evals are written against it. MIT.
-
Nonprofit evaluating frontier models for autonomous capability. Not cyber-specific, but its time-horizon work underpins most autonomy forecasting.
Simulation environments 3
Synthetic networks for training and studying automated agents.
-
Crossing the Cyber Divide: Sim-to-Sim and Sim-to-Real Transfer for RL Agents
A PPO attacker trained in CyberWheel is transferred zero-shot to NetSecGame, CyberBattleSim and NASim through a kill-chain action/observation wrapper and a domain-adversarial encoder. On NetSecGame the encoder reaches 45.2% wins, no better than the wrapper alone (47.4%). On one CyberBattleSim chain topology it wins all 20 episodes where the wrapper wins none. The sim-to-real claim rests only on action-type divergence from an emulator policy (JS 0.085); actual emulation runs are deferred.
contested# -
The environment behind the multinational CAGE Challenges for autonomous cyber defense agents. MIT. Challenge 4 adds multi-agent RL.
-
Abstract simulated network for agents moving laterally, built for reinforcement learning rather than realism. MIT. The abstraction is deliberate and it does limit transfer.
Prompt injection and agent hijacking 14
Turning an agent's inputs against it, as both an attack and a defence.
-
CounterSteer: Suppressing Indirect Prompt Injection with Activation Steering
Fits a per-model residual-stream direction and subtracts it from every tool-result token at prefill, with no detector or fine-tuning. Across five open-weights models, AgentDojo compromise falls from 0.10–0.49 to 0.006–0.079 at 93–100% benign utility. An adaptive AutoDojo attacker is held to about a quarter of its undefended success. Parameter manipulation survives: 13 of 18 samples cracked under framing search. The 0/2052 LLMail-Inject result replays attacks tuned against other models. Requires white-box serving; single-author Microsoft preprint.
-
Divide and Inject: Can Agents Reconstruct an Indirect Prompt Injection from Fragments?
AdaLCPI splits an injected instruction into two incomplete fragments buried in filler, plus a generic reconstruction cue, and evolves them with OpenEvolve against the target agent. Across seven models and 81 Email, GitHub and Slack task–objective pairs, 8k-token filler reaches 61.4% macro attack success against 32.8% for a matched explicit-instruction baseline. Fixed fragmentation scores 5.1%, and fragments without filler score 0%. Gains are uneven: GPT-5.6-Luna stays at 8.6%, and TAP does better on Ministral. Each search takes up to 129 black-box runs, and a GPT-4o judge decides success. Preprint.
preprint# -
A2A-TIBA chains indirect prompt injection with a persistent callback program delivered over the agent-to-agent protocol, so that after one implant subsequent commands bypass the agent entirely. Tested across 15 front-end x back-end combinations (900 attacks) with a four-class metric separating LLM refusal from agent-layer interception; penetration ran 12% for hermes to 45% for Claude Code. An ablation argues untrusted-source label headers on A2A, tool and memory channels sharply raise refusal. Single-author preprint submitted to IEEE TDSC; testbed only, decoy files, China-market back-ends.
preprint# -
Prompt-injection defense that gates each tool call on where its arguments came from, rather than trying to recognise malicious text. The author reports that a text-matching rule layer catches 19.8% of a corpus of real attacks, motivating the switch. Measured on the author's own corpus with no model in the loop, so agent behaviour is untested. Single-author Zenodo preprint.
preprint# -
Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents
Fifteen open prompt-injection detectors, including Prompt Guard 2, and two LLM judges, tested on replayed AgentDojo and τ-bench tool outputs and on BIPIA. Rankings barely transfer: PIGuard catches 95.1% of BIPIA injections at 1% FPR, having trained on BIPIA inputs, but 2.1% on AgentDojo. False-positive rates on tool outputs (0 to over 90%) stay consistent across both agent benchmarks. Single-author, unreviewed preprint; Claude wrote the code, ran the experiments and drafted the text. Static attacks only, so detection figures are upper bounds.
preprint# -
Self-Evolving Defense: Continual Security Policy Learning for LLM Agents
Training-free defense in which an LLM judge scores each completed agent trajectory, distils harmful ones into a tree of retrievable security policies, and injects the closest matches into later system prompts. On DeepSeek-V4-Flash it cuts AgentDojo targeted injection from 49.0% to 0.42% (DRIFT: 3.7%) and adaptive X-Teaming from 50% to 7.8%. Memory is learned online from each benchmark's own episode stream, with the agent's base model as judge, and poisoning of that memory is not tested. Unreviewed preprint.
-
Where Do LLMs Decide to Break the Rules? Mechanistic Localization of Prompt Injection Compliance
Activation patching across five dense models (4B–32B) finds injection presence decodable from layer 1, but causal leverage over compliance only in the final third of the network. Patching there reverses 77–92% of compliant cases in a synthetic single-turn customer-service setup, about 12% on naturalistic benchmarks. In InjecAgent and AgentDojo rollouts no single-token patch beats a random control. A probe at the causal peak reaches 0.993 AUROC, and a white-box suffix halves its recall. Unreviewed preprint.
-
Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications
Self-replicating prompts that propagate between RAG-connected agents without a user clicking anything, plus a proposed guardrail. Demonstrated in a lab; no such worm has been observed operationally.
-
Data package for an audit of prompt-only defenses against prompt injection in LLM-assisted peer review: a 56,700-row experiment matrix, a held-out bank of 20 attack templates, adaptive-attacker results against three reviewer models, defense baselines including a Prompt-Guard filter-then-review pipeline, and blinded human annotation. The target is a reviewing model, not a tool-using agent. Anonymous authors; companion paper under revision.
preprint# -
Code and data for a study of prompt injection against LLM task planners in a robot-tended finish-turning cell, defended by checking planned actions against process constraints rather than filtering text. Includes a discrete-event simulation of the cell. The cell, attacks and defense are all the authors' own, in simulation. Companion manuscript under review.
preprint# -
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
Ninety-seven realistic agent tasks paired with 629 security test cases, built so attacks and defenses can be swapped independently — which is what makes it reusable rather than a one-off result. The standard reference environment for agent hijacking work.
preprint# -
Reading list built around the lethal trifecta framing: MCP CVEs, defensive tooling, documented incidents. A fork with almost no history, so treat the selection as one person's.
living# -
Technical Blog: Strengthening AI Agent Hijacking Evaluations
Hijacking success against one model rose from 11% to 81% once the red team adapted its attacks. That gap is the methodological warning for anyone quoting a single injection-resistance number, including the good ones.
-
Simon Willison: Prompt injection
Twenty-five posts, September 2022 to November 2025, by the person who named the attack class. Tracks it from first description through multi-modal variants to defensive design patterns. Has a feed.
Defensive agents 3
Agents that patch, triage or defend rather than attack.
-
NeuroGuard: An Interpretable Cyber Defense Agent Through Reinforcement Learning
Reinforcement-learning agent for network defense, designed so that its decisions can be explained. RL defenders are usually trained and scored inside one simulator, so transfer beyond it is the caveat to check; see saika-2026-cyber-sim-to-real-transfer. Springer LNCS chapter.
paywalled# -
Towards Hierarchical Cyber Defense with Large Language Models: From Planning to Execution
Frozen LLMs replace PPO policies as planner, executor or both in a hierarchical defender in ORNL's Cyberwheel, facing a scripted Atomic Red Team attacker on 15-, 100- and 1,010-host networks. Llama-3.3-70B at both levels holds compromise near 1% and impact near 0% at every scale. The per-scale-retrained RL baseline reaches 17.95% and 15.20% on the large network. An LLM planner over a PPO executor barely helps (10.75–13.05% compromise). Preprint, with one seed, 20 episodes and three actions.
-
Introducing CodeMender: an AI agent for code security
The defensive counterpart to the discovery agents: reports 72 security fixes upstreamed to open source over six months, including into a 4.5-million-line codebase. Vendor-authored, with no external audit of fix quality.
vendor-authored#
Threat reporting 11
Vendor and government reporting on observed misuse of AI systems.
-
Threat Intelligence (Anthropic)
Index of Anthropic's dated misuse reports, March 2025 onward. No feed, so it needs checking by hand.
-
AI as tradecraft: How threat actors operationalize AI
Two years on from the 2024 baseline: AI embedded across phishing, malware development and post-compromise work, while humans keep targeting decisions. Notes DPRK use of fabricated identities for employment fraud.
vendor-authored# -
Amazon Threat Intelligence observes threat actors running cybercrime campaigns with ARTEX AI agents
Amazon's MadPot decoys posed as LLM inference APIs and, from 3 October to 5 November 2026, caught requests from ARTEX agents carrying their own campaign memory. Two campaigns are described, one stealing AI provider keys from exposed gateways to fund more hunting and one building deepfakes to pass exchange identity checks, but no counts of requests, sensors, keys or targets are given. Every outcome is the agents' own claim, which Amazon did not observe and cannot verify. Vendor threat report with indicators.
vendor-authored# -
Detecting and countering misuse of AI: September 2026
Case studies of actors disrupted between December 2025 and August 2026 across seven harm categories, including cyber espionage and influence operations. Vendor-authored, and covers only misuse of one company's models.
vendor-authored# -
GTIG AI Threat Tracker: From Prompting to Autonomy - The Evolution of Adversarial AI
September 2026 tracker documenting a shift from prompt-assisted work to agentic workflows, including a cloud compromise and mass credential harvesting completed in under six hours. Names specific actors and malware families.
vendor-authored# -
What we learned mapping a year's worth of AI-enabled cyber threats
Maps 832 banned accounts from March 2025 to March 2026 onto MITRE ATT&CK, concluding AI use concentrated in post-compromise operations and that ATT&CK lacks categories for agentic orchestration.
vendor-authored# -
Adversarial Misuse of Generative AI
GTIG's January 2025 baseline on APT and information-operations use of Gemini, concluding threat actors gained speed and volume but no new capability. Useful as the deliberately deflationary counterweight to later autonomy claims.
-
Cyber threat actors: AI-assisted intrusion research
Account of terminated DPRK-linked accounts researching intrusion tooling, phishing and malware, noting the requests mostly sought publicly available information. A useful deflationary data point against autonomy narratives.
vendor-authored# -
Disrupting the first reported AI-orchestrated cyber espionage campaign
Anthropic's account of a campaign it attributes to a Chinese state-sponsored group that drove Claude Code through roughly 80-90% of an intrusion chain against about thirty targets. Vendor-authored and self-reported; no independent corroboration of the attribution.
-
First known AI-powered ransomware uncovered by ESET Research
PromptLock generates Lua payloads at runtime using a local gpt-oss-20b model. ESET states plainly it was never seen in an attack and is believed to be a proof of concept, which most coverage omitted.
-
Staying ahead of threat actors in the age of AI
The first joint Microsoft and OpenAI disclosure, naming five state-linked groups using models for reconnaissance, scripting and social engineering, and finding no new AI-enabled attack. The baseline everything since is measured against.
vendor-authored#
Social engineering at scale 1
Model-generated phishing and pretexting, and the evidence about its effectiveness.
-
Assessing AI-Generated vs. Human-Authored Spear Phishing SMS Attacks: An Empirical Study
Pilot study of 25 participants comparing GPT-4 and student-written phishing SMS, finding 28% versus 21% click intention, a difference the authors themselves call statistically uncertain. Cite it carefully.
contested#
Governance and policy 4
National guidance, standards and policy on AI-enabled cyber threat.
-
Deploying AI Systems Securely: Best Practices for Deploying Secure and Resilient AI Systems
Seven-nation joint cybersecurity information sheet: governance, hardened deployment environments, access control, continuous monitoring. Written for operators. Nothing in it addresses offensive agent capability, which is worth knowing before citing it in that context.
-
The reference vocabulary for adversarial ML: attacker goals, capabilities, mitigations. This page is the E2023 final, published January 2024. A later 2025 edition exists and I could not find a stable landing page for it.
-
OWASP Top 10 for Agentic Applications for 2026
Community risk list for autonomous agent systems. Practitioner-oriented, not empirical. The download page asks for your details.
-
Impact of AI on cyber threat from now to 2027
UK national assessment using calibrated probability language, judging that AI will almost certainly make intrusion operations more efficient and that a widening gap will separate defended from undefended estates.
Tooling 2
Scanners, harnesses and utilities for testing AI systems.
-
pikit: A Composable Toolkit for Indirect Prompt Injection Research and Evaluation
Open-source harness that composes 13 injection wordings, 16 carriers (web pages, PDF metadata, MIME headers, calendar events and others) and 9 prompt-level defenses, so each axis can be varied independently. Tested on the pi coding agent with an undisclosed model, defenses cut pooled attack success from 6.00% to 1.69% over 2,817 matched runs. The '71.8% reduction' is relative to that low baseline, on a pre-selected high-risk subset. Heuristic detectors reached at most 17.78% recall. Tencent Zhuque Lab preprint.
-
NVIDIA's LLM vulnerability scanner. Prompt injection, jailbreaks, training-data leakage, glitch tokens. Apache-2.0.
living#
Honeypots and honeynets 1
Decoy systems built to be attacked and observed.
-
LLM in the Shell: Generative Honeypots
shelLM, an LLM-backed Linux shell honeypot reporting a 0.90 true negative rate against security experts asked to tell it from a real host. The paper that started the generative-honeypot line.
LLM-backed honeypots 4
Decoys whose responses are generated by a language model rather than scripted.
-
LLM Honeypot: Leveraging Large Language Models as Advanced Interactive Honeypot Systems
Fine-tunes an open-weights model on captured attacker sessions to generate honeypot responses, then evaluates realism and deploys it live. Preprint; evaluation is thinner than shelLM's.
preprint# -
SoK: Honeypots & LLMs, More Than the Sum of Their Parts?
Systematizes both directions at once: LLMs used to build honeypots, and honeypots built for LLM attackers. Includes a taxonomy of honeypot detection vectors and a critique of how the area evaluates itself.
preprint# -
LLM-powered web honeypot: generates a plausible HTTP response to whatever arrives instead of emulating fixed applications. Go, Apache-2.0.
living# -
Low-code deception runtime, Go, GPL-3.0. The interesting part is the MCP bait tooling: decoys aimed at AI agents rather than human intruders. A commercial product sits on top.
Deception against machine attackers 9
Deception designed for automated adversaries rather than human ones.
-
AgentTrap: Stateful Feedback Deception against Autonomous Penetration Testing Agents
An LLM-driven honeypot endpoint that adapts its responses to an attacking agent and escalates to asking for the agent's own API key. Tested against eight open-source pentest agents with two models each, on a single SQL-injection target at three runs per configuration. Real-target success falls only from 95.8% to 79.2%, and keys leak in 18.8% of runs, all from CAI and PentestAgent. Other agents' models refused, or Strix's sandboxing blocked the leak. A four-page unreviewed preprint.
-
Honeyquest for LLMs: Rethinking Cyber Deception for AI Attackers
Twenty-one models against 174 reconnaissance queries, finding LLMs take deceptive bait far more often than humans, show no attention-diversion effect, and act on traps 73.4% of the time despite naming them in their reasoning.
preprint# -
Four cooperating agents divert jailbreak attempts into decoy responses, reporting a 68.77% average reduction in attack success while leaving legitimate queries intact. Preprint; the threat model is model-level, not network-level.
preprint# -
Say Shibboleth: Fingerprinting the LLMs and Harnesses Behind Agentic Attacks
Proposes honeypot services that hand an attacking agent credentials only after it names its model and writes free-text notes, plus probes that read the harness from how it masks secrets, enforces timeouts and schedules tool calls. Across about 800 lab runs it names the right model among 15 about 90% of the time and the right harness among four in 92% of runs. Those runs used containers on a private network, and agents told to give a fake name cut accuracy to 8%. Unreviewed v0.1 preprint extending pasquini-2024-mantis.
preprint# -
Stinging the Swarm: Extracting Attacker Intelligence from a Twin-Controller OT Honeypot
Describes Talos, an inline gate that screens each agent request to an industrial controller and reroutes agents it judges hostile to a mirror twin, where their attacks are mined for intelligence. Against 2,400 attacker agents in four concealment tiers of 600, no write reached the plant. The gate sent only 8% to 55% of each tier to the mirror and let 1,427 writes through, which a device-level check rejected, and every agent ran one abliterated Llama model against one simulated pump line. Unreviewed preprint with no repository link.
preprint# -
Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks
Plants adversarial text in responses an attacking agent will read, disrupting it or compromising the attacker's own machine, with over 95% reported effectiveness. The cleanest statement of deception aimed at machines rather than people.
-
Artifact for a paper testing whether adversarially crafted smart-contract bytecode misleads LLMs used as forensic analysts: frozen model responses, judge outputs, bytecode and source inputs, gold labels and analysis scripts. It inverts this directory's usual direction, with the deceived machine on the defending side; read with prinos-2026-honeyquest-llms. CCS 2026 acceptance is claimed.
-
The Mantis decoys as running code: tarpitted FTP, deliberately vulnerable web apps, weak telnet, injection payloads, reverse-shell listeners.
living# -
Short essay predicting that agents will learn to spot honeypots and dodge fingerprinting, so traps should instead test for the mix of intelligence and speed that hiding would sacrifice. It reports no new measurements; the evidence is the author's 2024 honeypot, whose emulated Cowrie shell frontier models later saw through within seconds, plus figures cited from others. The author co-wrote pasquini-2026-shibboleth, whose fingerprinting approach this post expects to lose force. Individual blog post, not peer reviewed.
Frameworks and doctrine 1
Structured references for planning and running deception programmes.
-
ATLAS tactics, techniques, mitigations and case studies as versioned YAML. Easier to cite than the client-side-rendered website.
Order effects 1
Question-order and sequence phenomena, and the QQ equality that predicts them.
-
Takes the parameter-free QQ equality from quantum cognition and applies it to model log-probabilities, finding most item pairs saturate into near-determinism and so cannot support a distribution-level test. The only direct link between these two literatures found.
preprint#
Machine belief states 1
Applying human-cognition models and bias inventories to model behaviour.
-
Cognitive Bias in High-Stakes Decision-Making with LLMs
BiasBuster, a 16,800-prompt framework for measuring and mitigating cognitive bias in model decisions. Relevant here because deception doctrine assumes exploitable biases; this is the closest thing to an inventory of them in machines.
preprint#
Surveys and taxonomies 4
The fastest way into a literature.
-
Intelligent interactive honeypots: A systematization of AI-driven cyber deception
Systematizes forty studies on AI-driven interactive honeypots, mapping interaction level to attack stage and calling out unstandardised datasets and evaluation as the field's main weaknesses. Carries a 2027 issue date.
-
A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypes
Covers both directions of agentic AI in security and names systemic risks specific to agents: collusion between them, cascading failure, oversight evasion and memory poisoning. Preprint, with working prototypes included.
preprint# -
Synthesises over 70 academic and industry sources on offensive and defensive LLM use, including AI-generated malware. Preprint and broad rather than deep; useful mainly as a bibliography.
preprint# -
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
Threat taxonomy for agents acting across digital and physical environments, with a review of evaluation methods and both technical and governance defenses. Revised through April 2026; still a preprint.
preprint#
Ongoing trackers 3
Sources that keep publishing on the topic.
-
scadastrangelove/awesome-ai-security-tools
Curated index of AI security tools covering finding triage, agent security, pentest agents, LLM-driven fuzzing, SOC triage, reverse engineering and red teaming, with a section on LLM honeypots and deception. It catalogs tools rather than measuring them, and the page gives no figures on whether listed tools work. Entries are described, not tested, so use it for discovery and judge tools elsewhere. Single-maintainer list under a CC0 license, broader than awesome-ai-agent-security.
living# -
Threat-Actors-use-of-Artificial-Intelligence
Documented threat-actor AI use mapped to ATT&CK, from Microsoft, OpenAI, Anthropic, GTIG and Kaspersky reporting. Narrower than it looks: attacks on AI systems and influence operations are excluded by design.
living# -
Weekly security newsletter, free, with steady AI-security coverage. The feed avoids the signup.
Courses 1
Structured teaching material.
-
Eight free modules with Docker labs: prompt injection, RAG exploitation, multi-agent attacks, automated red teaming.
living#
Venues 2
Where this work is published and presented.
-
DEF CON community track on adversary simulation, emulation tactics and purple teaming. The programme page links no recordings at all, so it is hard to follow remotely.
living# -
Conference on Applied Machine Learning in Information Security (CAMLIS)
Practitioner venue for applied ML in security; 2024 proceedings open through CEUR-WS. The main site was serving stale cached content when I checked, so use the proceedings link.