ABRACADABRA Labs Resource directory

Topic · 89 links

Autonomous cyber attackers

AI systems and agents that find and exploit vulnerabilities, the evidence on how much of an intrusion they now run without a human, and the defensive work responding to them.

The argument has moved on from whether models can hack. Frontier models now find real vulnerabilities at scale, from DARPA's AI Cyber Challenge to restricted-access programs such as Anthropic's Project Glasswing, and threat intelligence teams describe state-linked intrusions in which the model did most of the work. The open questions are narrower and harder: how much of an operation actually runs without a human, whether faster discovery means faster exploitation, and whether defenders can use the same tools to patch what is found before attackers weaponize it. The evidence splits where the arguments do. The strongest claims about autonomy come from vendor threat reporting. Vendors hold the telemetry, but they also sell the models and the defenses, so their reports are flagged throughout. Benchmarks and exploitation data are more reproducible but measure narrower things. So far they show the number of vulnerabilities found going up, not a clear collapse in time-to-exploit. The harness around a model matters as much as the model itself: the same model can go from 3 of 40 to 37 of 40 in a different scaffold, so any capability figure describes a whole system, not a model.

Start here

Eight readings, best read in order. You'll start with what was claimed about AI attackers, then see how those claims held up when tested, and finish with what security companies say they've seen in real attacks. Each reading has a short note on why it's in that spot. The first four are also on the home page, and you'll find more reading lists onPaths.

  1. Staying ahead of threat actors in the age of AI

    The 2024 baseline: state actors using models, no novel attacks. Everything after is a claim of departure from this.

    The first joint Microsoft and OpenAI disclosure, naming five state-linked groups using models for reconnaissance, scripting and social engineering, and finding no new AI-enabled attack. The baseline everything since is measured against.

  2. LLM Agents can Autonomously Exploit One-day Vulnerabilities

    The paper that started the alarm, read with its own caveat in view: 87% with the CVE writeup, 7% without.

    Reports GPT-4 exploiting 87% of 15 one-day CVEs when handed the CVE description, dropping to 7% without it. The headline number is widely cited; the caveat that it needs the writeup is the part most citations omit.

  3. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models

    How capability actually gets measured, and why subtask decomposition matters for reading any score.

    Forty professional CTF tasks from four competitions, each decomposed into subtasks so partial progress is measurable. The benchmark most frequently cited when labs and AI safety institutes report offensive cyber capability.

  4. CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities

    The same question against real CVEs rather than CTFs. Up to 13% is the number to hold onto.

    Sandboxed reproductions of critical-severity web CVEs with automated success checks. Leading agent frameworks resolved up to 13%. That figure is the most defensible number currently available for end-to-end autonomous exploitation, and it is a long way below what the threat reporting implies.

  5. Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks

    Then see the number move when scaffolding changes: 3 of 40 becomes 37 of 40. Capability claims are about harnesses, not models.

    Inserts a high-level attack abstraction layer between the model and the network, reaching critical assets in 37 of 40 benchmark environments where baseline agents managed 3. The clearest evidence that scaffolding, not model scale, is the constraint.

  6. SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned

    The largest funded test of autonomous find-and-patch, written up by the people who ran and entered it.

    Camera-ready systematization of AIxCC by participants and organizers, covering competition design, the finalist architectures, and execution traces. The single best account of what autonomous cyber reasoning systems could and could not do by 2025.

  7. Disrupting the first reported AI-orchestrated cyber espionage campaign

    The first claimed end-to-end autonomous campaign. Read it as a vendor's account, because that is what it is.

    Anthropic's account of a campaign it attributes to a Chinese state-sponsored group that drove Claude Code through roughly 80-90% of an intrusion chain against about thirty targets. Vendor-authored and self-reported; no independent corroboration of the attribution.

  8. GTIG AI Threat Tracker: From Prompting to Autonomy - The Evolution of Adversarial AI

    A second vendor reaching similar conclusions independently, which is the closest thing to corroboration available.

    September 2026 tracker documenting a shift from prompt-assisted work to agentic workflows, including a cloud compromise and mass credential harvesting completed in under six hours. Names specific actors and malware families.

Everything in this topic

Grouped by subtopic, then by source type within each subtopic.

Agentic exploitation 2

Agents carrying an exploit chain end to end against real or reproduced targets.

Vulnerability discovery 4

Models and agents finding flaws in code rather than exploiting known ones.

Autonomous penetration testing 3

Systems that run an engagement with little or no human steering.

Multi-agent orchestration 1

Several coordinated agents running phases of an operation together.

Cyber reasoning systems 3

Automated find-patch-exploit systems, from the Cyber Grand Challenge onward.

  • SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned

    Camera-ready systematization of AIxCC by participants and organizers, covering competition design, the finalist architectures, and execution traces. The single best account of what autonomous cyber reasoning systems could and could not do by 2025.

    seminalopen version
  • AI Cyber Challenge marks pivotal inflection point for cyber defense

    DARPA's results announcement. Team Atlanta's system found 54 of 63 synthetic vulnerabilities and patched 43, across 54 million lines of code, and all seven finalist systems were released under OSI-approved licences — which is the part with lasting value, since the systems themselves are now inspectable.

  • Cyber Grand Challenge (CGC)

    The 2016 competition for machines that find, patch and exploit flaws unaided. It predates LLMs entirely — which is the reason to read it before treating agentic exploitation as new.

    datedseminalopen version

Benchmarks 7

Task suites built to measure offensive or defensive capability reproducibly.

  • BypassGym: A Benchmark for Evaluating AI Agents' Endpoint Defense Evasion Capabilities

    Benchmark in which an agent receives command-and-control shellcode and must build a loader that runs it on a real Windows endpoint without being stopped by the installed EDR product. Across 90 runs covering five lightweight models, six EDR products and three payloads, 71 succeeded (78.9%), and Windows Defender stopped none of its 15. Each combination ran once, and an injected evasion skill matters: GLM-5.3 Flash fell from 15 to 9 of 18 without it. Unreviewed preprint; code and data are promised only after review.

    preprint
  • Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models

    Forty professional CTF tasks from four competitions, each decomposed into subtasks so partial progress is measurable. The benchmark most frequently cited when labs and AI safety institutes report offensive cyber capability.

  • APTInvestBench: Evaluating Autonomous APT Investigation under Varying Telemetry

    Benchmark for LLM agents doing SOC-style APT investigation: 370 cases built from 56 report-informed attack reconstructions (16.4 million synthetic log records generated with Cisco Talos's EvidenceForge) under seven telemetry conditions. Across eleven models, agents acquire sufficient evidence for 44.3% of recoverable attack actions but cite it for only 25.0%. Endpoint-only telemetry cuts coverage by just 1.6 points while 35.5% of previously covered actions lose support. The main study uses 10 of the 56 scenarios; unreviewed preprint.

    preprint
  • CyberPersistBench: Evaluating LLM-Based Cyber Attackers on Installation and Persistence

    Starts agents from a restricted foothold and tests whether they can keep it: 203 tasks over 46 persistence mechanism families on Linux and Windows, scored at six prefix levels from writing the Artifact to surviving a reboot. Five frontier models reach 27.6–42.4% at pass@3, and most failures fall between writing a valid config and getting the host to register it. Native defenses on 128 tasks cut success to 5.5–13.3%, not to zero. The 44.8% scaffold figure excludes tasks refused on safety grounds. Preprint.

    preprint
  • CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities

    Sandboxed reproductions of critical-severity web CVEs with automated success checks. Leading agent frameworks resolved up to 13%. That figure is the most defensible number currently available for end-to-end autonomous exploitation, and it is a long way below what the threat reporting implies.

    open version
  • Cybench

    Task descriptions across six CTF categories, run instructions, evaluation logs, a frontier-model leaderboard. Start here rather than the paper if you mean to run it.

    livingopen version
  • agmi: Agent Memory Integrity, a conformance test suite for tamper evidence in AI agent memory and checkpoint stores

    Conformance suite testing whether AI agent memory and checkpoint stores reveal tampering. Release 0.6.3 runs nine storage-level edits against sixteen stores, eight of which claim tamper evidence, with a control confirming each edit changed only what was intended. The threat model is an attacker with write access to storage, not memory poisoned through the agent's own writes. Single-author Zenodo release, not peer reviewed.

    living

Capability evaluation 4

Methodology for eliciting and measuring what a model can actually do.

  • No One Architecture Fits All: A Cross-Environment Evaluation of Hierarchical Red Team Agents

    All-RL and all-LLM planner–executor red agents against expert RL defenders in CybORG CAGE-4 and Cyberwheel at 100 and 1010 hosts (18 configurations, 3,600 episodes). RL wins CAGE-4 (78.5% vs 18.0%) and Cyberwheel-100 (81.0% vs 50.5%) but scores 0.0% at 1010 hosts, where Foundation-Sec-8B reaches 55.0%. Every configuration compromises a host; the failures come at escalation or root-to-impact. LLM results depend on the model (DeepHat scores 0% everywhere), and on CAGE-4 the RL planner proves not load-bearing. Simulator-only preprint.

    contestedpreprint
  • Threat-Preserving Representation Sensitivity in Agent-Security Benchmarks

    Across 28,904 runs on ASB, MCPTox and AgentDojo, renaming tools while holding task, harmful action and scorer fixed shifts attack success. Neutral names on ASB's attacker tools raise committed ASR by 11.67–13.21 points; threat-explicit names on MCPTox lower it by 11.00 (GPT-5-mini) and 4.11 (Haiku 4.5); AgentDojo barely moves (−0.50). A form-matched neutral name reproduces 8.54 of MCPTox's 11 points, so threat vocabulary is not the main driver. Unreviewed preprint covering three small models.

    preprint
  • UKGovernmentBEIS/inspect_ai

    The UK AI Security Institute's evaluation framework. CVE-Bench and several other cyber evals are written against it. MIT.

    livingopen version
  • METR

    Nonprofit evaluating frontier models for autonomous capability. Not cyber-specific, but its time-horizon work underpins most autonomy forecasting.

    livingopen version

Simulation environments 3

Synthetic networks for training and studying automated agents.

  • Crossing the Cyber Divide: Sim-to-Sim and Sim-to-Real Transfer for RL Agents

    A PPO attacker trained in CyberWheel is transferred zero-shot to NetSecGame, CyberBattleSim and NASim through a kill-chain action/observation wrapper and a domain-adversarial encoder. On NetSecGame the encoder reaches 45.2% wins, no better than the wrapper alone (47.4%). On one CyberBattleSim chain topology it wins all 20 episodes where the wrapper wins none. The sim-to-real claim rests only on action-type divergence from an emulator policy (JS 0.085); actual emulation runs are deferred.

    contested
  • cage-challenge/CybORG

    The environment behind the multinational CAGE Challenges for autonomous cyber defense agents. MIT. Challenge 4 adds multi-agent RL.

    livingopen version
  • microsoft/CyberBattleSim

    Abstract simulated network for agents moving laterally, built for reinforcement learning rather than realism. MIT. The abstraction is deliberate and it does limit transfer.

    livingopen version

Prompt injection and agent hijacking 14

Turning an agent's inputs against it, as both an attack and a defence.

  • CounterSteer: Suppressing Indirect Prompt Injection with Activation Steering

    Fits a per-model residual-stream direction and subtracts it from every tool-result token at prefill, with no detector or fine-tuning. Across five open-weights models, AgentDojo compromise falls from 0.10–0.49 to 0.006–0.079 at 93–100% benign utility. An adaptive AutoDojo attacker is held to about a quarter of its undefended success. Parameter manipulation survives: 13 of 18 samples cracked under framing search. The 0/2052 LLMail-Inject result replays attacks tuned against other models. Requires white-box serving; single-author Microsoft preprint.

    preprintvendor-authored
  • Divide and Inject: Can Agents Reconstruct an Indirect Prompt Injection from Fragments?

    AdaLCPI splits an injected instruction into two incomplete fragments buried in filler, plus a generic reconstruction cue, and evolves them with OpenEvolve against the target agent. Across seven models and 81 Email, GitHub and Slack task–objective pairs, 8k-token filler reaches 61.4% macro attack success against 32.8% for a matched explicit-instruction baseline. Fixed fragmentation scores 5.1%, and fragments without filler score 0%. Gains are uneven: GPT-5.6-Luna stays at 8.6%, and TAP does better on Ministral. Each search takes up to 129 black-box runs, and a GPT-4o judge decides success. Preprint.

    preprint
  • From A2A Attacks to Envelope-Layer Defense: Red-Teaming Evaluation of LLM Agents and a Three-Layer Isomorphic Attack-Defense Model

    A2A-TIBA chains indirect prompt injection with a persistent callback program delivered over the agent-to-agent protocol, so that after one implant subsequent commands bypass the agent entirely. Tested across 15 front-end x back-end combinations (900 attacks) with a four-class metric separating LLM refusal from agent-layer interception; penetration ran 12% for hermes to 45% for Claude Code. An ablation argues untrusted-source label headers on A2A, tool and memory channels sharply raise refusal. Single-author preprint submitted to IEEE TDSC; testbed only, decoy files, China-market back-ends.

    preprint
  • Gating Tool Calls by Where Their Arguments Came From: A Prompt-Injection defense Measured Without a Model

    Prompt-injection defense that gates each tool call on where its arguments came from, rather than trying to recognise malicious text. The author reports that a text-matching rule layer catches 19.8% of a corpus of real attacks, motivating the switch. Measured on the author's own corpus with no model in the loop, so agent behaviour is untested. Single-author Zenodo preprint.

    preprint
  • Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents

    Fifteen open prompt-injection detectors, including Prompt Guard 2, and two LLM judges, tested on replayed AgentDojo and τ-bench tool outputs and on BIPIA. Rankings barely transfer: PIGuard catches 95.1% of BIPIA injections at 1% FPR, having trained on BIPIA inputs, but 2.1% on AgentDojo. False-positive rates on tool outputs (0 to over 90%) stay consistent across both agent benchmarks. Single-author, unreviewed preprint; Claude wrote the code, ran the experiments and drafted the text. Static attacks only, so detection figures are upper bounds.

    preprint
  • Self-Evolving Defense: Continual Security Policy Learning for LLM Agents

    Training-free defense in which an LLM judge scores each completed agent trajectory, distils harmful ones into a tree of retrievable security policies, and injects the closest matches into later system prompts. On DeepSeek-V4-Flash it cuts AgentDojo targeted injection from 49.0% to 0.42% (DRIFT: 3.7%) and adaptive X-Teaming from 50% to 7.8%. Memory is learned online from each benchmark's own episode stream, with the agent's base model as judge, and poisoning of that memory is not tested. Unreviewed preprint.

    contestedpreprint
  • Where Do LLMs Decide to Break the Rules? Mechanistic Localization of Prompt Injection Compliance

    Activation patching across five dense models (4B–32B) finds injection presence decodable from layer 1, but causal leverage over compliance only in the final third of the network. Patching there reverses 77–92% of compliant cases in a synthetic single-turn customer-service setup, about 12% on naturalistic benchmarks. In InjecAgent and AgentDojo rollouts no single-token patch beats a random control. A probe at the causal peak reaches 0.993 AUROC, and a white-box suffix halves its recall. Unreviewed preprint.

    contestedpreprint
  • Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications

    Self-replicating prompts that propagate between RAG-connected agents without a user clicking anything, plus a proposed guardrail. Demonstrated in a lab; no such worm has been observed operationally.

    contestedpreprintopen version
  • Data and materials for: Auditing prompt-only defenses against prompt injection in LLM-assisted scientific review

    Data package for an audit of prompt-only defenses against prompt injection in LLM-assisted peer review: a 56,700-row experiment matrix, a held-out bank of 20 attack templates, adaptive-attacker results against three reviewer models, defense baselines including a Prompt-Guard filter-then-review pipeline, and blinded human annotation. The target is a reviewing model, not a tool-using agent. Anonymous authors; companion paper under revision.

    preprint
  • process-aware-verification-mfg: code and data for Process-Aware Verification Against Prompt Injection in Manufacturing Cells

    Code and data for a study of prompt injection against LLM task planners in a robot-tended finish-turning cell, defended by checking planned actions against process constraints rather than filtering text. Includes a discrete-event simulation of the cell. The cell, attacks and defense are all the authors' own, in simulation. Companion manuscript under review.

    preprint
  • AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

    Ninety-seven realistic agent tasks paired with 629 security test cases, built so attacks and defenses can be swapped independently — which is what makes it reusable rather than a one-off result. The standard reference environment for agent hijacking work.

    preprint
  • awesome-ai-agent-security

    Reading list built around the lethal trifecta framing: MCP CVEs, defensive tooling, documented incidents. A fork with almost no history, so treat the selection as one person's.

    living
  • Technical Blog: Strengthening AI Agent Hijacking Evaluations

    Hijacking success against one model rose from 11% to 81% once the red team adapted its attacks. That gap is the methodological warning for anyone quoting a single injection-resistance number, including the good ones.

  • Simon Willison: Prompt injection

    Twenty-five posts, September 2022 to November 2025, by the person who named the attack class. Tracks it from first description through multi-modal variants to defensive design patterns. Has a feed.

    livingfeed

Defensive agents 3

Agents that patch, triage or defend rather than attack.

  • NeuroGuard: An Interpretable Cyber Defense Agent Through Reinforcement Learning

    Reinforcement-learning agent for network defense, designed so that its decisions can be explained. RL defenders are usually trained and scored inside one simulator, so transfer beyond it is the caveat to check; see saika-2026-cyber-sim-to-real-transfer. Springer LNCS chapter.

    paywalled
  • Towards Hierarchical Cyber Defense with Large Language Models: From Planning to Execution

    Frozen LLMs replace PPO policies as planner, executor or both in a hierarchical defender in ORNL's Cyberwheel, facing a scripted Atomic Red Team attacker on 15-, 100- and 1,010-host networks. Llama-3.3-70B at both levels holds compromise near 1% and impact near 0% at every scale. The per-scale-retrained RL baseline reaches 17.95% and 15.20% on the large network. An LLM planner over a PPO executor barely helps (10.75–13.05% compromise). Preprint, with one seed, 20 episodes and three actions.

    contestedpreprint
  • Introducing CodeMender: an AI agent for code security

    The defensive counterpart to the discovery agents: reports 72 security fixes upstreamed to open source over six months, including into a 4.5-million-line codebase. Vendor-authored, with no external audit of fix quality.

    vendor-authored

Threat reporting 11

Vendor and government reporting on observed misuse of AI systems.

  • Threat Intelligence (Anthropic)

    Index of Anthropic's dated misuse reports, March 2025 onward. No feed, so it needs checking by hand.

    livingvendor-authored
  • AI as tradecraft: How threat actors operationalize AI

    Two years on from the 2024 baseline: AI embedded across phishing, malware development and post-compromise work, while humans keep targeting decisions. Notes DPRK use of fabricated identities for employment fraud.

    vendor-authored
  • Amazon Threat Intelligence observes threat actors running cybercrime campaigns with ARTEX AI agents

    Amazon's MadPot decoys posed as LLM inference APIs and, from 3 October to 5 November 2026, caught requests from ARTEX agents carrying their own campaign memory. Two campaigns are described, one stealing AI provider keys from exposed gateways to fund more hunting and one building deepfakes to pass exchange identity checks, but no counts of requests, sensors, keys or targets are given. Every outcome is the agents' own claim, which Amazon did not observe and cannot verify. Vendor threat report with indicators.

    vendor-authored
  • Detecting and countering misuse of AI: September 2026

    Case studies of actors disrupted between December 2025 and August 2026 across seven harm categories, including cyber espionage and influence operations. Vendor-authored, and covers only misuse of one company's models.

    vendor-authored
  • GTIG AI Threat Tracker: From Prompting to Autonomy - The Evolution of Adversarial AI

    September 2026 tracker documenting a shift from prompt-assisted work to agentic workflows, including a cloud compromise and mass credential harvesting completed in under six hours. Names specific actors and malware families.

    vendor-authored
  • What we learned mapping a year's worth of AI-enabled cyber threats

    Maps 832 banned accounts from March 2025 to March 2026 onto MITRE ATT&CK, concluding AI use concentrated in post-compromise operations and that ATT&CK lacks categories for agentic orchestration.

    vendor-authored
  • Adversarial Misuse of Generative AI

    GTIG's January 2025 baseline on APT and information-operations use of Gemini, concluding threat actors gained speed and volume but no new capability. Useful as the deliberately deflationary counterweight to later autonomy claims.

    vendor-authoredopen version
  • Cyber threat actors: AI-assisted intrusion research

    Account of terminated DPRK-linked accounts researching intrusion tooling, phishing and malware, noting the requests mostly sought publicly available information. A useful deflationary data point against autonomy narratives.

    vendor-authored
  • Disrupting the first reported AI-orchestrated cyber espionage campaign

    Anthropic's account of a campaign it attributes to a Chinese state-sponsored group that drove Claude Code through roughly 80-90% of an intrusion chain against about thirty targets. Vendor-authored and self-reported; no independent corroboration of the attribution.

    contestedvendor-authoredopen version
  • First known AI-powered ransomware uncovered by ESET Research

    PromptLock generates Lua payloads at runtime using a local gpt-oss-20b model. ESET states plainly it was never seen in an attack and is believed to be a proof of concept, which most coverage omitted.

    contestedvendor-authored
  • Staying ahead of threat actors in the age of AI

    The first joint Microsoft and OpenAI disclosure, naming five state-linked groups using models for reconnaissance, scripting and social engineering, and finding no new AI-enabled attack. The baseline everything since is measured against.

    vendor-authored

Social engineering at scale 1

Model-generated phishing and pretexting, and the evidence about its effectiveness.

Governance and policy 4

National guidance, standards and policy on AI-enabled cyber threat.

Tooling 2

Scanners, harnesses and utilities for testing AI systems.

  • pikit: A Composable Toolkit for Indirect Prompt Injection Research and Evaluation

    Open-source harness that composes 13 injection wordings, 16 carriers (web pages, PDF metadata, MIME headers, calendar events and others) and 9 prompt-level defenses, so each axis can be varied independently. Tested on the pi coding agent with an undisclosed model, defenses cut pooled attack success from 6.00% to 1.69% over 2,817 matched runs. The '71.8% reduction' is relative to that low baseline, on a pre-selected high-risk subset. Heuristic detectors reached at most 17.78% recall. Tencent Zhuque Lab preprint.

    contestedpreprintvendor-authored
  • NVIDIA/garak

    NVIDIA's LLM vulnerability scanner. Prompt injection, jailbreaks, training-data leakage, glitch tokens. Apache-2.0.

    living

Honeypots and honeynets 1

Decoy systems built to be attacked and observed.

  • LLM in the Shell: Generative Honeypots

    shelLM, an LLM-backed Linux shell honeypot reporting a 0.90 true negative rate against security experts asked to tell it from a real host. The paper that started the generative-honeypot line.

LLM-backed honeypots 4

Decoys whose responses are generated by a language model rather than scripted.

  • LLM Honeypot: Leveraging Large Language Models as Advanced Interactive Honeypot Systems

    Fine-tunes an open-weights model on captured attacker sessions to generate honeypot responses, then evaluates realism and deploys it live. Preprint; evaluation is thinner than shelLM's.

    preprint
  • SoK: Honeypots & LLMs, More Than the Sum of Their Parts?

    Systematizes both directions at once: LLMs used to build honeypots, and honeypots built for LLM attackers. Includes a taxonomy of honeypot detection vectors and a critique of how the area evaluates itself.

    preprint
  • 0x4D31/galah

    LLM-powered web honeypot: generates a plausible HTTP response to whatever arrives instead of emulating fixed applications. Go, Apache-2.0.

    living
  • beelzebub-labs/beelzebub

    Low-code deception runtime, Go, GPL-3.0. The interesting part is the MCP bait tooling: decoys aimed at AI agents rather than human intruders. A commercial product sits on top.

    livingvendor-authoredopen version

Deception against machine attackers 9

Deception designed for automated adversaries rather than human ones.

  • AgentTrap: Stateful Feedback Deception against Autonomous Penetration Testing Agents

    An LLM-driven honeypot endpoint that adapts its responses to an attacking agent and escalates to asking for the agent's own API key. Tested against eight open-source pentest agents with two models each, on a single SQL-injection target at three runs per configuration. Real-target success falls only from 95.8% to 79.2%, and keys leak in 18.8% of runs, all from CAI and PentestAgent. Other agents' models refused, or Strix's sandboxing blocked the leak. A four-page unreviewed preprint.

    contestedpreprint
  • Honeyquest for LLMs: Rethinking Cyber Deception for AI Attackers

    Twenty-one models against 174 reconnaissance queries, finding LLMs take deceptive bait far more often than humans, show no attention-diversion effect, and act on traps 73.4% of the time despite naming them in their reasoning.

    preprint
  • HoneyTrap: Deceiving Large Language Model Attackers to Honeypot Traps with Resilient Multi-Agent Defense

    Four cooperating agents divert jailbreak attempts into decoy responses, reporting a 68.77% average reduction in attack success while leaving legitimate queries intact. Preprint; the threat model is model-level, not network-level.

    preprint
  • Say Shibboleth: Fingerprinting the LLMs and Harnesses Behind Agentic Attacks

    Proposes honeypot services that hand an attacking agent credentials only after it names its model and writes free-text notes, plus probes that read the harness from how it masks secrets, enforces timeouts and schedules tool calls. Across about 800 lab runs it names the right model among 15 about 90% of the time and the right harness among four in 92% of runs. Those runs used containers on a private network, and agents told to give a fake name cut accuracy to 8%. Unreviewed v0.1 preprint extending pasquini-2024-mantis.

    preprint
  • Stinging the Swarm: Extracting Attacker Intelligence from a Twin-Controller OT Honeypot

    Describes Talos, an inline gate that screens each agent request to an industrial controller and reroutes agents it judges hostile to a mirror twin, where their attacks are mined for intelligence. Against 2,400 attacker agents in four concealment tiers of 600, no write reached the plant. The gate sent only 8% to 55% of each tier to the mirror and let 1,427 writes through, which a device-level check rejected, and every agent ran one abliterated Llama model against one simulated pump line. Unreviewed preprint with no repository link.

    preprint
  • Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks

    Plants adversarial text in responses an attacking agent will read, disrupting it or compromising the attacker's own machine, with over 95% reported effectiveness. The cleanest statement of deception aimed at machines rather than people.

    preprintseminalopen version
  • The Deception Delta: Artifact for Adversarial Evaluation of LLM-Based Smart Contract Bytecode Forensics

    Artifact for a paper testing whether adversarially crafted smart-contract bytecode misleads LLMs used as forensic analysts: frozen model responses, judge outputs, bytecode and source inputs, gold labels and analysis scripts. It inverts this directory's usual direction, with the deceived machine on the defending side; read with prinos-2026-honeyquest-llms. CCS 2026 acceptance is claimed.

  • pasquini-dario/project_mantis

    The Mantis decoys as running code: tarpitted FTP, deliberately vulnerable web apps, weak telnet, injection payloads, reverse-shell listeners.

    living
  • Future of AI Honeypots

    Short essay predicting that agents will learn to spot honeypots and dodge fingerprinting, so traps should instead test for the mix of intelligence and speed that hiding would sacrifice. It reports no new measurements; the evidence is the author's 2024 honeypot, whose emulated Cowrie shell frontier models later saw through within seconds, plus figures cited from others. The author co-wrote pasquini-2026-shibboleth, whose fingerprinting approach this post expects to lose force. Individual blog post, not peer reviewed.

Frameworks and doctrine 1

Structured references for planning and running deception programmes.

  • mitre-atlas/atlas-data

    ATLAS tactics, techniques, mitigations and case studies as versioned YAML. Easier to cite than the client-side-rendered website.

    livingopen version

Order effects 1

Question-order and sequence phenomena, and the QQ equality that predicts them.

Machine belief states 1

Applying human-cognition models and bias inventories to model behaviour.

  • Cognitive Bias in High-Stakes Decision-Making with LLMs

    BiasBuster, a 16,800-prompt framework for measuring and mitigating cognitive bias in model decisions. Relevant here because deception doctrine assumes exploitable biases; this is the closest thing to an inventory of them in machines.

    preprint

Surveys and taxonomies 4

The fastest way into a literature.

Ongoing trackers 3

Sources that keep publishing on the topic.

  • scadastrangelove/awesome-ai-security-tools

    Curated index of AI security tools covering finding triage, agent security, pentest agents, LLM-driven fuzzing, SOC triage, reverse engineering and red teaming, with a section on LLM honeypots and deception. It catalogs tools rather than measuring them, and the page gives no figures on whether listed tools work. Entries are described, not tested, so use it for discovery and judge tools elsewhere. Single-maintainer list under a CC0 license, broader than awesome-ai-agent-security.

    living
  • Threat-Actors-use-of-Artificial-Intelligence

    Documented threat-actor AI use mapped to ATT&CK, from Microsoft, OpenAI, Anthropic, GTIG and Kaspersky reporting. Narrower than it looks: attacks on AI systems and influence operations are excluded by design.

    living
  • tl;dr sec

    Weekly security newsletter, free, with steady AI-security coverage. The feed avoids the signup.

    livingfeed

Courses 1

Structured teaching material.

  • AIRT — AI Red Team Academy

    Eight free modules with Docker labs: prompt injection, RAG exploitation, multi-agent attacks, automated red teaming.

    living

Venues 2

Where this work is published and presented.