AI systems change the attack surface. Threat modeling them does not mean inventing new theory every time. It means adapting classical threat modeling to the unique assets, data flows, and operational cadence of ML and agentic systems. This tutorial walks you through a repeatable process you can use inside an MLOps pipeline or product security review.

1) Set the scope and stakeholders Start by naming the AI system, its purpose, and the people who care. Include product managers, ML engineers, data engineers, SREs, privacy or compliance, and a security reviewer. Define success criteria for the model and what constitutes user harm. Keep scope tight: is this a model bundle, an API, a model-inference endpoint, or a whole agentic pipeline?

2) Inventory the AI-specific assets to protect Treat these items as first-class assets in your model of the system: training data stores and sources, data pipelines and provenance metadata, model checkpoints and artifacts, inference endpoints and APIs, prompt templates, feature stores, third-party pretrained models and model zoos, compute and container images, and dashboards or telemetry that expose model behavior. Do not forget human-operated components such as labelers and external data providers; they are attack vectors too. Microsoft’s threat-modeling guidance explicitly calls out training data stores and their hosts as part of scope because data poisoning is a primary risk to ML systems.

3) Map trust boundaries and dependencies Draw a simple dataflow diagram that marks trust boundaries: where does untrusted input enter (public API, user uploads, scraped corpora)? Which services are on your internal network? Which are third-party? Mark model dependencies like third-party checkpoints or tokenizers as external trust boundaries. Assume untrusted inputs can arrive anywhere a human or third-party can write data. This reframing helps you find poisoning, supply-chain, and model-extraction risks early.

4) Use established ML threat catalogs to enumerate threats Don’t invent every attack. Use MITRE’s ATLAS to translate attacker tactics and techniques into your context, and use OWASP’s ML Top Ten to prioritize common failure modes such as poisoning, backdoors, model theft, and inference attacks. These artifacts give concrete examples you can map to your assets and controls.

5) Create concrete threat scenarios (examples)

  • Data poisoning: An external data feed used in augmentation is poisoned with mislabeled samples that cause a targeted misclassification for a high-value input.
  • Prompt injection: A plugin or web page included in a chain-of-thought contains instructions that cause a deployed LLM to leak secrets from a system prompt.
  • Model extraction: An attacker queries a hosted inference API repeatedly to train a surrogate model that replicates proprietary behavior. For each scenario document: attacker goal, attacker capabilities (local, remote, authenticated), affected assets, likely impacts, and indicators you could detect.

6) Rate risk and decide mitigations Use a simple risk matrix: likelihood vs impact. ML-specific signal increases likelihood categories where public query surfaces exist or where data comes from uncurated sources. For common mitigations consider:

  • Data provenance and lineage, immutably logged (prevent and investigate poisoning).
  • Input validation and anomaly detection on incoming samples and training batches.
  • Rate limiting, query fingerprinting, and output truncation to reduce model extraction and abuse.
  • Differential privacy or synthetic-data generation to reduce privacy leakage from training data.
  • Access control and key rotation for model artifacts and checkpoints.
  • Adversarial training, robust optimization, and certified defenses where applicable.
  • Monitor distribution shift and concept drift metrics in production and create alert thresholds. These controls map directly to the threats in MITRE ATLAS and to OWASP mitigations for ML-specific weaknesses.

7) Operationalize testing and red teaming You want continuous validation, not a one-time gate. Integrate adversarial tests into CI: fuzz inputs, simulate poisoning in a sandbox, run membership inference and extraction probes against staging models. Tools and integrations exist to help emulate adversarial behavior; Microsoft’s Counterfit and related Arsenal plug-ins connect adversarial libraries to red-team frameworks so security teams can run realistic checks without becoming ML researchers overnight. Use ATLAS case studies to design test harnesses that replicate known techniques.

8) Map to governance: use the NIST AI RMF to structure programs NIST’s AI Risk Management Framework divides activities into functions you can operationalize: govern, map, measure, and manage. Use the framework to connect your threat-model outcomes to policy and lifecycle controls. For example, put your asset inventory and provenance policy under Governance; your threat model and attack mappings under Map; detection thresholds and metrics under Measure; and incident response and mitigation playbooks under Manage. This makes threat modeling an ongoing program rather than a one-off checklist.

9) Build an incident playbook for AI-specific failures Draft runbooks for the most likely high-impact incidents: model backdoor discovered, large-scale extraction detected, or training dataset compromise. Include immediate containment actions such as revoking keys, disabling endpoints, rolling to a known-good model checkpoint, and snapshotting affected datasets for forensics. Ensure legal and privacy teams are in the loop if sensitive training data may be exposed.

10) Sample checklist you can apply now

  • Diagram dataflows and mark trust boundaries.
  • Inventory third-party models and datasets.
  • Add production monitors for distribution shift, input anomalies, and query volume.
  • Add rate limits and authentication for public inference APIs.
  • Run membership-inference and model-extraction probes in staging weekly.
  • Keep immutable metadata for dataset lineage and labeler identities.
  • Create a one-page incident playbook for model compromise and test it.

11) Practical tips from the lab

  • Start small. Threat model one high-value model or endpoint first and expand the pattern.
  • Use template scenarios to accelerate reviews, but always adapt attacker capabilities to the deployment context.
  • Treat telemetry as code. Capture the exact inputs that triggered anomalies, along with timestamps and model versions. That data is the single most useful artifact when tracing poisoning or backdoor activity.
  • Automate what you can. CI gates that run extraction and privacy probes will save weeks of manual reviews at scale.

Closing note AI threat modeling is a repeatable engineering practice, not a research paper. Use established catalogs and tools to bootstrap your scenarios, wire the results into a governance framework like NIST AI RMF, and operationalize continuous testing. The adversary landscape will evolve, but the fundamentals of asset inventory, trust boundary mapping, realistic threat scenarios, and measurable controls will keep your systems resilient.