AI security checklist for production

An AI security checklist is a list of checks a team runs before an LLM application goes to production, covering how it can be attacked, how it can be wrong, what it remembers and how it behaves under load.

Last updated: 09 Oct, 2026 · Python 3.12

Horizontal pod autoscaling (HPA) ended with a deployment that looked scaled and was not. Most production problems of LLM applications have that shape: a control exists on paper and nobody checked that it works. A checklist turns each control into one question with a yes or no answer.

Four groups of checks

An agent in production has four kinds of risk: it can be attacked, it can be wrong, it forgets or remembers the wrong thing, and it falls over under load. The checks below are grouped the same way.

Guardrails: it can be attacked

Evaluation: it can be wrong

Memory: it forgets, or remembers the wrong thing

Operations: it falls over under load

Scoring a configuration against the checklist

A checklist that lives in a document is read once. One that lives in code runs on every release. The example keeps a few of the checks above as small functions over a configuration dictionary and prints what fails.

A check is a name and a rule

Each check is a group, a sentence, and a function that takes the configuration and returns true or false. Checks with a number in them carry their own threshold.

CHECKS = [
    ("Operations", "traffic is encrypted (TLS)", lambda c: c["tls"]),
    ("Operations", "failure share at peak load at most 1%",
     lambda c: c["failure_share_at_peak"] <= 0.01),
]

In the sample configuration the six operations values describe the deployment in the video: no credentials on the public API, no TLS, a 6.0 GiB memory request against 3.12 GiB in use, no node autoscaler, and 186 failed requests of 629 at 50 users. The other values are an invented starting point. The thresholds (20 goldens, 5% false positives, 1.5 times the measured memory, 1% failures) are this example's; set your own.

CHECKS = [
    ("Guardrails", "input check before the model", lambda c: c["input_check"]),
    ("Guardrails", "output check after the model", lambda c: c["output_check"]),
    ("Guardrails", "indirect injection tested on retrieved text", lambda c: c["indirect_injection_tested"]),
    ("Guardrails", "false-positive rate measured, at most 5%",
     lambda c: c["false_positive_rate"] is not None and c["false_positive_rate"] <= 0.05),
    ("Evaluation", "at least 20 goldens", lambda c: c["goldens"] >= 20),
    ("Evaluation", "faithfulness at or above its threshold", lambda c: c["faithfulness"] >= c["faithfulness_threshold"]),
    ("Evaluation", "an eval gate runs in CI", lambda c: c["eval_gate_in_ci"]),
    ("Memory", "history is bounded", lambda c: c["history_limit_tokens"] is not None),
    ("Memory", "long-term memory is separated per user", lambda c: c["memory_per_user"]),
    ("Memory", "a check runs before a memory is saved", lambda c: c["memory_write_check"]),
    ("Operations", "the public API asks for credentials", lambda c: c["public_api_requires_auth"]),
    ("Operations", "traffic is encrypted (TLS)", lambda c: c["tls"]),
    ("Operations", "memory request within 1.5x of measured use",
     lambda c: c["memory_request_gib"] <= 1.5 * c["memory_used_gib"]),
    ("Operations", "something adds nodes when pods do not fit", lambda c: c["node_autoscaler"]),
    ("Operations", "failure share at peak load at most 1%", lambda c: c["failure_share_at_peak"] <= 0.01),
]

config = {
    "input_check": True, "output_check": True, "indirect_injection_tested": False, "false_positive_rate": None,
    "goldens": 5, "faithfulness": 0.75, "faithfulness_threshold": 0.70, "eval_gate_in_ci": True,
    "history_limit_tokens": 2000, "memory_per_user": True, "memory_write_check": False,
    # the last six values describe the deployment in the video
    "public_api_requires_auth": False, "tls": False, "memory_request_gib": 6.0, "memory_used_gib": 3.12,
    "node_autoscaler": False, "failure_share_at_peak": 186 / 629,
}

failed, groups = [], {}
for group, text, rule in CHECKS:
    passed = bool(rule(config))
    done, total = groups.get(group, (0, 0))
    groups[group] = (done + passed, total + 1)
    if not passed:
        failed.append(f"{group}: {text}")

for group, (done, total) in groups.items():
    print(f"{group:11} {done} of {total}")
print(f"score: {sum(d for d, _ in groups.values())} of {len(CHECKS)}")
print("ship:", "yes" if not failed else "no, fix these first")
for line in failed:
    print("  -", line)

Guardrails  2 of 4
Evaluation  2 of 3
Memory      2 of 3
Operations  0 of 5
score: 6 of 15
ship: no, fix these first
  - Guardrails: indirect injection tested on retrieved text
  - Guardrails: false-positive rate measured, at most 5%
  - Evaluation: at least 20 goldens
  - Memory: a check runs before a memory is saved
  - Operations: the public API asks for credentials
  - Operations: traffic is encrypted (TLS)
  - Operations: memory request within 1.5x of measured use
  - Operations: something adds nodes when pods do not fit
  - Operations: failure share at peak load at most 1%

What the score says

• 6 of 15 checks pass, and the answer to "ship?" is no.
• Operations scores 0 of 5. Each of the five comes from a reading of the video's screen or files: an open API, plain HTTP, a request almost twice the measured use, no way to add a node, and a failure share of about 30% at the peak.
• The other groups each miss one or two. Indirect injection was never tested, the false-positive rate was never measured (None fails the check, it does not pass it), there are 5 goldens where the rule asks for 20, and nothing checks a memory before it is saved.
• A missing measurement is a failure. The rule for the false-positive rate tests for None first. A check that passes when the number is absent rewards not measuring.

What this checklist leaves out

Before launch vs after launch

Where you use an AI security checklist

• As a release gate, in CI, next to the deterministic evaluation tests.
• In a design review, before the first line of code, to decide which checks apply.
• After an incident, to add the check that would have caught it.

Related

• Previous: Horizontal pod autoscaling (HPA)
• Reference: OWASP Top 10 for LLM Applications

Start again from the overview: AI security