Idempotency Is the Thing AI-Generated Infrastructure Code Keeps Getting Wrong

Idempotent means running an operation twice produces the same result as running it once. It’s a simple definition and an easy one to nod along to, which is probably why it’s the property I most consistently find missing from AI-generated infrastructure scripts — not because the agent doesn’t “know” what idempotency means if you ask it directly, but because the property doesn’t show up as a bug on a first run. It shows up on a second run, on a retry, on a re-deploy of a script that already ran successfully once. By the time it matters, the code already looks done.

Why this is specifically an AI-generation problem

It isn’t, really — humans write non-idempotent infra scripts constantly. What’s specific to AI-generated code is the volume and the plausibility. An agent asked to “write a script that creates an S3 bucket and sets this policy” will, most of the time, produce something that reads cleanly and runs correctly on a clean account. The failure mode isn’t bad code, it’s code optimized to satisfy the prompt on a first pass, from a model with no persistent memory of what state the target system is already in. Ask for the script again in a different session and you may not get the same defensive checks twice, because there’s no institutional memory carrying “last time this bit someone” forward the way there is on a team that’s been burned by it before.

What to actually check

Four checks, in order of how often I find something:

1. Is it create-or-fail, or check-then-act?

# not idempotent — errors on the second run
def create_bucket(client, name):
    client.create_bucket(Bucket=name)

# idempotent — safe to run any number of times
def create_bucket(client, name):
    existing = client.list_buckets()["Buckets"]
    if any(b["Name"] == name for b in existing):
        return name
    client.create_bucket(Bucket=name)
    return name

This is the most common gap by far, and it’s the easiest to miss on review because the non-idempotent version is shorter and reads more cleanly. There’s a real tension here: the defensive version is more code, and more code looks less elegant in a diff, even though it’s the correct version.

2. Insert, or upsert?

Database seed scripts and config-sync scripts are the repeat offenders. INSERT fails or duplicates on a second run; INSERT ... ON CONFLICT DO UPDATE (or the equivalent upsert for whatever’s being written to) doesn’t. If a script’s job is “make sure this row exists with these values,” it should be written as exactly that, not as “add this row.”

3. Does retrying a partial failure make things worse?

This is the subtler one. A script that creates three resources in sequence and dies after the second one, then gets re-run from the top, needs the first two steps to be no-ops the second time — otherwise “retry” and “duplicate the first two resources” are the same action. Every step in a multi-step provisioning script needs the check from #1 individually, not just the script as a whole.

4. Is “success” actually verified, or just assumed from a lack of error?

An API call that returns 200 doesn’t always mean the thing you asked for is now true — some APIs accept an async request and return success before the operation completes. A script that moves on immediately, rather than polling for the resource to actually reach the expected state, will look correct on a well-timed test run and fail intermittently in real usage, in a way that’s genuinely hard to reproduce.

The test that actually catches this

Reading the code for these four patterns catches most of it, but the check that catches everything else is mechanical and takes thirty seconds: run the script, then run it again immediately, against the same target, and diff the resulting state. A truly idempotent script produces an empty diff on the second run — nothing changed, because everything it would have changed was already correct. This is a better test than “did it error,” because plenty of non-idempotent scripts don’t error on a second run, they just quietly create a duplicate or overwrite something they shouldn’t have touched.

I run this check on every AI-generated infra script before it goes anywhere near a shared environment now, the same way I’d run a test suite before merging application code. It’s not a trust issue with the tool — it’s that idempotency is a property about state over time, and nothing about reading a diff once tells you how the code behaves the second time it runs.