Recently I took over a project someone dumped on me. I pulled it down with git log and paused for a second — 9 commits, all authored by the same AI Dev <ai-dev@local>, not a single human commit. I won’t paste hashes for the commits I mention below; I’ll just refer to them by what they did. The repo has no CLAUDE.md, no AGENTS.md, no .claude/, no .cursor/, no CI, lint is just tsc --noEmit, and tests are nonexistent. I spent a day going through it and jotted down what it did well and what it botched.

First, what this project is: a pure client-side React SPA, backend is Express packaged into a single Vercel Function entry, api/index.ts plus six modules under api/lib/, database is Neon Postgres, object storage via an S3-compatible API. That’s all it is — not complicated.

First, what it did well

Some commits included verifiable validation results

4 of the 9 commits ended with validation results, and they carried checkable numbers.

The one wiring up object storage ended with “local end-to-end smoke passed: presign → PUT → public GET returns a 1×1 PNG” — a 1×1 PNG means it really ran the whole chain~ The one doing the existing-data migration ended with “done: 19 objects / 1.77 MB. DB has 0 data: leftovers” — how much was migrated and how much remained, both pinned down.

Of course, not every commit was like that. The validation section of the go-live acceptance commit was just one line: “Production end-to-end test passed.” — about the same as writing nothing; the deployment docs commit had a completely empty body, just a title. So don’t get too excited the moment you see a “three-part commit”…

That said, the ones with numbers were honestly better than what I write by hand…

It wrote its own ops manual, and knew the limits of its own permissions

I can’t see anywhere in the repo that anyone asked it to write docs — it just produced a DEPLOY.md on its own, and it even covered pure infrastructure knowledge in there. For example, in the section on attaching a custom domain, it specifically flagged a warning: the CDN Proxy must be turned off and use grey cloud, because enabling the proxy prevents Vercel’s SSL certificate from getting the ACME challenge. That has nothing to do with business code — it’s pure ops experience…

At the end of the doc it also maintained a 5-item roadmap checklist of todos, which later commits kept revising — not a one-shot artifact left to rot.

Along with that came a sense of boundaries: the object storage commit explicitly noted at the end that the custom domain needed a human to attach it manually in the console, because the token it held lacked the corresponding zone permission; the next commit recorded the state after the human finished. Knowing what it can’t do and saying so honestly beats stubbornly trying anyway…

It proactively subtracts

This one impressed me the most. It had earlier wired up a Redis service based on the tech stack I listed, then later on self-review found zero imports in the code, so it rolled back: removed the dependency, cleaned the related text out of the README and deployment docs. The closing line of that commit was: “When Redis is actually needed later, just reopen the db + add the env back, 5 minutes’ work.”

The same message also claimed it deleted envs across three environments and removed the remote Upstash instance via the management API — I can’t verify that part from the diff, since the diff only touched 5 files (.env.example, DEPLOY.md, README.md, package.json, lockfile).

“Cleaning up thoroughly” is usually what AI is worst at; most of the time it only adds, and even when it adds wrong it doesn’t dare touch it. This time at least it cleaned up the code half properly…

It chose to build a compatibility layer instead of touching call sites

When swapping the database, the frontend had twenty-plus components using the old SDK’s .from() chained calls. It didn’t go change those twenty-plus call sites; instead it wrote an interface-compatible shim that routed requests to the self-hosted gateway, with zero changes on the caller side. There’s not much technical depth to this trick — you’d think of it after a bit of thought — but if I’d done it myself, I’d probably have dived headfirst into editing files one by one…

It also wrote whitelist validation for the gateway, and the comment was pretty amusing, verbatim:

*   - A small allowlist of (table, op) tuples is required — anything else is
*     rejected. This keeps the proxy from turning into a "drop my whole DB"
*     button.

Now what it botched

All green locally, dead in production

tsconfig.json enabled allowImportingTsExtensions with noEmit, so every import in the source was written with the .ts suffix, and it ran without errors locally no matter what.

But Vercel compiled the entry into .js, and from './lib/sb-shim.ts' couldn’t find the corresponding file in production at all — straight to ERR_MODULE_NOT_FOUND, and that’s how production went down.

The fix commit itself stated the reason: the standard ESM TS approach is to reference the .js extension, and both tsx and tsc can transparently resolve .js to the corresponding .ts source file. This time, 7 imports across 4 files were fixed (api/index.ts accounts for 4, while sb-shim.ts/auth.ts/db-proxy.ts each account for 1). Incidentally, .vercelignore was only created this time around — before that, deployment didn’t have this file at all…

Running locally doesn’t mean it’s ready to ship, especially when the shape of the build output changes…

Hallucinations in commit messages

The commit message for the database switch said “vercel.json: cleanUrls + www→root domain 301 + 30s function timeout”, but when I used git show to look at that diff, there was no cleanUrls key in vercel.json at all. Checking the current file, it wasn’t there either — zero hits across the entire git log -S'cleanUrls' history.

Don’t fully trust a changelog written by AI — the diff is the truth…

Editing the docs created a contradiction

In the commit that removed Redis, I also changed the README line “redirect 301 to the root domain via the config file” to “converge to the root domain via the platform’s default 308”. But the two explicit 301s for permanent: true in vercel.json had always been there — they weren’t removed.

After the change, the README and the config file were in direct conflict, and nobody could tell which one to trust…

Credentials ended up in git history

In the “daily development” section of the deployment docs, the line for the local admin entry directly included the admin account and plaintext password (I won’t repeat the exact values), and this password was the one db:seed seeded into the database, so it’s not as harmless as “it’s just localhost anyway”. The default password in .env.example is also an obviously weak password that no one has ever changed.

What’s interesting is that the AI itself wrote as the last item on the roadmap: “Rotate credentials: JWT_SECRET / admin password — recommend rotating once deployment is stable” — it was aware, but didn’t handle it on the spot. This is probably the most typical behavior pattern of AI: it will note down the risk in the docs, and then keep writing…

Authentication was loosened for legacy compatibility

In the gateway’s JWT verification, there was this block in the catch branch:

} catch {
  // Legacy code uses the raw user id as a "token" — accept it as a user token
  // so unmigrated frontend code keeps working until the full JWT rollout.
  const maybeId = m[1].trim();
  if (/^[0-9a-fA-F-]{32,36}$/.test(maybeId)) {
    return { user: { sub: maybeId } };
  }
}

JWT verification failed? No problem — as long as it’s a 32-36 character hex/hyphen string, it’s accepted as a valid user token. That means knowing a user’s UUID is enough to impersonate them and pass through the gateway. This too was listed as a TODO on the roadmap, but the code shipped as-is…

A feature was downgraded into a silent no-op

The realtime channel in the frontend shim is an empty implementation, and the comment says it plainly:

*   - channel(name) — no-op shim (Realtime is on the roadmap; signaling
*     callsites in VideoChat won't throw but won't deliver either)

The call site won’t throw an error, but nothing will ever be delivered either. This kind of problem is the hardest to debug, because it pretends to be working…

When there’s no conventions file, where do human instructions go?

Traces of the human are scattered across several commit messages: the baseline commit mentioned “the initial version generated by the PM using AI Studio”, the next one mentioned “boolean→numeric in the PM data” and “PM stale column names”, and I quoted the line “to be manually hooked up by the user in the dashboard” earlier too.

The two sentences in the Redis-removal commit that best show that “instructions were given” are: one explaining why Redis was connected in the first place — “connecting Upstash was to align with the stack you listed” — showing the tech stack was specified by a human. The other is “Left: task 8 (VideoChat signaling) renamed to generic ‘Realtime alternative’”.

In other words, a numbered task list existed. But it’s not in the repo — grep across the whole codebase finds no task 8. It only lived in the conversation context, and once the session closed, it was gone…

This is probably the direct reason why, later on, I insisted on leaving a conventions file in the repo no matter what — even if it’s just one line, at least it won’t evaporate along with the session…

Summing it up in a table

After going through these 9 commits, I can roughly map out the boundaries of what “AI working alone” can do:

What it does wellWhat it doesn’t
Scaffolding, boilerplate codeSecurity boundaries (auth, credentials)
Compatibility layers / shims, avoiding large-scale changes to call sitesCross-file consistency (docs vs config)
Ops runbooks, even writing about infrastructure pitfallsIts own changelog (hallucinates)
Migration scripts: dry-run by default, only actually runs with --apply, idempotentRisks it notes down never get fixed later

That migration script deserves a special mention: by default it only prints a dry-run, and only actually uploads when --apply is added, and objects already starting with http are skipped directly, so running it repeatedly won’t cause problems. If I wrote this design myself, it’d come out about the same…

By the way: a comment written specifically for agents to read

While going through vite.config.ts, I came across a comment:

server: {
  // HMR is disabled in AI Studio via DISABLE_HMR env var.
  // Do not modify—file watching is disabled to prevent flickering during agent edits.
  hmr: process.env.DISABLE_HMR !== 'true',
},

Meaning: file watching was turned off by an environment variable, because the high-frequency writes from an agent editing files would make HMR flicker like crazy.

To be clear, this one wasn’t written by it; git log -- app/vite.config.ts only has the baseline import, a scaffolding comment that shipped with the initial version when the PM exported it from AI Studio, and nobody has touched it since. Conversely, that’s even more telling: it’s not some agent leaving a note for itself — the toolchain is already adapting by default to the scenario of “an AI is modifying these files.” In the past, this kind of comment was written for the next person to take over; now it’s also, in passing, written for the next model to take over…

Anyway, the conclusion is that it won’t stop on its own — someone, or a CI, has to force it to pause…