What 15 Years of Software Engineering Taught Me About AI Engineering

The model is a dependency, and you already know how to treat those

Fifteen years of integrating third-party services teaches one reflex: never trust the other side of an interface. Networks partition, APIs change under you, vendors return garbage with a 200 status. You wrap the dependency, you validate at the boundary, you time out, you design the fallback before you need it.

A language model is that dependency with one property no vendor API ever had: it fails persuasively. A flaky API returns errors that look like errors. A model returns errors that look like answers: well-formatted, confident, wrong. The reflex transfers unchanged; the boundary discipline tightens. Model output gets parsed against a contract before anything touches it, and near-miss output is rejected rather than repaired, because silently repairing a dependency’s output means debugging a guess later. In my experience, the engineers who take to this work fastest aren’t the ones with the most ML background; they’re the ones who already learned, somewhere painful, to distrust a dependency politely.

Contracts still beat intentions

Long before AI, every integration project taught the same lesson: prose specifications rot, typed contracts don’t. An interface written down as a schema can be validated, versioned, and tested; an interface that lives in a design document lives nowhere.

The same line runs through an AI system. A prompt is an intention. A schema the output must parse against is a contract. Prompts are worth engineering carefully, but nothing downstream should depend on a prompt having been obeyed; it should depend on the contract having been enforced. Everything the system does flows from parsed structure, never from raw model text, and the moment I see downstream code reading model prose with a regex, I can tell you where the incident report will come from. The architecture of enforcement (which checks, in what order, with what authority) is its own article; the fifteen-year-old lesson is just that the checks exist outside the thing being checked.

Testing: from examples to distributions, and the return of QA thinking

Classical testing leaned on a working assumption of reproducibility: same input, same output, assert equality. It was always an idealisation (anyone who has debugged a race condition or a flaky integration suite knows deterministic software isn’t), but it held well enough to build a testing culture on. A stochastic component breaks the assertion style without breaking the discipline, and the deeper reason isn’t randomness; it’s that correctness became semantic. Two differently worded outputs can both be right, and a word-perfect output can be wrong, so equality assertions give way to adjudicated outcomes: golden sets scored over the full set, compared release to release. The unit test’s job (catch regressions before users do) survives intact; what changed is that the oracle became a judgment instead of a string comparison. How that evaluation actually works in a RAG system is its own methodology.

What genuinely returns from the past is old-school QA thinking, the kind that treated software as guilty until proven innocent. The most valuable cases in an AI evaluation are the hostile ones: the ambiguous, the unanswerable, the almost-answerable. And one requirement with no classical ancestor: test what the system does when it should do nothing. Classical code rarely needed a test asserting that a function declines to run. AI systems need exactly that, because their signature failure is doing something anyway, fluently.

Observability: logs were always for the argument

Anyone who has faced an auditor, a regulator, or a furious client knows what logs are really for: reconstructing what happened, in order, attributably, when someone with authority demands the story. Debugging is the everyday use; the argument is the design case.

AI systems raise the stakes for a precise reason: asking the model afterwards is not evidence. A language model’s explanation of its own earlier output is a fresh generation, produced on demand, not a record of the original decision. So the system around the model has to keep the story: what came in, what was retrieved, what was proposed, what was checked, who approved, what executed. A decision trace, replayable end to end. I built exactly this for compliance platforms years before touching a model; the only change is that the component under scrutiny now genuinely cannot testify about its own past.

Review and rollback: the human parts survive intact

Code review was never really about syntax; machines took that early. It was about keeping accountable judgment between a change and its consequences, and recording that someone said yes. The structure transfers to AI systems whole: the model proposes, checks advise, a person approves what matters. What the volume of AI-generated work forces is honesty about an old truth: review attention is finite, so it gets spent where reversibility is lowest, not spread evenly out of ritual.

And rollback: the releases that hurt, across fifteen years of deployments, were the ones I couldn’t take back. In an AI system more things are releases than a classical deployment pipeline admits: the model version, the prompts, the retrieval index, the policy and configuration around them all change behaviour, so each deserves the same principle, a tested, cheap way back. Improvement is welcome; irreversible improvement is a bet the system’s users never agreed to.

What is actually new

If the essay stopped there it would be dishonest, because it would read as “AI is just software”. It isn’t, and the differences are not corner cases; they are why this discipline needed new architecture and not just old habits.

Failures are plausible instead of malformed. Classical failure had a smell: the stack trace, the null, the garbled payload. Model failure arrives well-dressed, which is why independent verification (a checker that never sees the generator’s framing) had to become a load-bearing layer rather than a nicety, and why “it looks right” stopped being weak evidence and became no evidence.

The component’s self-assessment is not a signal you can build on. I don’t treat a model’s stated confidence as evidence its output is correct, and repeated sampling agrees with itself on stable errors as readily as on truths. Classical components rarely offered opinions about their own correctness; now one does, fluently, and the dashboards have to be designed to ignore it.

Behaviour shifts without a deploy. A model version bump, a longer context, a slightly different retrieval mix: outputs move without a single line of your code changing. Pin what can be pinned, evaluate across every swap, and prefer architectures where the model is a replaceable part rather than load-bearing glue, because the part gets replaced monthly whether you like it or not.

Interactions between retrieval and generation fail jointly. Two components can each pass their own checks and still produce a confident answer built on the wrong records. Layered evaluation exists because component-level testing stopped being sufficient.

Abstention became a behaviour to engineer. Classical systems declined by crashing. AI systems have to decline on purpose: refusal, “insufficient evidence”, “not proven within this scope” are outcomes that need design, testing, and product surface of their own. That one still feels genuinely new to me: fifteen years of making software do things, and the hardest recent work has been making it not do things, well.

That is the honest split. The disciplines survived: boundaries, contracts, tests, traces, review, rollback. The rules tightened, and a handful of problems arrived that classical engineering never had to solve. Engineers who respect both halves (what transferred and what didn’t) are the ones I’d want building the systems that now sit in front of real decisions. After fifteen years, this doesn’t feel like a career change. It feels like the stakes finally matching the discipline.