Rules Before Model: How to Choose Between Automation and AI

The pricing calculation in our proposal generator wasn't done by people or by a model. It was done by rules: line items, discounts, taxes, extra costs — all pulled from a table. Rules don't make arithmetic mistakes, don't get tired by Friday, and don't invent a line item that isn't in the price list.

It's the boring part of the system, and it delivered most of the value. The question worth asking before you pick a tool is this: does this task have a single correct answer you can write down as a rule — or not? The answer determines not just the cost of building it, but who is accountable for the error.

Three Signs a Task Doesn't Belong to a Model

Three questions make a task easy to check, and you can answer all three before choosing a vendor. Each one rules out an entire class of projects that would otherwise go down the expensive path.

There's one correct answer, and it's verifiable. The cost of an order given a set of line items and a discount is a single number. A deadline date is a single date. Whether a counterparty has an active enforcement proceeding against it is yes or no. In tasks like these, a model doesn't add quality — it adds variance: it might give the right answer, or it might give a plausible one. A rule gives the same answer on the same input every time, and that's its defining property, not a limitation.

An error has a cost and an owner. A calculation error comes out of margin, a deadline error comes as a penalty, a wrong-details error sends a payment to the wrong place. When a rule makes the error, the post-mortem takes minutes: you open the rule, read it, fix it, and the error won't recur. When a model makes the error, the post-mortem runs into a question with no short answer — why it answered differently this time.

The result has to match on repeat. An internal document that goes out to a client, a report for a manager, an entry in a system of record — anywhere a result gets compared to the previous one, reproducibility matters more than flexibility. Two similar requests should produce two identical answers, or the difference between them becomes the subject of discussion instead of the work itself.

The third column matters more than the first two. In the deterministic part, accountability is named and locatable; in the probabilistic part, it's distributed and needs a mechanism of its own. Projects that skip this question at the start get stuck on it later.

It's worth naming what this test doesn't settle. It doesn't answer whether automation is worth doing at all: a task with one correct answer might run once a quarter and not justify a single hour of development. And it doesn't remove the hardest part of the work — figuring out what the rule actually is. In our case, formalizing the cost calculation took longer than coding it: it turned out three managers each had a different idea of what base the discount was calculated from, and nothing had ever surfaced that fact before we built the system.

That's a typical finding. A rule that "everyone already knows" splits into several conflicting versions the moment you try to write it down, and that exercise alone delivers value — regardless of what ends up handling the task.

Why Most AI Pilots Never Reach the Bottom Line

The gap between a demo and an actual return has been measured by several independent groups, and the numbers agree on direction even though the methods differ.

This report deserves a separate note. It got cited widely in the press, and the retellings distorted the sample size — different numbers replaced the actual 52 interviews and 153 surveyed executives. Check it against the PDF itself, not against news coverage of it; and it doesn't compare generative AI to deterministic automation — that connection is mine, not the report authors'.

The limitation here is obvious, and I'm the one flagging it, not a critic of the piece: these are executive self-reports, and the firm running the survey sells AI implementation consulting. The method and sample are disclosed transparently, so the number is usable — but as market participants' testimony about themselves, not as a measurement.

What Deterministic Automation Cost Before Any AI

Rules have their own measured track record, and it's useful precisely as a baseline. RPA got deployed in the same places — document workflows, back office, request processing — where language models are being introduced today.

This source is four years old, and that's a deliberate choice: I need a structural fact about how mature rules-based automation was before the generative AI boom, not breaking news. The data is self-reported and not independently audited, and the payback period growing from 16 to 22 months shows that rules have limits too — the report doesn't dress up the picture.

How Rules and Models Work Together in One Workflow

In practice the choice is rarely either/or. In a working system both classes sit side by side, and the value comes from an explicit boundary between them, not from one winning out over the other.

The order of these stages isn't decorative. Rules come before the model because it's cheaper to reject an obviously incomplete input before you've paid for a model call on it. Human-in-the-loop (a setup where a person approves or rejects the decision) belongs where an error costs money, not where it merely looks scarier.

And a separate point about measurement. Corrections a human makes should feed back into the evals (a set of reference tasks used to test the model). Otherwise, six months from now, no one will be able to say whether things got better or worse — there will be nothing to compare against.

A Sequence of Steps That Saves Budget

Frequently Asked Questions About Rules vs. AI