Process Observability: What You Should See a Month After Launch

In our proposal generator, we tracked two kinds of statistics from day one: performance by account manager, and which proposal types generated the most demand. It was the simplest part of the service to build — and the only part that let us answer "is this working?" with numbers instead of impressions.

A month after any process goes live, a leader asks the same question. If the answer comes back as "the team likes it" and "it feels smoother," what got launched wasn't a process — it was a demo. Process observability (the property of a system that lets you reconstruct its internal state from external data) gets built in before launch. It almost never gets added afterward.

The Three Numbers Every Process Needs

The set stays minimal not to save effort, but because practice demands it: every extra number needs an owner, and it opens conversations that don't lead anywhere near a decision. The three metrics below each answer a different question, and none of them substitutes for another.

Number of completed scenarios. Not the number of users, not the number of logins — the number of cases where the process ran from start to finish. This is the only metric that tells work apart from activity: a user can log in every day and never complete a single scenario.

Failure rate. A failure is a case where the work got done on paper but the result never moved forward: the document wasn't sent, the request wasn't assigned, the answer wasn't logged. This rate drops when you fix something and rises under load — it's how you tell whether a process is resilient or just held together by people pushing harder.

Time from start to result. Median, not average: the distribution almost always has a long tail, and the average describes none of the real cases in it. The median answers "how long does this usually take"; the tail answers "what's breaking."

The third column matters: the substitute almost always looks plausible, and it always makes things look better than they are. Active user counts rise on curiosity alone, support tickets fall once people stop expecting help, and average time improves the moment you drop the outliers.

Why Your Process Model Diverges From Actual Execution

There's another reason observability matters: the process you describe on paper and the process that actually happens are two different things, and the gap between them is measurable. The method for reconciling event logs against a formal model — conformance checking — has been around for a while, and it produces gaps larger than most people expect.

The study is nearly twenty years old, and it covers the Dutch public sector — that's its limitation. The method, though, is still a cornerstone of process mining, and the underlying result holds up in any organization where someone actually bothers to compare the documented procedure against the logs. The practical takeaway: a gap covering up to half of all cases isn't an anomaly. It's the norm for a process nobody is watching.

The Performance Gap Between Teams That Measure and Teams That Don't

The link between the ability to measure and actual results is best documented in software engineering, where metrics are standardized and have been collected for years.

Let me name the limitation directly: the report doesn't prove that observability alone causes the result — it shows that groups at different maturity levels also differ in their ability to count their own failures. Causality isn't established here, and I'm using these numbers to illustrate the scale of the gap, not as proof of a causal link.

Observability for AI Scenarios: A Special Case

For scenarios built on language models, a fourth number joins the baseline three: the share of results a human had to correct. Without it, any claim about a better prompt or a better model stays a matter of faith.

This needs the precision that usually gets lost in summaries: the report measures whether organizations deployed AI system monitoring at all, not whether they specifically monitor answer quality. There's no separate figure for accuracy control or the share of erroneous outputs. And this is a survey run by a vendor of observability tools — the commercial interest is obvious. What's still useful is the direction: nearly half of companies push AI scenarios into production without built-in control over how those systems actually behave.

Why Your KPI Set Should Stay Small

The temptation to measure everything shows up immediately, and it wins almost every time. The argument against it is older than any dashboard: a metric that becomes a target stops being a metric — the mechanism widely known today as Goodhart's law.

One caveat: the full text is paywalled, and I worked from the abstract — the famous phrasing itself doesn't appear in the open portion, though it's widely attributed to this paper. I'm using it as a conceptual argument, not a direct quote. The practical consequence is straightforward: any metric people get evaluated on starts steering their behavior, which is exactly why a set of three numbers is safer than a set of fifteen.

How to Build Observability Before Launch

The fourth decision is the only one you can't undo later. A baseline measurement takes half a day and gets done manually on the last twenty results before launch. Companies that skip this step spend their one-month review discussing memories of how things used to be, not an actual effect. The same logic runs through this breakdown of how to split an effect into its components: without a baseline, that kind of decomposition isn't possible in the first place.

The first three decisions look technical, but the process owner is the one who makes them. Deciding what counts as a failure isn't an architecture question — it's a question of where the company draws the line between work that's done and work that isn't.

Four Steps to Take Before Launch

Frequently Asked Questions About Process Observability