The failure mode people plan for is the one that shouts. A job crashes, an alert fires, somebody fixes it. That failure is annoying and basically safe, because it is visible.
The failure that costs money is the quiet one. A field was renamed upstream and the automation has been writing an empty value for three weeks. A webhook endpoint started returning 200 while silently dropping the payload. A classification step began sending a whole category of requests down the wrong branch. Nothing crashed. The dashboard is green. The work stopped being done correctly some time ago and nobody can say when.
Observability is the property of a system that lets you answer "is this still doing its job, and if not, since when" without guessing. For an automated workflow that means something quite specific, and it is not a monitoring tool licence.
Three questions a workflow should be able to answer
Strip away the tooling and every useful instrument on an automation exists to answer one of these:
- Did it run? On schedule, triggered by the right event, the expected number of times.
- Did it do the right thing? The output matches what the input implied, and the downstream system now holds what it should.
- What happened to this particular record? For any single item, the full path it took, every decision made about it, and where it ended up.
Most automations are built to answer the first and assume the other two. The second is where silent failure hides. The third is what you need at nine in the morning when someone asks about one specific order.
An automation you cannot interrogate record by record is not observable, however many charts it produces. Aggregate health and individual truth are different questions, and customers always ask the second one.
Instrument the work, not just the machine
Infrastructure metrics (CPU, memory, request latency) tell you whether the process is alive. They say nothing about whether the business process is correct. A perfectly healthy service can produce wrong results indefinitely.
The signals worth capturing describe the work itself:
| Signal | What it answers |
|---|---|
| Items received per run | Did the trigger fire and did it bring anything? |
| Items processed, skipped, failed | Where did the volume go? |
| Outcome distribution | Are the branch proportions roughly what they were last week? |
| Exception queue depth and age | Is the human side keeping up? |
| End-to-end duration per item | Is the workflow getting slower before it gets broken? |
| Dependency success rate per integration | Which third party is degrading? |
| Reprocessed or duplicated items | Is idempotency actually holding? |
The outcome distribution deserves particular attention. If a classification step normally routes roughly seventy percent of items down one branch and that proportion moves sharply over a few days, something changed, the input, the upstream format, or the model. No error was raised. The shape of the output is the only evidence.
Correlation IDs make the record-level question answerable
Attach a single identifier to an item the moment it enters the workflow and carry it through every log line, every outbound API call, every queue message and every record written. Log it as a structured field, not inside a sentence:
With that field present, "what happened to order 8f2a41" is one query. Without it, the answer is assembled by hand from four systems and a lot of timestamp arithmetic, and it takes an afternoon.
Two rules make this worth doing:
- Log structured fields, not prose.
reason: "customer_record_missing"can be counted, grouped and alerted on."Couldn't find the customer, skipping"cannot. - Log decisions, not only errors. Every branch taken, every rule applied, every value that determined a route. The quiet failures are all decisions that went the wrong way without erroring.
Alert on outcomes, not on events
The fastest way to make monitoring useless is to alert on every error. A retried timeout that succeeded on the second attempt is not an incident; it is normal operation. Page people often enough for non-events and they stop reading.
A workable split:
- Alert when a human needs to decide something: the exception queue is growing faster than it is being cleared, a run produced zero output when output was expected, the success rate dropped out of its normal band, a dependency has failed repeatedly, or an item has been stuck longer than its agreed limit.
- Log everything else: individual errors, retries that succeeded, routine skips, every decision made.
- Report on the slow signals: weekly volume, outcome mix, exception reasons by category, time saved against the baseline.
Each alert should carry a named owner, a sentence explaining what it means, and a first action. An alert that nobody knows what to do with is a notification, and notifications get muted.
Monitor for absence
This is the single most commonly missing check. Error-based monitoring cannot see a workflow that never started. A cancelled schedule, a revoked credential, a disabled webhook and an upstream system that simply stopped sending all look identical to an error monitor: silence.
Every scheduled or triggered workflow needs a liveness expectation, "this runs at least once an hour and processes at least one item on a working day", and an alert when the expectation is not met. It takes ten minutes to add and it catches the failures that otherwise run for weeks.
Keep a baseline you can compare against
Observability is comparative. "Four hundred items today" means nothing on its own; "four hundred against a weekday average of twelve hundred" means something immediately.
Record a baseline during the pilot, before full dependency: typical volume by hour and day, the normal outcome distribution, the usual exception rate and its common reasons, median and tail duration. Store it somewhere durable rather than in someone's memory, and revisit it after any significant change to the workflow or its inputs.
Seasonality matters here. A quiet Monday in August is not an incident. A baseline that ignores the business calendar generates false alarms, and false alarms are how real alarms get ignored.
Make it legible to the people who own the process
The operations lead who owns a workflow rarely has access to the engineering tooling, and would not find the answer there if they did. Give them one page that answers their questions in their language:
- How many items went through today, yesterday, this week.
- How many need a human right now, and how long the oldest has been waiting.
- What the common exception reasons were this week.
- Whether anything is currently broken, in one sentence.
- A search box that takes a customer or order reference and returns that item's history.
That last one removes most of the requests that otherwise arrive as engineering interruptions. It also changes the relationship with the automation: the team stops treating it as an opaque thing that occasionally misbehaves and starts treating it as a process they operate.
A pragmatic starting set
If you are retrofitting observability onto a workflow that already runs, this order gets the most value soonest:
- Add a correlation ID and make every log line structured.
- Count items in, items out, and items failed per run, and chart them.
- Add the absence alert, zero output when output was expected.
- Give every exception a reason code from a fixed list.
- Expose a record-level lookup to the people who answer customer questions.
- Record a baseline and review the outcome distribution weekly.
None of that requires a new platform. Most of it is a few days of work on a workflow that already exists, and it converts "I think it's fine" into something you can actually check.
The goal is not a wall of charts. It is to shorten the distance between the moment a workflow stops doing its job and the moment someone knows about it, because in an automated process, that interval is the whole cost.
If you are running automations nobody can currently inspect, see how we build and support production workflows or book a strategy call to review what yours is telling you.