# Automation errors at scale: the cost ROI leaves out

> Automation multiplies the gain per run, and it multiplies the error by the same count. Here is what a wrong rule costs before anyone can stop the routine.

- Author: Marlon Trettin
- Published: 2026-11-17 · Updated: 2026-11-17
- Language: en
- Canonical: https://yowpi.com/en/blog/automation-errors-at-scale

---

An automation that gets something wrong repeats the mistake at the same speed it would repeat the right answer. The ROI model multiplies the gain by the number of runs and almost never does the same math for the wrong rule. The missing question is an architecture question: how many wrong runs happen before someone, or the system itself, can stop the routine?

## TL;DR

- In the ROI spreadsheet, the gain per run is multiplied by volume. The loss from a wrong rule follows the same multiplication, so it belongs in the math before approval.
- On August 1, 2012, according to the SEC, Knight Capital processed 212 small customer orders and generated more than 4 million executions in about 45 minutes, losing more than $460 million. The SEC order records that automated limits and a procedure to halt the system were missing.
- Exposure depends on three numbers an operation can estimate: runs per hour, time to detection, and the cost of each wrong run.
- Four design decisions reduce that exposure: release the change gradually, cap each batch, let the system stop itself, and record each run so it can be undone.

## Why doesn't automation ROI show this risk?

Because the ROI spreadsheet treats every run as a success. It compares the cost to build with the time saved per run, multiplies by expected volume, and finds the month the project pays for itself. We have covered how [the number of process variants decides that break-even point](/en/blog/count-process-variants-before-you-automate). The other side of the same math is missing.

In a manual process, error has a natural brake: the speed of the person. Someone applying a wrong price increase by hand gets dozens of records wrong per hour, and often notices along the way. Automation removes that brake. The wrong rule is applied to every eligible record, at machine speed, with nobody looking at each one.

This is not an argument against automating. It is the reason the risk has to enter the project as a number, next to the gain.

## What happened at Knight Capital?

Knight Capital Americas was a US equities market intermediary, and the case is useful because the [SEC order of October 2013](https://www.sec.gov/litigation/admin/2013/34-70694.pdf) describes the failure in detail. The firm replaced old code, unused since 2003, with new functionality, and repurposed a flag that used to activate the old code. A technician did not copy the new version to one of the eight servers, and the deployment did not go through a second review.

At the market open on August 1, 2012, that server ran the old code. In about 45 minutes, 212 small customer orders turned into more than 4 million executions in 154 stocks. The firm built a long position of about $3.5 billion and a short position of about $3.15 billion, and the loss exceeded $460 million. The SEC imposed a $12 million penalty, according to its [official press release](https://www.sec.gov/newsroom/press-releases/2013-222).

Two details in the order matter to any operation. The system sent 97 automated rejection emails before the open, but they had not been designed as alerts and staff generally did not review them. And the firm had no aggregate capital thresholds linked to automated controls, and no procedure to halt the system in response to its own aberrant activity. The signal existed. What was missing was the architecture that turned it into a stop.

## How much can a wrong rule cost your operation?

The size of the damage fits in one multiplication: runs per hour, times hours until someone detects and stops it, times the cost of each wrong run, including the cost of undoing it. The math is approximate, but it forces you to estimate the term that usually gets left out, the time to detection.

That term depends less on code quality and more on how the operation is designed. If the error only shows up when a customer complains, detection time is however long the customer takes to complain. If the routine runs overnight and nobody checks the output before business hours, it is hours. It is the same problem as [finding out an automation has stopped](/en/blog/your-automation-stopped-when-will-you-find-out), with one aggravation: an automation that stops stops producing, and one that is wrong keeps producing.

The example below is a hypothesis to illustrate the math, not a measured case. A distributor with 4,000 active customers automates its annual price list update. A misconfigured rule applies the percentage twice. By hand, an analyst would update a few hundred records a day and would have a good chance of noticing the odd value early. Automated, the update reaches all 4,000 records in a single run, and the next day's orders go out at the wrong price until someone calls.

Following the same hypothesis, with a correct 8% increase on a $2,000 average order, each order goes out about $170 above the right price. If 300 orders are invoiced before anyone notices, that is about $52,000 in invoices to correct, before counting customer service time and the strain on the customer relationship. The numbers are illustrative; the math with yours is what matters.

## How do you limit the size of an error before it happens?

With four decisions made in the design, before the routine goes live. None of them depends on a specific tool.

The first is to release the change gradually. [Google's SRE Workbook](https://sre.google/workbook/canarying-releases/) calls this canarying: apply the change to a small fraction of production, evaluate, and only then continue. The chapter's own example shows the effect: if a change fails 20% of requests and is applied to 5% of the population, the overall error rate is 1%. For the price update, that means applying it to 200 customers first, checking, and then releasing the rest.

The second is to cap each batch. Every routine that changes a value, sends a charge, or creates records in bulk needs a quantity or value limit above which it stops and asks for confirmation. The cap does not prevent the error. It limits how many records the error reaches.

The third is to let the system stop itself. An alert nobody reads is Knight's 97 emails. If the rejection rate, the average value, or the volume leaves the expected range, the routine halts and notifies a named owner, instead of logging the fact and moving on.

The fourth is to record each run reversibly: what was changed, the previous value, and which run made the change. Without that record, undoing the error becomes a manual reconstruction from backups and memory.

## Does this apply to companies that are not in tech?

It does, because exposure does not depend on whether the company wrote the code that failed. On July 19, 2024, at 04:09 UTC, CrowdStrike released a configuration update to its Windows security software and reverted it at 05:27 UTC, according to the company's [preliminary post incident review](https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/). [Microsoft estimated](https://blogs.microsoft.com/blog/2024/07/20/helping-our-customers-through-the-crowdstrike-outage/) 8.5 million affected devices, less than 1% of all Windows machines.

Among the changes CrowdStrike itself announced are a staggered rollout for this type of update, starting with a canary, and more customer control over when and where it is applied. The affected companies did not write a line of the faulty code. Their exposure depended on how much outside change could enter the operation without passing through a step under their own control, and that control was not in their hands.

For a mid-sized business, the lesson is the same at a smaller scale. Every routine that touches prices, billing, commissions, or bulk records deserves the question: if the rule is wrong, how many records does it reach before someone notices? At [UniTrust](/en/cases/unitrust), a US insurance IMO, commissions are calculated instantly on every paid policy. In a routine with that frequency, the answer to that question needs to be in the design before the first run, not after the first error.

## Frequently asked questions

**Don't these controls reduce the return on automation?**

They have a cost, and it goes in the same spreadsheet as the gain. The honest comparison is between the cost of building the canary, cap, and stop, and the estimated exposure without them. In low-volume routines with low value per run, exposure may be small and simple controls are enough. In routines that touch prices or billing across the whole customer base, a single error can erase months of gains.

**Isn't it safer to keep the manual process the team already knows?**

The manual process makes mistakes too, just more slowly and less visibly. The person typing is the brake and also the source of error, and that error is rarely recorded. Well-designed automation replaces the human brake with explicit limits and a record of what was done, which makes it possible to measure and undo.

**Do I need a technical team to implement this?**

The hard part is deciding, not coding: which fraction gets the change first, which cap makes the routine stop, who gets notified and how fast. Those decisions come from the people who know the operation. Implementation is the job of whoever builds the system, and it costs less when it is part of the project from the start than when it is retrofitted after the first incident.

## The next step

If nobody in your company can say today how many records an automated routine reaches before someone can stop it, the Operational Architecture Diagnostic is a 30-minute conversation to map that exposure. [Talk to an architect](/en/contact).
