Your automation stopped. When will you find out?
The person who finds out an automation stopped is usually the customer calling to complain. The routine does not announce failure, it simply stops running, and silence looks exactly like normal operation. Making the operation speak up is an architecture decision, not a monitoring tool purchase.
Why is silent failure the normal state of an automation?
Because the error that shows up on a screen has a person looking at it, and the error that happens at three in the morning does not. That is the whole difference.
When a user clicks and the screen freezes, someone is annoyed and a ticket exists within minutes. When the nightly routine that pushes orders from the portal into the ERP fails, nobody is in the room. The symptom is a slightly smaller number the next day, and a smaller number always finds a plausible explanation.
The problem is older and more structural than your operation. Yuan and colleagues studied 198 real, user-reported failures in five widely used data systems, among them Cassandra, HDFS and Redis, and published the result at OSDI 2014. In 92% of the catastrophic failures, the cause was incorrect handling of a non-fatal error the software itself had already detected and signaled.
Read that again, because it breaks the common intuition. The error was seen. It just never reached anyone.
The next finding is worse. In 35% of the catastrophic failures, the code that should have handled the error fell into three trivial patterns, among them being empty or containing only a log line. A log nobody reads is silence with a timestamp. The authors also calculate that simple testing of the error handling code would have caught 58% of those catastrophic failures.
The study is from 2014 and it covers infrastructure software, not your order integration. The logic is the same every time someone has to decide what the program does when the other side does not answer, and that someone is not the person who will feel the consequence.
What in your operation has an expiry date?
More things than are written down anywhere with an owner. Digital certificates, API keys, access tokens, service passwords, vendor contracts and the card that pays for the tool subscription.
The certificate that holds a link between two systems is the one that shrank fastest. The CA/Browser Forum, where certificate issuers and browser makers set these rules, approved a progressive reduction of the validity period on April 11, 2025, with 25 votes in favor, none against and 5 abstentions among issuers, and 4 to zero among browsers. According to the effective date table in the requirements themselves, the maximum fell from 398 to 200 days on March 15, 2026, goes to 100 days on March 15, 2027 and to 47 days on March 15, 2029.
The same table shrinks a second clock most teams never look at. The reuse period for domain validation data drops to 200 days in 2026, 100 days in 2027 and 10 days in 2029, which means the check behind the certificate has to be redone more often too.
One task a year became two, then almost four, and by 2029 it will be close to eight. In the same window the reminder was switched off. Let's Encrypt ended its expiration notification emails on June 4, 2025, citing, among other reasons, that many subscribers now automate renewal.
The argument is right for anyone who automated. For anyone who relied on that email, the alarm went off the air nine months before the task doubled in frequency.
How long does the report keep counting a gain that already stopped?
Until someone checks execution, and that check is on nobody's calendar. The business case was approved with a formula of hours per run times runs per month, and that formula keeps producing a number in the spreadsheet long after the routine died.
The distortion runs both ways. The gain stays reported at full value while the manual rework that came back is diluted into payroll with no name of its own. That is reasoning about how the indicator works, not a field measurement, and it is one reason why the return on an automation has to be measured in the flow rather than in the approval estimate.
An example is worth it, and it is a hypothesis: the details below are illustrative and describe no client. A mid-sized distributor connects its order portal to the ERP, so that an order placed in the portal creates the matching order in the ERP every fifteen minutes. The portal vendor changes the format of one date field. The routine starts failing only on orders with a scheduled delivery date and keeps working on everything else.
Nobody notices. The drop in volume fits inside the normal variation of the month, and the team solves the problem at its own level: they go back to typing those orders by hand, because that is faster than opening a ticket. The automation is dead on one slice of the flow, the report shows the full gain, and the team absorbed the difference without telling anyone.
Look at the shape of that failure. A full stop turns into a complaint within hours. A partial failure fits inside normal variation and survives for months.
What makes the operation speak up when something stops?
Four decisions, all of them before any tool is chosen.
The first is a proof of life. The routine has to record that it ran, not only shout when it fails, because a routine that died does not shout. If the execution record does not arrive inside the expected window, that is the alarm. No news becomes news.
The second is a count on both ends. How many orders left the portal and how many entered the ERP in the same period. That comparison does not depend on the error having been handled correctly inside the code, and it is what catches the partial failure in the example above.
The third is an inventory of expiry dates with an owner. One list of every link, with what expires, when it expires and who answers for it. As long as that date lives in the inbox of whoever configured the integration, it leaves the company with that person. It is the same habit behind documenting and versioning automations, applied to expiry.
The fourth is the destination of the error. Every error needs a human address and an expected action, otherwise it goes back to being a log line. Defining it means deciding who gets called, within how long, and what that person has the authority to do. The absence of that definition is one of the failures that break internal system integrations.
None of the four is a monitoring software purchase. Server monitoring answers whether the machine is up. The business question is different: did the order arrive on the other side. No tool answers that unless someone has defined in advance what should have arrived. UniTrust, a US insurance IMO, has run policies, agents and commissions since 2022 and Reatop records hospital waste in the field, offline, syncing later. In operations like these the design starts from the same question: how does the system prove the record arrived.
Frequently asked questions
Will implementing this take too much time from my team?
Three of the four decisions are design, not construction. Deciding who receives the error, what has to match on both ends and who answers for each expiry date is meeting work. The part that becomes code is two items, the execution record and the compared count, and neither requires touching the logic of an integration that is already running.
How do I measure the result of an alarm, if all it does is tell me something stopped?
With two numbers your operation can already count. The time between the stop and the discovery, and how many incidents were found by the system instead of by the customer. If the second number does not improve after the change, the design is wrong, and that is a useful answer too.
Is this not just one more dashboard nobody will open?
It would be, if the output were a dashboard. A dashboard depends on someone remembering to open it, and silent failure happens precisely when nobody remembered. What changes the operation is the opposite path: the discrepancy goes looking for a named person, with a deadline and the authority to act.
The next step
If you cannot say today how many links your operation runs on and which of them speaks up when it stops, the Operational Architecture Diagnostic is a 30-minute conversation to map that.
Related case studies
Did you recognize your operation in this article?
Book the Operational Architecture Diagnostic: 30 minutes to map where your operation's bottleneck is. No proposal in the first conversation.
Book a diagnostic