# Failures that look like success By [Anubhav Choubey](https://anubhavchoubey.com/) · Cloud & DevOps · 2026-11-04 · 4 min read > Deploys that reported green and changed nothing, and AI features that failed with no alert. How I made silent failures loud. ```stats Silent no-op deploys · unapplied monitors · one alert per customer-visible error · store in every title ``` ## The problem The worst failures are not the ones that crash. They are the ones where every signal says things are fine. Two of the systems I worked on had exactly that shape. The first was shared plugin code that shipped to every store on a managed hosting platform. A deploy could report success and change nothing. The second was the platform's AI features: a builder and an analytics assistant. They could fail in front of customers, again and again, and nothing anywhere would say so. ## Deploys that did nothing The shared plugin code reached every store over a shared network mount, in a folder per version. The deploy read the version from a header in the plugin's main file, wrote the code to that version's folder, and **skipped the folder if it already existed**. ```text read version header ──▶ versions/ exists? ──yes──▶ skip (reported as success) └──no───▶ copy files, switch stores to ``` So if someone changed the code but not the header, the deploy was a no-op. It reported green, and the stores kept running the old code. The fix that time was a version bump. The lasting fix was a habit: every change to shared code had to bump its version, and I verified deploys from the outside. To remove some noisy console logs, I deployed once *with* them and checked they appeared, then *without* them and checked they were gone. That proved the deploy mechanism worked, not just that the code had changed. I also split the shared code into a lean base for every store and a separate package for the AI builder's preview tools, shipped only inside the builder's image. Code that no real store needed could no longer reach one. ## A slot that could not be reclaimed The builder's preview environment swapped some folders for links into each user's own working copy. If removing the real folder failed, the code gave up on that slot and moved on, so the preview kept serving whatever was there before. Then a package updater, running as root inside a preview, replaced one of those links with a real folder owned by root. The preview process did not have permission to remove it, so that user's preview served the wrong code, permanently and silently. I changed the step to reclaim the slot, or fail loudly if it could not. ```callout warn A "best effort" step that sets up an invariant does not fail loudly. It quietly falls back to the previous state, and the previous state is usually the wrong one. ``` ## Monitors that had never run For the AI features, monitors had already been written in Terraform for another monitoring tool. They were reviewed and merged. They had **never been applied**: a flag was off, and there was no integration to deliver them to Slack even if it had been on. Unapplied infrastructure code reads exactly like applied infrastructure code. Instead of repairing that path, I built on the cloud provider's own monitoring, which already delivered other alerts to Slack. Most of the work was not the alerts. It was making failures visible at all: - **Log the failure where it happens.** The analytics assistant could refuse a turn, tell the user why, and write nothing to any log. A policy over a log line nobody writes can never fire. The fix started with one structured log line in the service. - **Alert on every error a customer sees.** A customer saw "model at capacity" and the channel stayed quiet, because the alert waited for several errors per store. Both policies moved to a **single error**, grouped by store, error code and cause, posting once per incident. Errors the user could see but that wrote no error line now wrote one. ## Alerts people can act on Every alert title had been identical for every store. So the first step on every incident was opening it to find out which store it was about. I put the **store in every title**: ```text BUILDER turn errors: () ``` When a second source started firing one of the alerts, its runbook still described only the first. Every key sentence was wrong for the new source. A runbook that sends the responder to the wrong place is worse than none, so I rewrote it. And I removed an alert for a component that no longer existed, because alerts that fire for nothing teach people to ignore the channel. ## What I'd do differently - **Make "applied" checkable.** A CI check that compares the monitors in code with the monitors that exist would have caught the unapplied ones on day one. - **Stage shared code rollouts.** A deploy that goes to a test environment by default, and to every store only on purpose, would turn a mistake into a non-event. These failures are close cousins of the ones in [point-and-edit AI builder](/blog/point-and-edit-ai-builder), where a feature looked alive in the UI and was dead underneath. ## Takeaway Green is not proof. A deploy has to show it changed something, a monitor has to show it can fire, and an alert has to say which customer it is about. Make every failure loud at its source, then make the alert easy to act on.