I Grep the Repo, Ship It, and Break Someone Else's Service


The change looked safe. Rename a field, tidy up a response, remove something nobody uses.

I searched the repository I was working in. Nothing came back. Shipped it.

The thing that breaks in this story is never in the repo you searched. It's in a service another team owns, written in a different language, calling your endpoint over HTTP where no compiler or linter was ever going to notice.

The failure mode is silence

What makes these bad isn't the crash. A crash is fine, you see it immediately and roll back.

The dangerous ones don't crash. A useful way to frame it is that the warnings are worse than the errors, because they corrupt data quietly instead of failing loudly.

Take a field that changes from an integer to a string. Any client expecting an integer now gets a string. Some languages coerce it. Some concatenate instead of adding. The endpoint still returns 200, the status page stays green, and something downstream is now writing wrong values into a database.

There's a live example doing the rounds: a field that goes null rather than disappearing. Nothing errors. The consumer just starts recording nothing where it used to record something, and you find out weeks later when a report looks odd.

GitHub's own definition of a breaking change is worth borrowing because it's blunt: removing an operation, removing or renaming a parameter, removing or renaming a response field, adding a new required parameter. That last one catches people, because adding something feels safe by definition.

Staging won't tell you

I used to treat a green staging run as the answer to this.

There's a January 2026 postmortem that's a clean example of why it isn't. A stable endpoint was refactored for performance. It passed development. It passed staging. Under production load and production configuration it behaved differently, and transactions started failing across every client of that API wrapper. They rolled back.

Staging tests your code. It doesn't contain every consumer of your code, and it definitely doesn't contain the internal tool somebody built eighteen months ago that still polls you every five minutes.

The cost, per one 2026 industry roundup, is 15 to 20 hours per incident in emergency fixes, with unmanaged API changes accounting for a large share of integration failures. That matches what I've seen, and most of those hours go on finding out who was affected rather than on the fix itself.

Search the organisation, not the repo

This is the habit I actually changed, and it's the least clever fix imaginable.

Vörr searches code across every connected repository at once, so before I remove or rename anything I look for it everywhere rather than where I happen to be.

Three things I check now:

Every reference to the symbol. Not just calls, but imports and re-exports too, because a re-export means somebody else's module is exposing your thing under a different name and the trail continues from there.

Who calls the function, and who it calls. Tracing the callers upward tells you the blast radius. Tracing downward tells you what you might disturb on the way. Both matter and I used to only do the first.

The literal string, across everything. This is the one that catches the cross-language cases. If the field is customer_id, search for customer_id as text across all repos, including config files, SQL, and frontend code. Compilers won't help you here and neither will a language-aware search. Plain text will.

That third one has caught more for me than the other two combined, mostly because HTTP consumers don't reference your code, they reference your strings.

What plain search still misses

Being honest about this, because it's the thing that will get you.

Dynamic access. If a consumer builds the field name at runtime, or reads it from a config file, or maps it through a schema, the string may never appear literally anywhere.

Anything outside your repositories. A Retool dashboard, a spreadsheet somebody wired up, a partner integration. If it isn't in your source control, no code search finds it.

Compiled or vendored clients. An SDK generated six months ago, sitting in a repo nobody has touched since.

So code search shrinks the unknown considerably. It doesn't empty it, and I don't think anything does.

What I'd do around it

Deprecate rather than remove. Keep the old field, add the new one, log whenever the old one gets read. After a few weeks you have an actual list of consumers rather than a guess, which is far better information than any search gives you.

Diff the spec in CI. If you publish an OpenAPI spec, comparing versions on every PR and failing the build on a breaking change is a genuinely small amount of setup for what it catches.

And announce with a date. Not "we're planning to remove this eventually" but a specific sunset date, on a channel people read. Half the incidents in this category are consumers who genuinely didn't know.

The lesson I took from breaking that endpoint isn't that I should search more carefully. It's that my repository was never the right scope for the question I was asking.


Author note

Manjunaathaa, Associate DevOps Development Engineer at Frigga Cloud Labs.

The bit that still gets me about this class of bug is how confident you feel. You searched, you found nothing, and finding nothing feels like evidence. It isn't. It's the absence of evidence in one particular place you happened to look.

I've started treating an empty search result as a question about my search rather than an answer about the code.

If you've found a way to track down consumers outside source control, I'd like to hear it, because that's still the gap I don't have a good answer for. LinkedIn

Post a Comment

Previous Post Next Post