Side Project or Service? Two Ways to Run TestOps

Most teams run their automated tests one of two ways, and the difference between the two quietly decides whether engineers trust their own pipeline. In the first way, the suite is a side project: written once, left to accumulate, checked only when something breaks. In the second, the suite is run like a small service with users and a service level — the working core of TestOps, a discipline built around one question: when the pipeline speaks, does anybody believe it? This piece puts the two models side by side and lets the published evidence arbitrate, because opinions about testing are free while reruns are not.

The Side Project Model, Described Fairly

To its credit, the side project model costs nothing up front. Somebody writes tests, somebody else adds more, and the CI job runs them all on every change. There are no dashboards, no owners, no budgets — and no honesty either. When a red result appears, the first move is a rerun, then shrug, then merge anyway. Nobody is lying; the model simply never promised that red means anything. The suite behaves like an untended mailing list: mostly quiet, occasionally screaming, never accountable.

The hidden bill arrives as waiting time. A suite nobody curates grows until it takes an hour, so developers batch their changes, so each change gets bigger, so failures get harder to read. The loop feeds itself.

The Service Model, Described Fairly

The service model treats each verdict as a deliverable with a deadline and a reliability target. Results are stored with the code revision they judged, so history can be audited instead of guessed. Unreliable checks are identified and benched with a named owner rather than tolerated. Stage times are budgeted the way a publication budgets editing passes: quick checks early, deep checks later, nothing blocking for longer than the value of the answer. It is more bookkeeping — that part of the sales pitch is true. The claim worth examining is whether the bookkeeping buys trust. There is actual data on that.

What the Noise Data Says About TestOps

Start with the largest suite anyone has published numbers for. Examining Google’s tests in 2016, researchers measured a flake rate near 1.5 percent of runs — small per test, until you multiply. Around one test in six showed flaky behaviour at some point in its life. Most striking: when a previously passing test suddenly failed, roughly 84 percent of those turnarounds were flakes, not bugs. Read that again from the developer’s seat. Under the side project model, four out of five alarms are false, and the natural human response to a five-in-six false alarm rate is to ignore the alarm. The side project does not merely lack polish; it actively trains its users to disregard it.

The Evidence on Selection

The second charge against the service model is cost: if every verdict needs curation, surely it needs more compute too. The selection data says otherwise. A team at Meta taught a model to pick the tests a given change actually required, feeding it only results history from which flaky entries had been stripped. The result: fewer than 0.1 percent of regressions slipped through, while the model skipped roughly two of every three dependent tests. That training detail is the whole story — teach a selector from noisy records and it confidently learns to run the wrong things. Curation is not overhead on top of intelligence; it is the prerequisite for intelligence at all.

TestOps Repair: A Before-and-After

Can a distrusted pipeline be repaired, or is badwill permanent? Slack’s mobile crew published their own before-and-after: the main branch now goes green on 96 percent of builds, where before the flake work it managed just 19.8. A pipeline that fails four times out of five is not really a pipeline, as we saw above — it is a slot machine with extra steps. After the fix, roughly one build in twenty-five misreports. Same product, same engineers, different operating model. That gap is the entire return on the bookkeeping.

One honest caveat belongs here. The service model can curdle into ceremony — dashboards nobody reads, quarantine lists that turn into a museum, meetings about metrics with no owner behind them. The difference between operations and bureaucracy is whether somebody acts when a number moves. Every team we have seen succeed kept the loop small: measure, name a person, fix or delete, repeat. Keep that loop or the bookkeeping becomes its own side project.

The Scorecard, and Where the Models Fit

Summing the published record:

  • Trustworthiness of a red result: side projects manufacture false alarms at industrial rates; service model teams measure and suppress them.
  • Cost per change: unmanaged suites grow until everyone batches work; curated suites can shrink execution by two-thirds through selection.
  • Speed of recovery: with stored, revision-tagged history, a bad verdict is diagnosable in minutes instead of folklore.

The fair conclusion is not that side projects are stupid — they are simply unaudited, and the data says unaudited suites drift toward noise. If you want the practice laid out in full, the Ops Atlas TestOps entry is the densest honest summary we have found.

There is a publishing parallel our readers already know. Anyone who has followed our walkthroughs on asking a good question on Stack Overflow or publishing on Substack has met the same law: channels that accumulate unreplied noise stop earning attention, and only deliberate maintenance keeps an audience listening. Even the DEV Community publishing routine works because somebody tends the commons. Test suites are audiences of one — the developer waiting for a verdict — and they respond to neglect the same way.

Run your tests however you like. Just be honest about which model you picked, because the rerun button has already voted.

Related Posts

How Writers Learn From Published Posts: A State-of-Practice Look

More contributors now dissect one finished post instead of copying a favourite author. A level-headed look at what the habit reveals and where its limits lie.

Read More

How to Rewrite a Post in Plain Words, One Step at a Time

A seven-step method for turning a jargon-heavy draft into something strangers can follow, with a few simple swaps and a quick outsider read.

Read More

Content Marketing Within Limits: An FAQ on Creative Constraints

Answers to common questions about planning content when the blank page freezes you, using self-set limits such as one reader, one takeaway and one platform.

Read More

Why Logging Every Guest Post the Same Day Saves Hours

A friendly explainer on filing, logging and clearing small bits of content work the day they finish, so a pile of drafts never builds up.

Read More

Content Audit Checklist: Fast, Cheap or Good, Which Do You Drop?

A seven-step checklist for auditing your content routine, spotting which of speed, budget or quality you are really sacrificing, and recording that choice.

Read More

Eight Content Patterns to Check Before You Rewrite Any Post

Before you rewrite a flop, count. These eight repeating signals in your headlines, timing and feedback show where the real weakness sits.

Read More

Leave a Reply

Your email address will not be published. Required fields are marked *