For a long time, I could have told you good stories about the quality of our software. We talk to customers before we build, reviews catch problems early, and critical bugs get fixed fast. What I could not have shown you were software quality metrics that backed any of this up.

That changed when our yearly ISO 9001 audit asked how software development had performed against the quality goals we had set ourselves. To answer, I had to put numbers next to our stories. It was more useful than I expected, and a little uncomfortable.

What the Numbers Did to Our Stories

The first story was that we validate requirements with customers, and we do. Our product managers talk to customers and test designs with them before something is built. But when I looked for evidence, only two out of several hundred tickets said anything about it, because the field was not even part of the form. We could not show something we were sure we did.

The second story was that we had more bugs because we review more carefully. The number of bugs had more than doubled, and a systematic security review and code reviews, some of them supported by AI, explained a large part of that.

But not all of it. Even without those findings, the number of bugs per finished feature ticket was still slightly higher than before, and at least one import error had only been found in production. The story was mostly true, which made it easy to stop asking.

The third case went the other way. Support tickets had dropped by about two thirds, which looked like happier customers. In practice, customers had started writing directly to their customer success contact, and those requests went straight into our backlog without ever becoming a ticket. Here, the number would have misled us without the story behind it.

Not every story failed. Critical bugs really were fixed quickly. But I only knew that for sure once I had counted.

Why a Number Makes You Question the Story

In a small company, many of us were in the room when a customer said something memorable. Those moments are easy to recall, and what is easy to recall feels common. Amos Tversky and Daniel Kahneman described this as the availability heuristic: people judge how often something happens by how easily examples come to mind.

A number is less vivid, but it can be checked. It does not tell you what is true. It tells you which of your stories need a second look.

After our quarterly team ratings, I wrote that the scores needed the observations behind them to mean anything. This is the other half of the same lesson. Numbers need observations to be interpreted, and observations need numbers to be checked.

How Many Software Quality Metrics Does a Startup Need?

Fewer than you might think, but more than none. One or two numbers are enough to start, as long as you write down how each one is calculated and treat that definition as a draft. The point is not a dashboard. The point is to have something that can disagree with your stories.

Our old goals show why the definition matters. The previous bug rate could not be recalculated, because nobody had written down how it was calculated. Instead of defending the old number, we are now writing the calculation into the goal itself. We also made the validation field mandatory, and in the first week every new feature ticket was assessed.

Next, we are adding a field for where a bug was found: during development, in review, in testing or in production. That distinction says more about quality than the total count, and it would have answered our second story much faster.

My practical takeaway: pick one or two software quality metrics early, define each in a sentence, and change the metric when it stops being useful. A rough number you revise is worth more than a precise story you cannot check.

Structured Data Points Make AI Analysis Useful

There is a second reason I now care more about simple data points. We use language models more and more for analysis like this. With a few structured fields, a model can prepare our quarterly review across hundreds of tickets: which bugs reached production, which features were validated with customers, and where patterns repeat. Without those fields, it can only summarise the text in the tickets.

A summary of anecdotes is still an anecdote, only better written. The more of our summaries AI writes, the more we need numbers to check them against. In When Building Gets Faster Than Learning, I argued that the real limit is the number of decisions a team can back with solid evidence. Counting is one of the cheapest ways to add to that evidence.

I am still learning which numbers are worth keeping. But I no longer trust a story about our quality that I have never tried to count.

Numbers do not replace the stories. They tell you which stories to check.