How to Reduce Defects in Production

Liviu Lupei

Founder & Solutions Architect, Endtest · August 12, 2026

Production has a strange reputation in software.

A defect shows up there and everyone talks about it as if production somehow created it.

"It happened in production."

"We had an issue in production."

"An edge case appeared."

Then comes the technical explanation. A race condition. An unexpected state. A browser-specific problem. A service returned something we didn't expect.

But in many cases, the defect didn't suddenly appear in production. Production was simply the first place where someone found it.

The bug was probably sitting there in development. Then it moved to the test environment. Then staging.

Nobody encountered the exact combination of data, permissions, browser, timing, actions, or previous state required to trigger it.

Then the release went live.

A real user happened to take that path.

And now it's a production defect.

There are obviously genuine production-only problems. Traffic, infrastructure, configuration, third-party systems, and scale can create conditions that don't exist elsewhere.

But a surprising number of production defects aren't like that.

They're functional defects that made it through several environments without anyone finding them.

So if you want fewer defects in production, don't start with "how do we fix production defects faster?"

Start with:

Why didn't we find them earlier?

1. Get automated tests into the sprint

Here's a development process I've seen variations of many times.

Developers work on features until the final days of the sprint.

Meanwhile, the one or two people responsible for automation have their own backlog of test scenarios to automate.

Development moves faster than that backlog.

So the automated tests are always one release behind.

The framework is perpetually "almost ready."

The test suite is going to cover the new functionality soon.

Probably next sprint.

Then the end of the current sprint arrives and everyone realizes the new functionality hasn't really been tested.

So the entire team spends a few frantic hours manually clicking around before the release.

That's manual testing with an automation side project, not a test automation strategy.

And when a defect later reaches production, the team is surprised.

The most useful automated test is generally one that exists while the feature is still being developed.

That's the window where it can catch a regression before release, run repeatedly while the code changes, and give the team confidence about what they're shipping.

Creating a beautiful automated regression test three weeks after the feature has already gone to production is still useful.

But you've missed its most valuable moment.

This is also why I don't think unit tests alone are a reasonable answer to functional quality.

Unit tests are excellent for validating small pieces of behavior. They're fast, deterministic, and extremely valuable.

But they don't magically prove that a real user can log in, change a subscription, upload a document, complete checkout, receive the right confirmation, and see the correct result afterward.

We've known this for decades.

Different tests catch different categories of problems.

If production defects are happening in real user flows, you need tests that actually exercise those flows.

AI doesn't fix a bad testing process

AI makes automated testing faster.

It does not remove the need to think.

There's an increasingly common workflow that looks like this:

"Here is my application. AI, generate some tests."

That's easy.

It's also lazy.

AI will usually produce reasonable scenarios. It will cover the obvious paths. It may even find cases you didn't immediately think about.

But the creative part of testing still matters.

A good tester doesn't only ask:

What should happen when the user does the correct thing?

They ask:

What happens when the user has two tabs open?

What if this account was created three years ago?

What if the payment succeeds but the next request fails?

What if the user navigates backward?

What if this field contains something unexpected?

What if the session expires halfway through?

What if the same action is performed twice?

And there's another problem with outsourcing the thinking to AI. People get less creative when they stop exercising that part of the process.

A better workflow is the opposite.

First, have a human think about the functionality and write down the important scenarios.

Then ask AI:

What did I miss?

Now you're combining human understanding of the product with the breadth and speed of AI.

That's much more powerful than starting with an empty prompt and accepting whatever test cases a model generates.

Don't let the developer test their own assumptions

There's another rule I strongly believe in.

The person who implements a feature shouldn't be the only person designing the functional tests for it.

Ideally, someone else creates or reviews those tests after the developer explains how the feature works.

The reason is simple.

The developer has assumptions.

Some of those assumptions are embedded in the implementation.

If that same person creates all the tests, those assumptions can easily become embedded in the tests too.

Now you have code and tests sharing the same blind spot.

Independent verification is an important principle in high-assurance and mission-critical software for exactly this reason.

You want another brain looking at the problem.

And this doesn't need to become some giant bureaucratic QA process.

A developer can explain a feature to a colleague in ten minutes.

That colleague can ask three questions the original developer never considered.

Those three questions might be more valuable than another hundred lines of test code.

The ideal test automation team is the whole team

If only one person knows how to maintain the automated tests, you don't really have a test automation system.

You have a test automation person.

That's fragile.

And it creates a throughput problem.

Five developers can produce new functionality much faster than one automation engineer can reproduce everything in a separate automation framework.

The obvious solution is getting more people involved.

Developers. Testers. Product-minded engineers. Anyone who understands the functionality.

But there's a problem.

Most people don't want to learn another framework, maintain another repository, debug another dependency tree, and spend half a day figuring out why some selector generated nine months ago no longer works.

You might think AI solves this.

Just ask Claude to generate Playwright tests and now everyone can contribute.

I don't think that's the answer either.

You can very quickly create a black box containing thousands of lines of AI-generated test code that nobody really understands.

The tests pass.

Great.

But does everyone know what they're actually checking?

And when they fail six months later, who understands why?

This becomes even more dangerous if your team is already using AI heavily to build the product itself.

AI writes the implementation.

Then AI writes tests for the implementation.

You can end up with a double blind spot where neither side received enough independent human reasoning.

The better approach is to make the tests themselves understandable.

This is one of the reasons we built Endtest.

Tests in Endtest are represented as readable steps that people across the team can inspect, modify, run, and maintain without living inside a test framework repository.

AI can help create tests, generate assertions and variables, repair broken elements with Self-Healing Tests, and even analyze failed test steps.

But the test doesn't disappear behind the AI.

The human-readable steps remain there for the team to review and control.

That's an important distinction.

AI should make it easier for humans to maintain good tests. It shouldn't make humans irrelevant to testing.

And when contributing to tests doesn't require everyone to become an expert in a separate testing framework, you have a much better chance of getting the whole team involved.

That's how you get tests written during the sprint instead of three weeks later.

2. Bring accountability back to software quality

Here's the second way to reduce production defects.

Accountability.

Nobody likes that word.

Software teams have become extremely careful about avoiding anything that could sound like blame.

And there's a good reason for that. A culture where people are afraid to make mistakes produces terrible software organizations.

But we've sometimes overcorrected.

A major defect reaches production and the language immediately becomes strangely passive.

"An issue occurred."

"The system allowed an invalid state."

"The validation wasn't triggered."

Then there's a technical postmortem, everyone agrees that software is complicated, and life continues.

That's not enough.

When a notable defect reaches production, the people involved in developing and testing that functionality should discuss it.

Not so someone can be punished, and not so a manager can point at a developer and say, "You caused this."

The goal should be to understand:

What did we miss, and why did we miss it?

Was there no automated test?

Was the scenario never considered?

Did the test exist but run too late?

Did nobody understand how customers actually used the feature?

Was one person's assumption accepted by everyone else?

Was Safari excluded from testing even though 18% of customers use it?

Was there a test backlog that everyone already knew was two sprints behind?

Those are process problems you can fix.

Accountability actually helps here.

When people know that important mistakes will be discussed, they pay more attention.

That's not necessarily negative pressure.

It can work almost like gamification.

Professional football teams don't pretend individual mistakes never happen.

If a player repeatedly loses possession in a certain situation, they review it.

They watch the footage.

They understand what went wrong.

The player learns from it. The coach learns from it. The rest of the team learns from it.

Nobody says:

"Don't worry about it. The ball was lost. These things happen."

The final score is still what matters most.

But football teams measure an absurd number of individual statistics because individual performance affects the final result.

Software teams can learn something from that.

You can have a blameless culture without having an accountability-less culture.

Those are not the same thing.

3. Make developers understand the business

There's one more thing that has an enormous effect on software quality, and it's rarely listed under "testing."

Your developers need to understand the business.

The whole business, beyond the Jira ticket.

The product.

The customers.

That means talking to product owners.

Business analysts.

Marketing.

Sales.

Customer support.

And yes, that means spending engineering time in meetings instead of writing code.

I think that's one of the highest-return investments a software company can make.

I've seen situations where technically excellent developers knew surprisingly little about the product they were building.

This becomes especially common in large products where teams own narrow areas.

Imagine you're working on warehouse software for an online retailer.

Your system receives an order, finds the inventory, creates a picking task, and eventually marks the parcel as dispatched.

You could understand that component perfectly.

But you'd make better decisions if you also understood what happened before and after it.

How was the order created?

When is the customer charged?

When is the invoice generated?

What information gets sent to the courier?

How does the customer track the package?

What happens if an item becomes unavailable?

What happens when the customer requests a cancellation while the warehouse is already processing the order?

Suddenly, an edge case inside "your" small area stops being a weird state in a database.

It's a customer who was charged for an order they can't receive.

Understanding users matters just as much.

Suppose you discover a blocking defect that only affects Safari.

"Can't they just use Chrome?"

Technically, maybe.

Commercially, maybe not.

As I wrote in What Browsers Should You Test Your Website On?, browser coverage should be driven by your actual users and the risk associated with those users, not by the browser your development team happens to prefer.

And the same principle applies everywhere.

A checkout defect is not equivalent to a typo on a settings page.

If users encounter a cosmetic problem, many of them will keep going.

If they encounter an error when they're about to pay you, they probably aren't going to contact Support, clear their cache, switch browsers, and try again.

They're going to buy from somebody else.

A developer who understands that will make different testing decisions.

Production quality starts long before production

Reducing production defects is mostly about moving discovery earlier, not about building a better incident-response process.

You want relevant functional tests running while the feature is being built.

You want someone other than the developer challenging the assumptions behind the feature.

You want the entire team capable of contributing to tests instead of maintaining a permanent automation backlog.

You want AI helping people think and execute faster, not replacing the thinking.

You want production defects reviewed with accountability but without finger-pointing.

And you want engineers who understand what their software actually means to the business and the people using it.

None of these ideas are particularly exotic.

That's part of the problem.

Most teams already know some version of them.

The hard part is organizing development so they actually happen.

Because the objective isn't an impressive automation framework, or 10,000 automated tests, or proudly announcing that 94% of your unit tests pass.

The objective is much simpler:

Find defects before your customers do.

Everything else is implementation detail.

If your current automation setup makes it difficult for the whole team to create, understand, and maintain tests during the sprint, you can explore how Endtest works.