How to Handle Defects in Production

Liviu Lupei

Founder & Solutions Architect, Endtest · June 18, 2026

At some point, every software company gets the message nobody wants to receive:

Something is broken in production.

Sometimes it comes from an automated alert. Sometimes it comes from support. And sometimes it comes from a customer who has already spent 20 minutes trying to understand whether they are doing something wrong.

The technical defect might be small. The damage rarely is.

A production bug interrupts someone while they are trying to get work done. It can make them question whether their data is safe, whether an important operation completed successfully, and whether they can continue relying on your product.

For a software company, reliability is part of the product.

Customers do not separate your features from your uptime, browser compatibility, error messages, and response to incidents. It all becomes part of the same judgment:

Can I trust this product?

The way your team responds to a production defect can either reinforce that trust or make the situation worse.

First, determine what is actually broken

The first report is usually incomplete.

A customer might say that a page does not work, a button does nothing, or their data disappeared. That is enough to start investigating, but not enough to start changing production code.

Before discussing causes, determine the scope of the problem.

Try to answer a few basic questions:

Which users are affected?
Which workflows are affected?
When did the issue begin?
Does it happen consistently or intermittently?
Is it limited to a browser, operating system, account type, region, or configuration?
Was there a recent deployment or infrastructure change?
Is data being lost or corrupted?
Is there a security or privacy impact?

This is the isolation stage.

The goal is to reduce the size of the unknown.

A defect affecting one browser version requires a different response from a defect corrupting data for every customer. A cosmetic problem on an internal settings page is different from a broken checkout, login flow, or deployment process.

Teams often lose time because they jump immediately to the first plausible explanation. Someone remembers a recent frontend change, so everyone assumes that change caused the problem. Another engineer sees a database timeout and concludes that the database is responsible.

Those clues may be useful, but they are not conclusions.

Reproduce the issue using the same conditions as the affected user whenever possible. Use the same browser, operating system, permissions, account state, feature flags, and data.

The closer your reproduction environment is to the customer's real environment, the less likely you are to spend hours fixing the wrong problem.

Contain the damage before finding the perfect fix

When a defect is actively harming customers, the first objective is containment.

That might mean rolling back a release, disabling a feature flag, temporarily blocking a specific operation, restoring a previous configuration, or routing traffic away from a failing service.

A rollback is a tool for reducing the consequences of failure, not an admission that the team failed.

The temptation is to keep the new release live while the team works on a clean fix. That can make sense when the impact is minor and well understood. It makes much less sense when customer data, billing, authentication, or a critical workflow is involved.

A useful question is:

If we do nothing for the next two hours, what happens?

If the answer includes more corrupted records, more failed payments, more locked-out customers, or more support tickets, containment should happen before a detailed root-cause investigation.

The elegant solution can come later. Stop the damage first.

Reproduce the defect before patching it

A patch created without a reliable reproduction is partly a guess.

It may work. It may also hide the original symptom while leaving the underlying problem in place.

Before changing the implementation, create the smallest repeatable scenario that demonstrates the failure. Ideally, that scenario should become an automated functional test.

This gives the team three things:

Proof that the defect exists.
A way to verify that the patch fixes it.
Protection against the same defect returning later.

The test should fail before the patch and pass after the patch.

This sounds obvious, but it is frequently skipped during urgent incidents. Someone changes a condition, refreshes the page twice, sees the expected result, and declares the problem fixed.

That is a promising observation, not verification.

A production defect has already demonstrated that the system behaved differently from what the team expected. That is exactly when assumptions deserve more scrutiny, not less.

Find the root cause, not only the visible symptom

The root cause is not always the line of code where the error appeared.

Suppose a customer cannot save a form because a new validation rule rejects an empty optional field. The immediate fix might be to change the condition.

But the deeper questions are more useful:

Why was the optional field treated as required?
Why did the existing test suite not detect the change?
Was the requirement unclear?
Did the developer test only the most common path?
Was the frontend tested without the real API response?
Did different browsers serialize the value differently?
Was the affected workflow missing from the release checklist?

The root cause might be a coding error. It might also be a missing test, an inaccurate assumption, an incomplete specification, a browser difference, or a review process that focused on implementation details while overlooking user behavior.

Stopping at "the condition was wrong" makes it likely that a similar problem will appear somewhere else.

Root-cause analysis should not become a search for the person who made the mistake. That encourages people to hide uncertainty and defend their decisions.

The better question is:

What allowed this defect to reach users?

That question can produce an actionable answer.

Apply the smallest safe patch

Urgency creates a dangerous incentive to rewrite too much.

An engineer sees the defect, notices that the surrounding code is messy, and decides to clean up the entire area while fixing it. The resulting change becomes larger, harder to review, and more likely to introduce another problem.

During an incident, prefer the smallest change that safely corrects the behavior.

That does not mean inserting a random workaround and forgetting about it. The idea is to separate immediate risk reduction from broader architectural work.

A good production patch should be:

Narrow enough to understand quickly
Easy to review
Covered by a regression test
Reversible if the result is unexpected
Tested against adjacent workflows
Monitored after deployment

The regression test should cover the exact scenario that failed in production. The team should also test nearby paths that could be affected by the patch.

A change to authentication should not be verified only by logging in once. Check failed logins, password resets, sessions, permissions, and any related account flows.

A change to checkout should not be verified only with one successful payment. Check declined payments, retries, discounts, taxes, duplicate submissions, and confirmation messages.

The patch solves the known defect. Regression testing helps ensure that it does not purchase one fix by creating three new problems.

Test the patch like a real user

One of the most dangerous phrases during an incident is:

It works on my machine.

That statement is not useless, but it is incomplete.

The software does not exist only on the developer's machine. It exists on the combinations of browsers, operating systems, screen sizes, network conditions, permissions, and account states used by customers.

Testing a patch with unit tests and component tests is important. It is not sufficient.

At Endtest, we see aggregate patterns from millions of automated test executions performed through the platform each year. One pattern remains consistent: defects still appear at the functional level even when lower-level tests pass.

This is not an argument against unit or component testing. Both are useful. Unit tests provide fast feedback about isolated logic. Component tests help verify individual pieces of the interface.

But neither can fully replace a functional end-to-end test that opens the application, interacts with it through the browser, sends requests to real services, and verifies the outcome from the user's perspective.

There are always defects that appear only when the pieces are connected.

A button can render correctly while sending the wrong payload. An API can return the correct response while the frontend fails to display it. Two components can work independently but fail when they share state. A workflow can pass in Chrome and break in Safari.

The closer the test gets to actual user behavior, the more of these integration boundaries it can verify.

A headless Linux container is not the entire internet

Headless browsers running in Linux containers are convenient. They are fast, inexpensive, and easy to add to a CI pipeline.

They are also only one testing environment.

It is extremely risky to treat a passing headless test suite inside a Linux container as proof that a release works for all users.

A production release should also be tested in real browser processes running on full operating-system environments similar to the ones used by customers. Those machines may be physical or properly virtualized, but they should reproduce the browser and operating-system combinations that matter to the product.

Headless testing can miss differences involving:

Browser rendering
Fonts and layout
File downloads and uploads
Permissions
Popups and new tabs
Clipboard behavior
Authentication flows
Video and audio
Browser extensions
Operating-system dialogs
Focus, scrolling, and keyboard input
GPU-dependent behavior

The answer is to understand what headless tests prove, not to stop using them.

A headless Chrome test on Linux proves that the tested scenario worked in headless Chrome on Linux under those conditions. It does not prove that it works in Safari on macOS, Chrome on Windows, Firefox, or Edge.

That distinction matters when a release can affect paying customers.

Safari is not the same thing as WebKit

Cross-browser testing is often reduced to running Chrome and then running a WebKit-based test in a container.

That is better than testing only Chrome, but it is not the same as testing Safari.

WebKit is the rendering engine used by Safari. Safari is the complete browser, with its own networking behavior, privacy features, media handling, permissions, browser interface, and integration with macOS and iOS.

A WebKit build running on Linux can help identify compatibility problems. It should not be presented as a complete substitute for testing the real Safari browser on an Apple operating system.

This distinction becomes especially important for products with a meaningful number of Safari users.

Teams should not choose browsers based on personal preference or convenience. Look at real production analytics and determine what customers actually use.

That might include:

Chrome on Windows
Chrome on macOS
Safari on macOS
Mobile Safari on iOS
Edge on Windows
Firefox on Windows or Linux

The exact matrix will vary by product.

A B2B application used primarily by corporate customers may have a large Edge and Windows audience. A consumer product may have significant mobile Safari traffic. A developer tool may have more macOS and Firefox users than expected.

Your test strategy should follow your customers, not browser market-share headlines.

For a practical way to choose that browser matrix, see What Browsers Should You Test Your Website On?.

Functional tests must belong to the whole team

Many companies technically have functional tests, but only one person understands them.

That creates a fragile system.

The tests live in a separate repository. They use a custom framework. They require specialized setup. Most developers avoid touching them. Product managers cannot read them. Manual testers have to ask an automation engineer to update a simple workflow.

Eventually, the test suite falls behind the product.

New functionality is released every sprint, while test updates remain in a permanent state of "almost ready." The suite gradually becomes less representative of the application. Failures are ignored. Tests are disabled. The CI job remains green because fewer meaningful things are being tested.

For functional tests to remain useful, all relevant team members should be able to contribute to them.

That includes developers, testers, and, where appropriate, product or support team members who understand the workflow being tested.

Not everyone needs to become an automation-framework specialist. The tests need to be understandable and editable without requiring one specific person to translate every product change into test code.

This is one of the reasons we believe functional automation should be accessible to the broader team. Tests that cannot be maintained during the sprint are unlikely to keep up with the product.

A large test suite that represents the application from six months ago is worth less than a smaller suite that accurately protects today's critical workflows.

This is also why test maintenance has to happen alongside product development rather than in a separate phase after the sprint. How Testing Keeps Up With Development explores that problem in more detail.

Communicate with affected customers

When a production defect affects customers, silence creates its own damage.

Customers do not expect software to be literally incapable of failing. They do expect the company operating it to respond honestly and competently.

The communication should match the severity of the incident.

For a small issue affecting a few known users, direct communication may be enough. For a widespread outage, use the status page, in-app notifications, email, or the channels customers already rely on.

A useful incident update explains:

What users are experiencing
Which users or workflows are affected
Whether there is a temporary workaround
What the team has done so far
Whether data or security is affected
When the next update will be provided

Avoid speculation. Do not announce a root cause before it has been confirmed.

Also avoid empty language such as "some users may be experiencing degraded performance" when the application is clearly unavailable. Customers recognize language designed to minimize the appearance of a problem.

Trust is better served by accuracy.

After the issue has been resolved, tell affected customers what changed. The explanation does not need to expose sensitive implementation details, but it should demonstrate that the team understood the problem and took steps to prevent a recurrence.

For serious incidents, a service credit or other concrete gesture may be appropriate. The correct response depends on the impact, the contract, and the relationship with the customer.

Production defects create brand damage

A defect is not only an engineering cost.

It creates support tickets, interrupts sales conversations, distracts product teams, delays other releases, and forces leadership to explain what happened.

Repeated defects also change how customers use the product.

They begin double-checking results manually. They avoid recently released features. They keep backup tools available. They hesitate before expanding usage to another team.

At that point, the product may still technically retain the account, but trust has already started eroding.

For subscription software, this can eventually increase churn.

Customers rarely cancel because of one harmless visual issue. They do cancel when recurring defects make the product feel unreliable, especially when those defects affect core workflows.

Reliability compounds in the same way product quality does.

A company that responds well, communicates clearly, and prevents repeated failures can strengthen customer trust after an incident. A company that repeatedly ships avoidable defects and provides vague explanations teaches customers not to depend on it.

Run a blameless post-incident review

After the patch is deployed and customers are no longer affected, the team should review what happened.

Do this while the details are still fresh.

A useful review covers:

The incident timeline
The initial customer impact
How the issue was detected
How it was isolated
The confirmed root cause
Why existing tests did not catch it
What slowed down the response
What limited the impact
Which preventive actions will be taken
Who owns each action and when it should be completed

Avoid turning the review into a document that says "be more careful."

That is not a preventive action.

Useful actions might include adding a functional test, changing a deployment check, improving browser coverage, adding monitoring, clarifying ownership, updating a feature flag, or making a critical workflow easier to test.

The best post-incident actions change the system so that future teams do not need to remember the lesson manually.

Build the regression test into the release process

A production defect should leave behind a permanent test whenever the scenario can be automated.

The test should not sit on someone's laptop or in a branch that will be merged later. It should become part of the normal release process.

For critical workflows, functional tests should run automatically before deployment or immediately after deployment in a controlled environment.

A green check mark has no value when the test avoids the conditions that real customers face. It should not exist merely so that the team can say, "The pipeline was green," after something breaks in production.

The purpose of an automated test is to reduce uncertainty about the product.

That means interacting with the application like a real user, in a real browser, on representative machines, and verifying meaningful outcomes.

Did the order actually get created?

Did the user receive the correct permissions?

Was the file really uploaded?

Did the confirmation appear?

Was the record stored correctly?

Does the workflow still work in Safari?

These questions matter more than the number of tests in the dashboard.

For a deeper explanation of how these tests protect complete workflows, see What Is End-to-End Testing?.

A practical production-defect checklist

When a defect reaches production, use this sequence:

Confirm the report. Capture the exact symptoms and affected workflow.
Assess severity. Determine the impact on users, data, security, and revenue.
Contain the issue. Roll back, disable, or isolate the affected functionality when necessary.
Reproduce it. Match the user's browser, operating system, account state, and data.
Create a failing test. Preserve the scenario before changing the implementation.
Identify the root cause. Explain both the technical failure and why the process allowed it through.
Apply a narrow patch. Avoid unrelated rewrites during the incident.
Run regression tests. Cover the exact defect and adjacent workflows.
Test representative browsers. Include the real browser and operating-system combinations used by customers.
Communicate clearly. Inform affected users without minimizing or speculating.
Monitor the release. Verify that the patch behaves correctly under production conditions.
Review the incident. Assign specific preventive actions with owners and deadlines.
Keep the regression test. Add it permanently to the automated release process.

The real goal is confidence, not perfect software

No testing strategy can guarantee that production will never contain another defect.

The goal is to reduce the number of avoidable failures, limit their impact, respond quickly, and prevent the same defect from returning.

Unit tests, component tests, monitoring, and code review all help.

But none of them removes the need to test the complete product from the perspective of a real user.

That means opening the application in actual browsers, interacting with real workflows, and checking the outcome that matters to the customer.

It also means making those tests accessible enough that the whole team can keep them current throughout each sprint.

When functional tests become an outdated side project owned by one specialist, they stop protecting the release. When they become part of everyday product development, they provide evidence that the software customers depend on still works.

That is worth more than another green icon.

That is also the principle behind Endtest: functional tests should reflect real user behavior, run in the environments customers actually use, and remain maintainable by the wider team.