# The 5 Stages of Test Automation Maturity A practical framework for judging test automation maturity, from fragile local scripts to reliable, shared, cross-browser release confidence. Published June 17, 2026 by Liviu Lupei on the Endtest blog. Canonical URL: https://endtest.io/blog/the-5-stages-of-test-automation-maturity --- Most teams talk about test automation as if it were binary. You either have automated tests or you do not. A manager asks, "Do we have automated testing?" Someone says yes because there is a Playwright repository with 180 test files in it. A dashboard exists. A few green screenshots were shown in a demo six months ago. The box gets checked. But the team still spends two days manually testing every release. The automated suite is not part of the deployment pipeline. Only one person understands it. Half the tests are skipped. Nobody trusts the results enough to delay a release. Production defects continue to appear in workflows that were supposedly covered. Technically, the company has test automation. Operationally, it does not. Test automation maturity is a progression. A team can have thousands of automated steps and still be at an early stage. Another team can have a smaller suite that reliably protects its most important customer journeys and be much further ahead. A better question than "Do we have automated tests?" is: > What stage is our test automation in, and what would move it to the next one? I have worked in software testing since 2012, first as a manual tester and later building Endtest. Over that time, I have seen the same progression repeat across small startups, growing SaaS companies, and large organizations. The tools change. Selenium gives way to Cypress. Cypress gives way to Playwright. AI gets added to the conversation. But the organizational problems remain surprisingly consistent. Here is the five-stage framework I use to think about test automation maturity. ## The five stages at a glance | Stage | Description | CI/CD integration | Ownership | Browser coverage | Release confidence | |---|---|---|---|---|---| | **1. Almost Ready** | Tests exist, but they do not influence releases | None | One person | Inconsistent | Very low | | **2. Useful on Demand** | Tests occasionally provide useful feedback | Partial or scheduled | One or two people | Usually Chrome | Low | | **3. Part of the Pipeline** | Critical tests run automatically with deployments | Yes | Small automation group | Mostly Chrome | Moderate | | **4. Broad Release Confidence** | Reliable coverage extends across browsers and teams | Strong | Shared across QA and development | Chrome, Firefox, Edge, Safari | High | | **5. Automation as a Team Capability** | Testing is a dependable part of how software is built | Continuous | Broad, shared ownership | Full relevant coverage | Very high | These stages are not formal industry standards. The indicators are intentionally practical rather than academic. A team can also sit between stages. You might have Stage 4 browser coverage but Stage 2 ownership. You might have excellent CI integration but poor failure analysis. The goal is to find the weakest part of the system, the one limiting confidence, rather than to obtain a perfect label. ## Stage 1: Almost Ready This is the most common stage. The team has automated tests, but the tests live beside the real development process rather than inside it. ### Typical indicators - Automated tests are not required in the CI/CD pipeline. - One person created and maintains most of the suite. - Tests are run manually before selected releases. - A large percentage of failures require rerunning the suite. - The automation project is described as "almost ready." - The team still performs broad manual regression testing. - Production defects regularly escape from supposedly covered areas. - Most automation conversations are about tools, frameworks, and architecture. ### What it looks like There is usually a repository. It may contain Playwright, Selenium, Cypress, or a mix of older experiments. There are page objects, helper functions, environment files, custom fixtures, reporting libraries, and a README that has not been updated since the second developer joined the project. The suite appears substantial when measured by files or lines of code. But it does not reliably answer the question the business actually cares about: > Can we release this version safely? The tests may work on the automation engineer's laptop. They may even work in a dedicated environment after a few retries. But they are not dependable enough to become part of the release process. Only one team member knows how to investigate failures. When that person is on vacation, the suite effectively stops existing. Meanwhile, manual testers carry the real release risk. They repeat login checks, account creation, checkout flows, subscription upgrades, permissions tests, and other regression scenarios by hand. The organization tells itself that automation will soon reduce this work. It has often been telling itself that for a year. ### What to focus on The goal at Stage 1 is to make a small number of tests operationally useful rather than to add more tests. Start by identifying five to ten workflows that would cause meaningful customer or business damage if they stopped working. Examples might include: - A new customer creating an account - An existing customer logging in - A payment completing successfully - A subscription being upgraded - A user inviting a colleague - A critical report being generated - A customer receiving a confirmation email Get those tests running automatically in a stable environment. The suite does not need to cover everything. It needs to produce a result that another team member can understand and trust. Stage 1 teams often make the mistake of treating framework completeness as progress. They spend weeks deciding how page objects should inherit from base classes, how fixtures should be organized, or whether the company should move from Selenium to Playwright. Those decisions are not meaningless. But they are secondary. A mediocre test that runs on every deployment and catches a real regression is worth more than a beautifully abstracted test framework that is still "almost ready." ### Common traps **Debating Playwright versus Selenium instead of shipping useful coverage** Tool debates feel productive because they are technical and measurable. But choosing the perfect framework does not solve unclear ownership, unstable environments, missing test data, or the absence of CI integration. **Automating low-value scenarios first** Teams often begin with whatever is easiest to automate rather than what is most important to protect. A hundred tests for settings screens can create impressive numbers while checkout remains manual. **Building an internal platform too early** At this stage, the team needs reliable feedback, not a generalized testing framework capable of serving every future project. **Treating manual testing as the problem** Manual testing is carrying the release because automation is not ready. Removing manual coverage before automation becomes dependable only increases risk. ## Stage 2: Useful on Demand At Stage 2, the suite sometimes earns its keep. It is still not a dependable release gate, but it catches real issues and saves time in specific situations. ### Typical indicators - Tests run nightly, on demand, or for selected pull requests. - One or two people maintain most of the suite. - The team has a small set of reasonably stable critical tests. - Failed tests still generate a meaningful number of false alarms. - Results are reviewed, but they do not automatically block deployment. - Browser coverage is usually limited to Chrome or Chromium. - Manual regression remains the final source of release confidence. ### What it looks like The team has moved beyond experimentation. A developer can trigger the suite. A scheduled run produces a report. Some failures are real and lead to bugs being fixed before release. This is an important improvement. For the first time, test automation is producing visible outcomes rather than only code. But trust is inconsistent. When the suite is green, everyone feels better. When it is red, the first assumption is often that the test is wrong. The tests may also be disconnected from normal development timing. A nightly run that fails at 2:00 a.m. might not be investigated until the afternoon. By then, several unrelated changes have been merged and identifying the cause becomes slower. The team has automation, but feedback arrives too late or too inconsistently to shape daily behavior. ### What to focus on The next step is to shorten the distance between a code change and a trustworthy test result. Move the most important and stable tests into the CI/CD pipeline. They do not need to run after every small commit if execution is still slow, but they should run automatically at a meaningful point before release. At the same time, establish simple ownership rules: - Who investigates a failure? - How quickly should a failed test be reviewed? - When is a flaky test fixed? - When is a test removed? - Who can update tests after a product change? A test suite with no response process is only a notification system. Teams should also begin measuring outcomes instead of activity. Useful indicators include: - How many releases received automated feedback? - How long did failures take to investigate? - How many real defects were caught before production? - Which critical workflows remain entirely manual? - How often was a failure dismissed as flaky? - How much manual regression time was actually removed? The number of automated test cases can be useful, but it is not the primary measure of maturity. ### Common traps **Moving every test into CI at once** This often creates a wall of red failures that developers learn to ignore. Start with the tests that deserve trust. **Using retries to hide instability** Retries can help with intermittent infrastructure problems. They can also make an unreliable suite look healthier than it is. **Keeping automation isolated inside QA** Developers do not need to write every test, but they should understand how product changes affect the suite. If automation remains a separate department's problem, maintenance will always lag behind development. **Assuming Chrome coverage means web coverage** Chrome may represent most of your users, but it does not prove that the application works correctly in Safari, Firefox, or Edge. That becomes the central limitation of the next stage. ## Stage 3: Part of the Pipeline At Stage 3, automated testing becomes part of how the team ships software. Critical tests run automatically. Failures are investigated. A red result can delay a deployment. This is where automation begins to create real release confidence. ### Typical indicators - Critical end-to-end tests run in CI/CD. - Failures can block or delay releases. - Results are available soon enough to influence development. - Several team members can investigate common failures. - Test data and environments are more predictable. - The team has reduced some manual regression work. - Most automated browser coverage still runs in Chrome. - Complex workflows involving email, SMS, or third-party systems remain manual. ### What it looks like The tests are no longer a side project. A pull request or deployment triggers a meaningful set of checks. Developers know when they have broken a critical journey. Product managers and testers can look at a result and understand which customer flow failed. The organization starts to rely on the suite during sprints. This changes behavior. Developers become more careful about breaking stable selectors. Product changes include test updates more often. Test failures are discussed alongside build failures rather than after them. Manual testers also gain time for exploratory work, new-feature analysis, usability, edge cases, and risks that are difficult to automate. But there is often a hidden confidence gap. The suite runs almost entirely in Chrome. The team knows that Chrome works. It assumes the rest. Safari bugs are discovered by customers. Firefox problems are dismissed as uncommon. Edge is treated as Chrome with a different icon, even when enterprise policies, extensions, permissions, or browser-specific behavior create differences. At this stage, automation protects a significant part of the product, but only through a narrow technical lens. ### What to focus on The goal at Stage 3 is to broaden confidence without destroying execution speed. Start by identifying the browsers and devices that matter to your actual users. You do not need to run every test in every browser after every commit. A sensible strategy might look like this: - Run a fast Chrome suite on pull requests. - Run critical flows in Chrome, Firefox, Edge, and Safari before deployment. - Run the broader cross-browser suite on a schedule. - Add targeted browser-specific tests for known risks. - Review production analytics to decide where coverage is justified. This is also the point where the team should make failures easier to understand. A useful failure result should include enough context to answer basic questions quickly: - Which step failed? - What was visible on the page? - Was the expected element missing or merely delayed? - Which browser and environment were used? - What network or console errors occurred? - Did the same test pass in other browsers? - Was the failure caused by the application, test data, infrastructure, or the test itself? The faster the team can classify a failure, the more willing it will be to rely on automation. ### Common traps **Running the entire suite in every browser on every commit** This can make feedback too slow and expensive. Cross-browser maturity is about meaningful risk coverage, not brute force. **Treating WebKit emulation as identical to real Safari** Browser engines provide useful coverage, but real operating systems, devices, permissions, fonts, networking, and browser behavior can still matter. **Leaving test readability behind** A suite can become technically sophisticated while becoming inaccessible to most of the team. If only specialists can understand a failed scenario, ownership remains narrow. **Measuring pass rate without measuring trust** A 98% pass rate sounds excellent. It means less if the remaining 2% fails unpredictably and is routinely ignored. ## Stage 4: Broad Release Confidence At Stage 4, the test suite protects the product across the environments customers actually use. Automation is integrated into delivery, browser coverage is deliberate, and ownership is no longer concentrated in one person. ### Typical indicators - Critical tests run in all relevant major browsers. - Chrome, Firefox, Edge, and Safari are covered intentionally. - Test failures are usually investigated by the same day. - Developers, testers, and other team members contribute. - Tests are readable enough for non-authors to modify. - Manual regression is narrower and risk-based. - The suite covers important API interactions and external dependencies. - Automated results are trusted during release decisions. ### What it looks like The suite has become a shared product asset. A developer changing a workflow updates the relevant tests. A tester can add coverage without redesigning the framework. A failed Safari scenario is not postponed indefinitely because only one person knows how to reproduce it. The team has also stopped thinking about automation purely as browser clicking. Real customer journeys cross system boundaries. A signup flow may involve: 1. Submitting a form in the browser 2. Creating a record through an API 3. Sending a verification email 4. Opening a link from that email 5. Receiving an SMS code 6. Confirming the account 7. Displaying the correct permissions after login At Stage 4, the team starts automating these journeys as complete business processes rather than isolated pages. This produces a different kind of confidence. The question becomes whether the customer can achieve the outcome the product promises, rather than whether a button can be clicked. ### What to focus on The focus now shifts from creating coverage to keeping the system understandable and economical. As suites grow, teams should regularly review: - Which tests protect important risks? - Which tests duplicate lower-level coverage? - Which scenarios are slow without providing unique value? - Which tests fail often enough to create fatigue? - Which flows should run on every change? - Which flows can run before release or on a schedule? - Which manual checks still exist because automation is difficult? - Are those manual checks useful, or merely habitual? Stage 4 teams should also reduce key-person dependencies. Readable test design matters here. So does documentation, but documentation alone will not solve ownership. The test itself should communicate intent clearly enough that another person can understand what business behavior it protects. This is one reason no-code and low-code tools can be a good fit for some teams, while code-first frameworks remain better for others. The right choice depends on who needs to contribute and how much low-level control the organization requires. Whether the test is code matters less than whether the right people can safely maintain it. ### Common traps **Automating every possible variation** More coverage eventually creates diminishing returns. A large suite can become slower, more expensive, and harder to maintain without materially improving release confidence. **Confusing broad participation with no ownership** Shared ownership does not mean nobody is accountable. Someone still needs responsibility for suite health, standards, and long-term direction. **Replacing all manual testing** Exploratory testing, usability review, visual judgment, and investigation of new risks still matter. Mature automation makes manual testing more focused. It does not make human testing obsolete. **Ignoring execution economics** A cross-browser suite that takes four hours and blocks every deployment may be comprehensive but operationally unusable. ## Stage 5: Automation as a Team Capability At Stage 5, automated testing is part of the company's software delivery system rather than a project owned by QA. ### Typical indicators - Tests run multiple times per day. - Critical checks are fully integrated into CI/CD. - Most relevant team members can contribute or make safe updates. - Tests are easy to read, understand, and modify. - Failures contain enough evidence for fast diagnosis. - Chrome, Firefox, Edge, and Safari are covered based on real risk. - Native mobile coverage is included when the product requires it. - End-to-end tests verify email, SMS, API, file, payment, and other external workflows. - Manual testing is focused on exploration and new risk. - Releases are faster because confidence arrives earlier. - Production defects still happen, but recurring critical regressions are uncommon. ### What it looks like The organization no longer celebrates the existence of automated tests. It expects them to work. A product change and its test coverage are usually delivered together. The suite runs throughout the day and provides feedback while the change is still fresh in the developer's mind. Instead of being hidden inside an automation specialist's personal framework, the tests are visible, understandable, and connected to business behavior. When a test fails, the team can usually determine whether it is: - A product regression - An intentional product change - A test-data problem - An environment failure - A browser-specific issue - A third-party service problem - A genuine flaky test The suite also reflects how the product actually works. It can verify that an email arrived, that an SMS contained the correct code, that an API response changed after a UI action, that a downloaded file contains the expected information, and that a user can complete the same critical journey in the browsers the company supports. No single person is holding the system together. That may be the clearest sign of maturity. ### What to focus on Stage 5 is not the end. A mature suite can still decay if the organization stops paying attention. The focus becomes preserving speed, trust, and maintainability as the product and team grow. Useful practices include: - Reviewing slow and flaky tests regularly - Removing coverage that no longer protects meaningful risk - Keeping fast checks separate from broader release suites - Monitoring browser and device usage - Updating tests as part of product changes - Maintaining clear ownership for suite health - Making failure evidence easy to access - Tracking production escapes and adding coverage when patterns emerge - Evaluating whether new tools reduce or merely relocate maintenance work AI can help at this stage by creating initial tests, suggesting assertions, explaining failures, and helping repair outdated steps. But AI does not replace the operating model. A suite with weak ownership remains weak even if an AI agent writes more of it. A flaky test does not become trustworthy because its locator was generated by a language model. A test that never runs in CI provides little release confidence, regardless of how it was created. Maturity comes from the whole system: people, process, coverage, execution, diagnosis, and maintenance. ### Common traps **Assuming maturity is permanent** A fast-growing suite can move backward. New products, acquisitions, team turnover, and architectural changes can reintroduce fragile ownership and unreliable coverage. **Optimizing the test suite instead of the delivery system** Automation exists to help the organization deliver reliable software with less wasted effort, not to produce beautiful reports. **Adding AI everywhere without measuring the result** The relevant question is whether tests are created faster, maintained with less effort, and trusted more often, not whether AI is present. **Allowing success to create bloat** Once automation becomes respected, every team wants more of it. Mature teams remain selective. ## Test automation maturity is not the number of tests Many teams try to measure progress with test counts. They report that the company has: - 2,000 automated test cases - 14,000 automated steps - 93% automation coverage - A 97% pass rate Those numbers can be useful, but they can also hide weak maturity. Two thousand tests that run once per month and require one engineer to interpret them may provide less value than 100 critical tests that run on every deployment and are trusted by the entire team. A better maturity assessment looks at outcomes. Ask: - How quickly does the team learn that a critical flow is broken? - Can the result stop a bad release? - How many people can understand and update the test? - How much repetitive manual regression has actually been removed? - Which customer environments are covered? - Do tests verify complete customer outcomes? - How much time is lost investigating false failures? - Are production defects becoming less repetitive? - Can the suite keep working when its original author leaves? These questions are harder to place on a dashboard. They are also much closer to the reason test automation exists. ## A note about tools Every stage can be reached with several tools. You can build a mature system with Playwright, Selenium, Cypress, WebdriverIO, or a commercial platform. You can also build an unusable system with any of them. Tools influence the cost of moving between stages, but they do not determine maturity by themselves. A code-first framework may be ideal when developers own automation, unusual workflows require low-level control, and the company is willing to maintain the infrastructure. A platform such as [Endtest](https://endtest.io/product) may be a better fit when the team wants broader participation, built-in cross-browser execution, and less framework maintenance. The honest comparison is the total cost of producing and maintaining release confidence, not open source versus paid. That includes engineering time, infrastructure, debugging, adoption, training, execution, and the risk of a suite that never becomes dependable. For teams evaluating code-first options, the article on [Playwright alternatives](https://endtest.io/blog/top-7-playwright-alternatives-2026) covers the tradeoffs in more detail. Teams considering a managed approach can also compare the current [Endtest pricing](https://endtest.io/pricing) with the ongoing cost of an internal framework. ## Where is your team? The point of this framework is to decide what work matters next, not to award yourself a maturity badge. If you are at Stage 1, stop expanding the framework and get a few critical tests running automatically. If you are at Stage 2, improve trust, ownership, and feedback speed. If you are at Stage 3, broaden browser coverage and make failures easier to diagnose. If you are at Stage 4, extend automation across real customer journeys and reduce key-person dependencies. If you are at Stage 5, protect the speed and clarity of the system as it grows. Every stage has a different bottleneck. Trying to solve a Stage 4 problem while you are still at Stage 1 creates impressive architecture and very little confidence. The goal is to make automated testing a reliable part of how your team delivers software.