Test Automation in AI Software Factories: A Practical Guide

Liviu Lupei

Founder & Solutions Architect, Endtest · October 20, 2026

An AI agent picks up a Jira ticket, fixes a bug, commits the code, and deploys it to a test environment. Nobody had to remind it to do any of that.

That's a pretty good result for a task that might otherwise have sat in the backlog for another six months.

But there's a question I'd want answered before the change gets anywhere near production: who checked that it actually works?

Generating code has become surprisingly cheap. Getting reliable evidence that the code behaves correctly is a different problem.

Key takeaways

• Test automation in an AI software factory is a continuous loop of planning, executing, observing, and adapting tests, so that code written by AI agents is verified through the interface, in real browsers and environments like the ones customers use, before release.
• A green checkmark is not evidence of correct behavior unless the assertions check what matters.
• Our results: using this approach, we resolved 680 Jira backlog issues, and roughly 95% passed human review. The other 5% went back with additional instructions, and no task was abandoned.
• Keep a human near the release switch. We run a software factory, not a fully dark one: every change still needs final human approval before production.

What is test automation in an AI software factory?

A software factory is a repeatable process for turning requirements or tickets into working software. AI agents can handle planning, implementation, testing, and parts of the delivery process, with humans stepping in where needed.

Test automation in a software factory is the part that verifies the agents' output. It covers four jobs: deciding what to test, running the tests in realistic environments, interpreting the results, and keeping the tests useful as the product changes.

The more ambitious version is sometimes called a dark factory. The term comes from manufacturing, where highly automated factories can operate without people on the production floor. In Dan Shapiro's description of the dark software factory, the agents produce software without a human reviewing the code.

That sounds appealing until the factory confidently ships a broken checkout flow.

The most practical approach I know is a continuous cycle: plan the tests, execute them, understand the results, and adapt as the application changes. An automated test run that reports a green checkmark is only one part of that cycle.

I'd also start small. A backlog of minor bugs and improvements is a much better place to learn what works than a completely new feature with half the business rules still undecided. If you're new to test automation itself, our guide to the easiest way to automate tests is a good starting point.

What results can an AI software factory deliver?

Here are the numbers from our own use of a software factory at Endtest, a cloud test automation platform:

A change counts as passing human review when it was merged without further instructions to the agent.

The roughly 5% that went back had no single pattern. The issues were diverse: some tickets weren't clear or detailed enough, some changes included UX details that weren't intuitive, and in some cases the agent made an incorrect assumption.

By our estimate, these tasks would normally have taken about 5 person-hours each. With the factory, they took 10 to 15 person-minutes each. Across 680 issues, that is roughly 3,200 person-hours saved (an estimate, not a measured figure).

These numbers describe backlog work, not autonomous development of major new features. We're still evaluating how far to take the approach for larger projects.

How do you plan tests for AI-generated code? (Stage 1: Planning)

It's tempting to point a coding agent at your repository, hand it the specifications, and ask it to generate Playwright tests. That can work, and Playwright MCP is useful for browser interaction. But a test is only as good as the agent's understanding of the product.

Anyone who has maintained a mature application knows how this goes. The docs describe the original design. The code contains exceptions added over several years. And somewhere in the UI there's a confirmation step that nobody thought to document.

Test discovery works better when the agent actually uses the application.

Endtest Bot can explore a website through its interface, including authenticated areas and supported signup flows with email verification. You can give it guidance, but it doesn't need the source repository to navigate the product. It produces regular Endtest tests with readable, editable steps, instead of leaving the entire scenario buried inside a prompt.

That distinction matters in an unattended workflow. If an agent creates 100 tests overnight, somebody should still be able to open one and understand what it's checking. Structured steps, reusable components, variables, and conditional logic are useful long after the initial AI session has ended.

The goal at this stage is to discover meaningful behavior. Generating a large number of shallow tests is easy. Finding the scenarios that might catch a genuine regression takes more thought.

Where should AI-generated tests run? (Stage 2: Execution)

Hypothetical example: an online retailer gets a support ticket saying a customer in Germany can't complete checkout in Safari.

The software factory has already run the checkout test in headless Chromium on a Linux container. It passed. That's useful information, but it doesn't tell you whether the reported problem exists in Safari.

A test environment is a choice, not a neutral detail. Real browsers, operating systems, screen sizes, and locations can expose different behavior. Payment redirects, file downloads, locale settings, and complicated front-end interactions are particularly good reasons to broaden coverage.

This is one reason we run Endtest tests on real browser and operating-system combinations in the cloud. An agent can use Endtest MCP to find existing test suites, start an execution with a particular browser, OS, resolution, and geolocation, then retrieve its result. MCP, the Model Context Protocol, is an open standard that lets AI agents call external tools through a consistent interface.

It also solves an orchestration problem. In a software factory you may not know in advance which tests an agent will create, or which existing suites a change might affect. The central agent needs a way to inspect available coverage and decide what to run next.

If the Safari checkout test passes from Germany, you've narrowed the investigation. You haven't proved the customer is wrong, but you have a much better starting point than "it works on Chrome."

Why can an automated test pass while the app is still broken? (Stage 3: Observation)

Hypothetical example: a test selects Spanish at the start of checkout, finishes the purchase, and finds the order confirmation heading. Every traditional assertion passes.

Except the application switched back to English halfway through.

A test that checks only for the presence of a button or a heading can miss that kind of failure. Deterministic assertions are still important, especially for things like payment totals and order status. But some checks need to evaluate the page more broadly.

AI Assertions can check conditions described in natural language, such as whether the page is still in the user's selected language. Visual regression and accessibility checks add other ways to catch problems that a few DOM assertions wouldn't notice.

Observation also means choosing the next tests intelligently. A small change to account settings may affect login, notification emails, or permissions. Running only the test for the edited screen isn't necessarily enough.

The factory should read the execution results through an API or MCP connection, inspect failures, and use that evidence to decide whether to retry, investigate, send the task back for another fix, or advance it to the next environment. A failure log that nobody reads is just an expensive file.

Why does test maintenance get expensive? (Stage 4: Adaptation)

Test creation gets most of the attention because it's easy to demonstrate. Ask an AI to generate a test, watch the browser move, and you're done in a few minutes.

Then the product changes.

A field moves into a modal. A login screen adds another verification step. An onboarding flow branches based on account type. Suddenly, the tests created last month don't describe the current application.

For a changed locator, self-healing tests can often recover without anyone editing the test. For a redesigned user journey, there's more work involved. The agent may need to explore the new flow, update the scenario, and prove that the revised test actually validates the intended behavior.

That's why maintainability matters more than the number of generated tests. Every test becomes something the factory has to keep useful. A thousand tests that require constant repairs can be worse than a smaller set of dependable ones.

It's also why readable test steps matter. If a flow does need attention, you can inspect and modify it directly instead of asking another AI agent to explain a pile of AI-generated code.

The four stages at a glance

What can go wrong in an autonomous QA loop?

Honest limitations are part of any useful guide, so here are the three problems I'd plan for first.

Cost. An agent that repeatedly generates, executes, diagnoses, and repairs tests can burn through a remarkable number of tokens, especially when it gets stuck. Give it execution limits, retry budgets, and clear conditions for stopping. On Endtest's Pro and Enterprise plans, unlimited AI usage makes that part of the cost more predictable, although the workflow still needs sensible limits.

False confidence. An agent can make a test pass by using a shortcut that a real customer couldn't use. It can also declare victory because every assertion passes, even though the assertions never checked the behavior that mattered. End-to-end tests should interact with the product through the interface wherever possible, with additional API or database checks used deliberately.

User experience. We've all seen applications where every button works and every form submits, yet the whole thing feels irritating to use. An AI might verify that a five-step process can be completed. It is much less reliable at judging whether the process should take five steps in the first place.

That's a good reason to involve a human. They don't necessarily need to click through the feature themselves. Reviewing a video of the automated test can be enough to notice a strange interaction or a flow that feels unnecessarily difficult.

How does our software factory work at Endtest?

We started with a problem that probably sounds familiar: a long backlog of small issues that kept losing priority to bigger projects. Some were minor bugs with workarounds. Others were modest improvements that never quite made it into a sprint.

We now keep a separate sprint for these tasks. A central agent coordinates work across Jira, Confluence, GitHub, Endtest, and other tools. It reads the ticket and relevant documentation, works through an implementation, and commits the changes to a test environment.

The testing work then follows the four stages above. We use Endtest Bot for test discovery and creation, and Endtest MCP to run those tests alongside relevant existing coverage and retrieve the results. If something fails, the agent has evidence to work from instead of simply assuming its code was correct.

Changes go through our development, test, staging, and pre-production environments. We still require final human approval before moving into production.

The outcome was the 680 resolved Jira issues and roughly 95% human-review pass rate described above. We're not running a completely dark factory. We deliberately kept the human approval step.

How does this compare with generating Playwright or Cypress tests?

Code-based generation and a platform with editable steps are different trade-offs, and the right choice depends on your team.

For a broader look at the market, see our roundup of the best AI-powered test automation tools. Whichever approach you choose, the key question is the same: when an agent creates tests overnight, can a person open one in the morning and tell what it checks?

How do you start with test automation in a software factory?

1. Pick a low-risk backlog. Minor bugs and small improvements teach you more than a new feature with undecided business rules.
2. Let the agent explore the real application, not only the repository and documentation.
3. Run on real browsers and locations that match your customers.
4. Combine deterministic and AI-based assertions. Use exact checks for totals and status, broader checks for language, layout, and accessibility.
5. Set retry budgets and stop conditions so a stuck agent can't loop indefinitely.
6. Keep tests readable and editable so people can audit what the agent created.
7. Review test videos for flows that pass but feel wrong.
8. Require human approval before production, at least until you have evidence for your own context that you can relax it.

To decide which numbers to track as the factory scales, see our guide to software quality metrics for CTOs in 2026.

Should a software factory run without human testing?

For repetitive, well-understood work, I'm increasingly comfortable letting agents do most of the implementation and testing. The payoff is easy to see when a backlog starts shrinking without pulling engineers away from more important projects.

But a software factory needs a way to challenge its own output. It has to test the behavior customers experience, recognize when its coverage has become outdated, and know when the evidence isn't good enough to ship.

AI will keep making code generation cheaper. That makes independent test automation more valuable, because someone, or something, still has to verify everything we're producing.

The lights can go out on a lot of routine work. Keep a person near the release switch.

Frequently asked questions

What is an AI software factory?

An AI software factory is a repeatable process in which AI agents turn tickets or requirements into working software, handling planning, implementation, testing, and parts of delivery. Humans step in where judgment or approval is needed.

How do you test code written by AI agents?

Use a continuous loop of planning, execution, observation, and adaptation. Let an agent explore the real application to discover what to test, run the tests on real browsers and operating systems, check the results with both deterministic and AI-based assertions, and keep the tests current as the product changes.

What is a dark software factory?

A dark software factory is a highly automated process in which AI agents produce software without a human reviewing the code. The name comes from manufacturing, where factories can run without people on the floor. Most teams today keep human approval before production.

Can AI agents replace QA engineers?

Not entirely. Agents can create and run tests at a scale that was impractical before, but they are less reliable at judging whether a flow is pleasant to use or whether coverage still matches the product. In our factory, a human still approves every change before it reaches production.

How do you stop AI-generated tests from passing falsely?

Require end-to-end tests to use the product through the interface, add deliberate API or database checks rather than shortcuts, and combine exact assertions with broader AI, visual, and accessibility checks. Humans should also review test videos for flows that pass but feel wrong.

Is Playwright enough for testing in a software factory?

Usually not on its own. Playwright is a strong open-source option, and Playwright MCP lets agents drive a browser, but generated tests tend to be linear scripts, which makes them hard to scale as the suite grows. Full end-to-end flows also need additional components, such as email testing and SMS testing, that you have to add and maintain yourself. And because the tests are generated code, they can become a black box that nobody on the team fully understands. An agentic test automation platform addresses these gaps with readable, editable steps, email verification flows, and cloud browsers.

How many tests should an AI agent create?

Fewer dependable tests are better than many fragile ones. Every generated test becomes something the factory has to maintain, so measure maintainability and the regressions a test can catch, not the raw count.

About the author

Liviu Lupei is a Solutions Architect at Endtest with 15 years of experience in Quality Assurance.

Last updated: October 20, 2026.