CI/CD Test Automation: The Real Maturity Check

Liviu Lupei

Founder & Solutions Architect, Endtest · August 25, 2026

There is a point where an automated test suite stops being a collection of scripts and starts becoming part of your engineering process.

That point is usually when you put it in the CI/CD pipeline.

Before that, things can look surprisingly good.

Someone creates a few Selenium or Playwright tests. They run them on their laptop. Everything passes. They show the team. Maybe they even run the entire suite a few times without a failure.

Great.

But a laptop is a very forgiving place to run automated tests.

You control the browser version. You control the screen resolution. You're probably already authenticated. Your IP address already has access to the staging environment. Your machine isn't competing with five other builds for CPU. And if something fails, you can just run it again.

CI/CD removes most of those conveniences.

That's why I think putting automated tests into the pipeline is one of the best maturity checks you can give a test suite.

The pipeline is where tests get audited

A test that works on one person's computer proves something.

A test that runs automatically after every build, on infrastructure nobody is babysitting, proves a lot more.

The pipeline asks questions your local environment doesn't:

Can this test start from a clean state?

Can it authenticate without a person helping it?

Can it find the right test data?

Can it survive a slower environment?

Can it upload and download files?

Can it run at the same time as other tests?

Can it produce enough evidence for somebody else to understand a failure?

And, perhaps most importantly:

Can the team trust it enough to let it influence a release?

That's a much higher bar.

I've seen plenty of automated test projects that looked healthy until they were connected to CI. Suddenly the pass rate dropped, execution times became unpredictable, and failures appeared that nobody had seen locally.

That's not CI breaking the tests.

That's CI exposing the tests.

Your CI runner should orchestrate tests, not become a browser lab

One pattern I don't love is putting the entire browser-testing infrastructure directly inside the CI runner.

A GitHub Actions job starts.

It installs Node or Python.

It installs Playwright or Selenium dependencies.

It downloads browsers.

It starts a headless browser inside a Linux container.

Then it runs the complete end-to-end suite there.

Technically, this works.

For some tests, it's perfectly reasonable.

But I wouldn't make that the only environment responsible for deciding whether a web application is ready for real users.

A better architecture is to let CI/CD orchestrate the tests while dedicated testing infrastructure executes them.

Something like this:

New build
    ↓
Deploy test environment
    ↓
CI/CD pipeline sends test command
    ↓
Dedicated Windows / macOS test machines
    ↓
Real browsers execute the tests
    ↓
Results, screenshots, logs and artifacts
    ↓
Pipeline decides whether to continue

Your GitHub, GitLab, Jenkins, Azure DevOps, or other runner doesn't need to become the browser infrastructure itself.

It can simply call an API or execute a lightweight command that says:

Run regression suite 42 against this environment.

That separation turns out to be useful.

Your build infrastructure can change without changing your browser infrastructure. Your test machines can be upgraded independently. You can control browser versions, resolutions, concurrency and operating systems in one place.

And your pipeline remains relatively boring.

Boring pipelines are good.

Run important tests where your users actually run the app

Headless Chrome on Linux is fast and useful.

I wouldn't get rid of it.

It's excellent for fast checks where you want feedback in a few minutes.

But there is a dangerous leap between:

Our application works in headless Chromium inside a Linux container.

and:

Our application works for our customers.

Those aren't always the same thing.

If your customers use Chrome and Edge on Windows, you should test meaningful flows there.

If they use Safari, at least part of your release process should involve real Safari on macOS.

This matters for things such as fonts, rendering, scrolling, browser APIs, file downloads, native behavior, viewport handling and browser-specific JavaScript behavior.

The goal of automated testing isn't to make the pipeline green.

The goal is to catch situations that would otherwise make users unhappy.

A test suite that passes beautifully in an environment none of your customers use can create a dangerous sense of confidence.

That's why, for release-critical end-to-end tests, I prefer real browser testing on Windows and macOS rather than relying exclusively on headless Linux containers.

You can still keep fast containerized tests earlier in the pipeline.

Think of them as different layers, not competing philosophies.

What usually breaks when tests move into CI/CD

This is where the maturity check gets interesting.

There are a few problems that show up over and over.

1. Rate limiting

Your application may happily accept a person clicking through checkout once.

Then your pipeline runs 40 tests in parallel.

Suddenly the authentication service, API, payment sandbox or third-party provider starts returning 429 Too Many Requests.

The test didn't necessarily discover a product bug.

It discovered that your automated testing architecture behaves differently from a human.

The fix might involve controlling concurrency, creating dedicated test accounts, coordinating test data, or working with the service's rate limits instead of blindly adding retries.

2. Access denied to test environments

The developer's laptop can reach staging.

The CI runner cannot.

Maybe staging is behind a VPN. Maybe there is an IP allowlist. Maybe a firewall rule only permits corporate addresses.

This is another reason remote execution architecture matters.

You want stable, known testing machines whose IP addresses and permissions can be managed intentionally.

Environment access shouldn't be rediscovered during every build.

3. File uploads and downloads

File testing has a habit of working perfectly until the tests leave somebody's laptop.

A local test might reference:

/Users/alex/Desktop/invoice.pdf

That file obviously won't exist inside a random CI container.

Downloads introduce another set of problems: temporary directories, browser permissions, generated filenames, cleanup and retrieving the downloaded artifact after the run.

If uploading and downloading files are meaningful parts of your product, they need to be treated as first-class parts of the testing infrastructure.

Not clever workarounds inside individual tests.

4. Flakiness

This is the big one.

The pipeline is often slower than the computer where the test was written.

Animations take slightly longer.

An API responds in 1.8 seconds instead of 700 milliseconds.

A JavaScript component renders in a different order.

The screen resolution changes.

A button moves below the fold.

Five browsers are running simultaneously.

Suddenly that innocent-looking two-second sleep isn't enough anymore.

And this is where teams often make the situation worse by adding more sleeps and more retries.

A mature test suite waits for conditions, not arbitrary amounts of time.

It understands when an element is actually ready.

And when it fails, the failure is treated as information rather than immediately being hidden behind three retries.

5. Test data collisions

Two pipeline jobs create a user with the same email address.

One deletes an account while another test is still using it.

A test assumes the database starts empty.

Another assumes last night's test data is still there.

Tests that depend on shared mutable state tend to become much more fragile as soon as parallel execution arrives.

Unique data, reliable setup and reliable cleanup matter much more in CI than they do on a laptop.

6. Secrets and authentication

Passwords, API keys and authentication tokens shouldn't live in test code.

In CI/CD, credentials need to come from a proper secrets system.

Authentication flows also need to survive clean sessions.

If a test only passes because the person who created it still has a valid browser cookie from Tuesday, it isn't really automated yet.

7. Resource contention

There is a strange class of flaky tests that aren't really flaky at all.

The machine is flaky.

CPU gets saturated.

Memory gets tight.

Four browsers compete for resources.

The page takes twice as long to render.

Then someone spends two days rewriting selectors.

Dedicated execution capacity can eliminate a surprising amount of this noise.

Retries are useful. Invisible retries are dangerous.

There is nothing inherently wrong with retrying an infrastructure failure.

Networks fail.

Browsers occasionally crash.

Services occasionally return something strange.

But retries should not turn a failing test into a passing test without leaving evidence behind.

If a test fails the first time and passes the second time, that is still useful information.

A team should be able to see:

• the original failure;
• whether a retry happened;
• screenshots or video from the failed attempt;
• the page state;
• logs;
• how frequently the same test needs retries.

Otherwise, your CI dashboard says everything is green while reliability quietly gets worse every month.

Eventually developers learn that "green" doesn't mean much.

And once engineers stop trusting automated tests, rebuilding that trust is much harder than building the tests in the first place.

Not every test needs to block every pull request

Putting automated tests in CI/CD doesn't mean running 900 browser tests before somebody can merge a typo.

The test strategy should match the stage of the pipeline.

A practical setup might look like this:

Pull request: fast smoke tests covering the most important functionality.

After deployment to staging: critical end-to-end user journeys.

Main branch or release candidate: broader regression coverage across your important browsers and operating systems.

Scheduled runs: longer combinations, edge cases and less critical regression scenarios.

You want feedback early without making developers wait unnecessarily.

But you also want a point before production where the application is tested like the application your customers will actually use.

This is where cross-browser testing becomes much more valuable than simply running the same headless browser 50 times.

CI/CD also tests whether your automation is keeping up

There is another reason I like using CI/CD integration as a maturity check.

It exposes whether the automated tests are actually keeping up with development.

A company can say:

We have 700 automated tests.

That sounds impressive.

But if those tests validate the application from three sprints ago, the number doesn't mean very much.

When the tests are part of the pipeline, there is pressure to keep them current.

A developer changes onboarding.

The onboarding tests need to change.

The checkout flow changes.

The checkout tests need to change.

A new feature ships.

Someone needs to decide what automated coverage it requires.

That is healthier than having one automation engineer maintaining a separate backlog that is permanently one release behind the product.

The pipeline makes outdated automation visible.

A failing test becomes harder to ignore

There is also an organizational effect that doesn't get discussed enough.

A local test can fail quietly.

Someone runs it.

It fails.

They run it again.

Maybe they fix it next week.

Maybe nobody ever knows.

A pipeline failure has a history.

Build 381 passed.

Build 382 failed.

Here is the test.

Here is when it started failing.

Here is the screenshot.

Here is the log.

Here is the commit associated with it.

Suddenly the test has accountability.

That's a good thing.

Putting automated tests in the pipeline puts both the tests and the people maintaining them in the spotlight.

Weak tests become obvious.

Outdated tests become obvious.

Flaky tests become annoying enough that somebody eventually has to fix them.

And strong tests become something the engineering team genuinely depends on.

Make failures cheap to investigate

If automated tests are going to stop releases, they need to explain themselves.

A message that says:

ElementNotFoundException

isn't enough.

A useful failure should give the team enough context to make a decision quickly.

Ideally that includes the failed step, browser and operating system, screenshot, video, logs, page source, timestamps and environment information.

You shouldn't need the original test author sitting next to you to understand what happened.

This is one of the reasons we've built Endtest so a CI/CD system can simply trigger executions through the API, while the actual browser execution and evidence collection happen on dedicated testing infrastructure.

The build system asks for the test.

The testing system handles the test.

That separation becomes increasingly valuable as suites grow.

What mature CI/CD test automation looks like

You probably have a reasonably mature setup when:

• tests can start from clean, repeatable conditions;
• important end-to-end flows run on the browsers and operating systems your customers use;
• your CI runner triggers tests instead of carrying the entire browser infrastructure itself;
• test data can survive parallel execution;
• uploads, downloads, authentication and external services are handled deliberately;
• flaky tests are investigated instead of endlessly retried;
• failures contain enough evidence for someone else to diagnose them;
• automated coverage changes during the same sprint as the product;
• the team trusts a red test enough to investigate it;
• and the team trusts a green build enough to ship it.

That last part is the entire point.

The pipeline is a test of the tests

The first milestone in test automation is creating a test.

The second is making it pass.

But neither one tells you whether you have a mature test automation system.

Put it in the pipeline.

Run it repeatedly.

Run it without the person who created it.

Run it on clean machines.

Run it after every meaningful product change.

Run important flows on the actual browsers and operating systems your customers use.

Then see what happens.

CI/CD has a habit of exposing every shortcut hiding inside a test suite.

That's useful.

Because the best automated tests aren't the ones that look impressive during a demo.

They're the ones your team forgets to worry about because they run every day, catch real regressions, leave useful evidence when something breaks, and quietly give everyone more confidence to ship.