A suite nobody trusts
is worse than no suite.
It just costs more.
Once a third of failures are noise, engineers re-run the build instead of reading it — and the suite stops catching anything. Our QA engineers fix the shape and the flake rate first. Full-time in your team, on the Virtus payroll.
What we usually find, and what we leave behind
Same product, same coverage of the same user journeys. The left-hand suite takes three and a half hours and lies to you a third of the time. The right-hand one takes sixteen minutes and does not.
740 end-to-end tests because it was the only tier anyone knew how to write. Nobody had run the full suite on a pull request in eight months.
The journeys still covered end to end are the ones that take money. Everything else moved down a tier, where it is faster, cheaper and more precise about what broke.
Four rules that get it from 31% to 1%
None of them are clever. They are just applied consistently, which is the part that requires someone whose job it actually is.
Quarantine, do not delete
A flaky test still knows something. It moves to a quarantine suite that does not block the pipeline, with an owner and a date.
Fix the wait, not the timeout
Nine times out of ten a flaky end-to-end test is waiting on a timer instead of on a condition. Raising the timeout hides it for a month.
Control the data
Tests that share a seeded database fail in an order-dependent way. Each test creates what it needs and cleans up after itself.
Track the rate publicly
Flake rate goes on the same dashboard as coverage. Once it is visible it gets fixed; while it is invisible it gets tolerated.
Six things they own
Test strategy, written down
What is covered where, and why. Two pages, agreed with engineering, so nobody writes the same assertion at three levels.
Playwright or Cypress, properly
Page objects, network stubbing where appropriate, parallel execution, and traces on failure so a red build is diagnosable in a minute.
API and contract tests
The tier that catches most regressions for the least runtime — and the one most suites are missing entirely.
Mobile automation
Appium or Maestro on a real device cloud, because the emulator passes tests that the phone in someone's pocket fails.
Performance and load
k6 or JMeter with a scenario that resembles real traffic, run in CI against a threshold rather than by hand before a launch.
Accessibility in the pipeline
axe checks on every page in the suite, so the WCAG conversation happens at pull-request time instead of at audit time.
Three bands
Mid-level
3–5 yrs$24–30/hrWrites and maintains tests in an existing framework. Solid on the tooling, still learning what to test where.
Senior
5–8 yrs$30–38/hrOwns the strategy and the framework. Drives the flake rate down and keeps the pipeline fast enough that nobody skips it.
Lead / SDET
8+ yrs$38–52/hrBuilds the test platform, embeds quality into the definition of done, and works as an engineer rather than a gatekeeper.
How long does your suite take, and how often is it wrong?
If you cannot answer the second half, that is the finding. Send us the last fifty CI runs and we will work it out.