// work / software qa & testing
Test automation and release QA for a SaaS platform
A QA audit, a Playwright and API automation suite running on every pull request, and a release process that cut escaped bugs by two thirds for a B2B SaaS team with no dedicated tester.
- Client
- B2B subscription software company
- Industry
- SaaS
- Year
- 2025
- Role
- QA lead and automation engineer
- Duration
- 6 weeks, then monthly retainer
Context
A B2B SaaS company with eight developers and no QA function was shipping weekly. Every release went out after a quick manual click-through by whoever had time, and every second release came back with a customer-reported regression. The team had tried automation twice; both suites had rotted into a folder of skipped tests within a few months.
The problem
Catch regressions before customers do, without slowing the weekly release down, and build something the developers would keep running after I left. The previous suites failed on the last point: they were flaky, slow and nobody owned them.
Approach
Audit first. A week walking the product with support and engineering, reading the last six months of production incidents and mapping them to features. Twelve user journeys accounted for almost all of the customer-facing bugs: sign-up and billing, the import pipeline, permissions, and the reporting screens. That became the priority list.
Test plan with deliberate gaps. A risk-based plan that said what would be automated end to end, what would be covered at the API level, what stayed manual, and what was consciously not tested. Writing the gaps down was as useful as the coverage; it ended the “why didn’t tests catch this?” conversation.
The suite. Playwright in TypeScript, structured around user journeys rather than pages. 42 end-to-end tests for the critical paths, each creating its own isolated test data through the API so tests never depend on each other or on a shared database state. API tests in Postman and code cover the contract and the negative paths the UI cannot reach. A k6 load test gives the reporting endpoints a baseline the team can compare against after each change.
Flake as a metric. Every test runs three times in isolation before it is allowed into the main suite; anything that is not deterministic is fixed or rejected. Flake rate is reported monthly, alongside pass rate and duration. It has stayed under 1%.
CI. The suite runs on every pull request in about eleven minutes, with Allure reports and a screenshot and trace attached to every failure. Nightly runs cover the full matrix on BrowserStack. A failing test blocks the merge, which was the cultural change that mattered most.
Release process. A one-page checklist, an exploratory pass on each release by me, and a short note on what was covered. Bugs go to Jira with steps, environment, severity and evidence attached, so the developer can start on the fix rather than on reproduction.
Outcome
Customer-reported regressions fell by about two thirds over the first quarter, measured against the previous six months of support tickets. The release cadence stayed weekly. Six months on, the developers are adding tests to the suite themselves as part of feature work, which is the outcome the two earlier attempts never reached. The engagement continues as a monthly retainer for release passes and suite maintenance.