A reliable CI test suite should be measured as an engineering system, not only as a green or red build. This monthly review template explains how to track test suite health metrics, estimate the cost of instability, identify recurring failures, and turn the results into a focused improvement plan. Use it with Playwright, Cypress, API tests, browser testing tools, GitHub Actions, GitLab CI, Jenkins, or another CI platform by adapting the data-collection details to your environment.
Overview
The purpose of a monthly test-suite review is to answer four practical questions:
- Can the team trust a passing result?
- How much developer and CI capacity does testing consume?
- Which failures are recurring, flaky, or blocking delivery?
- What maintenance work should be prioritized next?
A single pass rate cannot answer these questions. A suite can have a high pass rate while producing frequent false alarms, waiting in a long CI queue, or leaving important product risks untested. Combine reliability, speed, maintenance, and risk-coverage measures instead.
| Metric | Definition | What it reveals |
|---|---|---|
| Pass rate | Passing test runs divided by total completed runs | Overall outcome, but not whether failures are valid |
| Flaky-test rate | Runs that change outcome without a relevant product or code change | How often the suite creates uncertainty |
| Failure recurrence | Share of failures associated with an existing failure pattern | Whether the team is resolving or repeatedly rediscovering problems |
| Execution time | Time spent running tests after workers begin | Suite size, test efficiency, and feedback speed |
| Queue time | Time from job creation until test execution starts | Runner capacity and CI scheduling pressure |
| Quarantine age | Time a disabled or excluded test remains outside normal runs | Whether temporary isolation is becoming permanent |
| Risk-area coverage | Important product risks with at least one relevant automated check | Whether automation supports the right customer journeys |
Review these metrics by test type when possible. A combined number for unit, API, integration, end-to-end, and visual tests can hide the source of a problem. For example, a fast unit-test layer may make the overall execution time look healthy while browser tests are slow and unstable.
How to estimate
Start with a consistent reporting window, such as the previous calendar month. Export completed CI test jobs and retain enough fields to group results by repository, branch, workflow, test type, browser, operating system, commit, and failure reason. Do not mix pull-request runs, scheduled regression runs, and post-deployment smoke tests without labeling them.
Use these basic formulas:
- Pass rate = passing runs ÷ total completed runs × 100.
- Flaky-test rate = runs classified as flaky ÷ total test runs × 100.
- Failure recurrence = failures linked to an existing issue or signature ÷ total failures × 100.
- Median execution time = the middle execution duration after sorting completed runs from shortest to longest.
- Median queue time = the middle wait duration before a runner starts the job.
- Quarantine age = review date minus the date a test was quarantined.
- Risk-area coverage = mapped risk areas with an active automated check ÷ total mapped risk areas × 100.
Use medians for typical execution and queue time, then inspect the longest runs separately. Averages can be pulled upward by a small number of unusually slow jobs. Also report volume alongside percentages. A 10% flaky-test rate means something different across 20 runs and 20,000 runs.
For each metric, record the current value, the previous review value, the direction of change, the owner, and the next action. If your CI platform has limited historical retention, store a monthly summary outside the platform. A compact spreadsheet, database table, or test reporting tool is sufficient if the definitions remain stable.
Inputs and assumptions
Before calculating a scorecard, agree on what counts. Reliability reviews become difficult when teams change definitions from month to month.
Define the unit of measurement
Choose whether a result represents a whole workflow, a job, a shard, a test case, or an individual attempt. For example, a parallel test run may produce several shard results. Count shards consistently for infrastructure analysis, but use test-case outcomes for flaky-test analysis when your reporting data supports that distinction.
Separate retries from first attempts
Retries can improve delivery flow, but they can also conceal instability. Report the first-attempt outcome separately from the final outcome. A workflow that is green only after two retries should not be treated as equivalent to a workflow that passed immediately.
Use an explicit failure taxonomy
At minimum, classify failures as product defect, test defect, environment or infrastructure issue, data or configuration issue, timeout, and unknown. Unknown should be a temporary category with an owner, not a permanent conclusion. Link repeated failures to a common issue or failure signature so recurrence can be calculated.
Map tests to product risks
Coverage by risk area is more useful than a raw test count. Create a small inventory of critical journeys, such as authentication, checkout, permissions, data export, or deployment smoke paths. Map each journey to the tests that validate it, and note whether those tests run on every change, on a schedule, or only before release.
Set local thresholds rather than copying universal targets. A new suite, a mature product, and a heavily parallelized browser suite may have different reasonable baselines. The important practice is to establish a baseline, define an unacceptable trend, and require an explanation when the trend changes.
Worked examples
Assume a team records 1,000 completed test runs during a month. Of those, 930 pass on the first attempt, 50 fail because of known product defects, and 20 are later classified as flaky. The first-attempt pass rate is 93%. If the 20 flaky outcomes are included in the total, the flaky-test rate is 2%.
Now suppose the team had 100 total failures, and 65 were linked to failure patterns already seen earlier in the month. Failure recurrence is therefore 65%. This does not necessarily mean every recurring failure has the same root cause, but it does show that repeated symptoms are consuming attention. The review should prioritize the largest recurring clusters rather than treating every failed test as a new investigation.
For time, imagine that the median execution time is 14 minutes and the median queue time is 6 minutes. The developer-facing wait is approximately 20 minutes before a result is available, excluding setup and reporting overhead. If execution time remains stable but queue time rises over several reviews, adding or reallocating runners may be more effective than rewriting tests. If queue time is stable but execution time rises, inspect test growth, setup work, browser coverage, and parallelization.
A practical monthly scorecard might look like this:
| Metric | This month | Previous month | Action |
|---|---|---|---|
| First-attempt pass rate | 93% | 96% | Review the three largest failure groups |
| Flaky-test rate | 2% | 1% | Assign owners to the top five unstable tests |
| Failure recurrence | 65% | 48% | Prioritize fixes for repeated signatures |
| Median execution time | 14 min | 12 min | Inspect slow suites and shard balance |
| Median queue time | 6 min | 3 min | Check runner demand and scheduling |
| Oldest quarantine | 31 days | 18 days | Remove, repair, or restore the test |
End the meeting with no more than a few committed actions. Each action should name an owner, a due date or review point, the affected tests, and the metric expected to improve. Connect the work to your test suite maintenance checklist, and use a flaky test troubleshooting workflow for unstable cases.
When to recalculate
Run the scorecard monthly, but recalculate sooner after changes that can alter the inputs. Revisit it after changing CI runners, worker counts, parallel test sharding, retry policies, browser versions, test-reporting integrations, retention limits, or the structure of a test framework. Pricing changes for hosted CI or browser testing can also make execution and queue time more important to monitor from a capacity-planning perspective.
Revisit definitions after a major product release, repository split, migration between GitHub Actions, GitLab CI, Jenkins, or another CI platform, or a significant change in the balance of unit, API, integration, and end-to-end tests. Preserve the old definition in the report when possible so a change in measurement is not mistaken for a change in quality.
For the next review, take these steps:
- Export one consistent month of CI results and separate first attempts from retries.
- Calculate the seven core metrics using documented formulas.
- Compare each result with the previous month and inspect the largest changes.
- Group failures by signature, owner, and risk area.
- Choose a small set of corrective actions and record expected outcomes.
- Update the scorecard when CI configuration, tooling, or reporting inputs change.
Use the scorecard as a decision aid, not a ranking system. Its value comes from making reliability work visible, repeatable, and connected to developer time and product risk. For implementation details, pair this review with a CI test reporting workflow, a parallel test sharding guide, and a test tagging strategy for selecting the right tests at the right stage.