Parallel test execution can shorten a CI pipeline, but adding workers alone does not guarantee a faster or more reliable build. This guide shows how to estimate the useful worker count, compare file-based, duration-based, and work-stealing sharding, account for resource contention, and define rerun rules that do not hide failures. Use the calculations whenever the suite, runner size, test data, or build-time target changes.
Overview
Test sharding divides a test suite into separate groups, or shards, that run at the same time. Each shard normally has its own CI job, environment, browser process, or test worker. The goal is to reduce wall-clock time: the time from the first job starting to the last job finishing.
A simple model is:
Estimated pipeline time = queue time + setup time + longest shard duration + result collection time
This model is more useful than dividing the total test time by the number of workers and assuming the result will scale perfectly. Shards rarely contain exactly equal work. Jobs may spend time installing dependencies, starting services, loading test data, or waiting for an available runner. More workers can also increase CPU, memory, database, browser, and network contention.
There are three common ways to divide automated tests:
- File-based sharding: assigns test files to shards, often using a predictable order or round-robin distribution. It is simple and works well when files have similar runtimes.
- Historical-duration sharding: uses previous execution times to distribute files or tests so that each shard has a similar estimated total. It is usually a better fit for suites with uneven test durations.
- Work stealing: gives new work to whichever worker becomes available. This can balance variable test durations effectively, but it may require framework or test-runner support and careful control of shared resources.
Sharding is not the same as in-process parallelism. A CI matrix may create four jobs, while each job may also run multiple browser workers. Track both levels so that “four shards” does not accidentally become dozens of competing processes.
How to estimate
Start with a baseline from several representative CI runs rather than a single unusually fast or slow run. Record the following values:
- Serial test duration: the time required to execute the selected suite with one worker.
- Setup overhead: installation, service startup, fixture loading, and other work repeated by every shard.
- Candidate shard count: the number of parallel CI jobs you are considering.
- Observed balance: the duration of the fastest and slowest shards.
- Queue and capacity time: time spent waiting for a hosted or self-managed runner.
A first estimate for idealized execution is:
Ideal test time per shard = serial test duration ÷ shard count
Then add repeated setup and a balance allowance:
Practical time per shard ≈ ideal test time per shard + setup overhead + imbalance allowance
The imbalance allowance represents the work left in the longest shard after the other shards finish. You can calculate a simple balance ratio:
Balance ratio = average shard duration ÷ longest shard duration
A ratio closer to 1 indicates more even distribution. Do not treat the ratio as a quality score by itself. A balanced set of shards can still be slow if every job is resource-constrained, and a slightly uneven set may be acceptable when the total pipeline meets its target.
Compare candidate worker counts using a small table. For example:
| Shards | Ideal test time | Repeated setup | Estimated longest shard | Decision |
|---|---|---|---|---|
| 2 | Serial time ÷ 2 | 2 setup instances | Based on observed balance | Baseline parallel option |
| 4 | Serial time ÷ 4 | 4 setup instances | Often lower, if capacity is available | Candidate default |
| 8 | Serial time ÷ 8 | 8 setup instances | May be limited by contention or queueing | Use only if measurements support it |
Use measured wall-clock time to choose the winner. The best worker count is not necessarily the largest one; it is the point at which additional workers provide enough time savings to justify their setup, capacity, and troubleshooting cost.
Inputs and assumptions
Before changing a CI matrix or test-runner flag, make the assumptions explicit.
Suite composition
List the files or test groups included in the shard. Note tests that are unusually long, launch multiple browsers, upload large files, or depend on external services. A file-based strategy can become uneven when one file contains a large scenario or a serial-only group.
Timing data
Historical-duration sharding depends on usable timing data. Decide whether the source is the last successful run, a rolling set of recent runs, or a manually maintained estimate. A rolling view can reflect current behavior, while a single run may be distorted by a transient failure or slow service.
Keep the timing format stable and identify tests by a durable path or test ID. Renaming files, splitting test files, or moving tests between projects can make old timings misleading. When timing data is missing, define a fallback rule, such as assigning unknown tests to a separate group for review.
Runner and service capacity
Count the resources consumed by both CI jobs and in-job workers. Check CPU, memory, browser processes, database connections, test accounts, ports, and storage. If each shard starts a full application stack, the bottleneck may be service capacity rather than test execution.
Isolation model
Parallel tests must not unintentionally share mutable state. Use isolated databases, unique accounts, partitioned test data, or deterministic cleanup. For a practical approach, see the Test Data Management in CI guide. Tests that require a shared singleton resource should be isolated, serialized, or redesigned before increasing concurrency.
Rerun policy
Separate a failed shard from a flaky test. A rerun may help identify an intermittent failure, but it should not convert a failed first attempt into an invisible pass. Preserve the original result, record the rerun outcome, and apply a limit. For triage steps, use the Flaky Test Troubleshooting Guide.
Worked examples
Example 1: choosing between two and four shards
Assume a browser suite takes 40 minutes when run serially. A two-shard run produces jobs lasting 23 and 21 minutes, excluding five minutes of repeated setup. A four-shard run produces jobs lasting 14, 13, 12, and 11 minutes, with the same five-minute setup per job.
The observed wall-clock estimate is approximately 28 minutes for two shards and 19 minutes for four shards, before queue time and result collection. Four shards are faster in this example, but the comparison should also include runner capacity and the cost of starting four application environments instead of two.
Example 2: why equal file counts fail
Suppose 12 files are assigned evenly to four shards, with three files per shard. If one file contains a long checkout journey and the others contain short smoke checks, the shard containing checkout becomes the critical path. Historical-duration sharding can place that long file with shorter files and produce a more even total. If the file’s duration changes frequently, work stealing may be more effective than relying on stale timing data.
Example 3: deciding whether to rerun
A shard fails after 18 minutes. The CI policy permits one targeted rerun, but the original logs and artifacts remain available. If the rerun passes, report the result as “passed on rerun” and open or update a flaky-test record. If it fails again, retain the shard as a genuine failure. Do not rerun the entire suite automatically when only one shard failed unless the failure could affect shared state or invalidate the other results.
For result ownership and artifacts, connect the sharding workflow to a CI test reporting workflow. A dashboard should show shard duration, queue time, retry count, failed tests, and whether a result was original or retried.
When to recalculate
Revisit worker count and shard assignments when any input changes, not only when the pipeline becomes obviously slow. Recalculate after adding a substantial group of tests, changing runner hardware, changing browser versions, moving from self-hosted to hosted runners, modifying application startup, or introducing new database and service dependencies.
Review the model when the build-time budget changes. A team optimizing a pull-request check may prefer a smaller, predictable smoke suite, while a nightly regression run may accept more shards and higher resource use. Keep those objectives separate rather than forcing one concurrency setting onto every workflow. The Smoke Test Pipeline Checklist can help define a smaller release gate.
Use this practical review sequence:
- Export several recent runs, including successful and failed runs.
- Compare total duration, queue time, setup time, and longest-shard duration.
- Calculate the balance ratio and identify the files or tests that dominate the slowest shard.
- Check CPU, memory, browser, database, and service saturation during execution.
- Test one alternative shard count or assignment strategy.
- Compare wall-clock time, failure rate, rerun rate, and operational cost.
- Document the selected worker count, timing-data source, fallback behavior, and rerun limit.
Update the configuration when the evidence supports it, then schedule another review after the next meaningful suite or infrastructure change. Treat sharding as a measured CI design decision—not a permanent setting—and parallel test execution will remain a productivity tool instead of becoming another source of unstable builds.