A CI test report should do more than confirm that a job failed. It should show which tests ran, preserve the evidence needed to diagnose failures, identify ownership, and give reviewers a clear next action. This guide presents a practical reporting workflow for JUnit results, HTML reports, traces, screenshots, videos, and hosted dashboards, with a recurring review process for keeping the system useful as your test suite changes.
Overview
Automated test result reporting is the layer between a test runner and a development decision. Without it, engineers often search through long CI logs to answer basic questions: Did the test start? Which browser or environment was used? Is the failure new? Was the problem in the application, test data, infrastructure, or the test itself?
A useful CI reporting workflow separates three outputs:
- Machine-readable results: Structured files such as JUnit XML that CI platforms can parse and summarize.
- Human-readable reports: HTML, Allure, or hosted reports that provide steps, timings, retries, and failure details.
- Diagnostic artifacts: Screenshots, videos, traces, browser logs, API responses, and other files captured around a failure.
These outputs serve different audiences. A pull request needs a short pass-or-fail summary. A developer investigating a regression needs the exact failing assertion and reproduction context. A QA or engineering lead reviewing test health needs trends such as duration, failure frequency, retry rate, and flaky-test volume.
Start with a small, dependable workflow rather than enabling every report format at once. For most teams, that means publishing JUnit results on every relevant CI job, retaining failure artifacts, and linking to a richer report when the test framework supports one. The same approach works with browser testing tools, API testing in CI/CD, unit tests, and end-to-end testing frameworks.
What to track
1. Test results in a standard format
Use a machine-readable format, commonly JUnit XML, as the baseline. The report should distinguish passed, failed, skipped, and, where supported, retried tests. It should also include suite names, test names, durations, and failure messages.
Keep result files separate by job, browser, operating system, or shard when those dimensions matter. A single merged file can be convenient for a summary, but separate files make it easier to locate failures and compare environments. Use stable naming such as results/chromium/shard-1/junit.xml rather than names based only on timestamps.
2. A readable report for investigation
HTML reports are often sufficient for a local or internal workflow. They can expose steps, hooks, retries, timings, and error output without requiring engineers to interpret raw XML. Allure and similar reporting systems can add history, categories, attachments, and trend views. Hosted reporting options may add searchable runs, pull-request integration, and centralized retention.
Choose a report format based on the question your team needs to answer. If the main need is CI status, JUnit may be enough. If developers regularly investigate browser failures, a framework-specific HTML report or hosted dashboard can reduce diagnosis time. Avoid maintaining several overlapping reports unless each has a defined purpose.
3. Evidence from failed tests
Capture artifacts that explain what the test saw at the moment of failure. A practical default for browser-based end-to-end tests includes:
- A screenshot after a visible assertion fails.
- A trace or step timeline when the framework supports it.
- A video for selected failures or for tests that are difficult to reproduce.
- Browser console output and relevant network information.
- Application logs or server-side request identifiers when available.
Do not capture large files indiscriminately if they make CI slow or exceed artifact limits. A balanced policy is to keep lightweight metadata for every test and richer evidence for failures, retries, or a deliberately sampled set of successful runs.
4. Ownership and classification
A report becomes actionable when each failure has a clear next step. Add metadata for the owning component, test type, environment, browser, shard, and failure category where your tools allow it. Useful categories include:
- Product defect
- Test defect
- Test data or environment issue
- Dependency or service failure
- Timeout or performance issue
- Suspected flaky test
Ownership should describe who triages the failure, not necessarily who wrote the test. A checkout test may be owned by the checkout team even if the shared QA group maintains the test framework.
5. Pull-request summaries
Every CI test job should produce a concise summary that answers: what ran, what passed, what failed, how long it took, and where the detailed evidence is stored. Keep the summary short enough to scan in a pull request. Link to the full report and artifacts instead of copying complete stack traces into the discussion.
| Reporting option | Best used for | Strengths | Watch for |
|---|---|---|---|
| JUnit XML | CI summaries and release gates | Portable, structured, widely parsed | Limited diagnostic context on its own |
| HTML report | Local or CI investigation | Readable steps, errors, and timings | May require artifact publishing and retention planning |
| Allure | Rich history and attachments | Useful categories, trends, and evidence | Needs consistent result generation and report hosting |
| Hosted reporting | Shared dashboards and team workflows | Search, collaboration, and centralized history | Review access, retention, integrations, and cost before adoption |
Cadence and checkpoints
Reporting quality changes as the test suite, CI platform, and retention settings change. Treat it as an operational workflow, not a one-time configuration.
Every test run
- Publish a machine-readable result, including when tests fail.
- Upload the report and failure artifacts even if a later CI step is skipped.
- Record the commit, branch, environment, browser, shard, and test command.
- Make the failure summary visible from the job or pull request.
Weekly or during active development
- Review repeated failures and confirm that each has an owner.
- Check whether retries are hiding failures rather than helping diagnosis.
- Look for tests whose duration is increasing steadily.
- Remove obsolete artifacts, duplicate reports, and unused integrations.
For recurring failure analysis, use the workflow in the flaky test troubleshooting guide. A report should preserve enough evidence to support triage, but it should not replace a deliberate investigation.
Monthly
- Compare failure counts by suite, component, browser, and environment.
- Review the proportion of failures classified as product, test, data, or infrastructure issues.
- Check median and slowest job durations, including the impact of parallel execution.
- Confirm that artifact links still work for the period your team needs to investigate regressions.
- Review whether test data setup or shared environments are creating misleading failures.
If test setup is a recurring source of noise, revisit test data management in CI and verify that fixtures, seeds, and isolated environments are represented in the report context.
Quarterly or after a major workflow change
- Validate reporter integrations after changing the test framework, CI provider, or runner version.
- Review artifact size, storage limits, access permissions, and retention rules.
- Confirm that sharded jobs merge results without losing test names or attachments.
- Reassess which suites block a merge and which are informational.
- Check that reports still reflect the way the team owns and maintains the product.
Teams that add parallel test execution should make this review part of the rollout. The guidance in this parallel test execution guide can help separate faster feedback from confusing or incomplete reporting.
How to interpret changes
A change in the report is a signal, not automatically a defect. Start by comparing like with like: the same suite, environment, browser set, branch type, and reporting configuration. A higher failure count may reflect broader coverage, more parallel jobs, a newly enabled browser, or a change in retry behavior.
Use a simple investigation sequence:
- Confirm the scope. Determine whether the change affects one test, one suite, one environment, or the whole pipeline.
- Check the first occurrence. Compare the first failing commit with the last known successful run.
- Read the evidence. Open the screenshot, trace, logs, and network details before rerunning repeatedly.
- Classify the failure. Assign a product, test, data, infrastructure, or dependency category.
- Record ownership and next action. A failure without an owner tends to become permanent CI noise.
Watch for patterns rather than isolated numbers. A test that fails intermittently across unrelated commits may need flaky-test fixes. A group of tests failing after a schema or fixture change may point to test data. A sudden increase in timeouts across browsers may indicate an environment or application performance issue. When visual checks are involved, separate intended UI changes from rendering differences and use a documented approval process; see the visual regression testing guide for a broader workflow.
Also distinguish test failure rate from pipeline health. A pipeline can have a low test failure rate but still be difficult to use if reports are missing, artifacts expire too quickly, or failures are hidden behind retries. Track the usability of feedback as well as the outcome of the tests.
When to revisit
Revisit this reporting workflow on a monthly or quarterly cadence, and immediately after changes that affect what a test result means. The most important triggers are:
- A new test runner, browser, language, or CI provider is introduced.
- Suites are reorganized, renamed, split, or moved into parallel jobs.
- Artifact retention, storage quotas, permissions, or log policies change.
- The team adopts a new dashboard, hosted reporting tool, or pull-request integration.
- Retries are increased, reduced, or enabled for additional test types.
- A regression investigation shows that screenshots, traces, logs, or test data context were missing.
- Failure ownership changes because teams or product boundaries have been reorganized.
Use this short checkpoint at the next review:
- Run a representative CI job and confirm that results, reports, and artifacts are all published.
- Open a recent passing run and a recent failing run from the links shown to reviewers.
- Verify that a developer can identify the test, environment, owner, and next action without reading raw CI logs.
- Check one artifact from each supported browser or shard.
- Remove reports that no longer answer a defined question.
- Update the workflow documentation and notify the people who triage failures.
For teams building their first workflow, begin with JUnit results, failure screenshots or traces, and a pull-request summary. Add HTML history, Allure, hosted dashboards, videos, or richer analytics only when the team can maintain them. The goal of CI test reports is not to collect more files; it is to shorten the path from a failed check to a well-owned decision.