Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/accessibility/changelog.mdx
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ sidebar_position: 200

## Week of 2/24/2025

- [Cypress Accessibility Branch Review](/accessibility/core-concepts/comparing-reports) has entered beta. Read the docs to learn how to compare runs and automatically spot newly introduced issues.
- [Cypress Accessibility Branch Review](/accessibility/core-concepts/compare-reports) has entered beta. Read the docs to learn how to compare runs and automatically spot newly introduced issues.

## Week of 1/13/2025

Original file line number Diff line number Diff line change
@@ -29,7 +29,7 @@ Comparing the results from different runs is useful in multiple scenarios.
- **Detecting content issues**: Sometimes content editors can introduce accessibility issues unrelated to code changes. Seeing the example issues presented visually, in context, helps you quickly triage whether you are dealing with a recent code change issue, or a content authorship problem.
- **Reviewing AI-generated code changes**: The increased use of AI to generate and/or review front-end code creates some increased risks of accessibility regressions making it to production. The increase or decrease of accessibility issues when reviewing a pull request helps you understand the impact of the change.
- **Tracing the introduction of issues**: With dropdowns for each run, it's easy to rapidly compare different A and B runs to find the exact commit that introduced a problem.
- **Demonstrating the resolution of issues**: Confirm the effect of your improvements, and share overview with your team to more quickly review code changes.
- **Demonstrating the resolution of issues**: Confirm the effect of your improvements, and share the overview with your team to more quickly review code changes.

## Content of the report

@@ -49,47 +49,8 @@ This organization of the report brings the most significant results to the top -

## How to compare runs

The first step is to get to the Branch Review area of Cypress Cloud, which will let you compare one branch against another - or different runs on the same branch, if needed. We refer to be baseline fun for comparison as the `base` run, and the changes we are comparing with as the `changed` run.

There are a number of ways to get to Branch Review depending on where you are looking at your Cypress results. In all cases, once you've picked a `changed` run, you can adjust the automatically-selected `base` run to any branch or run in the project.

### From a Pull Request

<DocsImage
src="/img/accessibility/core-concepts/branch-review-pull-request.png"
alt="A pull request comment from Cypress with an arrow pointing to the 'View all changes introduced in this branch' link. Test results and some accessibility score and violation information is also visible in the comment. The project has an 86.86% accessibility score, with 9 failed elements. Failed rule counts are 1 critical, 1 serious, 3 moderate, and 0 minor."
/>

Click the "View all changes introduced in this branch" link at the bottom of the Test Results table. You will enter branch review with the current PR's branch and latest run pre-selected as the "changed" run.

### From the run list

Click the branch name associated with the run. This will take you to Branch Review with that branch's newest run pre-selected as the `changed` run.

<DocsImage
src="/img/accessibility/core-concepts/branch-review-run-card.png"
alt="A couple of cards showing various details about Cypress test runs, including pass/fail counts and metadata. An arrow points to the branch name."
/>

### From the project list

<DocsImage
src="/img/accessibility/core-concepts/branch-review-project-card.png"
width="40%"
alt="A card showing the name of an example project and related branches to choose."
/>

The project card shows three active branches for the project. You can click through to any of these to enter Branch Review with the newest run on that branch pre-selected as the `changed` run.

### From the main navigation

<DocsImage
src="/img/accessibility/core-concepts/branch-review-side-nav.png"
width="40%"
alt="The main Cypress Cloud navigation showing Latest Runs, Branches, and Analytics menu items. An arrow points to Branches."
/>

When inside of a project, you can select "Branches" in the main navigation to see a full, filterable list of available branches and choose one to set as the `changed` run.
The first step is to get to the [Branch Review](/cloud/features/branch-review.mdx) area of Cypress Cloud, which will let you compare one branch against another - or different runs on the same branch, if needed.
You can access this area by clicking the branch name associated with a run, or in several other ways. [Learn more about how to compare runs](/cloud/features/branch-review.mdx).

## FAQ

@@ -99,9 +60,11 @@ The best subjects to compare are passing runs that ran similar tests on the same

That said, it still possible and valid to compare runs from different points in time with different sets of test results, as long as you bear in mind all the potential sources of difference between the two runs, which you can evaluate for yourself as you explore the results.

In order to see unified changes for your entire test suite, you need to group all the tests together under a single Cypress run, for each report. Learn more about this in the [Branch Review Best Practices documentation](/cloud/features/branch-review#Best-Practices).

### What is the purpose of the Beta label?

This indicates the feature is ready for use and actively seeking feedback based on real usage of the current implementation. We have a few known issues to work through on our side before we consider this fully production-ready and remove the beta label. These issues only affect a subset of projects -- in most cases everything is working as intended. If you see anything unexpected, please hit the feedback button and let us know.
This indicates the feature is ready for use and actively seeking feedback based on real usage of the current implementation. We have a few known issues to work through on our side before we consider this fully production-ready and remove the beta label. These issues only affect a subset of projects -- in most cases everything is working as intended. If you see anything unexpected, please use the feedback button and let us know.

### Why do I see some views (pages or components) changing from run-to-run?

2 changes: 1 addition & 1 deletion docs/accessibility/core-concepts/how-it-works.mdx
Original file line number Diff line number Diff line change
@@ -15,7 +15,7 @@ Cypress Accessibility seamlessly integrates accessibility testing into your deve
1. **Processing**: When your run completes, the final specs are processed for accessibility checks. Earlier specs have already entered processing and are likely completed by the end of the run.
1. **Organizing Views**: Cypress Cloud groups Views (pages and components) rendered during the run, merging and deduplicating across tests. This ensures page-level results and live DOM snapshots are organized effectively.
1. **Generating reports**: A combined "rule-level" report is created, summarizing all detected accessibility violations.
1. **Displaying results**: Results are displayed in Cypress Cloud in the run details area as well as in [Branch Review](/accessibility/core-concepts/comparing-reports) for comparing runs. Accessibility is also flagged in integrations (e.g., Slack, GitHub/GitLab comments), and accessible via the [Results API](/accessibility/results-api) for optional CI pipeline actions (e.g., failing builds based on unmet standards).
1. **Displaying results**: Results are displayed in Cypress Cloud in the run details area as well as in [Branch Review](/accessibility/core-concepts/compare-reports) for comparing runs. Accessibility is also flagged in integrations (e.g., Slack, GitHub/GitLab comments), and accessible via the [Results API](/accessibility/results-api) for optional CI pipeline actions (e.g., failing builds based on unmet standards).

## About Axe Core®

4 changes: 2 additions & 2 deletions docs/accessibility/get-started/introduction.mdx
Original file line number Diff line number Diff line change
@@ -27,7 +27,7 @@ Cypress Accessibility adds detailed accessibility checks and workflows in Cypres
label="See a demo"
icon="action-play-small"
className="!mb-2 mr-1"
href="https://www.loom.com/share/eedb1d18c22942d39a5ed2471ae72788?sid=e39f791e-c0b3-4efc-9770-cbfac403c3b9"
href="https://on.cypress.io/accessibility-demo-video"
/>
<Btn
label=" Explore an example project"
@@ -75,7 +75,7 @@ From there, you can integrate with CI to set your own standards for handling the
href="/accessibility/guides/detect-changes"
aria-labelledby="card-title-8"
>
<Icon name="chart-line" />
<Icon name="shield-halved" />
<h3 id="card-title-8">Detect and manage changes</h3>
<p>
Compare reports in detail to review only _new_ accessibility issues, use
4 changes: 2 additions & 2 deletions docs/accessibility/guides/detect-changes.mdx
Original file line number Diff line number Diff line change
@@ -11,7 +11,7 @@ sidebar_position: 70

Cypress Accessibility supports three main paths to observing and acting on changes in the accessibility report for your projects:

- Run comparisons with [Branch Review](/accessibility/core-concepts/comparing-reports)
- Run comparisons with [Branch Review](/accessibility/core-concepts/compare-reports)
- The [Results API](/accessibility/results-api) to fail builds or trigger alerts when results don't meet your standards
- [Analytics](/cloud/features/analytics/enterprise-reporting#Cypress-Accessibility) for high-level trend-spotting and analysis

@@ -33,7 +33,7 @@ Use Branch review when reviewing a pull request made by another developer or bef

Since this reduces the amount of violations to review to just what was affected by the PR code changes, you won't spend any time looking for what's relevant -- and if there is no difference, you'll know your PR does not affect the accessibility score and is safe to merge.

See the main [Accessibility Branch Review docs](/accessibility/core-concepts/comparing-reports) for a video demo of this workflow.
See the main [Accessibility Branch Review docs](/accessibility/core-concepts/compare-reports) for a video demo of this workflow.

#### Compare reports during local development (without waiting for CI)

2 changes: 2 additions & 0 deletions docs/app/get-started/why-cypress.mdx
Original file line number Diff line number Diff line change
@@ -108,12 +108,14 @@ Below are listed some of the key features of each product.

- **Visualize Coverage:** [UI Coverage](/ui-coverage/get-started/introduction) provides a visual overview of test coverage across every page and component of your app, offering clear insights into uncovered areas that everyone can understand.
- **Results API:** Use the UI Coverage [Results API](/ui-coverage/results-api) to programmatically access test coverage data and integrate it into your existing workflows.
- **Branch Review:** Compare runs to see newly introduced elements in your application or unexpected reductions in test coverage.

### Cypress Accessibility

- **Accessibility Checks:** Maximize the value of your existing Cypress tests by instantly adding thousands of [accessibility checks](/accessibility/get-started/introduction) with no setup, code changes, or performance penalty.
- **Run-level reports:** Get a detailed report of accessibility issues found in your test runs with [Run-level reports](/accessibility/core-concepts/run-level-reports).
- **Results API:** Use the Cypress Accessibility's [Results API](/accessibility/results-api) to programmatically access Accessibility results in a CI environment.
- **Branch Review:** Compare any report against a baseline to review only the new violations, without any noise from existing issues.

## Solutions

8 changes: 4 additions & 4 deletions docs/app/guides/accessibility-testing.mdx
Original file line number Diff line number Diff line change
@@ -92,6 +92,10 @@ To learn more, you can read our [dedicated docs](/accessibility/get-started/intr

While automation like Axe Core® can detect missing attributes and other aspects of code quality that impact the experience of people with disabilities using the web, this kind of automation doesn't know anything about your specific application and the expectations you have for your users. That's where including accessibility in your specs comes in.

### Keyboard navigation

To test in-page navigation with the keyboard, you can use the [`cy.press()`](/api/commands/press) to dispatch native tab events. This can also be used to test custom behavior like autocomplete.

### Asserting alt text of images

To assert that the correct alternative text is present on your logo image, an explicit test can be written:
@@ -169,10 +173,6 @@ This means that, without some assertions about either the `button` element itsel

Developers who are unfamiliar with accessibility may assume that if a Testing Library `ByRole` locator can be made to pass before and after a code change, there has been no functional or accessibility-related change in the underlying element. As we've seen, this is not actually the case, because of the extra behavior browsers only implement for native HTML elements. For more about this difference and why semantic HTML elements are preferred, see the [first rule of Accessible Rich Internet Applications (ARIA)](https://www.w3.org/TR/using-aria/#rule1).

### Keyboard navigation

To test intra-page navigation with the keyboard, you can use the [`cy.press()`](/api/commands/press) to dispatch native tab events.

## Where to test accessibility

So what should you do in your test automation to help confirm the UI is accessible? First of all, for known critical areas like forms or checkout flows, ensure that the accessibility behavior is tested explicitly in at least one place. The means verifying that form fields and buttons have the correct labels and use the expected HTML elements, and other aspects of the DOM that communicate necessary information.
1 change: 0 additions & 1 deletion docs/app/references/trade-offs.mdx
Original file line number Diff line number Diff line change
@@ -37,7 +37,6 @@ We want to highlight some _temporary_ restrictions that Cypress hopes to
eventually address.

- [Workarounds for the lack of a `cy.hover()` command.](/api/commands/hover)
- [`cy.tab()` command.](https://github.com/cypress-io/cypress/issues/299)
- [There is not any native or mobile events support.](https://github.com/cypress-io/cypress/issues/311#issuecomment-339824191)
- [iframe support is somewhat limited, but does work.](https://github.com/cypress-io/cypress/issues/136)

143 changes: 80 additions & 63 deletions docs/cloud/features/branch-review.mdx

Large diffs are not rendered by default.

13 changes: 10 additions & 3 deletions docs/cloud/get-started/introduction.mdx
Original file line number Diff line number Diff line change
@@ -13,11 +13,13 @@ Cypress Cloud unlocks the full potential of Cypress test automation tools in you
<Btn
label="Sign up ➜"
variant="indigo-dark"
className="mr-1"
href="https://cloud.cypress.io/signup"
/>
<Btn
label="See a demo"
icon="action-play-small"
className="mr-1"
href="https://www.youtube.com/watch?v=vFLShoCM8pA"
/>
<Btn
@@ -33,7 +35,7 @@ Cypress Cloud unlocks the full potential of Cypress test automation tools in you

## Benefits

[Cypress Cloud](https://on.cypress.io/cloud) is our enterprise-ready companion
[Cypress Cloud](https://on.cypress.io/about-cypress-cloud?utm_medium=cloud-benefits&utm_source=docs.cypress.io&utm_term=&utm_content=Cypress+Cloud) is our enterprise-ready companion
to the open-source Cypress app, allowing you to:

- [View and debug past test results](#View-and-debug-past-test-results) from your CI environment.
@@ -45,6 +47,7 @@ to the open-source Cypress app, allowing you to:
- [View CI runs and test health](#Cypress-app-integration) directly from the
Cypress app.
- [Configure Cypress Cloud how you want](#Flexible-enterprise-configuration-and-single-sign-on) and enable enterprise single sign on.
- Add premium solutions like [UI Coverage](https://on.cypress.io/ui-coverage-trial?utm_medium=cloud-benefits&utm_source=docs.cypress.io&utm_term=&utm_content=UI+Coverage) and [Cypress Accessibility](http://on.cypress.io/accessibility-trial?utm_medium=cloud-benefits&utm_source=docs.cypress.io&utm_term=&utm_content=Cypress+Accessibility) to get new insights into test and application quality with no code, setup, or impact on your test runs.

### View and debug past test results

@@ -158,9 +161,13 @@ directly from specific test failures.

The [Cypress app](/cloud/features/recorded-runs#Cypress-App) integrates directly with
Cypress Cloud to provide developers with the latest test results without having
to switch tools.
to switch tools. This provides notifications about new failing tests in CI, so you can then
re-run them immediately in your local environment to debug.

<DocsImage src="/img/app/core-concepts/open-mode/debug-page.png" alt="Debug" />
<DocsImage
src="/img/app/core-concepts/open-mode/debug-page.png"
alt="Debug page showing test results and failure details in the Cypress App"
/>

### Flexible enterprise configuration and single sign on

97 changes: 97 additions & 0 deletions docs/ui-coverage/core-concepts/compare-reports.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
---
title: 'Comparing UI Coverage Reports | Cypress Documentation'
description: 'Review where coverage has changed in the application between two runs.'
sidebar_position: 40
sidebar_label: Compare reports
---

<ProductHeading product="ui-coverage" />

# Compare reports

UI Coverage reports from different runs can be compared in the [Branch Review](/cloud/features/branch-review#Getting-Started) area of Cypress Cloud. This allows you to understand the exact change in coverage between any two points in time, revealing the impact of application changes or test code updates on your overall coverage area.

<DocsImage
src="/img/ui-coverage/branch-review/uic-branch-review.png"
alt="UI Coverage Branch Review showing coverage changes between two runs, with sections highlighting new untested elements, resolved coverage gaps, and elements with changed coverage status"
/>

## Use cases

Comparing the results from different runs is useful in multiple scenarios.

**Key use cases:**

- **Pre-merge checks**: Know if any net-new untested elements are introduced by UI code changes.
- **Monitoring changes**: Compare nightly monitoring runs and track down changes in coverage caused by unexpected changes in the application.
- **Reviewing AI-generated code changes**: The increased use of AI to generate and/or review both tests and application code increases the risk of reducing coverage, or adding redundant coverage.
- **Tracing the introduction of issues**: With dropdowns for each run, it's easy to rapidly compare different A and B runs to find the exact commit that introduced a problem.
- **Demonstrating the resolution of issues**: Confirm the effect of your improvements, and share the overview with your team to more quickly review changes.

## Content of the report

The Branch Review report is organized into three sections:

### Untested links

<DocsImage
src="/img/ui-coverage/branch-review/new-untested-links.png"
alt="UI Coverage Branch Review showing new untested links that appear in the Changed run but were not present in the Base run"
/>

Untested links represent unopened doors in your application - pages that a user can reach through your UI, but that are not tested with Cypress. This report shows any new untested pages that appear in the Changed run that were not present in the Base run.

### Untested elements

<DocsImage
src="/img/ui-coverage/branch-review/new-untested-elements.png"
alt="UI Coverage Branch Review showing new untested elements that appear in the Changed run but were not present or tested in the Base run"
/>

Untested interactive elements can increase for two potential reasons:

1. In the Changed run, the tests are no longer interacting with an element that was interacted with in the Base run. The element is still present, but the **test code** has changed.
2. In the Changed run, new elements exist that were not present in the Base run, and they are not tested in the Changed run. This happens when the **application code** has changed but tests have not kept up.

Both changes are worth reviewing, and while the causes are different, the way to increase coverage is the same. The elements can be tested with Cypress to contribute towards your coverage score, or they can be ignored with configuration if it is not important to test the elements.

### Added links, elements, and views

<DocsImage
src="/img/ui-coverage/branch-review/added-links-and-elements.png"
alt="UI Coverage Branch Review showing a summary of new links, elements, and views added in the Changed run compared to the Base run"
/>

This is a summary of everything that appears to be new in the Changed run, and includes both tested and untested items. It's a useful way to compare two builds of your application and how the changes in the application itself relate to your overall coverage metrics.

### Detail View

<DocsImage
src="/img/ui-coverage/branch-review/detail-view.png"
alt="A list of only the new untested elements, displaying a log-out link in red with a tooltip pointing to it."
/>

Clicking on any untested link or element will take you into the Detail View, where you can see all the examples of the specific elements that have changed between the two runs. Elements that have the same status on both runs are removed here, so you can focus only on the changes.

## How to compare runs

The first step is to get to the [Branch Review](/cloud/features/branch-review.mdx) area of Cypress Cloud, which will let you compare one branch against another - or different runs on the same branch, if needed.
You can access this area by clicking the branch name associated with a run, or in several other ways. [Learn more about how to compare runs](/cloud/features/branch-review.mdx).

## FAQ

### How do I ensure a good comparison?

The best subjects to compare are passing runs that ran similar tests on the same set of content. This means that each run visited roughly the same pages and completed the same kinds of workflows. In this situation, any diff in the results is likely the result of changes present in the newer run. This is usually what happens out-of-the box when comparing a pull-request branch with your main branch.

That said, it is still possible and valid to compare runs from different points in time with different sets of test results, as long as you bear in mind all the potential sources of difference between the two runs, which you can evaluate for yourself as you explore the results.

In order to see unified changes for your entire test suite, you need to group all the tests together under a single Cypress run, for each report. Learn more about this in the [Branch Review Best Practices documentation](/cloud/features/branch-review#Best-Practices).

### What is the purpose of the Beta label?

This indicates the feature is ready for use and actively seeking feedback based on real usage of the current implementation. We have a few known issues to work through on our side before we consider this fully production-ready and remove the beta label. These issues only affect a subset of projects -- in most cases everything is working as intended. If you see anything unexpected, please use the feedback button and let us know.

### Why do I see some views (pages or components) changing from run-to-run?

URLs with dynamic slugs in them can appear as "new" pages in some situations. This behavior can be adjusted with [View configuration](/accessibility/configuration/views) to make sure page names will match across runs by wildcarding parts of the URL as needed.
2 changes: 1 addition & 1 deletion docs/ui-coverage/get-started/introduction.mdx
Original file line number Diff line number Diff line change
@@ -22,7 +22,7 @@ Easily track, monitor, and visualize the test coverage of your UI to prevent reg
label="See a demo"
icon="action-play-small"
className="mr-1"
href="https://go.cypress.io/hubfs/App%20Quality%20Products/UI%20Coverage/ui_coverage__overview_and_demo%20(1080p).mp4"
href="https://on.cypress.io/ui-coverage-demo-video"
/>
<Btn
label=" Explore an example project"
8 changes: 8 additions & 0 deletions docs/ui-coverage/guides/monitor-changes.mdx
Original file line number Diff line number Diff line change
@@ -14,3 +14,11 @@ Monitoring changes to your UI Coverage scores over time ensures that regressions
## Automate monitoring with the Results API

The UI Coverage [Results API](/ui-coverage/results-api) allows you to programmatically fetch UI Coverage data for integration into your CI/CD pipeline.

## Deep dive on changes with Branch Review

[Branch Review for UI Coverage](/ui-coverage/core-concepts/compare-reports) allows you to compare any two runs in detail and review new untested elements and links, as well as all new interactive elements added to the application.

## Spot trends overtime with cross-project analytics

In the [Enterprise Reporting](/cloud/features/analytics/enterprise-reporting#UI-Coverage) area of Cypress Cloud, you can view high-level trends across all projects and runs, as well as download reports or retrieve details with an API.
2 changes: 2 additions & 0 deletions src/theme/MDXComponents.js
Original file line number Diff line number Diff line change
@@ -79,6 +79,7 @@ import {
faPlus,
faQuestionCircle,
faSearch,
faShieldHalved,
faLaptopCode,
faStar,
faSyncAlt,
@@ -136,6 +137,7 @@ library.add(
faQuestionCircle,
faSearch,
faLaptopCode,
faShieldHalved,
faStar,
faSyncAlt,
faTerminal,
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.