Skip to content

UI Coverage and Branch Review updates #6167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
May 5, 2025
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/accessibility/changelog.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ sidebar_position: 200

## Week of 2/24/2025

- [Cypress Accessibility Branch Review](/accessibility/core-concepts/comparing-reports) has entered beta. Read the docs to learn how to compare runs and automatically spot newly introduced issues.
- [Cypress Accessibility Branch Review](/accessibility/core-concepts/compare-reports) has entered beta. Read the docs to learn how to compare runs and automatically spot newly introduced issues.

## Week of 1/13/2025

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Comparing the results from different runs is useful in multiple scenarios.
- **Detecting content issues**: Sometimes content editors can introduce accessibility issues unrelated to code changes. Seeing the example issues presented visually, in context, helps you quickly triage whether you are dealing with a recent code change issue, or a content authorship problem.
- **Reviewing AI-generated code changes**: The increased use of AI to generate and/or review front-end code creates some increased risks of accessibility regressions making it to production. The increase or decrease of accessibility issues when reviewing a pull request helps you understand the impact of the change.
- **Tracing the introduction of issues**: With dropdowns for each run, it's easy to rapidly compare different A and B runs to find the exact commit that introduced a problem.
- **Demonstrating the resolution of issues**: Confirm the effect of your improvements, and share overview with your team to more quickly review code changes.
- **Demonstrating the resolution of issues**: Confirm the effect of your improvements, and share the overview with your team to more quickly review code changes.

## Content of the report

Expand All @@ -49,47 +49,8 @@ This organization of the report brings the most significant results to the top -

## How to compare runs

The first step is to get to the Branch Review area of Cypress Cloud, which will let you compare one branch against another - or different runs on the same branch, if needed. We refer to be baseline fun for comparison as the `base` run, and the changes we are comparing with as the `changed` run.

There are a number of ways to get to Branch Review depending on where you are looking at your Cypress results. In all cases, once you've picked a `changed` run, you can adjust the automatically-selected `base` run to any branch or run in the project.

### From a Pull Request

<DocsImage
src="/img/accessibility/core-concepts/branch-review-pull-request.png"
alt="A pull request comment from Cypress with an arrow pointing to the 'View all changes introduced in this branch' link. Test results and some accessibility score and violation information is also visible in the comment. The project has an 86.86% accessibility score, with 9 failed elements. Failed rule counts are 1 critical, 1 serious, 3 moderate, and 0 minor."
/>

Click the "View all changes introduced in this branch" link at the bottom of the Test Results table. You will enter branch review with the current PR's branch and latest run pre-selected as the "changed" run.

### From the run list

Click the branch name associated with the run. This will take you to Branch Review with that branch's newest run pre-selected as the `changed` run.

<DocsImage
src="/img/accessibility/core-concepts/branch-review-run-card.png"
alt="A couple of cards showing various details about Cypress test runs, including pass/fail counts and metadata. An arrow points to the branch name."
/>

### From the project list

<DocsImage
src="/img/accessibility/core-concepts/branch-review-project-card.png"
width="40%"
alt="A card showing the name of an example project and related branches to choose."
/>

The project card shows three active branches for the project. You can click through to any of these to enter Branch Review with the newest run on that branch pre-selected as the `changed` run.

### From the main navigation

<DocsImage
src="/img/accessibility/core-concepts/branch-review-side-nav.png"
width="40%"
alt="The main Cypress Cloud navigation showing Latest Runs, Branches, and Analytics menu items. An arrow points to Branches."
/>

When inside of a project, you can select "Branches" in the main navigation to see a full, filterable list of available branches and choose one to set as the `changed` run.
The first step is to get to the [Branch Review](/cloud/features/branch-review.mdx) area of Cypress Cloud, which will let you compare one branch against another - or different runs on the same branch, if needed.
You can access this area by clicking the branch name associated with a run, or in several other ways. [Learn more about how to compare runs](/cloud/features/branch-review.mdx).

## FAQ

Expand All @@ -99,9 +60,11 @@ The best subjects to compare are passing runs that ran similar tests on the same

That said, it still possible and valid to compare runs from different points in time with different sets of test results, as long as you bear in mind all the potential sources of difference between the two runs, which you can evaluate for yourself as you explore the results.

In order to see unified changes for your entire test suite, you need to group all the tests together under a single Cypress run, for each report. Learn more about this in the [Branch Review Best Practices documentation](/cloud/features/branch-review#Best-Practices).

### What is the purpose of the Beta label?

This indicates the feature is ready for use and actively seeking feedback based on real usage of the current implementation. We have a few known issues to work through on our side before we consider this fully production-ready and remove the beta label. These issues only affect a subset of projects -- in most cases everything is working as intended. If you see anything unexpected, please hit the feedback button and let us know.
This indicates the feature is ready for use and actively seeking feedback based on real usage of the current implementation. We have a few known issues to work through on our side before we consider this fully production-ready and remove the beta label. These issues only affect a subset of projects -- in most cases everything is working as intended. If you see anything unexpected, please use the feedback button and let us know.

### Why do I see some views (pages or components) changing from run-to-run?

Expand Down
2 changes: 1 addition & 1 deletion docs/accessibility/core-concepts/how-it-works.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Cypress Accessibility seamlessly integrates accessibility testing into your deve
1. **Processing**: When your run completes, the final specs are processed for accessibility checks. Earlier specs have already entered processing and are likely completed by the end of the run.
1. **Organizing Views**: Cypress Cloud groups Views (pages and components) rendered during the run, merging and deduplicating across tests. This ensures page-level results and live DOM snapshots are organized effectively.
1. **Generating reports**: A combined "rule-level" report is created, summarizing all detected accessibility violations.
1. **Displaying results**: Results are displayed in Cypress Cloud in the run details area as well as in [Branch Review](/accessibility/core-concepts/comparing-reports) for comparing runs. Accessibility is also flagged in integrations (e.g., Slack, GitHub/GitLab comments), and accessible via the [Results API](/accessibility/results-api) for optional CI pipeline actions (e.g., failing builds based on unmet standards).
1. **Displaying results**: Results are displayed in Cypress Cloud in the run details area as well as in [Branch Review](/accessibility/core-concepts/compare-reports) for comparing runs. Accessibility is also flagged in integrations (e.g., Slack, GitHub/GitLab comments), and accessible via the [Results API](/accessibility/results-api) for optional CI pipeline actions (e.g., failing builds based on unmet standards).

## About Axe Core®

Expand Down
4 changes: 2 additions & 2 deletions docs/accessibility/get-started/introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Cypress Accessibility adds detailed accessibility checks and workflows in Cypres
label="See a demo"
icon="action-play-small"
className="!mb-2 mr-1"
href="https://www.loom.com/share/eedb1d18c22942d39a5ed2471ae72788?sid=e39f791e-c0b3-4efc-9770-cbfac403c3b9"
href="https://on.cypress.io/accessibility-demo-video"
/>
<Btn
label=" Explore an example project"
Expand Down Expand Up @@ -75,7 +75,7 @@ From there, you can integrate with CI to set your own standards for handling the
href="/accessibility/guides/detect-changes"
aria-labelledby="card-title-8"
>
<Icon name="chart-line" />
<Icon name="shield-halved" />
<h3 id="card-title-8">Detect and manage changes</h3>
<p>
Compare reports in detail to review only _new_ accessibility issues, use
Expand Down
4 changes: 2 additions & 2 deletions docs/accessibility/guides/detect-changes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ sidebar_position: 70

Cypress Accessibility supports three main paths to observing and acting on changes in the accessibility report for your projects:

- Run comparisons with [Branch Review](/accessibility/core-concepts/comparing-reports)
- Run comparisons with [Branch Review](/accessibility/core-concepts/compare-reports)
- The [Results API](/accessibility/results-api) to fail builds or trigger alerts when results don't meet your standards
- [Analytics](/cloud/features/analytics/enterprise-reporting#Cypress-Accessibility) for high-level trend-spotting and analysis

Expand All @@ -33,7 +33,7 @@ Use Branch review when reviewing a pull request made by another developer or bef

Since this reduces the amount of violations to review to just what was affected by the PR code changes, you won't spend any time looking for what's relevant -- and if there is no difference, you'll know your PR does not affect the accessibility score and is safe to merge.

See the main [Accessibility Branch Review docs](/accessibility/core-concepts/comparing-reports) for a video demo of this workflow.
See the main [Accessibility Branch Review docs](/accessibility/core-concepts/compare-reports) for a video demo of this workflow.

#### Compare reports during local development (without waiting for CI)

Expand Down
2 changes: 2 additions & 0 deletions docs/app/get-started/why-cypress.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -108,12 +108,14 @@ Below are listed some of the key features of each product.

- **Visualize Coverage:** [UI Coverage](/ui-coverage/get-started/introduction) provides a visual overview of test coverage across every page and component of your app, offering clear insights into uncovered areas that everyone can understand.
- **Results API:** Use the UI Coverage [Results API](/ui-coverage/results-api) to programmatically access test coverage data and integrate it into your existing workflows.
- **Branch Review:** Compare runs to see newly introduced elements in your application, or unexpected reduction in test coverage

### Cypress Accessibility

- **Accessibility Checks:** Maximize the value of your existing Cypress tests by instantly adding thousands of [accessibility checks](/accessibility/get-started/introduction) with no setup, code changes, or performance penalty.
- **Run-level reports:** Get a detailed report of accessibility issues found in your test runs with [Run-level reports](/accessibility/core-concepts/run-level-reports).
- **Results API:** Use the Cypress Accessibility's [Results API](/accessibility/results-api) to programmatically access Accessibility results in a CI environment.
- **Branch Review:** Compare any report against a baseline to review only the new violations, without any noise from existing issues.

## Solutions

Expand Down
8 changes: 4 additions & 4 deletions docs/app/guides/accessibility-testing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,10 @@ To learn more, you can read our [dedicated docs](/accessibility/get-started/intr

While automation like Axe Core® can detect missing attributes and other aspects of code quality that impact the experience of people with disabilities using the web, this kind of automation doesn't know anything about your specific application and the expectations you have for your users. That's where including accessibility in your specs comes in.

### Keyboard navigation

To test in-page navigation with the keyboard, you can use the [`cy.press()`](/api/commands/press) to dispatch native tab events. This can also be used to test custom behavior like autocomplete.

### Asserting alt text of images

To assert that the correct alternative text is present on your logo image, an explicit test can be written:
Expand Down Expand Up @@ -169,10 +173,6 @@ This means that, without some assertions about either the `button` element itsel

Developers who are unfamiliar with accessibility may assume that if a Testing Library `ByRole` locator can be made to pass before and after a code change, there has been no functional or accessibility-related change in the underlying element. As we've seen, this is not actually the case, because of the extra behavior browsers only implement for native HTML elements. For more about this difference and why semantic HTML elements are preferred, see the [first rule of Accessible Rich Internet Applications (ARIA)](https://www.w3.org/TR/using-aria/#rule1).

### Keyboard navigation

To test intra-page navigation with the keyboard, you can use the [`cy.press()`](/api/commands/press) to dispatch native tab events.

## Where to test accessibility

So what should you do in your test automation to help confirm the UI is accessible? First of all, for known critical areas like forms or checkout flows, ensure that the accessibility behavior is tested explicitly in at least one place. The means verifying that form fields and buttons have the correct labels and use the expected HTML elements, and other aspects of the DOM that communicate necessary information.
Expand Down
1 change: 0 additions & 1 deletion docs/app/references/trade-offs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,6 @@ We want to highlight some _temporary_ restrictions that Cypress hopes to
eventually address.

- [Workarounds for the lack of a `cy.hover()` command.](/api/commands/hover)
- [`cy.tab()` command.](https://github.com/cypress-io/cypress/issues/299)
- [There is not any native or mobile events support.](https://github.com/cypress-io/cypress/issues/311#issuecomment-339824191)
- [iframe support is somewhat limited, but does work.](https://github.com/cypress-io/cypress/issues/136)

Expand Down
Loading