attempt number is added to screenshot name, causing retries of failures to inappropriately succeed

[froodian]

Describe the bug
Similar to meinaart/cypress-plugin-snapshots#217 (and the open pull that fixes it), cy.state('runnable')._currentRetry (as documented here) should be incorporated into the nameCacheCounter logic so that a retried test is not seen as a new test.

To Reproduce
Steps to reproduce the behavior:

1. Have a test, with an accepted and committed baseline image
2. Have retries enabled in the cypress config (docs)
3. Change the code under test so that the test will go red
4. Run the test

Expected behavior
Test should go red. Instead test goes red, then on retry it goes green because the plugin is generating an actual with #1.png in the name instead of #0.png and therefore treating it as a new test.

Screenshots

Please complete the following information:

• Mac OS 12.6
• Chrome 107
• Cypress 10.8.0

Additional context
Add any other context about the problem here.

[github-actions]

🎉 This issue has been resolved in version 3.1.0 🎉

The release is available on:

• GitHub release
• npm package (@latest dist-tag)

Your semantic-release bot 📦🚀

[FRSgit]

Hey @froodian,

First of all - thank you for such detailed issue! ❤️ I never have used Cypress retries option, so you description made it much, much easier for me to understand what's the issue and how it can be covered.
I've just merged a PR that should cover your case (see #171). The implementation is slightly different than the one from cypress-plugin-snapshots (I do not use separated map for retries tracking), but I think it will do.

Please try the new release 3.1.0 and come back with a feedback (simple "it works!" is enough 😄 )

[froodian]

Hi @FRSgit - thanks for the fast fix!

It looks like with 3.1.0, it mostly works - but, if the first attempt fails for unrelated reasons (before the screenshot), the retry attempt creates Rendered IAM HTML Built Via Grapes DnD HTML IAM Composer Rendered IAM #-1.png (instead of #0 as would be expected), and so still goes green when the screenshot comparison should fail.

I think this could probably be fixed by making the code a little more like

  // it's a retry of the same image that was run previously, so let's decrease the counter
  if (currentRetryNumber > 0 && nameCacheCounter[screenshotPath] > -1) {
    --nameCacheCounter[screenshotPath];
  }

to make sure nameCacheCounter[screenshotPath] actually saw the image previously, so it doesn't go below -1?

[FRSgit]

Hey @froodian,

You're right - this condition is not enough. But after giving it a though solution proposed by you (and the implementation for cypress-plugin-snapshots) also does not cover all of the cases. For example this won't work when there were more than 2 screenshots done before test fail & retry, because we're now decreasing nameCacheCounter only by one. Also - what if names of screenshots done before retry were different?

I need to think about it for a moment, if you have any ideas - don't hesitate to post them here