Add additional details on Gradle Check failures autocut issues

[prudhvigodithi]

Is your feature request related to a problem? Please describe

Coming from #11217 and #3713 we now have the the failure analytics for OpenSearch Gradle Check failures. For more details check the Developer Guide. The next step is to highlight the flaky failures as GitHub Issues so they can be prioritized and fixed. Additionally, when a test fails during the creation of a new PR but is not related to the PR's code changes, this process will help make the contributor aware of the existing flaky test.

Describe the solution you'd like

From the OpenSearch Gradle Check MetricsOpenSearch Gradle Check Metrics dashboard create an issue (with specific format) on failing tests that are part of post merge actions. The failing tests from post merge actions which are executed after the PR is merged are for sure the flaky tests.

Related component

Build

Describe alternatives you've considered

Today the gradle check failure issues are created from https://github.com/opensearch-project/OpenSearch/blob/main/.github/workflows/gradle-check.yml#L161-L168 which sometimes fails to execute https://github.com/opensearch-project/OpenSearch/actions/runs/9320653340/job/25657907035, so clean up the issues and disable the functionality in the gradle-check.yml.

Additional context

Coming from @dblock #11217 (comment) we should also

1. For any failed test, lookup an existing flaky test issue, if it exists, comment.
2. For any new failure, highlight it in comments with something like "new flaky test? please check and open one manually".

[msfroh]

I added a comment on #11217 (comment), highlighting that we need to avoid tainted data from tests broken by changes in an open PR, but that get fixed before the PR is merged. (Short story: I added a commit to fix 1 test and broke over 1000 other tests. Nobody else saw it though, because my mistake was limited to my open PR.)

Maybe a good heuristic could look at tests that fail across multiple PRs in some time window? Alternatively, if we have a job that just runs Gradle check continuously (without changing any code), it could be a good canary to collect "true" failures.

[prudhvigodithi]

Short story: I added a commit to fix 1 test and broke over 1000 other tests. Nobody else saw it though, because my mistake was limited to my open PR.

Thanks @msfroh, the question I have is for I added a commit to fix 1 test and broke over 1000 other tests are you referring after the PR was merged and it broke 1000 other tests? AFIAK it should be caught while the PR is open itself because today we run gradle check against the commit part of the open PR and against the commit that was merged (after the PR was merged and part of post merge action).

[msfroh]

are you referring after the PR was merged and it broke 1000 other tests? AFIAK it should be caught while the PR is open itself because today we run gradle check against the commit part of the open PR

Correct -- it was caught while the PR was still open. I noticed that those 1000+ test failures showed up on the dashboard (and made it look like ClientYamlTestSuiteIT is the biggest source of failures in the past week).

[peternied]

[Triage - attendees 1 2 3 4 5 6 7]
@prudhvigodithi Thanks for creating this issue