Skip to content

fix flaky test PemTrustConfigTests#testTrustConfigReloadsFileContents #17984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

guojialiang92
Copy link
Contributor

Description

This PR is trying to fix a flakey test org.opensearch.common.ssl.PemTrustConfigTests#testTrustConfigReloadsFileContents.

Related Issues

Resolves #[17983]

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Contributor

❕ Gradle check result for 0ee096c: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Copy link

codecov bot commented Apr 17, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.51%. Comparing base (cbaddd3) to head (75c5b9f).
Report is 231 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##               main   #17984    +/-   ##
==========================================
  Coverage     72.51%   72.51%            
+ Complexity    67108    67101     -7     
==========================================
  Files          5475     5478     +3     
  Lines        309916   310034   +118     
  Branches      45060    45066     +6     
==========================================
+ Hits         224725   224823    +98     
+ Misses        66895    66844    -51     
- Partials      18296    18367    +71     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Contributor

@gaobinlong gaobinlong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This failed test and another one testBadFileFormatFails always fail with the same AssertionError in my local, could you help to check if it's caused by the dependency upgrade so the error message thrown is changed? If so I think we can change the assertion in assertFailedToParse() directly.

@guojialiang92
Copy link
Contributor Author

guojialiang92 commented Apr 18, 2025

Thanks, @gaobinlong.
You are right! PemTrustConfigTests#testBadFileFormatFails seems to have the same problem.
I reproduced it locally, but it is not stable. About one exception will occur every 100 runs.
测试报错图

I abstracted method PemTrustConfigTests#assertFailedToParseOrInvalidFileFormat and used it for these two flaky tests.

Copy link
Contributor

❌ Gradle check result for 981586d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: guojialiang <[email protected]>
@guojialiang92 guojialiang92 force-pushed the dev/fix-PemTrustConfigTests-testTrustConfigReloadsFileContents branch from 981586d to 37f824e Compare April 18, 2025 14:39
Copy link
Contributor

❌ Gradle check result for 37f824e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for 67d72db: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for d910aaf: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: guojialiang <[email protected]>
Copy link
Contributor

✅ Gradle check result for 75c5b9f: SUCCESS

final SslConfigException exception = expectThrows(SslConfigException.class, trustConfig::createTrustManager);
assertThat(exception.getMessage(), Matchers.containsString(file.toAbsolutePath().toString()));
assertThat(exception.getMessage(), Matchers.containsString("cannot create trust"));
assertThat(exception.getMessage(), Matchers.containsString("PEM"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we merge the two lines 167 and 168 into one?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could combine 166-168 actually.

boolean isFailedToParse = false;
boolean isInvalidFileFormat = false;
try {
assertFailedToParse(trustConfig, file);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know when will Failed to parse happen and when will Invalid format happen? I think it's better to make the trigger condition clear rather than depending on try...catch.

@andrross
Copy link
Member

@guojialiang92 thanks for the PR! Are you still working on this?

@opensearch-trigger-bot
Copy link
Contributor

This PR is stalled because it has been open for 30 days with no activity.

@opensearch-trigger-bot opensearch-trigger-bot bot added stalled Issues that have stalled and removed stalled Issues that have stalled labels Jun 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants