Skip to content

ci: add workflow for rust bench crate #4210

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Dec 7, 2023
Merged

Conversation

jmayclin
Copy link
Contributor

The bench crate contains interop testing that would be useful to have in our ci runs. This commit adds a github workflow to run that. It also adds an options to generate the bindings without running the tests, since the tests take several minutes to run.

Description of changes:

  • adds CI workflow for bench crate
  • adds option to skip tests in bindings generation

Testing:

The workflow should pass.

After this workflow has been added, we should make it a required success for merging.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

The bench crate contains interop testing that would be useful to have in
our ci runs. This commit adds a github workflow to run that. It also
adds an options to generate the bindings without running the tests,
since the tests take several minutes to run.
@github-actions github-actions bot added the s2n-core team label Sep 20, 2023
@jmayclin jmayclin marked this pull request as ready for review September 20, 2023 01:21
@jmayclin jmayclin enabled auto-merge (squash) September 20, 2023 01:24
Comment on lines 62 to 64
# our benchmark testing includes interop tests between s2n-tls, rustls, and
# openssl
bench-tests:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mentioned this concern a while back, but you might have forgotten: if I see the name "bench / benchmark tests", I don't assume interop testing. I assume performance benchmarks. Does this actually run any performance benchmarks? Or is it just running interop tests?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/aws/s2n-tls/blob/main/bindings/rust/bench/src/harness.rs#L426

It runs these tests, which for all of OpenSSL, Rustls, s2n-tls, will test the product of

  • handshake type
  • cipher suite
  • kx type
  • cert type

Also session resumption interop tests.

And then checks that the certificate file paths are valid.

So this doesn't do any performance benchmarks.

Do you have any name suggestions? I was just sticking to the lowest denominator in that these are literally the tests in the bench crate. The longer term plan that has been tossed around is that we will shift all of our test harnesses to use this, at which point these tests could live in the integration crate?

Copy link

This PR has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

* adjust test name to remove "bench" and mention interop
@jmayclin jmayclin requested a review from lrstewart November 30, 2023 00:48
* create certs directory if it doesn't exist
The required bounds for traits have recently changed.
@jmayclin jmayclin merged commit d6d06bf into aws:main Dec 7, 2023
@jmayclin jmayclin deleted the bench-ci-workflow branch December 22, 2023 01:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants