8346106: Verify.checkEQ: testing utility for recursive value verification #22715

eme64 · 2024-12-12T15:42:29Z

In testing, we often generate "golden" values, and then compare the results with it. This requires comparison loops etc in every test. I would like to create a dedicated facility for this, to simplify testing in the future. This is also preparation for JDK-8344942, the Template framework.

I have written code like this in various tests before, see:
test/hotspot/jtreg/compiler/loopopts/superword/TestAlignVectorFuzzer.java
./test/hotspot/jtreg/compiler/loopopts/superword/TestMemorySegment.java
test/hotspot/jtreg/compiler/loopopts/superword/TestIndependentPacksWithCyclicDependency.java.

It is now time to make a proper facility, so I can save time when writing tests in the future.

A related PR, for value generation: #22716

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8346106: Verify.checkEQ: testing utility for recursive value verification (Enhancement - P4)

Reviewers

Vladimir Kozlov (@vnkozlov - Reviewer)
Theo Weidmann (@theoweidmannoracle - Author)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/22715/head:pull/22715
$ git checkout pull/22715

Update a local copy of the PR:
$ git checkout pull/22715
$ git pull https://git.openjdk.org/jdk.git pull/22715/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 22715

View PR using the GUI difftool:
$ git pr show -t 22715

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/22715.diff

Using Webrev

Link to Webrev Comment

bridgekeeper · 2024-12-12T15:43:16Z

👋 Welcome back epeter! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2024-12-12T15:43:58Z

@eme64 This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8346106: Verify.checkEQ: testing utility for recursive value verification

Reviewed-by: kvn, tweidmann

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 329 new commits pushed to the master branch:

4d77dba: 8346576: Remove vmTestbase/gc/memory/Nio/Nio.java from test/hotspot/jtreg/ProblemList.txt
2649a97: 8332827: [REDO] C2: crash in compiled code because of dependency on removed range check CastIIs
2344a1a: 8345732: Provide helpers for using PartialArrayState
572ce26: 8345266: java/util/concurrent/locks/StampedLock/OOMEInStampedLock.java JTREG_TEST_THREAD_FACTORY=Virtual fails with OOME
f6e7713: 8339356: Test javax/net/ssl/SSLSocket/Tls13PacketSize.java failed with java.net.SocketException: An established connection was aborted by the software in your host machine
23d6f74: 8346463: Add test coverage for deploying the default provider as a module
484229e: 8346306: Unattached thread can cause crash during VM exit if it calls wait_if_vm_exited
b0c40aa: 8340401: DcmdMBeanPermissionsTest.java and SystemDumpMapTest.java fail with assert(_stack_base != nullptr) failed: Sanity check
6b89954: 8346475: RISC-V: Small improvement for MacroAssembler::ctzc_bit
00d8407: 8346016: Problemlist vm/mlvm/indy/func/jvmti/mergeCP_indy2manyDiff_a in virtual thread mode
... and 319 more: https://git.openjdk.org/jdk/compare/28ae281b42cd00f471e275db544a5d23a42df59c...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

openjdk · 2024-12-12T15:44:48Z

@eme64 The following label will be automatically applied to this pull request:

hotspot-compiler

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

eme64 · 2024-12-12T15:43:53Z

test/hotspot/jtreg/compiler/lib/verify/Verify.java

+    /**
+     * Verify the content of two Objects, possibly recursively. Only limited types are implemented.
+     */
+    public static void checkEQ(Object a, Object b) {
+        checkEQ(a, b, "");
+    }


Note: this is the only entry point to the Utility.

Maybe this note could be added to the documentation comment?

I could. But it is also the only public method, so it is already given that way. And who knows: we might add more public methods, and then we could forget to remove that comment. I'd rather not add it.

eme64 · 2024-12-12T15:45:06Z

test/hotspot/jtreg/testlibrary_tests/verify/examples/TestVerifyInCheckMethod.java

+    @Setup
+    public static Object[] setup() {
+        // Must make sure to clone input arrays, if it is mutated in the test.
+        return new Object[] {INPUT_A.clone(), INPUT_B};
+    }
+
+    @Test
+    @Arguments(setup = "setup")
+    public static Object test(int[] a, float b) {
+        for (int i = 0; i < a.length; i++) {
+            a[i] = (int)(a[i] * b);
+        }
+        // Since we have more than one value, we wrap them in an Object[].
+        return new Object[] {a, b};
+    }
+
+    @Check(test = "test")
+    public static void check(Object result) {
+        Verify.checkEQ(result, GOLD);
+    }


Note: this is how we might generate Templates in the future: Using the @Check method with Verify.checkEQ. It allows the template to basically return whatever it wants, and it will be verified with the interpreter run.

Could we go even further and annotate this directly at the test method? For example:

@Test @Arguments(setup = "setup") @VerifyResult(expected = GOLD) public static Object test(int[] a, float b) { for (int i = 0; i < a.length; i++) { a[i] = (int)(a[i] * b); } // Since we have more than one value, we wrap them in an Object[]. return new Object[] {a, b}; }

We totally could. That would be a separate RFE to the IR framework. We have already filed similar RFEs.

JDK-8310533 [IR Framework] Add possibility to automatically verify that a test method always returns the same result

Adding this verify method could be the building block for that.

mlbridge · 2024-12-12T15:57:03Z

Webrevs

vnkozlov

Okay.

theoweidmannoracle · 2024-12-18T07:27:50Z

test/hotspot/jtreg/compiler/lib/verify/Verify.java

+    /**
+     * Verify the content of two Objects, possibly recursively. Only limited types are implemented.
+     */
+    public static void checkEQ(Object a, Object b) {
+        checkEQ(a, b, "");
+    }


Maybe this note could be added to the documentation comment?

theoweidmannoracle · 2024-12-18T07:29:02Z

test/hotspot/jtreg/compiler/lib/verify/Verify.java

+            System.err.println("ERROR: Verify.checkEQ failed: null mismatch");
+            print(a, "a " + context);
+            print(b, "b " + context);
+            throw new VerifyException("Object array null mismatch.");


Why does this message mention "array"? From what I understand, it can compare any two objects.

Good catch. I used to only have it work for arrays ;)

theoweidmannoracle · 2024-12-18T07:31:22Z

test/hotspot/jtreg/compiler/lib/verify/Verify.java

+                System.err.println("ERROR: Verify.checkEQ failed: type not supported: " + ca.getName());
+                print(a, "a " + context);
+                print(b, "b " + context);
+                throw new VerifyException("Object array type not supported: " + ca.getName());


This pattern repeats 3 times here and also below. Maybe you want to consider to move it to a reportError(String message, String details, a, b, context) method?

It does repeat somewhat, but not really identically. With different types for a, sometimes with context, sometimes without. Feel free to add code suggestions, because I'm not sure what you are exactly imagining.

theoweidmannoracle · 2024-12-18T07:46:11Z

test/hotspot/jtreg/testlibrary_tests/verify/examples/TestVerifyInCheckMethod.java

+    @Setup
+    public static Object[] setup() {
+        // Must make sure to clone input arrays, if it is mutated in the test.
+        return new Object[] {INPUT_A.clone(), INPUT_B};
+    }
+
+    @Test
+    @Arguments(setup = "setup")
+    public static Object test(int[] a, float b) {
+        for (int i = 0; i < a.length; i++) {
+            a[i] = (int)(a[i] * b);
+        }
+        // Since we have more than one value, we wrap them in an Object[].
+        return new Object[] {a, b};
+    }
+
+    @Check(test = "test")
+    public static void check(Object result) {
+        Verify.checkEQ(result, GOLD);
+    }


Could we go even further and annotate this directly at the test method? For example:

@Test @Arguments(setup = "setup") @VerifyResult(expected = GOLD) public static Object test(int[] a, float b) { for (int i = 0; i < a.length; i++) { a[i] = (int)(a[i] * b); } // Since we have more than one value, we wrap them in an Object[]. return new Object[] {a, b}; }

eme64 · 2024-12-18T10:21:12Z

@theoweidmannoracle thanks for reviewing! I think I have responded to all your comments :)

eme64 · 2024-12-19T08:00:26Z

@theoweidmannoracle Is there still anything you would change, or can I get your approval? ;)

theoweidmannoracle

Looks good to me!

vnkozlov

Last version looks good.

eme64 · 2024-12-20T06:51:38Z

@vnkozlov @theoweidmannoracle Thanks for the reviews!
/integrate

openjdk · 2024-12-20T06:52:26Z

Going to push as commit 35fafbc.
Since your change was applied there have been 330 commits pushed to the master branch:

b2811a0: 8340493: Fix some Asserts failure messages
4d77dba: 8346576: Remove vmTestbase/gc/memory/Nio/Nio.java from test/hotspot/jtreg/ProblemList.txt
2649a97: 8332827: [REDO] C2: crash in compiled code because of dependency on removed range check CastIIs
2344a1a: 8345732: Provide helpers for using PartialArrayState
572ce26: 8345266: java/util/concurrent/locks/StampedLock/OOMEInStampedLock.java JTREG_TEST_THREAD_FACTORY=Virtual fails with OOME
f6e7713: 8339356: Test javax/net/ssl/SSLSocket/Tls13PacketSize.java failed with java.net.SocketException: An established connection was aborted by the software in your host machine
23d6f74: 8346463: Add test coverage for deploying the default provider as a module
484229e: 8346306: Unattached thread can cause crash during VM exit if it calls wait_if_vm_exited
b0c40aa: 8340401: DcmdMBeanPermissionsTest.java and SystemDumpMapTest.java fail with assert(_stack_base != nullptr) failed: Sanity check
6b89954: 8346475: RISC-V: Small improvement for MacroAssembler::ctzc_bit
... and 320 more: https://git.openjdk.org/jdk/compare/28ae281b42cd00f471e275db544a5d23a42df59c...master

Your commit was automatically rebased without conflicts.

openjdk · 2024-12-20T06:52:34Z

@eme64 Pushed as commit 35fafbc.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

JDK-8344942

99c7510

openjdk bot changed the title ~~JDK-8344942~~ 8344942: Template-Based Testing Framework Dec 12, 2024

openjdk bot added the hotspot-compiler [email protected] label Dec 12, 2024

eme64 commented Dec 12, 2024

View reviewed changes

eme64 marked this pull request as ready for review December 12, 2024 15:45

eme64 marked this pull request as draft December 12, 2024 15:46

eme64 changed the title ~~8344942: Template-Based Testing Framework~~ 8346106: Verify.checkEQ: testing utility for recursive value verification Dec 12, 2024

eme64 marked this pull request as ready for review December 12, 2024 15:46

eme64 mentioned this pull request Dec 12, 2024

8344942: Template-Based Testing Framework #22483

Closed

3 tasks

fix whitespaces

2127416

openjdk bot added the rfr Pull request is ready for review label Dec 12, 2024

eme64 mentioned this pull request Dec 12, 2024

8346107: Generators: testing utility for random value generation #22716

Closed

3 tasks

vnkozlov approved these changes Dec 12, 2024

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Dec 12, 2024

theoweidmannoracle reviewed Dec 18, 2024

View reviewed changes

fix exception text

7a65b74

openjdk bot removed the ready Pull request is ready to be integrated label Dec 18, 2024

improve documentation

bf1377f

theoweidmannoracle approved these changes Dec 19, 2024

View reviewed changes

vnkozlov approved these changes Dec 19, 2024

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Dec 19, 2024

openjdk bot added the integrated Pull request has been integrated label Dec 20, 2024

openjdk bot closed this Dec 20, 2024

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Dec 20, 2024

theoweidmannoracle mentioned this pull request Jan 7, 2025

8346107: Generators: testing utility for random value generation #22941

Closed

3 tasks

eme64 mentioned this pull request Apr 30, 2025

8352869: Verify.checkEQ: extension for NaN, VectorAPI and arbitrary Objects #24224

Closed

3 tasks

8346106: Verify.checkEQ: testing utility for recursive value verification #22715

8346106: Verify.checkEQ: testing utility for recursive value verification #22715

Uh oh!

Conversation

eme64 commented Dec 12, 2024 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Reviewing

Uh oh!

bridgekeeper bot commented Dec 12, 2024

Uh oh!

openjdk bot commented Dec 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openjdk bot commented Dec 12, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mlbridge bot commented Dec 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

vnkozlov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eme64 commented Dec 18, 2024

Uh oh!

eme64 commented Dec 19, 2024

Uh oh!

theoweidmannoracle left a comment

Choose a reason for hiding this comment

Uh oh!

vnkozlov left a comment

Choose a reason for hiding this comment

Uh oh!

eme64 commented Dec 20, 2024

Uh oh!

openjdk bot commented Dec 20, 2024

Uh oh!

openjdk bot commented Dec 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

eme64 commented Dec 12, 2024 •

edited by openjdk bot

Loading

openjdk bot commented Dec 12, 2024 •

edited

Loading

mlbridge bot commented Dec 12, 2024 •

edited

Loading