8344942: Template-Based Testing Framework #24217

eme64 · 2025-03-25T08:31:36Z

Goal
We want to generate Java source code:

Make it easy to generate variants of tests. E.g. for each offset, for each operator, for each type, etc.
Enable the generation of domain specific fuzzers (e.g. random expressions and statements).

Note: with the Template Library draft I was already able to find a list of bugs.

How to get started
When reviewing, please start by looking at:

jdk/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestSimple.java

Lines 60 to 76 in d21a8aa

    
               // Generate a source Java file as String 
        
               public static String generate() { 
        
                   // Create a Template with two arguments. 
        
                   var template = Template.make("arg1", "arg2", (Integer arg1, String arg2) -> body( 
        
                       """ 
        
                       package p.xyz; 
        
                       public class InnerTest { 
        
                           public static int test() { 
        
                               return #arg1 + #arg2; 
        
                           } 
        
                       } 
        
                       """ 
        
                   )); 
        
                   // Use the template with two arguments, and render it to a String. 
        
                   return template.withArgs(42, "7").render(); 
        
               }

We have a Template with two arguments. They are typed (Integer and String). We then apply the arguments template.withArgs(42, "7"), producing a TemplateWithArgs. This can then be rendered to a String. And then that can be compiled and executed with the CompileFramework.

Second, look at this advanced test:

jdk/test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestAdvanced.java

Lines 102 to 119 in 7707980

    
                   var testTemplate = Template.make("typeName", "operator", "generator", (String typeName, String operator, MyGenerator generator) -> body( 
        
                       let("con1", generator.next()), 
        
                       let("con2", generator.next()), 
        
                       """ 
        
                       // #typeName #operator #con1 #con2 
        
                       public static #typeName $GOLD = $test(); 
        
                       @Test 
        
                       public static #typeName $test() { 
        
                           return (#typeName)(#con1 #operator #con2); 
        
                       } 
        
                       @Check(test = "$test") 
        
                       public static void $check(#typeName result) { 
        
                           Verify.checkEQ(result, $GOLD); 
        
                       } 
        
                       """ 
        
                   ));

And then for a "tutorial", look at:
test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java

It shows these features:

The body of a Template is essentially a list of Tokens that are concatenated.
Templates can be nested: a TemplateWithArgs is also a Token.
We can use #name replacements to directly format values into the String. If we had proper String Templates in Java, we would not need this feature.
We can use $var to make variable names unique: if we applied the same template twice, we would get variable collisions. $var is then replaced with e.g. var_7 in one template use and var_42 in the other template use.
The use of Hooks to insert code into outer (earlier) code locations. This is useful, for example, to insert fields on demand.
The use of recursive templates, and fuel to limit the recursion.
Names: useful to register field and variable names in code scopes.

Next, look at the documentation in. This file is the heart of the Template Framework, and describes all the important features.

jdk/test/hotspot/jtreg/compiler/lib/template_framework/Template.java

Lines 31 to 76 in d21a8aa

    
           /** 
        
            * {@link Template}s are used to generate code, based on {@link Token} which are rendered to {@link String}. 
        
            * 
        
            * <p> 
        
            * A {@link Template} can have zero or more arguments, and for each number of arguments there is an implementation 
        
            * (e.g. {@link ZeroArgs} for zero arguments and {@link TwoArgs} for two arguments). This allows the use of Generics 
        
            * for the template argument types, i.e. the template arguments can be type checked. Ideally, we would have used 
        
            * String Templates to inject these arguments into the strings. But since String Templates are not (yet) available, 
        
            * the {@link Template}s provide <strong>hashtag replacements</strong> in the Strings: the {@link Template} argument 
        
            * names are captured, and the argument values automatically replace any {@code "#name"} in the Strings. See the 
        
            * different overloads of {@link make} for examples. Additional hashtag replacements can be defined with {@link let}. 
        
            * 
        
            * <p> 
        
            * When using nested {@link Template}s, there can be collisions with identifiers (e.g. variable names and method names). 
        
            * For this, {@link Template}s provide <strong>dollar replacements</strong>, which automaticall rename any 
        
            * {@code "$name"} in the String with a {@code "name_ID"}, where the {@code "ID"} is unique for every use of 
        
            * a {@link Template}. The dollar replacement can also be captured with {@link $}, and passed to nested 
        
            * {@link Template}s, which allows sharing of these identifier names between {@link Template}s. 
        
            * 
        
            * <p> 
        
            * To render a {@link Template} to a {@link String}, one first has to apply the arguments (e.g. with 
        
            * {@link TwoArgs#withArgs}) and then the resulting {@link TemplateWithArgs} can either be used as a 
        
            * {@link Token} inside another {@link Template}, or rendered to a {@link String} with {@link TemplateWithArgs#render}. 
        
            * 
        
            * <p> 
        
            * A {@link TemplateWithArgs} can be used directly as a {@link Token} inside the {@link Template#body} to 
        
            * nest the {@link Template}s. Alternatively, code can be {@link Hook#insert}ed to where a {@link Hook} 
        
            * was {@link Hook#set} earlier (in some outer scope of the code). For example, while generating code in 
        
            * a method, one can reach out to the scope of the class, and insert a new field, or define a utility method. 
        
            * 
        
            * <p> 
        
            * A {@link TemplateBinding} allows the recurisve use of {@link Template}s. With the indirection of such a binding, 
        
            * a {@link Template} can reference itself. To ensure the termination of recursion, the templates are rendered 
        
            * with a certain amount of {@link fuel}, which is decreased at each {@link Template} nesting by a certain amount 
        
            * (can be changed with {@link setFuelCost}). Recursive templates are supposed to terminate once the {@link fuel} 
        
            * is depleated (i.e. reaches zero). 
        
            * 
        
            * <p> 
        
            * Code generation often involves defining fields and variables, which are then available inside a defined 
        
            * scope, and can be sampled in any nested scope. To allow the use of names for multiple applications (e.g. 
        
            * fields, variables, methods, etc), we define a {@link Name}, which captures the {@link String} representation 
        
            * to be used in code, as well as its type and if it is mutable. One can add such a {@link Name} to the 
        
            * current code scope with {@link addName}, and sample from the current or outer scopes with {@link sampleName}. 
        
            * When generating code, one might want to create {@link Name}s (variables, fields, etc) in local scope, or 
        
            * in some outer scope with the use of {@link Hook}s. 
        
            */

For a better experience, you may want to generate the javadocs:
javadoc -sourcepath test/hotspot/jtreg:./test/lib compiler.lib.template_framework

History
@TobiHartmann and I have played with code generators for a while, and have had the dream of doing that in a more principled way. And to hopefully make it more accessible for other VM developers, with the goal of improving test coverage.

@TobiHartmann started with an initial prototype. @tobiasholenstein took over, and worked with our intern @maasaid. Their approach was to take old tests and templetize them (draft PR). Their templates had holes for replacements, and a $ prefix for variables name replacement.

Once @tobiasholenstein left, I took over the project, and focused on nested template use, where templates could be passed arguments. I kept it string based, which worked, but the resulting syntax was a little cryptic. Debugging was difficult, as I had to produce custom stack traces, print available variables, etc. Here is the string syntax based version.

@theoweidmannoracle reviewed my draft, and had some very good feedback. He was frustrated with the complexity, and also the string syntax. Over several iterations, we kept most of the complexity (because code generation is a little complex), but changed the approach to use Java Generics. I took one of his prototypes and fleshed it out with all the other necessary features.

Related Work
There is lots of related work, for test generation:

Verify.java: verify results.
Generators.java: generate random inputs from "interesting distributions".
Compile Framework: take string source code, compile and class-load it for execution.
JDK-8352861 Template-Framework Library: future work. Provide lots of useful templates, generate Expressions and nested Statements, etc. Here a previous PR where I am experimenting with different Library features.. I decided to already include the Hooks from the library, as they are useful to have in the tests and examples already.

Note: with the Template Library draft I was already able to find a list of bugs.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8344942: Template-Based Testing Framework (Enhancement - P4)

Reviewers

Christian Hagedorn (@chhagedorn - Reviewer)
Roberto Castañeda Lozano (@robcasloz - Reviewer) Review applies to 72923879
Manuel Hässig (@mhaessig - Author)

Contributors

Tobias Hartmann <[email protected]>
Tobias Holenstein <[email protected]>
Theo Weidmann <[email protected]>
Roberto Castañeda Lozano <[email protected]>
Christian Hagedorn <[email protected]>
Manuel Hässig <[email protected]>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/24217/head:pull/24217
$ git checkout pull/24217

Update a local copy of the PR:
$ git checkout pull/24217
$ git pull https://git.openjdk.org/jdk.git pull/24217/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 24217

View PR using the GUI difftool:
$ git pr show -t 24217

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/24217.diff

Using Webrev

Link to Webrev Comment

bridgekeeper · 2025-03-25T08:32:14Z

👋 Welcome back epeter! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2025-03-25T08:33:05Z

@eme64 This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8344942: Template-Based Testing Framework

Co-authored-by: Tobias Hartmann <[email protected]>
Co-authored-by: Tobias Holenstein <[email protected]>
Co-authored-by: Theo Weidmann <[email protected]>
Co-authored-by: Roberto Castañeda Lozano <[email protected]>
Co-authored-by: Christian Hagedorn <[email protected]>
Co-authored-by: Manuel Hässig <[email protected]>
Reviewed-by: chagedorn, mhaessig, rcastanedalo

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 81 new commits pushed to the master branch:

dc96160: 8356159: RISC-V: Add Zabha
7838321: 8358496: Concurrent reading from Socket with timeout executes sequentially
42f48a3: 8350689: Turn on timestamp and thread metadata by default for java.security.debug
... and 78 more: https://git.openjdk.org/jdk/compare/90d6ad015714b81064dd16d0e64f1b774e68d4f3...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

openjdk · 2025-03-25T08:33:50Z

@eme64 The following label will be automatically applied to this pull request:

hotspot-compiler

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

eme64 · 2025-03-25T10:00:45Z

/contributor add @TobiHartmann @tobiasholenstein @maasaid @theoweidmannoracle

openjdk · 2025-03-25T10:01:55Z

@eme64 @TobiHartmann @tobiasholenstein @maasaid @theoweidmannoracle is not a valid user in this repository.

Syntax: /contributor (add|remove) [@user | openjdk-user | Full Name <email@address>]. For example:

/contributor add @openjdk-bot
/contributor add duke
/contributor add J. Duke <[email protected]>

User names can only be used for users in the census associated with this repository. For other contributors you need to supply the full name and email address.

eme64 · 2025-03-25T10:03:38Z

/contributor add @TobiHartmann

eme64 · 2025-03-25T10:03:51Z

/contributor add @tobiasholenstein

openjdk · 2025-03-25T10:03:55Z

@eme64
Contributor Tobias Hartmann <[email protected]> successfully added.

eme64 · 2025-03-25T10:04:17Z

/contributor add @maasaid

eme64 · 2025-03-25T10:04:24Z

/contributor add @theoweidmannoracle

openjdk · 2025-03-25T10:04:31Z

@eme64
Contributor Tobias Holenstein <[email protected]> successfully added.

openjdk · 2025-03-25T10:05:05Z

@eme64 @maasaid was not found in the census.

Syntax: /contributor (add|remove) [@user | openjdk-user | Full Name <email@address>]. For example:

/contributor add @openjdk-bot
/contributor add duke
/contributor add J. Duke <[email protected]>

User names can only be used for users in the census associated with this repository. For other contributors you need to supply the full name and email address.

openjdk · 2025-03-25T10:05:13Z

@eme64
Contributor Theo Weidmann <[email protected]> successfully added.

mlbridge · 2025-03-25T10:09:26Z

Webrevs

tobiasholenstein · 2025-03-25T10:18:19Z

/contributor add @TobiHartmann @tobiasholenstein @maasaid @theoweidmannoracle

Nice work @eme64 !
I think I should be in the census. But @maasaid @theoweidmannoracle might not be

galderz

Looks great though I'm not too familiar with the code to be able to do a reasonable review, but I had a question:

Have you got any practical use case that can show where you've used this and show what it takes to build such a use case? VectorReduction2 or similar type of microbenchmarks would be great to see auto generated using this?

The reason I ask this is because I feel that something that is missing in this PR is a small practical use case where this framework is put into action to actually generate some jtreg/IR/microbenchmark test and see how it runs as part of the CI in the PR. WDYT?

eme64 · 2025-03-27T06:51:19Z

@galderz Thanks for your questions!

Looks great though I'm not too familiar with the code to be able to do a reasonable review

Well the code is all brand new, so really anybody could review ;)

Have you got any practical use case that can show where you've used this and show what it takes to build such a use case?

I actually have a list of experiments in this branch (it is linked in the PR description): #23418
Some of them use the IR framework, though for now just as a testing harness, not for IR rules. Generating IR rules automatically requires quite a bit of logic...
I hope that is satisfactory for now?

Ah, but there was this test:
test/hotspot/jtreg/compiler/loopopts/superword/TestDependencyOffsets.java
I did now not refactor it, but it would not be too hard to see how to use the Templates for it. And I do generate IR rules in that one. I don't super like just refactoring old tests... there is always a risk of breaking it and then coverage is worse than before...

The reason I ask this is because I feel that something that is missing in this PR is a small practical use case where this framework is put into action to actually generate some jtreg/IR/microbenchmark test and see how it runs as part of the CI in the PR. WDYT?

I also have a few tests in this PR that just generate regular JTREG tests, without the IR framework, did you see those?

VectorReduction2 or similar type of microbenchmarks would be great to see auto generated using this?

I don't yet have a solution for microbenchmarks. It's mostly an issue of including the test/hotspot/jtreg/compiler/lib path... And I fear that JMH requires all benchmark code to be compiled beforehand, and not dynamically as I do with the class loader. But maybe there is a solution for that.

The patch is already quite large, and so I wanted to just publish the basic framework. Do you think that is ok?

galderz · 2025-03-31T07:21:15Z

@galderz Thanks for your questions!

Looks great though I'm not too familiar with the code to be able to do a reasonable review

Well the code is all brand new, so really anybody could review ;)

Right, what I meant is that developers that have past history with this work will be able to provide a more thorough review :)

Have you got any practical use case that can show where you've used this and show what it takes to build such a use case?

I actually have a list of experiments in this branch (it is linked in the PR description): #23418 Some of them use the IR framework, though for now just as a testing harness, not for IR rules. Generating IR rules automatically requires quite a bit of logic... I hope that is satisfactory for now?

Yes it is. Sorry I missed the linked PR when I read the description. The examples there look great, it's what I was looking for.

Ah, but there was this test: test/hotspot/jtreg/compiler/loopopts/superword/TestDependencyOffsets.java I did now not refactor it, but it would not be too hard to see how to use the Templates for it. And I do generate IR rules in that one. I don't super like just refactoring old tests... there is always a risk of breaking it and then coverage is worse than before...

The reason I ask this is because I feel that something that is missing in this PR is a small practical use case where this framework is put into action to actually generate some jtreg/IR/microbenchmark test and see how it runs as part of the CI in the PR. WDYT?

I also have a few tests in this PR that just generate regular JTREG tests, without the IR framework, did you see those?

Yeah I've seen them now.

VectorReduction2 or similar type of microbenchmarks would be great to see auto generated using this?

I don't yet have a solution for microbenchmarks. It's mostly an issue of including the test/hotspot/jtreg/compiler/lib path... And I fear that JMH requires all benchmark code to be compiled beforehand, and not dynamically as I do with the class loader. But maybe there is a solution for that.

The patch is already quite large, and so I wanted to just publish the basic framework. Do you think that is ok?

Yeah sure.

chhagedorn

Impressive work Emanuel! This is a very valuable framework to improve our testing coverage.

Here are some first comments after skimming some of the code and comments. I will deep dive more into the code and documentation later.

test/hotspot/jtreg/compiler/lib/template_framework/README.md

chhagedorn · 2025-03-31T08:09:35Z

test/hotspot/jtreg/compiler/lib/template_framework/README.md

+
+The basic functionalities of the Template Framework are described in the [Template Class](./Template.java), together with some examples. More examples can be found in [TestSimple.java](../../../testlibrary_tests/template_framework/examples/TestSimple.java) and [TestTutorial.java](../../../testlibrary_tests/template_framework/examples/TestTutorial.java).
+
+The [Template Library](../template_library/README.md) provides a large number of Templates which can be used to create anything from simple regression tests to complex fuzzers.


template_library/README.md does not exist.

Another thought: Should template_library be a subfolder in template_framework? Otherwise, it could suggest to be something separate from the Template Framework.

Removing that line, and moving the library to a subfolder.

test/hotspot/jtreg/compiler/lib/template_framework/README.md

chhagedorn · 2025-03-31T08:18:59Z

test/hotspot/jtreg/compiler/lib/template_framework/Template.java

+ * String Templates to inject these arguments into the strings. But since String Templates are not (yet) available,
+ * the {@link Template}s provide <strong>hashtag replacements</strong> in the Strings: the {@link Template} argument
+ * names are captured, and the argument values automatically replace any {@code "#name"} in the Strings. See the
+ * different overloads of {@link make} for examples. Additional hashtag replacements can be defined with {@link let}.


General Javadocs comment: You should preceed method names with # in order to create a proper link.

Without:

With (i.e. {@link make} and {@link #let}):

interesting. As we saw offline, javadoc does not need the hashtag, but some editors do, and it seems to be common practice to have the hashtag. I'll add them for every applicable link.

I think I fixed all, but I cannot easily confirm. Can you check if there are any left?

test/hotspot/jtreg/compiler/lib/template_framework/Template.java

chhagedorn · 2025-03-31T08:39:36Z

test/hotspot/jtreg/compiler/lib/template_framework/Template.java

+ * {@link Template}s are used to generate code, based on {@link Token} which are rendered to {@link String}.
+ *
+ * <p>
+ * A {@link Template} can have zero or more arguments, and for each number of arguments there is an implementation


Since the README refers to this file for more information (which is perfectly fine to avoid repetition), I naturally started to read here at the top. But in this paragraph, we are already explaining details of the implementation without giving a more general introduction and motivation for the Template Framework which can leave readers without background confused. I think it could be worth to spend some more time in the README (which you can then also just refer to when suggesting to use the framework in PRs). You could cover: Why is the framework useful/why should I care about it, some leading example/testing scenario and why it is really hard to cover that without the framework (i.e. what we've done so far until today), how does the framework roughly work, how easy is it to write tests (you can then reference example tests from there) etc.

You can still do many references from the README to classes and examples.

Side note: I admit that I could have extended the IR framework README with a more motivational introduction as well - maybe I will add that later.

I added an additional test, and copied some snippets to the java docs.
I also added some more motivation.
Let me know if that is better, or if you have any more suggestions :)

chhagedorn

Thanks for the update! Almost there, some last comments and then we're good to go :-)

test/hotspot/jtreg/compiler/lib/template_framework/Name.java

test/hotspot/jtreg/compiler/lib/template_framework/Renderer.java

test/hotspot/jtreg/compiler/lib/template_framework/CodeFrame.java

test/hotspot/jtreg/compiler/lib/template_framework/Hook.java

test/hotspot/jtreg/compiler/lib/template_framework/Token.java

chhagedorn · 2025-06-03T14:05:14Z

test/hotspot/jtreg/compiler/lib/template_framework/NameSet.java

+            if (w < 0) {
+                throw new RuntimeException("Negative weight not allowed: " + w);
+            }


I thought zero is also not allowed?

Well that should have been filtered out already earlier, when we added the individual names. Now we could have a total weight of zero if we have no names. Then we just return null here, and then throw an exception a little further up the use case chain, e.g. DataName.sample -> throw new RendererException("No variable: " + mutability + msg1 + msg2 + ".");

This here is just a sanity check, hence I throw a RuntimeException, and not a RendererException.

As asked for offline: added some more comments here.

test/hotspot/jtreg/compiler/lib/template_framework/Renderer.java

chhagedorn · 2025-06-03T14:31:39Z

test/hotspot/jtreg/compiler/lib/template_framework/Renderer.java

            int hashtag = s.indexOf("#", start);
+            // If the character was not found, we want to have the rest of the
+            // String s, so instead of "-1" take the end/length of the String.
            dollar  = (dollar == -1)  ? s.length() : dollar;


s.length() could be called once before the loop and then reused inside the loop.

You mean as a performance optimization? Is that not something we let the compiler do?

As discussed offline: I made s final, so we are sure that it is not mutated and the length should be moved out of the loop by the compiler. It would also only be a very small performance impact, as we are doing things like indexOf here, which are much more expensive anyway.

Co-authored-by: Christian Hagedorn <[email protected]>

eme64 · 2025-06-03T15:49:50Z

@chhagedorn Thanks for this batch of comments, they are all addressed!

chhagedorn

Thanks a lot for addressing everything and all the interesting and insightful discussions - also learnt quite a lot :-) There is nothing left to say other than: Ship it! 🚀 (if the other reviewers also agree with the latest changes)

eme64 · 2025-06-03T21:08:03Z

@chhagedorn Thank you very much! ☺️
This was surely my biggest patch, and most deeply reviewed. An intense but rewarding experience.
And I really learned so much from so many of the contributors here :)

mhaessig

I had a look at the changes since my last review. They look excellent.
Especially good to see the tutorial improving even further.

eme64 · 2025-06-04T08:04:55Z

@mhaessig Thank you very much for having another look!

robcasloz

Looks good, I just have a few language suggestions. Thanks for driving this Emanuel!

test/hotspot/jtreg/testlibrary_tests/template_framework/examples/TestTutorial.java

test/hotspot/jtreg/compiler/lib/template_framework/Template.java

Co-authored-by: Roberto Castañeda Lozano <[email protected]>

eme64 · 2025-06-04T12:14:52Z

@robcasloz Thanks for making another pass and for the suggestions! 😊

eme64 · 2025-06-04T12:18:39Z

@chhagedorn @robcasloz @mhaessig Thanks a lot for all the time you invested to see this through, I know it took a lot of effort to review this, so I am very thankful 😊

eme64 · 2025-06-04T12:30:37Z

I want to thank all the contributors here again before integration:
@TobiHartmann For the early experiments and getting this project started.
@tobiasholenstein For taking on the project, and mentoring Said. They came up with the $ idea for de-duplication of variable names.
@theoweidmannoracle For the fantastic idea using Generics and making it more functional. His prototype is what I then ran with.
@chhagedorn We had a lot of enlightening conversations, he was the one who invested the most effort reviewing this patch. He pushed me to improve a lot of API parts, and I learned a lot from his efforts on the Testing Framework, especially that even such frameworks should be tested thoroughly.
@robcasloz Also invested a lot of time, and played with it hands on. One of his ideas was to allow the curly brackets for ${name} and #{name}.
@mhaessig Even though he only just joined us, he already jumped on board quickly and is already reviewing, just fantastic!

eme64 · 2025-06-04T13:14:09Z

/integrate

openjdk · 2025-06-04T13:16:38Z

Going to push as commit 248341d.
Since your change was applied there have been 82 commits pushed to the master branch:

09ec4de: 8358066: Non-ascii package names gives compilation error "import requires canonical name"
dc96160: 8356159: RISC-V: Add Zabha
7838321: 8358496: Concurrent reading from Socket with timeout executes sequentially
... and 79 more: https://git.openjdk.org/jdk/compare/90d6ad015714b81064dd16d0e64f1b774e68d4f3...master

Your commit was automatically rebased without conflicts.

openjdk · 2025-06-04T13:17:18Z

@eme64 Pushed as commit 248341d.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

JDK-8344942

d21a8aa

openjdk bot changed the title ~~JDK-8344942~~ 8344942: Template-Based Testing Framework Mar 25, 2025

openjdk bot added the hotspot-compiler [email protected] label Mar 25, 2025

eme64 added 4 commits March 25, 2025 10:31

fix whitespace

8cdf16a

whitespace

8b6d7ae

whitespace

0263454

fix tests

a10d6fa

eme64 mentioned this pull request Mar 25, 2025

8352861: Template-Framework Library #23418

Closed

3 tasks

eme64 marked this pull request as ready for review March 25, 2025 09:56

openjdk bot added the rfr Pull request is ready for review label Mar 25, 2025

galderz reviewed Mar 27, 2025

View reviewed changes

Merge branch 'master' into JDK-8344942-TemplateFramework-v3

ededf45

chhagedorn reviewed Mar 31, 2025

View reviewed changes

add some hashes

6ef7127

chhagedorn reviewed Jun 3, 2025

View reviewed changes

Apply suggestions from code review

3efb9fc

Co-authored-by: Christian Hagedorn <[email protected]>

openjdk bot removed the rfr Pull request is ready for review label Jun 3, 2025

more for Christian

3f0beb4

fix whitespaces from applied suggestion

7292387

openjdk bot added the rfr Pull request is ready for review label Jun 3, 2025

chhagedorn approved these changes Jun 3, 2025

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Jun 3, 2025

mhaessig approved these changes Jun 4, 2025

View reviewed changes

robcasloz approved these changes Jun 4, 2025

View reviewed changes

Apply suggestions from code review

0ec0949

Co-authored-by: Roberto Castañeda Lozano <[email protected]>

mhaessig approved these changes Jun 4, 2025

View reviewed changes

openjdk bot removed the ready Pull request is ready to be integrated label Jun 4, 2025

chhagedorn approved these changes Jun 4, 2025

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Jun 4, 2025

openjdk bot added the integrated Pull request has been integrated label Jun 4, 2025

openjdk bot closed this Jun 4, 2025

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Jun 4, 2025

eme64 mentioned this pull request Jun 4, 2025

8358600: Template-Framework Library: Template for TestFramework test class #25643

Closed

3 tasks

	// Generate a source Java file as String
	public static String generate() {
	// Create a Template with two arguments.
	var template = Template.make("arg1", "arg2", (Integer arg1, String arg2) -> body(
	"""
	package p.xyz;
	public class InnerTest {
	public static int test() {
	return #arg1 + #arg2;
	}
	}
	"""
	));

	// Use the template with two arguments, and render it to a String.
	return template.withArgs(42, "7").render();
	}

	var testTemplate = Template.make("typeName", "operator", "generator", (String typeName, String operator, MyGenerator generator) -> body(
	let("con1", generator.next()),
	let("con2", generator.next()),
	"""
	// #typeName #operator #con1 #con2
	public static #typeName $GOLD = $test();

	@Test
	public static #typeName $test() {
	return (#typeName)(#con1 #operator #con2);
	}

	@Check(test = "$test")
	public static void $check(#typeName result) {
	Verify.checkEQ(result, $GOLD);
	}
	"""
	));

	/**
	* {@link Template}s are used to generate code, based on {@link Token} which are rendered to {@link String}.
	*
	* <p>
	* A {@link Template} can have zero or more arguments, and for each number of arguments there is an implementation
	* (e.g. {@link ZeroArgs} for zero arguments and {@link TwoArgs} for two arguments). This allows the use of Generics
	* for the template argument types, i.e. the template arguments can be type checked. Ideally, we would have used
	* String Templates to inject these arguments into the strings. But since String Templates are not (yet) available,
	* the {@link Template}s provide <strong>hashtag replacements</strong> in the Strings: the {@link Template} argument
	* names are captured, and the argument values automatically replace any {@code "#name"} in the Strings. See the
	* different overloads of {@link make} for examples. Additional hashtag replacements can be defined with {@link let}.
	*
	* <p>
	* When using nested {@link Template}s, there can be collisions with identifiers (e.g. variable names and method names).
	* For this, {@link Template}s provide <strong>dollar replacements</strong>, which automaticall rename any
	* {@code "$name"} in the String with a {@code "name_ID"}, where the {@code "ID"} is unique for every use of
	* a {@link Template}. The dollar replacement can also be captured with {@link $}, and passed to nested
	* {@link Template}s, which allows sharing of these identifier names between {@link Template}s.
	*
	* <p>
	* To render a {@link Template} to a {@link String}, one first has to apply the arguments (e.g. with
	* {@link TwoArgs#withArgs}) and then the resulting {@link TemplateWithArgs} can either be used as a
	* {@link Token} inside another {@link Template}, or rendered to a {@link String} with {@link TemplateWithArgs#render}.
	*
	* <p>
	* A {@link TemplateWithArgs} can be used directly as a {@link Token} inside the {@link Template#body} to
	* nest the {@link Template}s. Alternatively, code can be {@link Hook#insert}ed to where a {@link Hook}
	* was {@link Hook#set} earlier (in some outer scope of the code). For example, while generating code in
	* a method, one can reach out to the scope of the class, and insert a new field, or define a utility method.
	*
	* <p>
	* A {@link TemplateBinding} allows the recurisve use of {@link Template}s. With the indirection of such a binding,
	* a {@link Template} can reference itself. To ensure the termination of recursion, the templates are rendered
	* with a certain amount of {@link fuel}, which is decreased at each {@link Template} nesting by a certain amount
	* (can be changed with {@link setFuelCost}). Recursive templates are supposed to terminate once the {@link fuel}
	* is depleated (i.e. reaches zero).
	*
	* <p>
	* Code generation often involves defining fields and variables, which are then available inside a defined
	* scope, and can be sampled in any nested scope. To allow the use of names for multiple applications (e.g.
	* fields, variables, methods, etc), we define a {@link Name}, which captures the {@link String} representation
	* to be used in code, as well as its type and if it is mutable. One can add such a {@link Name} to the
	* current code scope with {@link addName}, and sample from the current or outer scopes with {@link sampleName}.
	* When generating code, one might want to create {@link Name}s (variables, fields, etc) in local scope, or
	* in some outer scope with the use of {@link Hook}s.
	*/


		The basic functionalities of the Template Framework are described in the [Template Class](./Template.java), together with some examples. More examples can be found in [TestSimple.java](../../../testlibrary_tests/template_framework/examples/TestSimple.java) and [TestTutorial.java](../../../testlibrary_tests/template_framework/examples/TestTutorial.java).

		The [Template Library](../template_library/README.md) provides a large number of Templates which can be used to create anything from simple regression tests to complex fuzzers.

8344942: Template-Based Testing Framework #24217

8344942: Template-Based Testing Framework #24217

Uh oh!

Conversation

eme64 commented Mar 25, 2025 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Contributors

Reviewing

Uh oh!

bridgekeeper bot commented Mar 25, 2025

Uh oh!

openjdk bot commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openjdk bot commented Mar 25, 2025

Uh oh!

eme64 commented Mar 25, 2025

Uh oh!

openjdk bot commented Mar 25, 2025

Uh oh!

eme64 commented Mar 25, 2025

Uh oh!

eme64 commented Mar 25, 2025

Uh oh!

openjdk bot commented Mar 25, 2025

Uh oh!

eme64 commented Mar 25, 2025

Uh oh!

eme64 commented Mar 25, 2025

Uh oh!

openjdk bot commented Mar 25, 2025

Uh oh!

openjdk bot commented Mar 25, 2025

Uh oh!

openjdk bot commented Mar 25, 2025

Uh oh!

mlbridge bot commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

tobiasholenstein commented Mar 25, 2025

Uh oh!

galderz left a comment

Choose a reason for hiding this comment

Uh oh!

eme64 commented Mar 27, 2025

Uh oh!

galderz commented Mar 31, 2025

Uh oh!

chhagedorn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chhagedorn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eme64 commented Mar 25, 2025 •

edited by openjdk bot

Loading

openjdk bot commented Mar 25, 2025 •

edited

Loading

mlbridge bot commented Mar 25, 2025 •

edited

Loading