Is it possible to increase available memory?

[mraible]

I ask because I'm getting the following error:

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

You can see my actions definition here.

[fniephaus]

Hi @mraible, thanks for reaching out. According to actions/runner#1051 (comment), memory seems more restrictive on public Windows runners for some reason.

I don't think there's much we can do about that as GitHub provides the service. However, we are working on making GraalVM Native Image more memory-efficient.

Anyway, I can think of three options for you:

1. Try Linux or macOS instead of Windows.
2. Reduce the amount of work happening in one build. The build you linked ran for 1.5 hours. You could split the build into different stages, e.g., one that builds the jars of your app and one that builds a native executable with GraalVM Native Image.
3. Set up your own Windows-based GitHub runner on a machine with additional memory resources.

Please feel free to share your experience and let us know if you found another way to make things work.

[mraible]

@fniephaus The job I linked to is on Linux. It works fine on Mac and the job completes in 28m. Linux gets this error after 1h and 29m. With Windows, I get a different error:

The command line is too long.

[fniephaus]

4839.7s (94.8% of total time) in 839 GCs | Peak RSS: 6.13GB | CPU load: 1.97

That's a lot of time spent in GC (~81min) and explains why the build ran for so long. Nonetheless, maybe splitting the builds could help to release some memory pressure.

[fniephaus]

@fniephaus The job I linked to is on Linux. It works fine on Mac and the job completes in 28m. Linux gets this error after 1h and 29m. With Windows, I get a different error:

The command line is too long.

Apologies for mixing that up. Let me look into this next week.

[fniephaus]

So, I looked into the two build failures:

1. Regarding the Windows issue: it seems you're using an older version of Spring Boot. Could you try bumping that? Use a classpath jar under Windows native-build-tools#126 seems to fix the same problem in our native build tools so if bumping doesn't work, I think it'd makes sense to file an issue against Spring Boot.
2. Regarding the OOM issue: it seems that for Linux and Windows, free GitHub runner only provide 7GB of RAM while macOS runners provide twice as much. That explains, why there's a lot more memory pressure in your Linux build. I see you already tried increasing the swap size. Did that help (some Linux builds were canceled because of the Windows problem). How much RAM (Peak RSS is printed as part of the Native Image build output) does the build process consume when you build your project locally?

[fniephaus]

Also, GitLab recommends adjusting vm.swappiness (via sudo sysctl vm.swappiness=10) in memory-constrained environments. Maybe that helps as well?

[fniephaus]

Adjusting vm.swappiness does not seem to help in your case: https://github.com/fniephaus/auth0-full-stack-java-example/runs/5275349582?check_suite_focus=true#step:5:10.

On macOS, peak RSS is around 9GB, which is not too far away from 7GB but apparently problematic on Linux and Windows. We also have to keep in mind that the Native Image builder is invoked from Maven, which also requires some memory.

I'm afraid I'm running out of ideas at the moment but will keep my eyes open.

[mraible]

Regarding the Windows issue: it seems you're using an older version of Spring Boot. Could you try bumping that?

The GitHub action is pointing to the spring-native branch:

https://github.com/oktadev/auth0-full-stack-java-example/blob/main/.github/workflows/publish.yml#L10

That branch is using Spring Boot 2.6.3.

[mraible]

I was able to fix the OOM error on Linux by specifying -J-Xmx7g in the build arguments:

<plugin>
    <groupId>org.graalvm.buildtools</groupId>
    <artifactId>native-maven-plugin</artifactId>
    <version>${native-buildtools.version}</version>
    <extensions>true</extensions>
    <executions>
        <execution>
            <id>test-native</id>
            <phase>test</phase>
            <goals>
                <goal>test</goal>
            </goals>
        </execution>
        <execution>
            <id>build-native</id>
            <phase>package</phase>
            <goals>
                <goal>build</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <imageName>native-executable</imageName>
        <buildArgs>
            <buildArg>--no-fallback --verbose -J-Xmx7g</buildArg>
        </buildArgs>
    </configuration>
</plugin>

I also found that changing it from 7g to 10g drops the build time by about 5 minutes.

I posted my problem with "command is too long" on Windows to Stack Overflow.

[fniephaus]

I was able to fix the OOM error on Linux by specifying -J-Xmx7g in the build arguments

Great, thanks for the update! Now that I think about it, it makes sense: the JVM allocates only a percentage of available RAM by default, so it probably never actually used the additional swap space that we set up.

902.9s (41.4% of total time) in 150 GCs

That's still a lot of time spent in GCs. Maybe some additional tuning will help but glad things work for now.

I posted my problem with "command is too long" on Windows to Stack Overflow.

Could you bump the native build tools to 0.9.10 and try again (see graalvm/native-build-tools#214)?

[mraible]

Could you bump the native build tools to 0.9.10 and try again (see graalvm/native-build-tools#214)?

I tried this here. The Windows build hasn't failed yet, but it has taken over an hour (so far).

Update: it almost worked.

[2/7] Performing analysis...  [*********]                                                             (2562.6s @ 5.97GB)
Warning: Could not register complete reflection metadata for org.springframework.boot.actuate.health.ReactiveHealthEndpointWebExtension. Reason(s): java.lang.NoClassDefFoundError: reactor/core/publisher/Mono
  33,826 (93.95%) of 36,005 classes reachable
  56,486 (79.62%) of 70,946 fields reachable
 170,797 (65.78%) of 259,658 methods reachable
   2,318 classes,   803 fields, and 11,795 methods registered for reflection
      82 classes,    78 fields, and    67 methods registered for JNI access
[3/7] Building universe...                                                                             (299.9s @ 6.08GB)
Error: Image build request failed with exit status 1

[fniephaus]

Update: it almost worked.

Good! I assume this ran with -J-Xmx7g? It's weird that we don't see an error, so maybe this is another OOM crash? Maybe try again with -J-Xmx8g?

[mraible]

I assume this ran with -J-Xmx7g

I'm currently using -J-Xmx10g.

I was able to fix the windows build by setting the minimum pagefile size to 10GB! 🎉

You can see the successful run for details.

Time spent building native images:

• macOS: 31m 30s
• Linux: 33m 50s
• Windows: 59m 45s