[Bug]: System.OutOfMemoryException at VirtualBox.IDisplay.TakeScreenShotToArray

[mike-friedrich]

Version

7.2.4

Host OS Type

Windows

Host OS name + version

Windows 11 Enterprise 23H2 build 22631.6060

Host Architecture

x86

Guest OS Type

Windows

Guest Architecture

x86

Guest OS name + version

WinPE from Windows 10 IoT Enterprise 21H2

Component

API

What happened?

We prepared a custom Windows 10 IoT installer ISO file. This is a normal bootable ISO with a lot extra files.

We try to detect a screen like this and when we see it, we retry the installation from ISO:

For the installation we simply wait until the VM shuts down and then would start capturing the VM's disk.
To catch and retry we take screen captures every 100ms and compare it with the attached image.
While we wait and try to detect problems, we also monitor the process resources of the tool (C#) calling the VirtualBox API. This is the output:
Memory Usage: WorkingSet=33MB | Private=24MB | Virtual=4713MB | Managed=1MB
Memory Usage: WorkingSet=33MB | Private=24MB | Virtual=4713MB | Managed=1MB
Memory Usage: WorkingSet=13MB | Private=24MB | Virtual=4713MB | Managed=1MB
Memory Usage: WorkingSet=10MB | Private=24MB | Virtual=4713MB | Managed=1MB
Memory Usage: WorkingSet=9MB | Private=24MB | Virtual=4713MB | Managed=1MB
Memory Usage: WorkingSet=9MB | Private=24MB | Virtual=4713MB | Managed=1MB
Memory Usage: WorkingSet=9MB | Private=24MB | Virtual=4713MB | Managed=1MB
Memory Usage: WorkingSet=8MB | Private=24MB | Virtual=4713MB | Managed=1MB
Memory Usage: WorkingSet=8MB | Private=24MB | Virtual=4713MB | Managed=1MB
Memory Usage: WorkingSet=8MB | Private=24MB | Virtual=4713MB | Managed=1MB
Memory Usage: WorkingSet=7MB | Private=24MB | Virtual=4713MB | Managed=1MB
Memory Usage: WorkingSet=8MB | Private=24MB | Virtual=4713MB | Managed=1MB

System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
at VirtualBox.IDisplay.TakeScreenShotToArray(UInt32 aScreenId, UInt32 aWidth, UInt32 aHeight, BitmapFormat aBitmapFormat)
at vboxrun.Program.DetectOnScreen(IMachine machine, UInt32 monitorId, Bitmap refBmp)
at vboxrun.Program.RunUntilPoweredOff(IMachine machine)
at vboxrun.Program.Run()
vboxrun exit code: 2

I suspect a memory leak within VirtualBox.IDisplay.TakeScreenShotToArray as the working set constantly shrinks until TakeScreenShotToArray gives up with an exception. We call the GC between each loop every 100ms but this is unable to free that memory, hence me believing the leak is in unmanaged code.

This screen detection was working fine in 6.1 and is occurring in 7.1 as well. Unsure if it also occurs in 7.0.
The memory leak appears to not affect the VM running status at first glance but cannot verify due to #426.

How can we reproduce this?

Repeat multiple screenshots in an endless loop (100ms sleep and GC in between) using VirtualBox.IDisplay.TakeScreenShotToArray with either PNG or RGB format.

The screen size is standard Windows Installer resolution or standard boot screen resolution, nothing remotely large. See attached picture we try to detect.

`
IDisplay display = session.Console.Display;

GuestMonitorStatus status;
display.GetScreenResolution(monitorId, out uint width, out uint height, out uint bits, out int xOrigin, out int yOrigin, out status);
if (status == GuestMonitorStatus.GuestMonitorStatus_Enabled)
{
// Request PNG format for screenshot
byte[] bytes = (byte[])display.TakeScreenShotToArray(monitorId, width, height, BitmapFormat.BitmapFormat_PNG);
...
bytes = null;
}
`

Did you upload all of your necessary log files, screenshots, etc.?

• Yes, I've uploaded all pertinent files to this issue.