feat: add --init-image flag for specifying custom init filesystem images per VM

[manuschillerdev]

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Motivation and Context

I've been tinkering with ways to extend container with a primitive for interacting with the VM layer. One approach: expose the internal init filesystem as a user-configurable option via --init-image.

What this enables:

• Custom boot-time logic before the OCI container starts
• Running additional processes and daemons (e.g., daemons to configure eBPF network filters, logging agents) inside the VM (not the container)
• Debugging/instrumenting the init process

--init-image vs using the system property
we could also use container system property set image.init vminit:latest (documented via #1032), but this affects all VMs. I want to be able to override this only for a single VM.

In my current approach, the custom image wraps the default vminitd, so it's currently behaving like an entrypoint.
Curious if this direction aligns with the project's goals, or if there's a better abstraction for these use cases.
If there is interest, I would add proper docs and tests.

Example for a minimal entry point binary
Note: We use Go here for its straightforward cross-compilation to Linux ARM64.

// custom-init/wrapper.go
package main

import (
    "fmt"
    "os"
    "syscall"
)

func main() {
    kmsg, err := os.OpenFile("/dev/kmsg", os.O_WRONLY, 0)
    if err == nil {
        kmsg.WriteString("<6>custom-init: === CUSTOM INIT IMAGE RUNNING ===\n")
        kmsg.Close()
    }

    err = syscall.Exec("/sbin/vminitd.real", os.Args, os.Environ())
    if err != nil {
        fmt.Fprintf(os.Stderr, "Failed to exec vminitd.real: %v\n", err)
        os.Exit(1)
    }
}

Containerfile

# custom-init/Containerfile
FROM ghcr.io/apple/containerization/vminit:0.20.0 AS base

FROM ghcr.io/apple/containerization/vminit:0.20.0
COPY --from=base /sbin/vminitd /sbin/vminitd.real
COPY wrapper /sbin/vminitd

• build the binary with cd custom-init && CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o wrapper wrapper.go
• build the image with bin/container build -t local/custom-init:0.20.0 custom-init/
• run the container with our custom init-image: bin/container run --name init-test --init-image local/custom-init:0.20.0 alpine:latest echo "hello"
• check the logs with cat "/Users/$USER/Library/Application Support/com.apple.container/containers/init-test/vminitd.log" | head -50

Testing

• Tested locally
• Added/updated tests
• Added/updated docs

[jglogan]

@dcantah see also #838

[manuschillerdev]

@dcantah @jglogan would that be of interest for the project? Would be totally open to other suggestions on how one can interact with the startup process. Exposing --init-fs seemed like a viable option here

[manuschillerdev]

@jglogan @dcantah

I've rebased the PR, added tests and documentation, and updated the description to clarify why container system property set image.init doesn't fit my use case (it applies globally rather than per-VM).

Happy to elaborate on any open questions or explore alternative approaches that would better align with the project's architecture.
I'd appreciate any feedback on whether this direction makes sense to pursue or if there are concerns that would make it worth closing in favor of a different approach.