libcnb-test: Expose Pack build output as combined stdout + stderr

[edmorley]

So in writing some libcnb-test integration tests now, I found it frustrating that there's no way to get a unified stdout+stderr output, since in most cases I don't really care about testing whether I've output to stdout vs stderr (after all, log_error will take care of that, so there isn't much room for user error).

However, I do care about how the output renders as a whole, which I can't check properly if half of it is on stdout, and the other half is on stderr. (Even if I stitch them together afterwards, the ordering will then be wrong.)

Also, relatedly, I seems to be regularly running into occasions where I'm debugging a test, and one of my output assertions is failing, but it's not clear why, because the useful part of the build log is on the other stream. But of course the test stops running after the first assertion failure (given it's a panic), and so I never get to see the assertion against the other stream. This means I then have to spend time thinking about the "sensible order" of asserting stdout vs stderr, and varying that order depending on whether the test is testing a success case, or a failure case.

All in all, it seems this is adding friction, for very little test coverage gain.

As such, I'm wondering if we shouldn't just combine the Pack CLI's stdout+stderr into one (tie the two together in the Command), and have libcnb-test only expose that as a single output?

Pros:

• Reduced mental burden of having to think about whether one is asserting against stdout or stderr
• Able to test visual UX of full info/warning/error output in cases that are currently untestable (if the output straddles stdout and stderr)
• Less frustrating debugging of test failures due to the assertion issue mentioned above
• Parity with Hatchet

Cons:

• Not able to test that particular log output is on the expected stream. (Though IMO this isn't a significant drawback given we plan on leaning heavily on shared utils like log_error(), which will be widely used/tested, so already outputting on the correct stream)

We could also explore providing both a unified output and the original separate streams, though that does seem to increase implementation and libcnb-test public API complexity, for potentially little benefit.

[Malax]

I feel taking away the separation would be a mistake - they are separate concepts and there might be edge-cases where we want to assert against them separately.

I also get your use-case though, in many cases we don't care and having to deal with separate steams in such cases is a hassle for sure.

Long story short, I'm in favour as an additional (maybe even as the default) feature.

[edmorley]

I've been looking into this today.

An initial first step seems like it would be to add an opt-in way to redirect stdout to stderr or vice versa. That way we'd still keep the same shape of API for the consumer (the pack_stdout and pack_stderr properties on TestContext) - rather than a pack_combined_output property that would either mean TestContext has to be an enum or something equally awkward.

However, when using Command, it's currently not simple to have the child processes' stdout/stderr both be sent to a single destination inherited from the parent.

The best approach I can find that works on current stable Rust is this crate:
https://crates.io/crates/os_pipe

Though in Rust 1.87 (the next stable version, due to be released on 2025-05-15) there is a new stabilised anonymous pipe API which I believe will mean we won't need the external crate:
rust-lang/rfcs#871 (comment)
rust-lang/libs-team#375
rust-lang/rust#127154

[schneems]

An alternative approach to fixing this in libcnb-test is to change the entire buildpack output and/or upstream:

• With current recommended usage, bullet stream proxies everything to stderr already which solves this problem. Regarding stdout/stderr: My preference is stderr since this is intended to be consumed by humans (rather than piped to another process).
• With that note: I'm vaguely interested in movingpack build output all to stderr (upstream RFC). This would allow us to emit useful information from the build to stdout and pipe it to something meaningful. For example:

$ pack build my-image-name --path . --print-json "buildpacks" | jq -r '.buildpacks[] | "\(.name)@\(.version)"'
heroku/ruby@5.0.0
heroku/procfile@1.2.3

I don't have a strong use case, though, that demands this workflow, but the idea remains. If we push everything to stderr for users, it opens up stdout for some piped data (if we/CNCF end up wanting that ability)