Built-in html formatter generates a blank html report if scenarios have not finished yet completely

[seleniumghost]

👓 What did you see?

When running tests using built-in html formatter, it generates a blank HTML file until all scenarios finished completely.

✅ What did you expect to see?

HTML report to be generated should not be blank after a scenario has been finished

📦 Which tool/library version are you using?

cucumber 8.0.0
cucumber-html-formatter 19.2.0
ruby 3.1.1
Windows 10

🔬 How could we reproduce it?

Steps to reproduce the behavior:

1. Have more than one scenario in a test.
2. Execute cucumber --format HTML --out report.html
3. Open the html report when test is still running and html file is blank.

📚 Any additional context?

As I have not found it in the issue list, I'm picking up the issue mentioned in comment cucumber/cucumber-ruby#1524 (comment) as it shows up as unacceptable drawback compared with older versions, especially when having a lot of scenarios or long scenarios or even both of this. Additionally, when issues like cucumber/cucumber-ruby#1524 occur, the whole report is affected and not only a part of it.

written by marnen:

Unfortunately, the way that’s done makes the Cucumber HTML report much less useful than it used to be. Back in the day, Cucumber used to use a similar approach to RSpec: it wrote straight HTML as scenarios ran, so that the file would render something useful in the browser before it was completely written. That makes it ideal for monitoring a long run with lots of scenarios.

Current Cucumber HTML output, though, renders nothing until the run is complete (I think because window.CUCUMBER_MESSAGES isn’t well-formed JSON/JS till the final closing bracket is written), which means I need to wait for the entire run to finish before I see any output. Is there an HTML formatter for current Cucumber that doesn’t have this drawback? I haven’t found one, and I really miss the old RSpec-like behavior. (And if this is off-topic, I’ll raise a new issue or ask on the list.)

answered by mattwynne:

In fact, this kind of live monitoring of a test run from a browser should now be easier to do, given that the results are streamed out as little ndjson packets. The problem would be getting that stream from your Cucumber process to your browser. It's a really cool idea, but yes it is off-topic for this thread I think :) If you're interested in pursuing it, let's create a new issue (in https://github.com/cucumber/cucumber) and discuss it there.

---

This text was originally generated from a template, then edited by hand. You can modify the template here.

[mpkorstanje]

I don't think process monitoring is reasonable to expect. There are plugins better suited to that. However displaying a message when CUCUMBER_MESSAGES is not defined would avoid future confusion.

[seleniumghost]

Well, just to avoid misunderstanding and to be competely clear on that, I do not expect process monitoring, but partial output in the html report when a part (scenario) is finished. This means I expect behaviour which existed in former Cucumber versions. The quotes should be seen as additional supporting opinions only, which shall mean that there are other users as well seeing a problem in this changed behaviour.

[ciaranmcnulty]

Just to check, is this in a case where cucumber is writing to the filesystem and you're opening the file in your web browser?

[seleniumghost]

A) It occurs at any time when the test is still running and not every scenario in it has been completed yet.

B) It occurs if the 1st scenario is finished and the test currently runs in the 2nd scenario.

C) It stops occurring when the complete test run is finished (Takes a few seconds after that of course).

The file size of the html report file is 22 KB at the start. This file size does not change for A) and not for B) and only changes with C) to e.g. 2327 KB. This means it occurs when cucumber currently is not writing to the html file.

[aurelien-reeves]

I don't think the actual html-formatter would be able to show any progress. It aims to generate an all-inclusive static report once the execution is done.

Yes, at some point, it kinda shows some progress, but that was only a side effect of how it was built before.

What you are requesting would require a brand new dedicated formatter which would actually be able to process messages stream as they arrive and thus monitor the execution based on that stream.

[mpkorstanje]

This means I expect behaviour which existed in former Cucumber versions.

Sounds like a case of Hyrums law in action.

That reports could be viewed while test execution was in progress is merely a coincidence. It is only by the willingness of a browser to interpret and render malformed html documents that this could be done.

I don't see a reason to support this.

https://www.hyrumslaw.com/

[seleniumghost]

What you are requesting would require a brand new dedicated formatter which would actually be able to process messages stream as they arrive and thus monitor the execution based on that stream.

I just request what was working before, for many many years. Nothing new from this retro-perspective. I understand that this might be quite a turn-around to get back there and if there were any alternatives I would have not taken the effort to open and discuss this issue, but I found no alternative and user marnen mentioned above did not either.

I don't see a reason to support this.

The reasons have been told at start:

1. If you have a lot of scenarios or long running scenarios or even both of this, you see nothing until all is finished. This is a major drawback.
2. But this also means that if the test execution aborts to whatever reason, all is lost.
3. If there is any issue in the formatter itself, not only a part, but the whole report is affected.

That reports could be viewed while test execution was in progress is merely a coincidence. It is only by the willingness of a browser to interpret and render malformed html documents that this could be done.

No, I just checked this. The former partial html report was not malformed, but containing the partial content.

Sounds like a case of Hyrums law in action.

Wow, how easy it can be to block an issue of expected behaviour which was working in former versions and where also other users see a problem within. This is kind of unexpected hybris which makes me feel really sad for this world. :-(

[mpkorstanje]

https://xkcd.com/1172/

[aslakhellesoy]

I think maybe this can be solved without too much effort. Rather than writing a window.CUCUMBER_MESSAGES array at the end of the run, we could write messages as they come in - wrapped in a JavaScript function call:

<script>processMessage({...})</script>
<script>processMessage({...})</script>
<script>processMessage({...})</script>

Every time another line is added, the UI would update. We'd have to make sure it doesn't impact performance when a finished page is loaded, but I don't think that's hard to do either.

[aslakhellesoy]

Thanks for raising this issue @seleniumghost - I think fixing this would be an improvement. I might take a crack at it one day, if I can find some time.

[mpkorstanje]

Great, so now we have:

1. A blank report when opened prematurely with terrible UX and with no fix
2. A proposed commitment to a solution that is unlikely to be met.

[mpkorstanje]

We'd have to make sure it doesn't impact performance when a finished page is loaded, but I don't think that's hard to do either.

Just because it happens to work doesn't mean we should do it. That's how you get complex bugs that are hard to solve for use cases nobody should use.

There are better solutions to progress monitoring and they don't involve relying on a browsers handling of malformed files.

Keep it simple, stupid.