Lighthouse verification step passes with error

[marklaatikainen]

Describe the bug
Step passes with runtime error

To Reproduce
Run check against url that loads slowly

Expected behavior
Step should fail if page could not load

Additional context
Our webstore preview link takes long time to load on first load after it was built. Step passes still with SUCCESS when it has runtimeError

lighthouseCheckResults: {"code":"SUCCESS","data":[{"url":"webstore-preview-url","emulatedFormFactor":"desktop","runtimeError":"The page did not paint any content. Please ensure you keep the browser window in the foreground during the load and try again. (NO_FCP)","scores":{}}]}

Actual result
lighthouse-check: Scores passed minimum requirement ✅

[adamhenson]

@marklaatikainen - could you please update this issue to provide more details in the "To Reproduce" section and "Expected behavior" section? It looks like you provided the output of the GitHub Action which looks correct.

Please provide your configuration or some similar example configuration. I would expect there to be no scores in the output and an error message per this recent PR. Do you not see an error message logged in the output?

Is the problem that you are using foo-software/lighthouse-check-status-action and the workflow doesn't fail when you would expect it to? If so, you would definitely need to at least specify the relevant configuration for that (it's actually a different GitHub Action altogether... but it is related to this, so we can keep this open).

Regardless, I will need a minimal example and more information to understand the issue better.

[marklaatikainen]

Configuration looks like this. There are separate jobs for desktop and mobile, with different minimum scores.

It gives following output and passes both steps:

[adamhenson]

I see... thank you @marklaatikainen. As I suspected - this GitHub Action is actually working as expected. The issue resides in lighthouse-check-status-action, which is a different repo altogether.

Let's keep this open here since the other repo isn't well known. I'm busy today and tomorrow, but should be able to get to the bottom of it on Friday.

[adamhenson]

@marklaatikainen thanks again for opening this issue. It has been resolved and released in foo-software/lighthouse-check-action@9.0.0.

We now "fast fail" a job when a runtime error occurs. Example

Closing this, but feel free to comment or re-open if needed.