Jest: Custom reporter vs results processor

[mohanraj-r]

Custom Jest Reporter to output consolidated a11y issues as JSON file.

1. Reporter and tests are executed in their own sandboxes - objects cannot be mutated or propagated across including 'global' namespace. But reporter can read exception info thrown from tests.
2. Values passed to reporter hooks are readonly. Modifications e.g. to test instance pass/fail status do not take effect in Jest.
3. JSDOM environment is not available in reporter. Hence invoking automatic checks e.g. from within reporter hooks is not possible.
4. Differences exist with default test runner vs Circus (next-gen) test runner (onTestCaseResult is not triggered with default test runner in Jest v26 jestjs/jest#11405)

[mohanraj-r]

(From #62): Replaced the Custom Jest reporter with a custom test results processor

• The Jest reporter was growing complex in trying to produce a JUnit XML compatible output - with duplication in representing/processing the XML struct.

• With the test results processor we have access to the test results before they are rendered to JSON (which is a different (untyped) format - not plain serialization)

  • hence it is more straightforward to process them. With this we don't have to duplicate effort in trying to produce a JUnit compatible output
  • we can reuse existing tools that convert Jest JSON to JUnit XML.

• Just to clarify the reporter is a uni-directional workflow where Jest hands the aggregated result output off to the reporter and doesn't expect anything to be returned back

  • leaving any results conversion, file handling etc completely to the reporter.

• This would be fine if we wanted to convert the results to a different format entirely but if we wanted to stick to Jest's JSON output schema, it becomes more work.

  • We might be able to reuse formatTestResults.ts but the structs behind the JSON output are not typed/defined explicitly ?
  • making it a bit trickier to test the conversion among other things.

• Whereas the test processor sits in between

  • receiving the aggregated results from Jest and returning it back to Jest after processing
  • leaving further processing of results such as conversion to JSON, writing to file etc to Jest thereby reducing complexity a little and improving testability.

• Resources

  • jest/runJest.ts at master · facebook/jest · GitHub
  • jest/packages/jest-test-result/src at master · facebook/jest · GitHub
  • Configuring Jest · testresultsprocessor