`hereby runtests --no-typecheck --tests 'Public API'` produces bad baselines if not preceded by `npm run build`

[andrewbranch]

I guess running the tests tends to skip .d.ts emit for speed, but that emit is necessary for saving the API baseline. Consequently it’s very easy to accept and commit out-of-sync API baselines. I’m not sure what the best fix is here. @jakebailey thoughts?

[jakebailey]

Hm, this shouldn't be happening at all. The runtests task depends on the dts task. The only "hole" I intentionally left where it can be out of date is when you use the launch task in vscode as that runs the test harness directly, which I deemed to be "fine" as that should be fast and running unit tests this way already doesn't work right. At the CLI, it should always be accurate.

[jakebailey]

Are you running with --no-typecheck? In that case, it will skip dts emit.

[andrewbranch]

Ohh, yes I am 😅

[jakebailey]

I could go either way about dts being a dependency of runtests when --no-typecheck is passed; it seems like both have their pros/cons. Totally happy to change it to whatever is the least surprising, I just went for "if you say --no-typecheck, it's never going to typecheck". But, clearly we have instances where that info is needed like tests.

[andrewbranch]

It feels like the API baseline tests make less sense than they ever did to be included as regular tests in the test suite. Since it has different dependencies, it feels like the cleaner long term solution is to make it a separate task/check.

[jakebailey]

I agree. I didn't go for that in the module conversion as I was trying to minimize the change, but mostly because I wasn't sure how to add a test like that. The bundler is a script, not something in src, so I wasn't quite sure how to structure it, when it should run, etc.

I guess our tests already depend on importing something from scripts (failed-tests.cjs), so it wouldn't be out of the question to just have it run the bundler and compare the output or something.

[jakebailey]

Given the bundler is not self contained and requires the build system to have run to produce the input d.ts files, I'm more inclined to just make runtests always depend on dts no matter if --no-typecheck is provided, and keep the status quo of this being a unit test. At the very least, this will do the absolute minimal amount of typechecking possible (just is needed for typescript.d.ts and tsserverlibrary.d.ts), which to me seems okay. But, I'm not sure where we want to draw the line in the sand; the reality is that these tests are going to have to require the typecheck somewhere and I don't know if I want to make the test harness able to run hereby...