Description
The test package supports running tests through the test runner with full feature support, and it also supports running a _test.dart library directly as an executable, what we call a "standalone" test suite. When running as a standalone suite none of the test runner features are available. Configuration is ignored (whether in yaml, or annotations), and no arguments can be passed (so no features that rely on arguments are available).
There are also some behavior differences. One is that with the full test runner tests can be declared in a main() async { method, and declarer will wait for the returned Future to complete before running tests. When running as a standalone file the first call to group or test marks the beginning of a synchronous block in which all tests must be declared. After that synchronous execution the tests start running and it becomes an error to declare another test. (There seems to be some looseness here, it's not strictly synchronous, but it doesn't allow arbitrary async delays)
We could investigate the feasibility of removing the restriction that tests are all declared before any starts running. I suspect this is would not be worth the effort, since the impact would cascade outside of the test runner. IDEs which use the JSON reporter output would also likely need work to handle this case, since we currently report metadata about all the tests before we run them.
We could disallow asynchronous test declaration in the full test runner so that the standalone suites don't behave differently. I think the pain this would cause users of the test runner (the vast majority of test users) much more aggregate pain than the sharp edges for standalone suites (a small minority of test users).
We could introduce an API to mark that tests will be declared asynchronously, and then an API to explicitly start running the suite. Or something like Future<void> declareTests(Future<void> Function()). I'd prefer not to have to add this to the API surface area, but if there are test suites where it's difficult to refactor and there becomes a need for them to run as standalone suites this might be our only reasonable escape hatch.
Once macros have landed we could implement some macro that is applied on main() and injects some behavior that surfaces the Future returned from an async main to the global declarer. If we land the above API the macro could use it.
See discussion in dart-lang/dart_style#1499