Warn when tests are accidentally disabled due to nesting within setUp() method

[rtannenbaum]

Hello,

We recently discovered that several tests were accidentally nested inside a setUp() method.

While the tests compile, they were not actually run by the test runner, and we had no way to detect this. We only happened to notice coincidentally.

It would be helpful if the test runner could warn that tests are disabled when this happens.

Thanks for your consideration.

Rachel

[jakemac53]

Normally we would error for any calls to test that are not directly created inside main (or a group in main).

However it looks like if you have a setUp but no actual tests, then we never run the setUp and therefore never throw the expected error:

import 'package:test/test.dart';

void main() {
  setUp(() {
    test('hello', () {
      expect(1, 2);
    });
  });
}

The above for example does not fail, but if you add a real test after the call to setUp, it will fail.

[jakemac53]

From a practical standpoint I am not sure how to really resolve this, we don't want to run all setUp functions in a file if no tests are being ran in that file (maybe they were skipped or were not expected to run on the current platform etc).

[jakemac53]

One possible fix could be to warn if we see any group which has no tests defined (transitively).

That should catch most cases as long as any of those tests are actually ran in some configuration.