Adds the unittest2ListTests compile time flag

[jmgomez]

Running:
nim c -d: unittest2ListTests -r tests/sampletests.nim

Will result in:

Suite: Sample Tests
  Test: Sample Test
    File: sampletests.nim:4

Test: Global test
    File: sampletests.nim:8


Suite: Sample Suite
  Test: Sample Test
    File: sampletests.nim:12

  Test: Sample Test 2
    File: sampletests.nim:15

  Test: Sample Test 3
    File: sampletests.nim:18

Allowing external tools (i.e. a test runner) to gather test information without actually running the tests

[arnetheduck]

won't fly - need to do it at runtime really since tests can be added dynamically as well, ie:

import unittest2

for i in 0..<10:
 test "test" & $i:
    echo "hello"

[arnetheduck]

see also the separate collect features which do half of the job

[jmgomez]

won't fly - need to do it at runtime really since tests can be added dynamically as well, ie:

import unittest2

for i in 0..<10:
 test "test" & $i:
    echo "hello"

Didnt know that was possible but running the test to discover them its a bad idea as it will make the discovery quite slow (i.e. you want to add new tests to the list when the user creates a new test with its own button to run it from the file).

[arnetheduck]

Didn't know that was possible but running the test to discover them its a bad idea as it will make the discovery quite slow

On the flip side you get to run the test faster when it's done since it's now already compiled so practically, there's no difference in edit-to-test-running time .. ie can display something like "compiling tests.." in the meantime.

Collect-then-run is how most xunit-style test frameworks work, with the option to generate test cases on the fly being common - ie it's part of the basic feature set they offer, unittest2 included. Compiled languages have this downside that it takes a while to run the suite but there's little to do to avoid it that is at the same time correct - having an incorrect test parser, or one that works only in trivial cases (where arguably you don't need a test framework or editor support) is kind of useless - ie the gui is only useful if it actually helps with the complex test suites that warrant an IDE to begin with.

[jmgomez]

On the flip side you get to run the test faster

Consider you have a test that takes 2 minutes to run but you only want to run via the runner a suite that takes < 1s to run because you are actually developing that suite.

Collect-then-run is how most xunit-style test frameworks work

Right, but they do it mostly at runtime because such languages doesnt have compile time capabilities. AFAIK most .NET frameworks didnt allow to add test to the runner at runtime.

Also consider that we are elevating the bar for other libraries to support the test runner (namely the std/unittest one)

[arnetheduck]

Consider you have a test that takes 2 minutes to run

We're not running the test - just running the test collector that prints names - compilation takes whatever it will take as usual.

languages doesnt have compile time capabilities

Dynamic tests are usually about reading files from disk that contain test data, and generating suite interactions based on that - nim also typically does not support this at compile time, except through staticExec which is cheating.

.net

https://docs.nunit.org/articles/nunit/writing-tests/attributes/testcasesource.html

Also consider that we are elevating the bar

"catching up"

for other libraries to support the test runner (namely the std/unittest one)

std/unittest works / allows this too (except it has some built-in limitations on the test suite size), though it lacks the capability to collect tests before running them (ie it doesn't have the collect feature which is one of the fundamental reasons it remains a dead end that's practically unusable beyond trivial suites).