Rewrite .test as .py

[ikonst]

Many of us are using editors that handle Python code (syntax highlighting, symbol browsing). Would it be a good idea to restructure the .test data-files as Python files?

I've been musing about it for a while, so hear me out, then let me know if it's a stupid idea :)

Example:

-[case testFoo]
-s1: str = 42  # E: Incompatible types in assignment (expression has type "int", variable has type "str")
+def test_foo():
+  s1: str = 42  # E: Incompatible types in assignment (expression has type "int", variable has type "str")

To clarify, this wouldn't be true Python code, i.e. it won't be fed directly to CPython or Mypy; rather, it'll be a Python-flavored data file — perhaps parsed by AST (if feasible and performant), but where the function bodies are passed to the current machinery as text.

Extra qualifies like # flags: --strict-optional could be represented as decorators, e.g.

@flags('--strict-optional')
def test_foo():
   ...

[case testFoo-xfail] would become:

@pytest.mark.xfail
def test_foo():
   ...

[out] would become:

@expected_output("""
main:1: Incompatible types in assignment (expression has type "int", variable has type "str")
""")
def test_foo():
   ...

[file a.py] would become:

@testcase
def test_foo():
   ...  # main goes here


@test_foo.file('a.py')
def test_foo():
   ...  # a.py goes here

Benefits:

• First-class support in IDEs, e.g.
  
  
• The disarray of testcase options, sections and modifiers could be formalized in a Pythonic syntax and a module that would document them, e.g. from mypy.testcase import expected_output and then

  # mypy/testcase.py
  
  def expected_output(output: str) -> Testcase:
      """Documentation goes here"""
      ... 

• Can use some Python tooling like Black
• A more flexible representation than INI, while providing first-class editor experience (unlike, say, converting to YAML and putting the Python test cases into a multiline YAML block).
• More natural for newcomers?

Cons:

• Harder to implement?
• Confusing that it's Python but not really?
• Less natural to newcomers?

[hauntsaninja]

Syntax highlighting seems useful! I think the proposed syntax is a little magical; I wouldn't want to overload decorators like this. Maybe just comments to mark test case dividers would work well... We already do that for flags.

Also see mypyc/mypyc#959 where dosisod wrote some syntax highlighting for mypyc test cases

[JelleZijlstra]

For what it's worth, in pyanalyze (https://github.com/quora/pyanalyze/blob/8d0e9d4fcfa0c5875dd5ff93c6ec1ae188bf9738/pyanalyze/test_name_check_visitor.py#L174) I use a system similar to what you propose, where a decorator marks test cases written in nested functions. The framework then uses inspect.getsource to get the code for the function and runs it as a test.

Mypy could use this kind of system too. It sometimes doesn't work in cases where the code cannot be placed within a function (e.g., import * is invalid in a function), and it's not obvious how to use this syntax for mypy cases that span multiple files.

[ikonst]

Syntax highlighting seems useful! I think the proposed syntax is a little magical; I wouldn't want to overload decorators like this.

Whichever mechanism we choose, I'd like it to be more ... structured? i.e. that annotations/options/modifiers would be defined in some place (call it a "schema") rather than have a loose set of conventions (like testcase.name.endswith("-xfail") -> must fail). More schema, less convention.

Then one can map (especially naively by using a "go to definition" in their editor) from an option to whatever it does, and discover what other options exist.

Screen.Recording.2023-05-31.at.11.00.31.AM.mov

The framework then uses inspect.getsource to get the code for the function and runs it as a test.

The intention is not to let pytest naively load the module and collect functions that begin with "test_". As I said, not CPython nor Mypy would be fed the module as is. Instead, we would still implement a custom pytest collector, which might leverage AST for parsing if that works out. The Pythonic syntax will only be used as a form of data-serialization language that gets favorable IDE treatment and familiar to Mypy contributors.

it's not obvious how to use this syntax for mypy cases that span multiple files.

See my proposal for a syntax above ("[file a.py] would become: ...").

[KotlinIsland]

@ikonst in PyCharm/Idea you can associate the .test file extension with Python, then you get some degree of language support.

Of course, this isn't a solution, but it's better than nothing.

Here is the setting: