Are there any specs for the typed AST?

[mbdevpl]

I've started experimenting with this project and it works really nice so far, but what I find that is missing is the description of the typed AST. For ordinary AST there is some effort going on here: https://greentreesnakes.readthedocs.io/en/latest/nodes.html . However, is there any kind of documentation with regard to how typed_ast.ast35 should look?

E.g. type_comment seems to be treated as just a string. However what is done e.g. in mypy is the type comment is treated as code. Is that intended?

my_string = None # type: typing.Optional[str]

Module(
  body=[Assign(
    targets=[Name(
      id='my_string',
      ctx=Store())],
    value=NameConstant(value=None),
    type_comment='typing.Optional[str]')],
  type_ignores=[])

Another example is here, where type annotations are not treated as strings, but are parsed normally:

def function(a: int, b: int) -> int:
    return a + b

Module(
  body=[FunctionDef(
    name='function',
    args=arguments(
      args=[
        arg(
          arg='a',
          annotation=Name(
            id='int',
            ctx=Load())),
        arg(
          arg='b',
          annotation=Name(
            id='int',
            ctx=Load()))],
      vararg=None,
      kwonlyargs=[],
      kw_defaults=[],
      kwarg=None,
      defaults=[]),
    body=[Return(value=BinOp(
      left=Name(
        id='a',
        ctx=Load()),
      op=Add(),
      right=Name(
        id='b',
        ctx=Load())))],
    decorator_list=[],
    returns=Name(
      id='int',
      ctx=Load()),
    type_comment=None)],
  type_ignores=[])

[ddfisher]

The difference between ast35 and Python 3.5's ast module are documented briefly in the module docstring.

At parse time, type comments and type annotations are very different beasts. Type annotations are parsed as expressions because they actually are expressions that are executed at function definition time. (This behavior is already part of the standard Python 3.5 ast module and is unchanged in ast35.) Type comments are always returned as strings in the initial parse, but mypy calls back into the parser (with mode='eval') to parse them as expressions.

That said, it seems like it might now make sense to parse the type comments as expressions. I'll think about doing that.

[mbdevpl]

Thanks a lot for the answer! This will do. Next time I should read the docstrings more closely.

As a side note, I've published an experimental unparser for your typed-ast, called typed-astunparse: https://github.com/mbdevpl/typed-astunparse - I use it for my research, but if it would have any use otherwise, I'd be very glad :)

BTW, since you seem to be on the forefront of Python development, do you happen to know of any discussion about introducing AST unparsing into core?

Actually, there has been a hidden unparse.py script in Python source distribution for a very long time now, someone even extracted it and repackaged into astunparse module... It's just that ast.unparse() is not that hard to imagine, and it would be so nice :) And compile() flag enabling type comments parsing would also be sooo nice :)

Or, question more related to typed-ast project: would typed ast unparser be welcome in this module, if there happens to be a volunteer to write it? It's ok to have ast/astunparse typed-ast/typed-astunparse - it's just that for me these really belong together.

Asking just out of curiosity, please answer if you happen to have some free time :)

[gvanrossum]

I'm curious about the use case for unparsing without preservation of non-type comments. How common is this need? Wouldn't you be better off saving the original source code?

[mbdevpl]

I'm doing a kind of source-to-source translation project. The original code (and sometimes target code) is not necessarily Python. I'm experimenting with type hints as a method of increasing performance.

[gvanrossum]

OK, so the output is not meant for human consumption, right? You're essentially "transpiling" some language to Python so it can be executed. Then the ast unparser makes total sense.