[0.530] AssertionError: Internal error: Unresolved forward reference to UnboundType

[Daenyth]

Traceback (most recent call last):
  File "/Users/daenyth/Curata/cmp/.tox/mypy/bin/mypy", line 11, in <module>
    sys.exit(console_entry())
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/main.py", line 103, in type_check_only
    options=options)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 198, in build
    graph = dispatch(sources, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 1841, in dispatch
    process_graph(graph, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 2091, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 2203, in process_stale_scc
    graph[id].write_cache()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 1822, in write_cache
    self.manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 1088, in write_cache
    data = tree.serialize()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 238, in serialize
    'names': self.names.serialize(self._fullname),
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2432, in serialize
    data[key] = value.serialize(fullname, key)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2371, in serialize
    data['node'] = self.node.serialize()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2194, in serialize
    'names': self.names.serialize(self.fullname()),
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2432, in serialize
    data[key] = value.serialize(fullname, key)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2371, in serialize
    data['node'] = self.node.serialize()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 533, in serialize
    'type': None if self.type is None else self.type.serialize(),
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 857, in serialize
    'arg_types': [t.serialize() for t in self.arg_types],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 857, in <listcomp>
    'arg_types': [t.serialize() for t in self.arg_types],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 496, in serialize
    data['args'] = [arg.serialize() for arg in self.args]
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 496, in <listcomp>
    data['args'] = [arg.serialize() for arg in self.args]
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 994, in serialize
    'items': [t.serialize() for t in self.items],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 994, in <listcomp>
    'items': [t.serialize() for t in self.items],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 1410, in serialize
    assert False, "Internal error: Unresolved forward reference to {}".format(name)
AssertionError: Internal error: Unresolved forward reference to UnboundType

I get this running on my codebase after updating to 0.530. It passes without error in 0.521. I'm not sure what's triggering it - if you can help me figure that out I can try to minimize the failure.

[ilevkivskyi]

This is something related to a forward reference. Far example:

x: 'A'
A = List[int]

There is a chance (but not 100%) that you have a # type: ignore on the offending line.

Can you actually share the code? Alternatively I can prepare a "diagnostic" branch and then you could show me the output.

[Daenyth]

Unfortunately I can't share the code since it's closed source. I'll try to figure out which sub module is causing it and try to find a minimal reproduction.

The codebase is python2, if that helps.

[Daenyth]

Running with -v I get this immediately prior to the traceback:

LOG:  Processing the last 247 queued SCCs
LOG:  Processing SCC singleton (cmp.reporting.types) as inherently stale
LOG:  Writing cmp.reporting.types /Users/daenyth/Curata/cmp/cmp/reporting/types.py .mypy_cache/2.7/cmp/reporting/types.meta.json .mypy_cache/2.7/cmp/reporting/types.data.json
LOG:  Build finished in 7.030 seconds with 794 modules, 55 types, and 0 errors

I can confirm that running it on that single file fails, I'll try to minimize

[Daenyth]

It's something a little more complicated than just that.

The file contains a line similar to Foo = NewType('NotFoo', int). When I fix that, mypy does not crash.
When I copy the same file contents to a foo.py not inside the package directory on disk, mypy also does not crash.

[ilevkivskyi]

The file contains a line similar to Foo = NewType('NotFoo', int)

This is a very good hint. I remember having problems with NewTypes I thought have cleaned-up them all, probably not.

[ilevkivskyi]

Could you please grep for the places where Foo was used, it looks like some of this places is not visited in the third pass of semantic analysis.

[Daenyth]

It looks like it's used in multiple files.

• The only direct use is as part of an alias for Bar = Tuple[Foo, Blah].
• Bar is always used in a typevar position, eg List[Foo] or some other generic type
• The files that it's used in have a mypy.ini section with ignore_errors = True

[ilevkivskyi]

In particular, does it appear in an overload function signature? I have found a repro, but it is not related to NewTypes, it can happen with any broken forward reference in an overload.

[Daenyth]

The codebase doesn't have any uses of overload

[ilevkivskyi]

OK, this means that we have at least two crashes :-)

[ilevkivskyi]

Could you please try the original crash with mypy installed from current master? How does the crash look like? (There are chances it will crash differently.)

[Daenyth]

LOG:  Processing SCC singleton (cmp.reporting.types) as inherently stale with stale deps (__future__ builtins datetime typing)
LOG:  Writing cmp.reporting.types /Users/daenyth/Curata/cmp/cmp/reporting/types.py .mypy_cache/2.7/cmp/reporting/types.meta.json .mypy_cache/2.7/cmp/reporting/types.data.json
LOG:  Build finished in 0.972 seconds with 12 modules, 1757 types, and 0 errors
Traceback (most recent call last):
  File "/Users/daenyth/Curata/cmp/.tox/mypy/bin/mypy", line 11, in <module>
    load_entry_point('mypy===0.540-dev-77ce5e206089e796cf3c563b52791e8a0b4d2d54', 'console_scripts', 'mypy')()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/main.py", line 103, in type_check_only
    options=options)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 199, in build
    graph = dispatch(sources, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 1865, in dispatch
    process_graph(graph, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 2115, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 2227, in process_stale_scc
    graph[id].write_cache()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 1846, in write_cache
    self.manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 1090, in write_cache
    data = tree.serialize()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 238, in serialize
    'names': self.names.serialize(self._fullname),
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2440, in serialize
    data[key] = value.serialize(fullname, key)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2379, in serialize
    data['node'] = self.node.serialize()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2170, in serialize
    'names': self.names.serialize(self.fullname()),
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2440, in serialize
    data[key] = value.serialize(fullname, key)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2379, in serialize
    data['node'] = self.node.serialize()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 510, in serialize
    'type': None if self.type is None else self.type.serialize(),
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 856, in serialize
    'arg_types': [t.serialize() for t in self.arg_types],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 856, in <listcomp>
    'arg_types': [t.serialize() for t in self.arg_types],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 496, in serialize
    data['args'] = [arg.serialize() for arg in self.args]
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 496, in <listcomp>
    data['args'] = [arg.serialize() for arg in self.args]
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 993, in serialize
    'items': [t.serialize() for t in self.items],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 993, in <listcomp>
    'items': [t.serialize() for t in self.items],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 1418, in serialize
    assert False, "Internal error: Unresolved forward reference to {}".format(name)
AssertionError: Internal error: Unresolved forward reference to Foo

[ilevkivskyi]

I tried to play with this for some time, but still can't get a repro. It is quite mysterious, since the code provoking the crash looks very simple, from the traceback I see that there is a method (most likely in a module level class) that has one of the argument types of the form OneClass[Tuple[Foo, OtherClass]] where Foo is the broken NewType definition.

@Daenyth maybe you could find one of such methods, and isolate the related code, i.e. keep the import structure between files, but only leave the necessary definitions in the corresponding files, i.e. something like this:

# file a.py
from c import C
Foo = NewType('NotFoo', int)

# file b.py
from a import Foo
Bar = Tuple[Foo, int]

# file c.py
from b import Bar
class C:
    def meth(self, x):
        # type: (List[Bar]) -> None
        pass

And then see if this causes the crash? This will be very helpful.