Skip to content

dmypy crashes every 2nd run when trying to update unlreated and unchanged numpy.compat #11009

New issue
New issue
Closed
Closed
dmypy crashes every 2nd run when trying to update unlreated and unchanged numpy.compat#11009
Labels
crashtopic-daemondmypy
@aberres

Description

@aberres
aberres
opened on Aug 23, 2021 · edited by aberres

Crash Report

When dmypy is run the second time it crashes when trying to update its internal state. No code has been changed between runs.

Traceback

Traceback (most recent call last):
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/dmypy_server.py", line 229, in serve
    resp = self.run_command(command, data)
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/dmypy_server.py", line 272, in run_command
    return method(self, **data)
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/dmypy_server.py", line 331, in cmd_run
    return self.check(sources, is_tty, terminal_width)
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/dmypy_server.py", line 393, in check
    messages = self.fine_grained_increment_follow_imports(sources)
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/dmypy_server.py", line 637, in fine_grained_increment_follow_imports
    messages = fine_grained_manager.update([], to_delete)
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/server/update.py", line 272, in update
    messages = self.manager.errors.new_messages()
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/errors.py", line 587, in new_messages
    msgs.extend(self.file_messages(path))
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/errors.py", line 575, in file_messages
    return self.format_messages(self.error_info_map[path], source_lines)
  File "/Users/berres/venv/xyx/lib/python3.9/site-packages/mypy/errors.py", line 548, in format_messages
    source_line = source_lines[line - 1]
IndexError: list index out of range

To Reproduce

I do not yet have a stand alone repo, but collected some logs what happens.

The problematic call is simply

dmypy run -- --config-file mypy.ini -p flask_rest_json_api

In the config we set pretty=True which causes the error to show up.

When the crash happens, dmypy tries to update the error messages for the system wide installed numpy. The detected error is absolutely bogus. The file python3.9/site-packages/numpy/core/overrides.py has only 227 lines while multiple errors are detected between lines 2936 and 3363 (see log below). Hence the crash.

The thing is: We are not at all interested in these numpy errors. Numpy should not be a package errors are reported for. And the first run actually does not report any numpy errors.

The verbose logs from the second run. As we see numpy.compat is considered to be deleted. The integrity_problem log has beed added by myself.

LOG:  fine-grained increment: find_changed: 0.004s
changed []
new_files []
to_delete: [('email.mime', '/Users/berres/venv/xyz/lib/python3.9/site-packages/mypy/typeshed/stdlib/email/mime/__init__.pyi'), ('html', '/Users/berres/venv/xyz/lib/python3.9/site-packages/mypy/typeshed/stdlib/html/__init__.pyi'), ('numpy.compat', '/Users/berres/venv/xyz/lib/python3.9/site-packages/numpy/compat/__init__.py'), ('numpy.compat.py3k', '/Users/berres/venv/xyz/lib/python3.9/site-packages/numpy/compat/py3k.py')] <class 'list'>
LOG:  fine-grained: ==== update 'email.mime', 'html', 'numpy.compat', 'numpy.compat.py3k' ====
LOG:  fine-grained: previous targets with errors: ['flask_rest_json_api.resource.ResourceDetail.patch', 'flask_rest_json_api.resource.ResourceList.get']
LOG:  fine-grained: --- update single 'email.mime' ---
LOG:  fine-grained: delete module 'email.mime'
LOG:  fine-grained: triggered: ['<email.mime>', '<email.mime[wildcard]>']
LOG:  fine-grained: triggered: []
LOG:  fine-grained: update once: email.mime in 0.001s - 3 left
LOG:  fine-grained: --- update single 'html' ---
LOG:  fine-grained: delete module 'html'
LOG:  fine-grained: triggered: ['<html.AnyStr>', '<html.escape>', '<html.unescape>', '<html>', '<html[wildcard]>']
LOG:  fine-grained: triggered: []
LOG:  fine-grained: update once: html in 0.000s - 2 left
LOG:  fine-grained: --- update single 'numpy.compat' ---
LOG:  fine-grained: delete module 'numpy.compat'
LOG:  fine-grained: triggered: ['<numpy.compat.Path>', '<numpy.compat._inspect>', '<numpy.compat.asbytes>', '<numpy.compat.asbytes_nested>', '<numpy.compat.asstr>', '<numpy.compat.asunicode>', '<numpy.compat.asunicode_nested>', '<numpy.compat.basestring>', '<numpy.compat.bytes>', '<numpy.compat.contextlib_nullcontext>', '<numpy.compat.formatargspec>', '<numpy.compat.getargspec>', '<numpy.compat.getexception>', '<numpy.compat.integer_types>', '<numpy.compat.is_pathlib_path>', '<numpy.compat.isfileobj>', '<numpy.compat.long>', '<numpy.compat.npy_load_module>', '<numpy.compat.open_latin1>', '<numpy.compat.os_PathLike>', '<numpy.compat.os_fspath>', '<numpy.compat.pickle>', '<numpy.compat.py3k>', '<numpy.compat.sixu>', '<numpy.compat.strchar>', '<numpy.compat.unicode>', '<numpy.compat>', '<numpy.compat[wildcard]>']
LOG:  fine-grained: process: numpy.compat
LOG:  fine-grained: Can't find matching target for numpy.compat (stale dependency?)
LOG:  fine-grained: process: numpy.core.overrides
LOG:  fine-grained: triggered: []
LOG:  fine-grained: update once: numpy.compat in 2.971s - 1 left
LOG:  fine-grained: --- update single 'numpy.compat.py3k' ---
LOG:  fine-grained: delete module 'numpy.compat.py3k'
LOG:  fine-grained: triggered: ['<numpy.compat.py3k.Path>', '<numpy.compat.py3k.abc>', '<numpy.compat.py3k.abc_ABC>', '<numpy.compat.py3k.asbytes>', '<numpy.compat.py3k.asbytes_nested>', '<numpy.compat.py3k.asstr>', '<numpy.compat.py3k.asunicode>', '<numpy.compat.py3k.asunicode_nested>', '<numpy.compat.py3k.basestring>', '<numpy.compat.py3k.bytes>', '<numpy.compat.py3k.contextlib_nullcontext>', '<numpy.compat.py3k.getexception>', '<numpy.compat.py3k.integer_types>', '<numpy.compat.py3k.io>', '<numpy.compat.py3k.is_pathlib_path>', '<numpy.compat.py3k.isfileobj>', '<numpy.compat.py3k.long>', '<numpy.compat.py3k.npy_load_module>', '<numpy.compat.py3k.open_latin1>', '<numpy.compat.py3k.os>', '<numpy.compat.py3k.os_PathLike>', '<numpy.compat.py3k.os_fspath>', '<numpy.compat.py3k.pickle>', '<numpy.compat.py3k.sixu>', '<numpy.compat.py3k.strchar>', '<numpy.compat.py3k.sys>', '<numpy.compat.py3k.unicode>', '<numpy.compat.py3k>', '<numpy.compat.py3k[wildcard]>']
LOG:  fine-grained: process: numpy.compat.py3k.asbytes_nested
LOG:  fine-grained: Can't find matching target for numpy.compat.py3k.asbytes_nested (stale dependency?)
LOG:  fine-grained: process: numpy.compat.py3k.contextlib_nullcontext
LOG:  fine-grained: Can't find matching target for numpy.compat.py3k.contextlib_nullcontext (stale dependency?)
LOG:  fine-grained: process: numpy.compat
LOG:  fine-grained: Can't find matching target for numpy.compat (stale dependency?)
LOG:  fine-grained: process: numpy.compat.py3k
LOG:  fine-grained: Can't find matching target for numpy.compat.py3k (stale dependency?)
LOG:  fine-grained: process: numpy.compat.py3k.asunicode_nested
LOG:  fine-grained: Can't find matching target for numpy.compat.py3k.asunicode_nested (stale dependency?)
LOG:  fine-grained: triggered: []
LOG:  fine-grained: update once: numpy.compat.py3k in 2.855s - 0 left
LOG:  fine-grained: process target with error: flask_rest_json_api.resource.ResourceList.get
LOG:  fine-grained: process target with error: flask_rest_json_api.resource.ResourceDetail.patch
LOG:  fine-grained: triggered: []
prev errors {'flask_rest_json_api.resource.ResourceList.get', 'flask_rest_json_api.resource.ResourceDetail.patch'}
errors {'/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py': [<mypy.errors.ErrorInfo object at 0x10b347820>, <mypy.errors.ErrorInfo object at 0x10b3477c0>, <mypy.errors.ErrorInfo object at 0x10b347910>, <mypy.errors.ErrorInfo object at 0x10b347940>, <mypy.errors.ErrorInfo object at 0x10b347970>, <mypy.errors.ErrorInfo object at 0x10b3478e0>, <mypy.errors.ErrorInfo object at 0x10b3479d0>, <mypy.errors.ErrorInfo object at 0x10b347a30>, <mypy.errors.ErrorInfo object at 0x10b347a00>, <mypy.errors.ErrorInfo object at 0x10b347a60>], '/Users/berres/Devel/mpptool/backend/flask-rest-json-api/flask_rest_json_api/resource.py': [<mypy.errors.ErrorInfo object at 0x109fe00a0>, <mypy.errors.ErrorInfo object at 0x108fcc820>]}
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 2936 Class numpy.number has abstract attributes "__init__"
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 2936 If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3088 Class numpy.integer has abstract attributes "__init__"
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3088 If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3244 Class numpy.inexact has abstract attributes "__init__"
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3244 If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3344 Class numpy.flexible has abstract attributes "__init__"
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3344 If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3363 Class numpy.character has abstract attributes "__init__"
integrity_problem /Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py 227 3363 If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
LOG:  fine-grained increment: update: 5.955s

Your Environment

  • Mypy version used: 0.920+dev.7576f659d42ee271c78e69d7481b9d49517a49f6
  • Mypy configuration options from mypy.ini (and other config files):
[mypy]
ignore_missing_imports = True
color_output = True
pretty = True
show_error_codes = True
check_untyped_defs = True
warn_unreachable = True
error_summary = True
  • Python version used: 3.9
  • Operating system and version: MacOS

Activity

aberres
added
crash
on Aug 23, 2021
aberres

aberres commented on Aug 23, 2021

@aberres
aberres
on Aug 23, 2021
Author

No idea if this is intended or not, but when I compare the dmypy and the mypy output I notice something else:

dmypy checks way more files, while mypy only cares about the passed package. When I do not pass pretty=True instead of a crash I get the bogus error messages for numpy as errors:

First run

> dmypy run -- --config-file mypy.ini -p flask_rest_json_api
backend/flask-rest-json-api/flask_rest_json_api/resource.py:146: error: Need type annotation for "schema_kwargs"  [var-annotated]
backend/flask-rest-json-api/flask_rest_json_api/resource.py:272: error: Need type annotation for "schema_kwargs"  [var-annotated]
Found 2 errors in 1 file (checked 405 source files)

Second run

> dmypy run -- --config-file mypy.ini -p flask_rest_json_api
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:2936: error: Class numpy.number has abstract attributes "__init__"  [misc]
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:2936: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3088: error: Class numpy.integer has abstract attributes "__init__"  [misc]
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3088: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3244: error: Class numpy.inexact has abstract attributes "__init__"  [misc]
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3244: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3344: error: Class numpy.flexible has abstract attributes "__init__"  [misc]
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3344: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3363: error: Class numpy.character has abstract attributes "__init__"  [misc]
/Users/berres/venv/xyx/lib/python3.9/site-packages/numpy/core/overrides.py:3363: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
backend/flask-rest-json-api/flask_rest_json_api/resource.py:146: error: Need type annotation for "schema_kwargs"  [var-annotated]
backend/flask-rest-json-api/flask_rest_json_api/resource.py:272: error: Need type annotation for "schema_kwargs"  [var-annotated

Pure mypy

> mypy -p flask_rest_json_api --config mypy.ini
backend/flask-rest-json-api/flask_rest_json_api/resource.py:146: error: Need type annotation for "schema_kwargs"  [var-annotated]
backend/flask-rest-json-api/flask_rest_json_api/resource.py:272: error: Need type annotation for "schema_kwargs"  [var-annotated]
Found 2 errors in 1 file (checked 24 source files)
aberres
changed the title [-]dmypy crashes every 2nd run when trying to update unchanged numpy.compat[/-] [+]dmypy crashes every 2nd run when trying to update unlreated and unchanged numpy.compat[/+] on Aug 23, 2021
aberres

aberres commented on Aug 23, 2021

@aberres
aberres
on Aug 23, 2021
Author

I am trying to come up with a minimal reproduction, but it is not really obvious for me, why the numpy error is raised here.

What I know is that I have a dependency chain flask_rest_json_api->OtherPackage->numpy (not nice, but...).

When trying to reproduce I noticed something else which might be related: dmypy has sometimes problems to only check for errors in the specified package.

Repro:

  1. Install a package which has no mypy errors
  2. Start dmypy server like dmypy daemon -- -p my_package --verbose
  3. Trigger check dmypy run -- -p my_package --verbose - everything is fine
  4. Add an import which has mypy errors when checked individually. E.g. import marshmallow.
  5. Rerun dmypy run -- -p my_package --verbose and observe a bunch of errors directly in marshmallow.
  6. Restart dmypy and observe that errors are gone.
JelleZijlstra

JelleZijlstra commented on Aug 23, 2021

@JelleZijlstra
JelleZijlstra
on Aug 23, 2021
Member

Does this reproduce on master? It sounds like the kind of issue that #10937 recently fixed.

aberres

aberres commented on Aug 23, 2021

@aberres
aberres
on Aug 23, 2021
Author

@JelleZijlstra Yeps, could reproduce with master. Only dmymy has the problem, could not crash things with mypy only.

bobwhitelock

bobwhitelock commented on Sep 2, 2021

@bobwhitelock
bobwhitelock
on Sep 2, 2021
Contributor

I'm getting the same issues the second time I run dmypy run ., after just updating to numpy 1.21.2 (I get the same extra errors reported for numpy, and I get that crash if I add pretty = True to my config). Using Python 3.8 and mypy 0.790.

97littleleaf11
added
topic-daemondmypy
on Dec 7, 2021
bobwhitelock

bobwhitelock commented on Apr 13, 2022

@bobwhitelock
bobwhitelock
on Apr 13, 2022
Contributor

I'm getting the same issues the second time I run dmypy run ., after just updating to numpy 1.21.2 (I get the same extra errors reported for numpy, and I get that crash if I add pretty = True to my config). Using Python 3.8 and mypy 0.790.

I no longer see this issue on numpy 1.22.3 and Python 3.9.7, so possibly one of these updates fixed the issue. Still using the same version of mypy.

Although if a change in numpy can cause the daemon to crash like this it's possible there's still an underlying issue that needs to be fixed, but this no longer seems as important if it has cleared up

oxidase
mentioned this on Jun 21, 2022
sterliakov

sterliakov commented on May 3, 2025

@sterliakov
sterliakov
on May 3, 2025
Collaborator

The root cause was likely fixed in #18906.

sterliakov
closed this as completedon May 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    crashtopic-daemondmypy

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @JelleZijlstra@bobwhitelock@97littleleaf11@aberres@sterliakov

        Issue actions
          dmypy crashes every 2nd run when trying to update unlreated and unchanged numpy.compat · Issue #11009 · python/mypy