Pylint slow when run on script with pandas

[parantapa]

Sample script

> cat hello.py                                                                                                                                               (hodgepodge) 
"""
Hello.
"""

import pandas as pd

def hello():
    """
    Hello.
    """

    test_pdf = pd.DataFrame([[1, 2, 3]])

Running pylint

> /usr/bin/time pylint hello.py                                                                                                                              
No config file found, using default configuration
************* Module hello
W: 12, 4: Unused variable 'test_pdf' (unused-variable)

------------------------------------------------------------------
Your code has been rated at 6.67/10 (previous run: 6.67/10, +0.00)

Command exited with non-zero status 4
48.05user 0.15system 0:44.80elapsed 107%CPU (0avgtext+0avgdata 193132maxresident)k
0inputs+8outputs (0major+72586minor)pagefaults 0swaps

pylint --version output

pylint --version No config file found, using default configuration
pylint 1.9.1,
astroid 1.6.4
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0]

Q1. Is this expected behaviour?
Q1a. If so is there a way to make pylint ignore pandas?

[PCManticore]

No, this is not the expected behaviour, there is probably a check that triggers a deep pandas inference leading to this result. You can try to ignore it with --ignored-modules=pandas but it seems it's not working, since it might be a check that doesn't consider this option.

[kodonnell]

I'm definitely seeing this on Python 3.6.5 :: Anaconda, Inc. with

astroid==1.6.5
atomicwrites==1.1.5
attrs==18.1.0
certifi==2018.4.16
coverage==4.5.1
isort==4.3.4
lazy-object-proxy==1.3.1
mccabe==0.6.1
mkl-fft==1.0.0
mkl-random==1.0.1
more-itertools==4.2.0
numpy==1.14.5
pandas==0.23.1
pluggy==0.6.0
py==1.5.4
pylint==1.9.2
pytest==3.6.2
pytest-cov==2.5.1
pytest-pylint==0.9.0
python-dateutil==2.7.3
pytz==2018.4
scikit-learn==0.19.1
scipy==1.1.0
six==1.11.0
wrapt==1.10.11

It's fine if I disable all checks. I tried finding the 'culprit' check ... but a variety of checks cause the issue, and I stopped after ten or so.

[parantapa]

Sorry, about that. I hit the close issue by mistake.

[parantapa]

Does this mean there are more than 10 checks that ignore the ignore-modules=pandas directive?

[kodonnell]

As mentioned by @PCManticore that directive seems to have no impact.

Some tests using the following:

checkers.txt

$ cat test.py
import pandas
pandas.DataFrame()

$ cat time.py
from datetime import datetime
import os

times = []
with open("checkers.txt") as f:
    for i in f:
        i = i.strip()
        print(i, end="\r")
        t0 = datetime.now()
        _ = os.system("pylint --disable=all --enable=%s test.py > /dev/null 2>&1" % i)
        times.append({"checker": i, "time": datetime.now() - t0})
print()
for i in sorted(times, key=lambda x: -x["time"].total_seconds()):
    print(i["checker"].ljust(80), i["time"])

Results with above code (i.e. no ignored-modules=pandas) - showing only the long ones (all the rest were < 1 second).

missing-kwoa                                                                     0:00:43.373460
redundant-keyword-arg                                                            0:00:42.945230
unexpected-keyword-arg                                                           0:00:41.520050
consider-iterating-dictionary                                                    0:00:40.463160
abstract-class-instantiated                                                      0:00:40.146653
invalid-metaclass                                                                0:00:38.494744
logging-too-few-args                                                             0:00:38.419155
unsupported-binary-operation                                                     0:00:37.847943
invalid-slice-index                                                              0:00:37.419736
truncated-format-string                                                          0:00:37.097308
missing-format-string-key                                                        0:00:36.772325
invalid-sequence-index                                                           0:00:36.551161
unsupported-delete-operation                                                     0:00:36.546976
bad-open-mode                                                                    0:00:36.512247
bad-format-character                                                             0:00:36.458570
no-member                                                                        0:00:35.928044
format-needs-mapping                                                             0:00:35.898208
bad-format-string                                                                0:00:35.860376
invalid-unary-operand-type                                                       0:00:35.613437
unsupported-membership-test                                                      0:00:35.588740
not-context-manager                                                              0:00:35.584344
unused-format-string-key                                                         0:00:35.542302
logging-unsupported-format                                                       0:00:35.339732
mixed-format-string                                                              0:00:35.305616
assignment-from-none                                                             0:00:35.211583
repeated-keyword                                                                 0:00:35.161800
logging-too-many-args                                                            0:00:35.054962
logging-format-truncated                                                         0:00:34.871881
too-many-function-args                                                           0:00:34.859482
not-callable                                                                     0:00:34.792770
unsubscriptable-object                                                           0:00:34.770419
bad-str-strip-call                                                               0:00:34.757161
assignment-from-no-return                                                        0:00:34.727277
no-value-for-parameter                                                           0:00:34.664874
logging-not-lazy                                                                 0:00:34.529719
too-few-format-args                                                              0:00:34.459010
unsupported-assignment-operation                                                 0:00:34.373915
too-many-format-args                                                             0:00:34.195957
bad-format-string-key                                                            0:00:33.899370
unused-format-string-argument                                                    0:00:33.887841
missing-format-argument-key                                                      0:00:33.785160
redundant-unittest-assert                                                        0:00:33.740884
missing-format-attribute                                                         0:00:33.455067
logging-format-interpolation                                                     0:00:33.449355
bad-thread-instantiation                                                         0:00:33.306790
format-combined-specification                                                    0:00:33.294820
c-extension-no-member                                                            0:00:33.291890
keyword-arg-before-vararg                                                        0:00:33.234156
stop-iteration-return                                                            0:00:33.176356
invalid-format-index                                                             0:00:33.068345
deprecated-method                                                                0:00:33.047958
shallow-copy-environ                                                             0:00:32.959599

with ignored-modules=pandas

assignment-from-no-return                                                        0:00:47.875978
missing-kwoa                                                                     0:00:42.181332
invalid-sequence-index                                                           0:00:42.111520
logging-too-many-args                                                            0:00:41.893597
invalid-slice-index                                                              0:00:41.541294
no-value-for-parameter                                                           0:00:41.099715
not-callable                                                                     0:00:39.675871
bad-format-character                                                             0:00:39.205697
unsupported-binary-operation                                                     0:00:38.976457
repeated-keyword                                                                 0:00:38.825842
logging-too-few-args                                                             0:00:38.704934
no-member                                                                        0:00:38.503515
unexpected-keyword-arg                                                           0:00:38.427282
bad-open-mode                                                                    0:00:38.288543
unsubscriptable-object                                                           0:00:38.138226
redundant-keyword-arg                                                            0:00:38.018166
redundant-unittest-assert                                                        0:00:37.861007
logging-unsupported-format                                                       0:00:37.601585
too-many-function-args                                                           0:00:37.465114
unsupported-assignment-operation                                                 0:00:37.417673
logging-format-truncated                                                         0:00:37.258374
deprecated-method                                                                0:00:37.193201
bad-thread-instantiation                                                         0:00:37.192946
not-context-manager                                                              0:00:37.158083
assignment-from-none                                                             0:00:37.151823
unsupported-delete-operation                                                     0:00:37.124939
invalid-metaclass                                                                0:00:36.988846
truncated-format-string                                                          0:00:36.970254
shallow-copy-environ                                                             0:00:36.970162
mixed-format-string                                                              0:00:36.951497
invalid-unary-operand-type                                                       0:00:36.896878
unused-format-string-argument                                                    0:00:36.888768
unsupported-membership-test                                                      0:00:36.557271
format-needs-mapping                                                             0:00:36.517421
too-few-format-args                                                              0:00:36.205480
bad-format-string-key                                                            0:00:35.988614
missing-format-attribute                                                         0:00:35.296619
format-combined-specification                                                    0:00:34.861939
consider-iterating-dictionary                                                    0:00:34.782166
unused-format-string-key                                                         0:00:34.754750
abstract-class-instantiated                                                      0:00:34.620736
bad-format-string                                                                0:00:34.261011
invalid-format-index                                                             0:00:34.254017
c-extension-no-member                                                            0:00:33.711384
too-many-format-args                                                             0:00:33.553213
missing-format-string-key                                                        0:00:33.182854
stop-iteration-return                                                            0:00:33.017166
missing-format-argument-key                                                      0:00:32.750723
bad-str-strip-call                                                               0:00:32.468865
keyword-arg-before-vararg                                                        0:00:32.018734
logging-not-lazy                                                                 0:00:31.984399
logging-format-interpolation                                                     0:00:31.922702

Note: i7-7700HQ, so reasonable CPU.

[PCManticore]

As mentioned earlier, ignored-modules is in fact used only by a handful of checks, as per its description:

List of module names for which member attributes should not be checked (useful for modules/projects where namespaces are manipulated during runtime and thus existing member attributes cannot be deduced by static analysis. It supports qualified module names, as well as Unix pattern matching.

It's not intended to ignore all the errors that happens to be with a given module.

Also if anyone wants to investigate which checks contribute to the slowness of pylint, the better way to do it is to use yappi or a different profiler. Here is an example of a PR where @nickdrozd used that profiler in order to determine some hotspots in astroid, an approach that should be far more reliable than running pylint for one check (as pylint already has an incurred overhead from both the subprocess start and from instantiating all the scaffolding needed for running the checks).

[SeppMe]

While I was as of yet unable to find the root cause, I nevertheless traced this problem down to the 1.6.2 release of astroid.

I took a simple test program like the one above and varied the installed versions of Pandas, Numpy, Pylint, and astroid. I found that the Pandas, Numpy, and Pylint versions do not matter at all, I tested several versions of each from Summer 2017 up until today. But astroid <= 1.6.1 took only about 20-30 seconds, whereas anything >= 1.6.2 took 8-10 minutes! This also applies to the 2.x.y releases of Pylint and astroid, they take forever to analyse the simple test program.

[PCManticore]

Thanks @SeppMe I don't out of the top of my head what features shipped with astroid 1.6, but most likely there's something odd going on with the inference, which triggers these abnormal running times.

[co-dh]

It's astroid.
strace pylint foo.py
Ctrl-c while you see a lot of mmap and munmap, you'll get a Keyboard exception with 1000 stacks.

[dickreuter]

Is there any update on this?