Tests not parallelized efficienctly by pytest

[JukkaL]

When I run pytest mypy, towards the end of a test run only 1 or 2 python processes are running and test progress is very slow -- each process is using much less than 100% of a CPU. This happens even though I requested 8 parallel processes. Initially I see 8 processes doing work, but something causes a slowdown towards the end of a test run.

This doesn't look like a recent regression, but I think that things have gotten worse in the last few months or so.

Some data (Linux, Intel 4-core/8-thread CPU):

• When I check out e378601 from May 2018, tests finish in 66s (6024 tests).
• When I check out 1606f0f from Oct 2018, tests finish in 133s (6671 tests).

This seems to happen both on Linux and macOS.

Updating to the latest pytest doesn't help.

It would be great if somebody could try to reproduce the issue.

[emmatyping]

Yes, I have seen this too. The problem seems to get worse the more cores you throw at it. I went ahead and checked out e378601 and it only takes 57 seconds on my 16 core machine, while master takes 240s.

It seems this is a known issue with pytest-xdist:
pytest-dev/pytest-xdist#40
pytest-dev/pytest-xdist#18

[ilevkivskyi]

I see this all the time for last few months (and mentioned it internally a while ago). I tried to play one day with different scheduling modes (xdist has some), but I was never able to get efficient load scheduling.

[JukkaL]

I'm not sure if the reported pytest-xdist issues are relevant. It looks like too many slow tests are scheduled for certain processes, and other processes are starved.

I have a workaround that seems to improve the runtime to about 81s on master (on the Linux system I mentioned above). It splits the test run into two pytest invocations. I don't love the approach but it's probably worth the performance increase. I'll prepare a PR once I've verified that it works generally.