No CI for Python 3.10 yet

[dgw]

Tin. We've still not enabled a CI job for Python 3.10 because of various errors encountered along the way while attempting to do so.

Ultimately, the cause is probably related to pytest-dev/pytest#9174, an issue that was closed but it seems not confirmed fixed. Uncertainty remains as to whether fixing this problem is up to the pytest team or the core Python developers. It looks like a regression (breaking something that worked in Python 3.9 and prior versions) but it appears to be taking longer to sort out than the argparse stuff we dealt with a while back.

While we believe Sopel should run just fine on Python 3.10 in most cases, it's not currently possible to test Sopel on that version.

Log from last attempt (CPython 3.10.4, pytest 6.2.5

coverage run -m py.test -v . --vcr-record=none
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.[10](https://github.com/sopel-irc/sopel/runs/6252478452?check_suite_focus=true#step:7:10).4/x64/lib/python3.10/site-packages/py/test.py", line 4, in <module>
    sys.exit(pytest.main())
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/config/__init__.py", line 143, in main
    config = _prepareconfig(args, plugins)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/config/__init__.py", line 318, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/helpconfig.py", line 100, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1003, in pytest_cmdline_parse
    self.parse(args)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1283, in parse
    self._preparse(args, addopts=addopts)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/config/__init__.py", line [11](https://github.com/sopel-irc/sopel/runs/6252478452?check_suite_focus=true#step:7:11)72, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/importlib/__init__.py", line [12](https://github.com/sopel-irc/sopel/runs/6252478452?check_suite_focus=true#step:7:12)6, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
    exec(co, module.__dict__)
  File "/home/runner/work/sopel/sopel/sopel/tests/pytest_plugin.py", line 12, in <module>
    from sopel import bot, loader, plugins, trigger
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
    exec(co, module.__dict__)
  File "/home/runner/work/sopel/sopel/sopel/bot.py", line 22, in <module>
    from sopel import db, irc, logger, plugin, plugins, tools
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
    exec(co, module.__dict__)
  File "/home/runner/work/sopel/sopel/sopel/irc/__init__.py", line 37, in <module>
    from .backends import AsynchatBackend, SSLAsynchatBackend
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
    exec(co, module.__dict__)
  File "/home/runner/work/sopel/sopel/sopel/irc/backends.py", line 20, in <module>
    from .abstract_backends import AbstractIRCBackend
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
    exec(co, module.__dict__)
  File "/home/runner/work/sopel/sopel/sopel/irc/abstract_backends.py", line 8, in <module>
    from .utils import safe
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
    exec(co, module.__dict__)
  File "/home/runner/work/sopel/sopel/sopel/irc/utils.py", line 8, in <module>
    from dns import rdtypes, resolver
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
    exec(co, module.__dict__)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/dns/resolver.py", line 38, in <module>
    import dns.query
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
    exec(co, module.__dict__)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/dns/query.py", line 42, in <module>
    import requests
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
    exec(co, module.__dict__)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/requests/__init__.py", line [13](https://github.com/sopel-irc/sopel/runs/6252478452?check_suite_focus=true#step:7:13)3, in <module>
    from . import utils
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line [17](https://github.com/sopel-irc/sopel/runs/6252478452?check_suite_focus=true#step:7:17)0, in exec_module
    exec(co, module.__dict__)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/requests/utils.py", line 42, in <module>
    DEFAULT_CA_BUNDLE_PATH = certs.where()
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/certifi/core.py", line 37, in where
    _CACERT_PATH = str(_CACERT_CTX.__enter__())
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/contextlib.py", line 1[35](https://github.com/sopel-irc/sopel/runs/6252478452?check_suite_focus=true#step:7:35), in __enter__
    return next(self.gen)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/importlib/_common.py", line 89, in _tempfile
    os.write(fd, reader())
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/importlib/abc.py", line [37](https://github.com/sopel-irc/sopel/runs/6252478452?check_suite_focus=true#step:7:37)1, in read_bytes
    with self.open('rb') as strm:
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/importlib/_adapters.py", line [54](https://github.com/sopel-irc/sopel/runs/6252478452?check_suite_focus=true#step:7:54), in open
    raise ValueError()
ValueError
make: *** [Makefile:17: test_norecord] Error 1
Error: Process completed with exit code 2.

[half-duplex]

Is there a reason not to just update to pytest 7? It seems to mostly work on 3.10, save 10 of these:

ERROR sopel/modules/currency.py::test_example_exchange_cmd_0 - AttributeError: module 'sopel.modules.currency' has no attribute 'config'
ERROR sopel/modules/ip.py::test_example_ip_0 - AttributeError: module 'sopel.modules.ip' has no attribute 'config'
ERROR sopel/modules/translate.py::test_example_tr2_0 - AttributeError: module 'sopel.modules.translate' has no attribute 'memory'

[dgw]

Those are pretty weird ones. WTF?

[half-duplex]

Yeah, no idea. I tried to dig into it but got confused trying to figure out how sopel tests @example stuff, and the pytest release notes are Long and I didn't know what I was looking for.

[dgw]

It sounds similar to what people were describing in that issue I linked, where pytest rewrites stuff it shouldn't or otherwise gets confused by namespaces. They seem to think it's importlib's fault.

[half-duplex]

Same errors on python 3.9.2 / pytest 7.1.2 though 🤔

[dgw]

🤔 Maybe we should ask @Exirel to take a look? He wrote the pytest plugin that generates/handles the @example tests, and would therefore probably have some idea what to look for in the Long release notes.

[dgw]

@half-duplex I dug around a bit and to me the errors seem monkeypatch-related. There aren't any notable (again, to me) changes to that part of pytest though, not since 6.2.x, so idk why it's broken. Nonetheless, that breakage doesn't appear to be py3.10-related. Upgrading to pytest==7.1.2 under py3.9 yields the same kind of error.

[Exirel]

Something something pytest support nose, and it considers any setup function to be a nose-style setup function.

Long story short: use -p no:nose option when running pytest. This should be fixed with Pytest 7.2 (to be released). I hope. I'm sure @dgw can summarize our conversation on IRC.

[dgw]

We (OK, mostly @Exirel) pinned down a way to get the tests working on Python 3.10 and pytest 7:

diff --git a/setup.cfg b/setup.cfg
index b64a8e4b..c3df75a1 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -69,5 +69,5 @@ show_error_codes = True
 
 [tool:pytest]
 python_files = *.py
-addopts = --tb=short
+addopts = --tb=short -p no:nose
 norecursedirs = contrib

That should let us upgrade pytest, add Python 3.10 to CI, and finally resolve the conflict that has kept this from happening sooner between "new pytest breaks stuff" and "old pytest is broken on 3.10".

The short story is, pytest added new behavior treating any module-level function named setup as a test setup fixture. Disabling pytest's nose plugin is the solution, as shown above. Meanwhile I/we will keep an eye on further discussion and developments in pytest-dev/pytest#9549, pytest-dev/pytest#9886, & pytest-dev/pytest#9907.

Said nose plugin is for backward compatibility with an older Python test library, which has been in maintenance mode for a long time, but I guess people might not feel like completely reworking their test suites for pytest. The plugin lets them run the old tests under pytest. Ideally that plugin wouldn't be loaded by default, and would only take action if the pytest configuration explicitly requests it, but at least it's fairly easy to turn off…