Providing fixtures to non-python tests

[iliakur]

This is more of a question, I couldn't quite figure it out from either the docs or a skim of the code.

I'm generating tests from yaml similarly to what's described here. Unfortunately, these test cases rely on some fixtures used in other tests (so I'd like to keep them as fixtures). Is there a way to feed them to the items I create?

I've considered the tried-and-true method of adding another layer of indirection and defining some functions that both the fixtures and my yaml-to-test code would use, but was wondering if there's a more canonical way to make this happen

Thanks in advance!

[pytestbot]

GitMate.io thinks possibly related issues are #1376 (allow fixtures to provide extra info when a test fails), #41 (looponfail ignores changes to non-python testfiles), #367 (Skip tests that require a fixture), #920 (Non-deterministic test collection order for deterministic fixtures makes xdist fail), and #3625 (Provide for including fixtures from common location [improvement]).

[iliakur]

Update

I have found a different way to skin the cat!
I created a util.py module in the root of my test directory and put all the yaml-parsing logic in there.
Here's a mock-up of what I mean by that.

class YamlSuiteBuilder:
    __test_suite__ = "path/to/file.yml"

    @pytest.fixture
    def very_useful_fixture(self):
         ...
    
    def pytest_generate_tests(self, metafunc):
        # load yaml files here from __test_suite__
        # at the end use `metafunc.parametrize` to generate the actual tests
        ...

    def test_my_stuff(self, very_useful_fixture, args_from_yaml):
        ...
        assert stuff

Now in several of my test files I simply inherit from YamlSuiteBuilder and override the path to the yaml file. For example:

from test_my_pkg.util import YamlSuiteBuilder

TestYamlBased(YamlSuiteBuilder):
    __test_suite__ = "/path/to/a/different/file.yml"

Unfortunately, whenever one of these tests failed, I wouldn't get a useful pytest diff of how the expected value diverged from what the code being tested produced. I solved it the following way.

In my top-level __init__.py file I have the following code adapted from the plugin-writing documentation page:

import pytest

pytest.register_assert_rewrite('test_textengine.util')

Now I get pytest's pretty diff feedback if test_my_stuff fails, which would not have happened if I had just simply imported that module.

This seems like a bit of a hacky solution to me, I'd love some feedback for how to make it nicer!
I'm also not sure I explained this well, even after edits, so please feel to ask me questions!

[gaudenz]

I have the exact same problem as the OP. Any help would be appreciated. I'm trying to collect and run shell scripts from a folder as additional tests. Running these scripts requires a fixture which is also used in other (Python) tests. It's unclear to me how I can get this fixture into the pytest.Item subclass and use in the runtest method.

[iliakur]

For @gaudenz and others interested. I have found yet another way that's perhaps less hacky than my original solution.

Let's say I have the following test file in yaml:

- name: one
  input: a
  output: b
- name: two
  input: 1
  output: 2

In the root of my test folder I have a conftest.py file with something along the following lines.

import pytest
from functools import partial


def pytest_collect_file(parent, path):
    if path.ext == ".yaml" and path.basename.startswith("test"):
        return YamlFile(path, parent)


class YamlFile(pytest.File):

    def collect(self):
        import yaml  # only import pyyaml if necessary
        yaml_tests = yaml.safe_load(self.fspath.open())       
        for test_case in yaml_tests:
            yield pytest.Function(
                name=test_case['name'],
                parent=self,
                callobj=partial(
                    my_custom_test_function, 
                    test_case['input'], 
                    test_case['output'],
                ),
            )


def my_custom_test_function(input, output, my_awesome_fixture):
    # assert stuff just like in a regular test function
    pass


@pytest.fixture
def my_awesome_fixture():
    # behaves just like a pytest fixture
    pass

The key for me was to realize that pytest.Function objects are somewhat better integrated into the existing fixture finding mechanism than pytest.Item objects.

[nicoddemus]

Thanks @copper-head for sharing.

Just to add another way to the mix: doctest items also handle fixtures, here's how that is done:

pytest/src/_pytest/doctest.py

Lines 374 to 389 in f858177

	def _setup_fixtures(doctest_item):
	"""
	Used by DoctestTextfile and DoctestItem to setup fixture information.
	"""
	def func():
	pass
	doctest_item.funcargs = {}
	fm = doctest_item.session._fixturemanager
	doctest_item._fixtureinfo = fm.getfixtureinfo(
	node=doctest_item, func=func, cls=None, funcargs=False
	)
	fixture_request = FixtureRequest(doctest_item)
	fixture_request._fillfixtures()
	return fixture_request

This unfortunately requires using private APIs.

[iliakur]

@nicoddemus yea that's a pity about the private apis. I tried going down that route but then got a bit confused by all that stuff and that's what prompted me to get it working "for free" by having my tests "pose" as pytest.Function objects.

What do you think about adding this as a documentation page? I'd be interested in that, even if it means brushing things up a bit.

At the same time, I personally don't know how "mature" all of what's been posted is. Maybe it's wiser to wait a bit until we as a community converge on something?
Maybe even a plugin instead of a documentation page?

[nicoddemus]

Hi @copper-head,

I share your concerns. Unfortunately the fixture mechanism is private, and not very open for extensions as it has grown organically since pytest 2.3. @RonnyPfannschmidt and I have been talking about a more stable and public API for fixtures, but this will take a awhile I'm afraid.

[saarwasserman]

Hi, @copper-head
I am encountering difficulties with your solution when usin pytest.Function.
I am getting: 'YamlFile' object has no attribute 'obj'. on your exact code
I tried some workarounds but didn't find a solution.

[iliakur]

@saarwasserman which pytest version are you using? That snippet was developed with pytest 4.1 or so and we haven't had a chance to upgrade the pytest of the project that uses it since then, so chances are it won't work in newer versions of pytest.

[saarwasserman]

I am using pytest 5.0.0
However, I changed 'parent=self' to something like parent=pytest.Module(fspath=os.path.abspath(file) parent=self).
seems to work but maybe there is a better way to handle this.

Nevertheless, now I am trying to see how to integrate 'pytest_generate_tests' hook with this setup as pytest doesn't recognize the fixtures unless they are explicitly written in the conftests with @pytest.fixtture/. so I can't use it at the moment.

[iliakur]

I won't have time to look into this for a while, so here's what I would do:

1. Quick fix: try to add an obj = None declaration inside your YamlFile class and see if that works.
2. Read up on the code of Function and File classes to see where and why the obj attribute is being used. Maybe even diff it with version 4 to get a quicker view of what changed.

As for your pytest_generate_tests situation, sounds like a separate issue/thread to me ;)

[blazewicz]

It's been a while since last update and this still seems to be an unresolved issue.

I'd love to see at least some official guide on how to share fixtures with non-python code, even if it would be a hacky way.

Ideally I'd like to share a module session scoped yield-fixture between python and non-python tests. I understand its not possible at the moment, but some guide on how to achieve that would be very appreciated.