Skip to content

Update docs on parametrization gotchas #4101

@nicoddemus

Description

@nicoddemus
Member

As #1075 and other issues show it is common, when writing parametrized tests, for users to put dynamic variables (such as current time) or build the values list from structures without deterministic ordering. This causes problems with xdist because all workers (which live in different processes) need to collect the exact test items, both in order and node id.

The parametrize docs should have a section titled "Parametrization and xdist" (or something like that) that explains this problem, with examples of possible causes:

  • Using non-deterministic parametrization values (for example datetime.utcnow()): each worker might get a slightly different value, which will result in different node ids.
  • Using non-deterministic order: obtaining the values from a data structure which doesn't guarantee stable ordering (for example dict on <=py35).

The section should also contain an example traceback, specially the "Different tests were collected between..." phrase, so users can easily find the section by searching.

cc @rawrgulmuffins

Activity

added
type: docsdocumentation improvement, missing or needing clarification
on Oct 10, 2018
leoskyrocker

leoskyrocker commented on Apr 20, 2022

@leoskyrocker

This doc could also give some workarounds like passing in the id to parametrize, e.g. @pytest.mark.parametrize(..., pytest.param(..., id="test A"), pytest.param(..., id="test B")) which will assign a unique node id even with dynamic params.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic: parametrizerelated to @pytest.mark.parametrizetype: docsdocumentation improvement, missing or needing clarification

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @nicoddemus@leoskyrocker

        Issue actions

          Update docs on parametrization gotchas · Issue #4101 · pytest-dev/pytest