marking pytest.fixture as a step in the final report about python-pytest-steps HOT 9 CLOSED

or, perhaps to rephrase:
Can I parametrize the individual steps?

from python-pytest-steps.

Hi @rplevka, thanks for the feedback !

I am not sure that you use the fixtures for what they are meant for: a fixture is a fixed object that is created before any test is run, and the tests can run against it. You can see it as the "starter kit" of your tests. So you are right to put here everything that should be done as "setup": fixture instanciation is not part of the test. In addition:

• By default a fixture has no parameters so it is created once, and then passed to all your tests requiring it. So in your example, if fixture1 has no parameters, there is no way it will appear in the test id as you showed.
• however you can also parametrize a fixture. In which case, all tests are run on all versions of your fixture. So if fixture1 has two parameters, your 4 tests (2 step x 2 params) will be run 2 times each (once for each fixture), making 8 tests

Now, if we forget the fixtures. You can simply do

@test_steps(['step1', 'step2'])
@pytest.mark.parametrize('param', [1, 2], ids=str)
def test_e2e(param):
    # this is step 1
    yield
    # this is step 2
    yield

And this will generate the two steps for each parameter value = 4 tests

You can also revert the order of @test_steps and @pytest.mark.parametrize if you wish to get the same order than in your example.

@pytest.mark.parametrize('param', [1, 2], ids=str)
@test_steps(['step1', 'step2'])
def test_e2e(param):
    # this is step 1
    yield
    # this is step 2
    yield

I hope that helps ?

from python-pytest-steps.

@smarie thanks for your response.
I know I tried to 'misuse' the fixtures there.
I did so as I wanted to make use of the fixture nesting ability, while each fixture might contain different parameters.
I wanted to do this to prevent the "full matrix execution" - meaning, that the common paths (from the top of the tree are not duplicated for each parameter, and the paths are split at the desired level). This is why I'm asking, whether i can parametrize the individual steps.
I'm not sure I made myself clear so I'll better paste a diagram of the desired "execution tree"

instead of:

Perhaps the way to go is to make the common steps a standalone tests (i just need to make pytest execute them in the same order, or to set some 'dependency' relationship between them not to distribute them to individual workers if xdist is used, etc).

The motivation behind this is that i write system tests (black box) and every path is really expensive, so I'd like to reuse the the common steps and mark them dependent on each other.

from python-pytest-steps.

I see your point. This scenario is clearly not what pytest-steps has been designed for. I think that the best workaround we can reach as of today (xdist might not work however) would be to

• split each group of steps that have different parameters (step 1+2 vs. step3+4 vs. step5+6) in a dedicated test as you suggest - that way, each step group will have its own parameters: test3+4 will have (a/b) and test5+6 will have (aa/ab/ba/bb). Inside a group you can still use pytest-steps to separate the steps (i.e. step 3 from step 4 ; step 5 from step 6).

• order the tests manually in the module so that test 5+6 is run after test 3+4, that is run after test 1+2. You will have in each step test group to first check if previous step group has run successfully, you can do so by checking if the state fixture is filled (see next bullet). If not, manually do pytest.skip("required dependency has failed"). You can try to use for example pytest-dependency to automate this, but I am not sure that they propose fine-grained dependency conditional to a parameter value. So the best is for you to write the if/else yourself.

• create a global session-scoped or module scoped "state_dct" fixture, with no parameters, that returns a dictionary. In each test (step group), store the required state that you want to pass along to next step group (the arrows in your diagram) in that dictionary, with a key = (test id + parameter).

Not ideal, but should work.
If you think there is a way to make this easier from a user API perspective, we can discuss on how to propose this in pytest-steps ; but my fear is that it would be quite difficult to find an easy way for users to declare all of that with decorators.

from python-pytest-steps.

@smarie
yeah, i can see more clearly that this should really be out of the scope of this plugin.
However thanks for the suggestions, they are really inspirational.

from python-pytest-steps.

It seems to work:

import pytest
from pytest_steps import test_steps

@pytest.fixture(scope='module')
def results_dct():
    return dict()

@test_steps('step1', 'step2')
def test_1_2(results_dct):
    # step 1
    results_dct['step1'] = 1
    yield
    # step 2
    results_dct['step2'] = 'hello'
    yield

@test_steps('step3', 'step4')
@pytest.mark.parametrize('p', ['a', 'b'], ids="p={}".format)
def test_3_4(p, results_dct):
    if 'step2' not in results_dct:
        pytest.skip("Can not start step 3: step 2 has not run successfuly")
    # step 3
    results_dct.setdefault('step3', dict())[p] = 'bla'
    if p == 'b':
        # let's make this step fail so that we see the rest of the tree fail
        assert False
    yield
    # step 4
    results_dct.setdefault('step4', dict())[p] = 'blabla'
    yield


@test_steps('step5', 'step6')
@pytest.mark.parametrize('q', ['a', 'b'], ids="q={}".format)
@pytest.mark.parametrize('p', ['a', 'b'], ids="p={}".format)
def test_5_6(p, q, results_dct):
    if 'step4' not in results_dct:
        pytest.skip("Can not start step 5: step 4 has not run successfuly")
    elif p not in results_dct['step4']:
        pytest.skip("Can not start step 5: step 4 has not run successfuly for this value of p=%s" % p)
    # step 5
    yield
    # step 6
    yield

from python-pytest-steps.

@smarie, now that is really cool. Thanks for trying that out. I think i'll try to use the pytest-dependency to have it more readable, but i wouldn't be surprised if it worked the same way under the hood.

I guess i can close the issue now. Thanks again.

from python-pytest-steps.

Thanks for the feedback! If you find a way to go with pytest-dpendency, would you be so kind to post here an equivalent example ? Indeed I plan to release shortly a "pytest-patterns" project where I will put example design patterns for typical pytest tasks. I think that this would perfectly fit!

from python-pytest-steps.

sure thing.

from python-pytest-steps.