fail when envlist is empty about tox-gh-actions HOT 8 OPEN

What about an option --fail-on-no-environment? That wouldn't be a breaking change.

from tox-gh-actions.

Agreed, it was surprising than tox running no testenv resulted in success.

from tox-gh-actions.

Sorry for a slow response but I just published tox-gh-actions v2.12.0 which adds an option to fail when no environments are matched. Please check details in README and give it a try: https://github.com/ymyzk/tox-gh-actions/blob/v2.12.0/README.md#fail-when-no-environments-are-matched

Thank you all for your patience!

from tox-gh-actions.

Thanks for sharing additional feedback. Adding an option to fail should be possible but let me think about it a bit deeper before actually implementing it as tox-gh-actions hasn't provided any CLI options so far.

from tox-gh-actions.

This is especially important since it is not very well documented that, in order to be eligible for selection by this plugin, the env must also be present in envlist. This may make sense to those who understand how Tox plugins work, but is not intuitive to mere users. A failure on an empty generated envlist would help users detect and diagnose this.

from tox-gh-actions.

Hello everyone, thanks for the feedback. At this point, I'd like to avoid actually failing the tox when it couldn't find no environments as it can be technically a breaking change for exiting users.

At the same time, I understand this issue can be a bit difficult to catch for users. I'd like to propose showing a warning message when this problem happens as a mitigation: #123

This is especially important since it is not very well documented that, in order to be eligible for selection by this plugin, the env must also be present in envlist.

As the first example in README only specifies envlist and gh-actions, I was assuming it's clear enough for users but if you have ideas to improve the README further, feel free to open a PR.

[tox]
envlist = py37, py38, py39, py310, mypy

[gh-actions]
python =
    3.7: py37
    3.8: py38
    3.9: py39
    3.10: py310, mypy

[testenv]
...

from tox-gh-actions.

Just having an example of how to do it right does not inform users about nuanced behaviors when they deviate from the example.¹ Even this would be better:

[tox]
# gh-actions only has access to environments explicitly listed in envlist
envlist = py38, py39, py310, mypy

[gh-actions]
python =
    # note that py37 and py38-legacy will silently never be run because they are missing from envlist
    3.7: py37
    3.8: py38, py38-legacy
    3.9: py39
    3.10: py310, mypy

[testenv]
...

I like the --fail-on-no-environment option, but I like the --fail-on-missing-environment better, which would error if there was an environment listed in the [gh-actions] section that was unreachable or not also present in envlist. Either way, there should likely be warnings for any unreachable environments listed in [gh-actions].

¹ Assuming otherwise is typical by people already familiar with the behaviors in question, but often flawed. Those people are not the audience of documentation. The mere existence of this issue is evidence that your assumptions deserve revisiting. Further, if you can solve your user's difficulty with poor ergonomics by improving ergonomics, that's almost always a better solution than relying on secondary sources like documentation.

from tox-gh-actions.

If it is easier, then maybe some environment variable could be used?

from tox-gh-actions.