tox won't pass about friendly HOT 14 CLOSED

from friendly.

Found the issue, it's simple:

in your setup.cfg you have version = attr: friendly.__version__, in this case setuptools has two ways to find your actual version:

• It tries to parse the __init__.py file in hope to find the __version__ value without importing it, if this fails:
• It tries to import the file.

In friendly case as __version__ is imported, and not physically in __init__.py the first try fail, and setuptools has to import your file. But importing your file mean importing all the file imports too.

To there's three ways to fix this:

• Add pure_eval and many dependencies as build dependencies in pyproject.toml (bad bad bad)
• Duplicate the version in version.py AND in init.py (bad bad)
• Tell setup.cfg the version is in friendly.version.__version__ (I don't know why it does not work)
• Move the actual version in the __init__.py

from friendly.

from friendly.

I did move version inside init and everything works correctly locally but the git workflow fails as it apparently does not install pure_eval (and likely other requirements). Leaving this open as I do not know currently how to fix it. I'll be focusing on other issues for now.

from friendly.

See https://github.com/aroberge/friendly/actions/runs/869377266 as an example.

from friendly.

I only kept Ubuntu since Windows was failing (and Mac OS didn't run) due to a tox-related error (not an error with my code). As I work on Windows, it gets thoroughly tested.

from friendly.

I only kept Ubuntu since Windows was failing (and Mac OS didn't run) due to a tox-related error (not an error with my code). As I work on Windows, it gets thoroughly tested.

According to https://github.com/aroberge/friendly/actions/runs/870518597 Windows and MacOS tests were successfull.

I think it's still good to test PRs on Windows, just to validate other contributors like me don't break something.

from friendly.

from friendly.

Let me check this :)

from friendly.

@aroberge found the issue, coverage was configured both in tox.ini and .coveragerc. .coveragerc was winning, but missing a line.

I deduplicated the config in #227 and test passes for Windows, MacOS, and Linux.

I choose to put coverage config in tox.ini to have one less file, but don't hesitate to move the configuration the other way around if you prefer, just don't duplicate it.

from friendly.

For information: putting the "omit" information in tox.ini vs having that information in .coveragerc instead is yielding a different set of files being included in coverage: using tox.ini leaves out files having 100% coverage omitted from the report.

• Using tox.ini

Module | statements | missing | excluded | branches | partial | coverage
-- | -- | -- | -- | -- | -- | --
Total | 5083 | 676 | 416 | 1875 | 275 | 84%
friendly\__init__.py | 68 | 6 | 6 | 12 | 3 | 89%
friendly\__main__.py | 79 | 19 | 0 | 22 | 9 | 70%
friendly\config.py | 99 | 2 | 48 | 24 | 1 | 98%
friendly\core.py | 359 | 19 | 57 | 134 | 18 | 92%
friendly\editors_helpers.py | 68 | 8 | 0 | 16 | 0 | 88%
friendly\info_generic.py | 121 | 8 | 0 | 8 | 1 | 93%
friendly\info_specific.py | 88 | 9 | 0 | 8 | 2 | 89%
friendly\info_variables.py | 232 | 22 | 0 | 115 | 14 | 89%
friendly\my_gettext.py | 32 | 3 | 0 | 2 | 1 | 88%
friendly\path_info.py | 70 | 17 | 0 | 32 | 10 | 70%
...

Note how console_helpers.py, debug_helper.pyandformatter.py` are not included above.

• Using .coveragerc

Module | statements | missing | excluded | branches | partial | coverage
-- | -- | -- | -- | -- | -- | --
Total | 5085 | 677 | 413 | 1875 | 275 | 84%
friendly\__init__.py | 68 | 6 | 6 | 12 | 3 | 89%
friendly\__main__.py | 79 | 19 | 0 | 22 | 9 | 70%
friendly\config.py | 99 | 2 | 48 | 24 | 1 | 98%
friendly\console_helpers.py | 65 | 0 | 104 | 12 | 0 | 100%
friendly\core.py | 359 | 19 | 57 | 134 | 18 | 92%
friendly\debug_helper.py | 8 | 0 | 11 | 0 | 0 | 100%
friendly\editors_helpers.py | 68 | 8 | 0 | 16 | 0 | 88%
friendly\formatters.py | 43 | 0 | 111 | 16 | 0 | 100%
friendly\info_generic.py | 121 | 8 | 0 | 8 | 1 | 93%
friendly\info_specific.py | 88 | 9 | 0 | 8 | 2 | 89%
friendly\info_variables.py | 232 | 22 | 0 | 115 | 14 | 89%
friendly\my_gettext.py | 32 | 3 | 0 | 2 | 1 | 88%
friendly\path_info.py | 70 | 17 | 0 | 32 | 10 | 70%
...

EDIT: the reported total coverage is the same (now); however some files with 100% coverage are that way because pragma: no cover is used on purpose - and it can be useful to see why when doing review of the code coverage.

from friendly.

using tox.ini leaves out files having 100% coverage omitted from the report.

TL;DR: Either put the whole coverage config in tox.ini, either in .coveragerc, but don't mix both.

Got it! If there's a .coveragerc, coverage use it, and won't try to search another config file, missing the [coverage:report] skip_covered = True from the tox.ini.

If there's no .coveragerc, coverage will search for the config elsewhere, and find it in tox.ini (no need to tell it).

I double checked by copying both sections to .coveragerc and then to tox.ini (beware of adding/removing the coverage: prefix, and beware of removing the empty .coveragerc during the tox.ini test.

I've noticed no difference between the two runs, tried using:

coverage run -m pytest; coverage combine; coverage report > a
# moved the conf
coverage run -m pytest; coverage combine; coverage report > b
dif a b

Tox just run the commands, there's no magic conf editing or nothing, but beware: there's a slight difference running coverage manually with a single Python version and running it via my tox config because in my tox config I use multiple Python versions and then merge the reports, see:

via tox:

TOTAL                                                6938    756   2149    415    85%

manually using python 3.9:

TOTAL                                                6938    822   2149    438    84%

This way if you have:

if sys.version > (3, 5):
    ...
else:
    ...

both the if and the else will be merged during the coverage combine step, which is nice.

from friendly.

Thanks, I'll try removing the [coverage:report] section.

Yesterday, I did remove the [coverage:run] section entirely from tox.ini and, for a while, all the tests WERE passing. Then, last night, one windows-latest test failed again.... But I still have the [coverage:report] section in tox.ini as you point out.

Instead of using tox to combine reports, this is what I do in a .bat file:

call venv-friendly3.6\scripts\activate & python -m pytest --cov=friendly --cov-report html tests/
call venv-friendly3.7\scripts\activate & python -m pytest --cov=friendly --cov-append --cov-report html tests/
call venv-friendly3.8\scripts\activate & python -m pytest --cov=friendly --cov-append --cov-report html tests/
call venv-friendly3.9\scripts\activate & python -m pytest --cov=friendly --cov-append --cov-report html tests/
call venv-friendly3.10\scripts\activate & python -m pytest --cov=friendly --cov-append --cov-report html tests/

Running full tests with coverage can be too long. I have 6 virtual environments (one note tested automatically, to be used with IPython/Jupyter) and can activate any one with a very simple call (e.g. ae 3.10). Then, I can:

• run pytest only; my default when I am working on adding a new case.
• create the entire tracebacks for this specific Python version to be put in the docs, and see if anything has changed; in the past, I caught some subtle bugs that were not captured by the (incomplete) unit tests -- including changes/missing translations for the French version, not directly tested by unit tests
• run pytest (no coverage) with all Python versions
• run pytest with coverage (as above)
• update the documentation for all Python versions

from friendly.

To reduce the test time I run tests in parallel, but pytest-cov have hard times with parallel runs, so I avoid using it.

On friendly, on my machine, tox -p all takes 22s, (as a reference, tox -e py39 takes 9s, pytest takes 6s).

from friendly.