Being able to run tests for a Blender Application Template would be a great feature for your module.

The main issues are:

• It will require some design changes.
• The name won't be accurate anymore.

Sometimes a download from blender servers fail, due to some server issue. A retry after 5-10 minutes should be implemented to give the server time to recover and not fail the test.

3 time retry should be enough.

help(blender_addon_tester) yields this for now:

Help on package blender_addon_tester:

NAME
    blender_addon_tester

PACKAGE CONTENTS
    addon_helper
    blender_load_pytest
    get_blender
    run_blender

FILE
    /home/AAAAAA/lib/python3.7/site-packages/blender_addon_tester/__init__.py

On linux, with blender-addon-tester 0.1 installed in a virtualenv, when trying to run BAT, if in my home directory, since the default download path is ../, I get a permission denied related to writing.
Some sort of temp directory should be decided cross-OSes. Python does have mktemp, but such a temp directory must be redetected / reused somehow every time the script starts again... which might mean using instead home settings.

Only happens with the python 3.5 version so 2.78 an 2.79. version 2.80 works fine and above, which has python3.7

Exception:
Traceback (most recent call last):
  File "C:\cygwin64\home\douglas\projects\blender-fake-addon\examples\blender-2.79b-windows64\2.79\python\lib\shutil.py", line 544, in move
    os.rename(src, real_dst)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'c:\\cygwin64\\home\\douglas\\projects\\blender-fake-addon\\examples\\blender-2.79b-windows64\\2.79\\python\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\__init__.cpython-35.pyc' -> 'C:\\Users\\douglas\\AppData\\Local\\Temp\\pip-e9ejge9c-uninstall\\cygwin64\\home\\douglas\\projects\\blender-fake-addon\\examples\\blender-2.79b-windows64\\2.79\\python\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\__init__.cpython-35.pyc'

I am struggling a bit with the test-*-example-from-github-pip.yml. I think it can't work always. Right now it points to a git-repo branch. So that limits other branches from doing iterative work.

When we get pypi support, there will always be a latency issue. Even if published, the module available on pypi might not update for 2 hours, or what ever, so if you have edited a core piece of code you want to test, you might not see it in the length of time it takes for the testing to complete, which looks pretty fast, sub 10 minutes.

This test feels like one that should live with the blender-fake-addon repo, and the other one, with the wheel, lives here.

There is potentially a third/replacement test. One where you just set the PYTHONPATH to the directory with the module lives, or you do a straight python setup install (not with a wheel)

Am I missing something?

setting sys.path in run_blender.py is working on windows but fails on linux when used in test_load_lwo.py

Suspect there is a divergence in operation on operating systems

A feature was added to change the location of the addon installation.

bpy.context.user_preferences.filepaths.script_directory = addon_dir

turns out this value does not take effect until there is a restart. I can not find a command to restart blender, so while this value does change successfully, it no effect on testing.

https://docs.blender.org/api/current/bpy.types.PreferencesFilePaths.html#bpy.types.PreferencesFilePaths.script_directory

for blender-addon-tester 0.1, when piping a non-plugin path (eg. pictures directory), blender gets downloaded first, then blender detects that it's not a plugin..
We could detect grossly that there are files in it (compressed or uncompressed dir), with one file exposing one type of comment or methods, with a proper directories layout.

Note that such an archive validator could be imitated also to validate that: any plugin is a plugin according to a spec (could be used to check hundreds of them), any blender release is a supported blender release (eg. tar.bz2, tar.gz, tar.xz, dmg, *lender.app with executable in Contents/MacOSX etc..).

In the function clean_file if there is a non-text file in the targeted directory, I get a UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 3131: invalid start byte (in my case it's for a .DS_Store OSX file). I also had the issue with image files.

I don't get what this function is supposed to do and if this issue is actually critical.

For now, we are targetting a Github branch in order for pip install blender_addon_tester to test itself.

Having blender-addon-tester written with hyphens instead of underscores prevents easy import from python, so possibly we should stick to underscores in the setup.py / python packaging as much as possible.

We should:

• decide on the name and URL on pypi.org: blender_addon_tester
• decide on a versioning. Eg. start with 0.1
• write maybe a Github Actions workflow file that pushes a release to pypi.org using Twine when a v* eg. v0.1 tag is pushed (inspired from this workflow file and this send_to_pypi_bash_function)
• make a more concise README if we plan to use the text for both the pypi.org project page and the Github project home README
• test that pip installing from upstream does work, from a local a machine and from other Github projects CIs jobs maybe.

For info, after pushing a release to pypi.org, it looks like 30 minutes-2 hours are needed before any pip install grabs that new version

get_blender, os, run_blender below are probably exported inside the module for no reasons

Python 3.7.5 (default, Nov 20 2019, 09:21:52) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import blender_addon_tester as BAT
>>> dir(BAT)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'get_blender', 'os', 'run_blender', 'test_blender_addon']
>>> BAT.get_blender()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'module' object is not callable
>>> BAT.get_blender
<module 'blender_addon_tester.get_blender' from '/home/jd/.virtualenvs/gmicpy290numpy/lib/python3.7/site-packages/blender_addon_tester/get_blender.py'>
>>> BAT.get_blender("2.80")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'module' object is not callable

Hello,
I am planning to try to use your work for https://github.com/myselfhimself/gmic-blender though I will not use Travis CI but the Github Actions instead.
I was wondering how to enable an add-on from outside Blender programmatically. It seems your SetupPlugin() class does the work... I will see in <1 month.
I do plan to use pytest as well.
Thanks for contributing your work online.
Let us keep in touch.

A comment in .travis.yml says that Python can not be run on Travis CI Windows. See: https://docs.travis-ci.com/user/languages/python/#running-python-tests-on-multiple-operating-systems The command choco install blender python might just work.

That branch shall not be needed anymore :)

Actions can call actions.

A action should be created here that can be called by clients.

This will allow the bender versions to be added to in the pypi, and this will flow straight through to client testing.

Right now our side maintains a list and their side maintains a list. It works, but can be improved.

maybe a readthedocs page, sphinx, etc

Code cover in this project is slightly irrelevant. Consider removing.

When this code is being used to enable addon testing, it is the addon that needs the code coverage, not anything from here.

It's presence here might be confusing.

It should definitely be in the new form blender-fake-addon.

Thought?

Right now if you try to test an addon that comes with blender there appears to be a race condition on which addon to use. Based on the age of the file on the file system, so the chances of the old addon winning is actually greater.

Need to move existing addon out of the way for testing and return once testing is complete. I would rather not smash delete the old one.

rebuild_pipreinstall_and_testexamples.sh does not work in windows land. cmd does not have shell (I am referring to local windows machines, not powershell)

A similar script written in python should work on all systems as python is the only prerequisite required to run the whole show.

@myselfhimself

OK I decided to test the tester. I introduced and error that should cause every test to fail. However I get green every where, both on travis and github actions:

https://github.com/douglaskastle/blender-addon-tester/runs/537749667?check_suite_focus=true
https://travis-ci.org/github/douglaskastle/blender-addon-tester/builds/667444567

Looking in the test themselves they are failing, it is just that the failure is not percolating out.

This was the whole effort of the original work I did. The error happens in python in blender. That error has to get out of the python, out of blender, out of the wrapper python to the CI tool.

I have not taken your flow apart yet, but I suspect it the the blender to the top level python is where this is not taking place.

I did this inside the pypi-test branch as it was the active one I was on. It has all the work from todays work in macosc-support branch merged into it.

tests/test_version.py::test_versionID_pass FAILED                        [ 50%]
=================================== FAILURES ===================================
_____________________________ test_versionID_pass ______________________________
bpy_module = 'fake_addon'
    def test_versionID_pass(bpy_module):
        expect_version = (0, 1, 1)
        return_version = get_version(bpy_module)
>       assert expect_version == return_version
E       assert (0, 1, 1) == (0, 0, 1)
E         At index 1 diff: 1 != 0
E         Full diff:
E         - (0, 0, 1)
E         ?     ^
E         + (0, 1, 1)
E         ?     ^
tests/test_version.py:13: AssertionError
=========================== short test summary info ============================
FAILED tests/test_version.py::test_versionID_pass - assert (0, 1, 1) == (0, 0...
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
============================== 1 failed in 0.05s ===============================

We need this for gmic-py x blender :)

Failing build at: https://github.com/myselfhimself/gmic-blender/runs/513968946?check_suite_focus=true

raise Exception(f"dmg Unsupported")

Need

Allowing the user to target a local/custom Blender executable path would make the local development of this addon faster (don't have to download it all the time --which is a great feature by the way--). It would also some CICD pipelines like Gitlab's to use a custom docker image where Blender is already bundled to not overload the bandwidth.

How

Add a blender_path="" parameter to the main function?

Related to #2 (comment)