Giter VIP home page Giter VIP logo

pytest-embedded's Introduction

pytest-embedded

Documentation Status Python 3.7+

A pytest plugin that has multiple services available for various functionalities. Designed for embedded testing.

Installation

pytest-embedded pytest-embedded-serial pytest-embedded-serial-esp pytest-embedded-idf pytest-embedded-qemu pytest-embedded-arduino pytest-embedded-wokwi

Packages under this repo mainly use semantic versioning. Sometimes a bug fix version may contain some non-breaking new features as well.

It is recommended to use ~=1.0 to get rid of breaking changes, and use the latest new features. For example,

pip install -U pytest-embedded~=1.0

Quickstart

  • pip install -U pytest-embedded~=1.0
  • Create a file test_basic.py
from pytest_embedded import Dut


def test_basic_expect(redirect, dut: Dut):
    with redirect():
        print('this would be redirected')

    dut.expect('this')
    dut.expect_exact('would')
    dut.expect('[be]{2}')
    dut.expect_exact('redirected')
  • Run the test with pytest, the result would be like:
collected 1 item

test_basic.py .                                                        [100%]

============================= 1 passed in 0.01s =============================
  • if run with pytest -s, the output would be as follows:
collected 1 item

test_basic.py 2022-01-01 12:34:56 this would be redirected
.

============================= 1 passed in 0.01s =============================

The print line is also duplicated to console output.

Extra Services

You can activate more services with pytest --embedded-services service[,service] to enable extra fixtures and functionalities. These services are provided by several optional dependencies. You can install them via pip as well.

Available services:

  • serial: serial port utilities.
  • esp: auto-detect target/port by esptool.
  • idf: auto-detect more app info with ESP-IDF specific rules, auto-flash the binary into the target.
  • jtag: openocd/gdb utilities
  • qemu: running test cases on QEMU instead of the real target.
  • arduino: auto-detect more app info with arduino specific rules, auto-flash the binary into the target.
  • wokwi: running test cases with Wokwi instead of the real target.

Resources

pytest-embedded's People

Contributors

adityahpatwardhan avatar alekseiapa avatar dependabot[bot] avatar driftregion avatar erhankur avatar harshal5 avatar hfudev avatar hmalpani avatar horw avatar igrr avatar jammukekkonen avatar l-kaya avatar ouss4 avatar p-r-o-c-h-y avatar ronmov avatar tomassebestik avatar urish avatar wangyz1997 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pytest-embedded's Issues

Add CLI switch to skip flashing (RDT-59)

Hi checking out the project as a possible future solution for CI testing esp32 application code.
One of the first thing I noticed is the device is flashed every time, regardless of actual changes to the binary.
This dramatically slows down test writing, especially when many small tweaks to the pytest side of the test code are required.
Would be nice to include a switch to skip flashing.

[Windows] DuplicateStdout failed without "pytest -s" (RDT-96)

error message without "-s"

pytest --junit-xml=./test_app_results.xml --port=COM25 --embedded-services esp --target=esp32
=============================================== test session starts ===============================================
platform win32 -- Python 3.8.0, pytest-7.0.1, pluggy-1.0.0
rootdir: C:\github_repos\idf-extra-components\test_app
plugins: embedded-0.6.0rc0
collected 1 item

test_app.py E                                                                                                [100%]

===================================================== ERRORS ======================================================
___________________________________________ ERROR at setup of test_app ____________________________________________

self = <pytest_embedded.plugin.PytestEmbedded object at 0x0000022693DC9E20>
fixturedef = <FixtureDef argname='serial' scope='function' baseid=''>
request = <SubRequest 'serial' for <Function test_app>>

    @pytest.hookimpl(tryfirst=True)
    def pytest_fixture_setup(self, fixturedef: FixtureDef[Any], request: SubRequest):
        if fixturedef.argname != 'serial':
            return

        # inject the cache into the serial kwargs
        kwargs = self._pytest_fixturedef_get_kwargs(fixturedef, request)
        _class_cli_options = kwargs['_fixture_classes_and_options']

        # compatible to multi-dut
        if isinstance(_class_cli_options, ClassCliOptions):
            iterable_class_cli_options = [_class_cli_options]
        else:
            iterable_class_cli_options = _class_cli_options

        for _item in iterable_class_cli_options:
            _item_cls = _item.classes.get('serial')
            _item_kwargs = _item.kwargs.get('serial')

            if _item_cls is None or _item_kwargs is None:
                continue

            if _item_cls.__name__ == 'IdfSerial':  # use str to avoid ImportError
                _item_kwargs['port_target_cache'] = self._port_target_cache
                _item_kwargs['port_app_cache'] = self._port_app_cache
            elif _item_cls.__name__ == 'EspSerial':  # use str to avoid ImportError
                _item_kwargs['port_target_cache'] = self._port_target_cache

>       return self._pytest_fixturedef_exec(fixturedef, request, kwargs)

<USER>\.espressif\python_env\idf5.0_py3.8_env\lib\site-packages\pytest_embedded\plugin.py:1089:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
<USER>\.espressif\python_env\idf5.0_py3.8_env\lib\site-packages\pytest_embedded\plugin.py:1052: in _pytest_fixturedef_exec
    result = call_fixture_func(fixturefunc, request, kwargs)
<USER>\.espressif\python_env\idf5.0_py3.8_env\lib\site-packages\pytest_embedded\plugin.py:335: in wrapper
    res = func(*args, **kwargs)
<USER>\.espressif\python_env\idf5.0_py3.8_env\lib\site-packages\pytest_embedded\plugin.py:904: in serial
    return cls(**_drop_none_kwargs(kwargs))
<USER>\.espressif\python_env\idf5.0_py3.8_env\lib\site-packages\pytest_embedded_serial_esp\serial.py:50: in __init__
    self.esp: esptool.ESPLoader = esptool.get_default_connected_device(
<USER>\.espressif\python_env\idf5.0_py3.8_env\lib\site-packages\esptool.py:118: in get_default_connected_device
    print("Serial port %s" % each_port)
<USER>\.espressif\python_env\idf5.0_py3.8_env\lib\site-packages\pytest_embedded\log.py:195: in write
    self.pexpect_proc.write(data)
<USER>\.espressif\python_env\idf5.0_py3.8_env\lib\site-packages\pytest_embedded\log.py:105: in write
    self.send(s)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_embedded.log.PexpectProcess object at 0x0000022693E516D0>, s = b'\n'

    def send(self, s: AnyStr) -> int:
        """
        Write to the pexpect process and log.

        Args:
            s: bytes or str

        Returns:
            number of written bytes.
        """
        if not s:
            return 0

        s = self._coerce_send_string(s)
        self._log(s, 'send')

        # for pytest logging
        _temp = sys.stdout
        sys.stdout = self.STDOUT  # ensure the following print uses system sys.stdout

        _s = to_str(s)
        prefix = ''
        if self.source:
            prefix = f'[{self.source}] ' + prefix
        if self._with_timestamp:
            prefix = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S') + ' ' + prefix

        if not self._added_prefix:
            _s = prefix + _s
            self._added_prefix = True
        _s = _s.replace('\n', '\n' + prefix)
        if _s.endswith(prefix):
            _s = _s.rsplit(prefix, maxsplit=1)[0]
            self._added_prefix = False

>       sys.stdout.write(_s)
E       OSError: [WinError 6] The handle is invalid

<USER>\.espressif\python_env\idf5.0_py3.8_env\lib\site-packages\pytest_embedded\log.py:89: OSError
------------- generated xml file: C:\github_repos\idf-extra-components\test_app\test_app_results.xml --------------
============================================= short test summary info =============================================
ERROR test_app.py::test_app - OSError: [WinError 6] The handle is invalid
================================================ 1 error in 0.44s =================================================

Everything is fine with "-s"

Run tests faster (RDT-94)

service: idf,esp

re-organize the test sequence by app.binary_path(), skip autoflash between these cases

[jtag] Add mi mode flag (RDT-290)

For now we're using string mode for gdb by default. there're two ways to support mi2 mode

  1. run gdbmiparser.parse_response to parse the output
  2. use gdbmi = GdbController() instead of bare subprocess.popen

Prefer to use the first one. make the gdb write blocking as well, like the openocd one.

consider re-implemented the duplicated stdout without `multiprocessing` (RDT-80)

multiprocessing behaves differently on windows, macos, and linux. For now we could workaround this by using fork inside macos and linux, and spawn in windows. but in macos, fork is NOT recommended since it would cause error: https://bugs.python.org/issue33725

consider re-implement with https://pexpect.readthedocs.io/en/stable/api/popen_spawn.html#popenspawn-class and pass the stdout of other popen objects into the pexpect popen's stdin

(just an idea, need to investigate)

Get Rid of Thread (RDT-106)

Thanks to GIL, using threads to redirect stdout internally could cause performance loss if the test script is also using Thread.

Try to find a way to stop using threads.

pytest-embedded-serial-esp not compatible with esptool v4.x (RDT-220)

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behavior is the source of the following dependency conflicts.
pytest-embedded-serial-esp 0.7.1 requires esptool~=3.1, but you have esptool 4.1 which is incompatible.

The recent IDF master branch has switched to use esptool 4.1, when install the IDF environment, I saw the above error message.

[esp] Fix unclosed file warning (RDT-282)

=============================== warnings summary ===============================
pytest-embedded-serial-esp/tests/test_esp.py::test_detect_port_with_cache
/usr/local/lib/python3.10/inspect.py:2414: ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/pytest-of-root/pytest-0/test_idf_serial_flash_with_erase_nvs_but_no_parttool0/hello_world_esp32/build/hello_world.bin'>
sig = obj.signature
Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

pytest-embedded-serial-esp/tests/test_esp.py::test_detect_port_with_cache
/usr/local/lib/python3.10/inspect.py:2414: ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/pytest-of-root/pytest-0/test_idf_serial_flash_with_erase_nvs_but_no_parttool0/hello_world_esp32/build/partition_table/partition-table.bin'>
sig = obj.signature
Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

pytest-embedded-serial-esp/tests/test_esp.py::test_detect_port_with_cache
/usr/local/lib/python3.10/inspect.py:2414: ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/pytest-of-root/pytest-0/test_idf_serial_flash_with_erase_nvs_but_no_parttool0/hello_world_esp32/build/bootloader/bootloader.bin'>
sig = obj.signature
Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

Qemu dut frequently locked after 'Press ENTER to see the list of tests' (RDT-328)

I have observed following problem when running unity tests on QEMU:

dut.expect_exact('Press ENTER to see the list of tests')
dut.write('[qemu]')
dut.expect_unity_test_output(timeout=300)

After sending "[qemu]" to dut, there's no response from dut. This happens around 25% of time.

I have tried to run test application directly as follows:

qemu-system-xtensa -nographic -no-reboot -machine esp32 -drive file=/esp-idf/components/esp_ringbuf/test_apps/build_esp32_default/flash_image.bin,if=mtd,format=raw

and this problem can't be reproduced -- dut is always responding correctly.

[idf,jtag,qemu] Support running tests built as loadable ELF files (RDT-236)

Some test applications are small enough to be loaded into RAM of the target chip, and don't have to be flashed. Loading into RAM is typically faster than flashing.

The task is to support running tests with loadable ELF files for idf, jtag and qemu plugins.

For the ESP chips, ELF files can be loaded into the chip RAM in two ways:

  • Using esptool.py. In this case, the ELF file first needs to be converted into a .bin file (using esptool.py elf2image). Then the resulting .bin file can be loaded into chip RAM using esptool.py load_ram. The program is loaded and immediately starts to execute.
  • Over JTAG. In this case, the program can be loaded into RAM using GDB load command. (This approach works for all chips which have OpenOCD support, not necessarily ESPs.)

Note 1: we need to be careful not to reset the chip after loading the program and before running the test. Unlike the case when the program is flashed to the chip, the program loaded into RAM will be lost after reset. Then the chip will execute whichever program is stored in flash (if any).

Note 2: not every ELF file can be loaded into ESP chip RAM using the options above. When building an app using ESP-IDF, the option CONFIG_APP_BUILD_TYPE_ELF_RAM should be enabled. If it is not enabled, the resulting ELF file can't be loaded into RAM.
It is also possible to build loadable ELF files without IDF using the latest esp-2022r1-rc1 GCC based toolchain. (For the ESP32 only, so far.) For instructions, see internal merge request idf/newlib-cygwin!44.

For the old TTFW based implementation, see https://github.com/espressif/esp-idf/blob/master/tools/test_apps/system/gdb_loadable_elf/app_test.py.

Allow setting session_tempdir by a command line argument (RDT-267)

Currently the logs are placed into the system-wide /tmp directory. This is a bit inconvenient because the /tmp directory can't be easily be used for artifacts in Gitlab.

IDF overrides the session_tempdir fixture to set a different path. Perhaps we could add a command line argument for this, so that each project doesn't have to override this fixture?

Record Unity test execution time into JUnit report (RDT-289)

Currently add_unity_test_cases function parses the complete output of all the test cases and produces the Junit report. One thing we aren't doing with this approach is getting the execution times of individual test cases. This can be helpful when analyzing CI bottlenecks, to find specific test cases which are running for a long time.

v1.1.0 breaks the quick example (RDT-352)

The commit from #172 adds a dependency on pytest-embedded-idf without updating the pytest-embedded package's dependencies.

Now if users are following the quick example in the readme they will be treated to an error like this:

pytest
Traceback (most recent call last):
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 190, in console_main
    code = main()
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 148, in main
    config = _prepareconfig(args, plugins)
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 329, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/_pytest/helpconfig.py", line 103, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1058, in pytest_cmdline_parse
    self.parse(args)
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1346, in parse
    self._preparse(args, addopts=addopts)
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1229, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/home/daltonv/.pyenv/versions/3.10.2/lib/python3.10/importlib/metadata/__init__.py", line 162, in load
    module = import_module(match.group('module'))
  File "/home/daltonv/.pyenv/versions/3.10.2/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
    exec(co, module.__dict__)
  File "/home/daltonv/scratch/pytest_embedded_test/.venv/lib/python3.10/site-packages/pytest_embedded/plugin.py", line 42, in <module>
    from pytest_embedded_idf import CaseTester
ModuleNotFoundError: No module named 'pytest_embedded_idf'

The pytest-embedded/setup.py file should be modified to explicitly add this new dependency:

REQUIRES = [
    'pytest>=7.0',
    'pexpect>=4.4',
    'pytest_embedded>=1.1.0' # not sure if that exact version is the one needed
]

support testing console applications with pytest-embedded (RDT-237)

Although somewhat outside of the embedded scope of pytest-embedded, it would be convenient if we could test console applications using pytest-embedded as well.

ESP-IDF now supports building for the linux target, and the result of such a build is a console application. Similar to the applications running on the chip target, we could use expect and other features of pytest-embedded to check that such an application runs correctly.

This is especially convenient if the given test application can be built in IDF both for linux and for a chip target. Then we could write a test case which works for both builds, the only difference being the service provided by pytest-embedded (idf or console). This is similar to how we can already use the same test for QEMU and for a chip target.

Junit Report Fail to Merge when using multi-dut (RDT-256)

The dut-1.xml cases are trying to replace the sub cases to the main JUnit report, but the main test case is already been replaced by the dut-0.xml.

try merge the multi-dut sub cases at first and replace to main Junit report together.

command line arguments and addopts (RDT-253)

It is great that you managed to add --baud option to the tool.

I'm using Windows 10 and a FT232 serial adapter.
I noticed minor issue while passing arguments to the service:

  • addopts = --embedded-services serial -s --port COM3 --baud 921600 tests executed with pytest command works fine
  • addopts = --embedded-services serial -s --port COM3 tests executed with pytest --baud 921600 command works fine
  • addopts = --embedded-services serial -s --baud 921600 tests executed with pytest --port COM3 command fails with error OSError: [WinError 6] The handle is invalid

My goal is to specify the port from the command line.

[idf] Make elf file optional (RDT-200)

Auto-flash should only require 4 files, flash_args.json, bootloader.bin, partition_table.bin, and the real app .bin file.

elf file is only required when panic'ed. Make this file optional.

flash-image.bin not updated even if build/*.bin files have changed (RDT-87)

There are a couple issues related to generation of flash image when qemu is using together with idf:

  • flash-image.bin is created from .bin files in build/ directory. However if the project is rebuilt and files in build/ directory are updated, the flash-image.bin file isn't regenerated. It's possible to force a rebuild by deleting the file manually. However it's easy to forget to do this, and then we may spend time debugging some issue that is already fixed in the app, just the flash-image.bin isn't updated.
  • flash-image.bin is created in project directory, not build/ directory. So it's not excluded by default .gitignore rules, and it's not deleted by idf.py fullclean. It would be better to generate this file in build/

TypeError: cannot pickle 'EncodedFile' object (RDT-89)

When running pytest --port=/dev/cu.usbserial-01C3154E with pytest-embedded-serial plugin, the following error message is thrown:

args(classes={'app': <class 'pytest_embedded.app.App'>, 'serial': <class 'pytes... object at 0x10ac2f5b0>}})), 'app': <pytest_embedded.app.App object at 0x10ac2f640>, 'gdb': None, 'openocd': None, ...}
_close_or_terminate = <function apply_count_generator.<locals>.wrapper.<locals>._close_or_terminate at 0x10ac50700>, res = []

    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        def _close_or_terminate(obj):
            try:
                obj.close()
            except AttributeError:
                try:
                    obj.terminate()
                except AttributeError:
                    del obj
    
        res = []
        if COUNT == 1:
>           res = func(*args, **kwargs)

/.pyenv/versions/3.9.4/lib/python3.9/site-packages/pytest_embedded/plugin.py:150: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/.pyenv/versions/3.9.4/lib/python3.9/site-packages/pytest_embedded/plugin.py:824: in dut
    return cls(**kwargs)
/.pyenv/versions/3.9.4/lib/python3.9/site-packages/pytest_embedded_serial/dut.py:27: in __init__
    self.serial.create_forward_io_process(self.pexpect_proc, source='serial')
/.pyenv/versions/3.9.4/lib/python3.9/site-packages/pytest_embedded/log.py:167: in create_forward_io_process
    self._forward_io_proc.start()
/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/context.py:284: in _Popen
    return Popen(process_obj)
/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <Process name='Process-1' parent=30711 initial>, file = <_io.BytesIO object at 0x10ac4cc70>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       TypeError: cannot pickle 'EncodedFile' object

/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/reduction.py:60: TypeError

Environment:
OS: MacOS 12.0.1
Python version: Tested with 3.9.4 and 3.8.12
Connected hardware: esp32-wroom-32d
pytest.ini:

[pytest]
addopts = --embedded-services serial

# log related
log_cli = True
log_cli_level = INFO
log_cli_format = %(asctime)s %(levelname)s %(message)s
log_cli_date_format = %Y-%m-%d %H:%M:%S

log_file = test.log
log_file_level = INFO
log_file_format = %(asctime)s %(levelname)s %(message)s
log_file_date_format = %Y-%m-%d %H:%M:%S

Windows 10 echo example can't receive data (RDT-230)

Hi,

I modified echo example a little bit to make it work with Windows 10. Since I can't really use socat so I used USB-UART adapter with shorted TX and RX connected to COM12. The problem is that the dut.write() is working but dut.expect() can't read anything.

I also verified that the same setup works on Ubuntu 20.04, I can both write and receive data.

Here is the test case:

import pytest

@pytest.mark.parametrize(
    'port',
    ['COM12'],
    indirect=True,
)
def test_echo_tcp(dut):
    dut.write(b'aaa')
    dut.expect('aaa')  # will decode automatically

pytest.ini:

[pytest]
addopts = --embedded-services serial

# log related
log_cli = True
log_cli_level = INFO
log_cli_format = %(asctime)s %(levelname)s %(message)s
log_cli_date_format = %Y-%m-%d %H:%M:%S

In addition to this changes I removed the conftest.py

Here is the output after I use pytest command in the test case directory:

================================================= test session starts =================================================

platform win32 -- Python 3.10.0, pytest-7.1.2, pluggy-1.0.0

rootdir: C:\rnd\pytest-rnd\first_test, configfile: pytest.ini

plugins: embedded-0.7.3

collected 1 item

 

test_echo.py::test_echo_tcp[COM12]

--------------------------------------------------- live log setup ----------------------------------------------------

2022-06-24 08:52:31 INFO Logs recorded under folder: C:\Users\user1\AppData\Local\Temp\pytest-embedded\2022-06-24_06-52-31\test_echo_tcp[COM12]

FAILED                                                                                                           [100%]

 

====================================================== FAILURES =======================================================

________________________________________________ test_echo_tcp[COM12] _________________________________________________

 

self = <pytest_embedded_serial.dut.SerialDut object at 0x00000168C441F1F0>, pattern = 'aaa', expect_all = False

args = (), kwargs = {}, patterns = ['aaa'], res = [], wrapped_buffer_bytes = ''

debug_str = 'Not found "aaa"\nBytes in current buffer: \nPlease check the full log here: C:\\Users\\user1\\AppData\\Local\\Temp\\pytest-embedded\\2022-06-24_06-52-31\\test_echo_tcp[COM12]\\dut.log'

 

    @functools.wraps(func)  # noqa

    def wrapper(

        self, pattern, *args, expect_all: bool = False, **kwargs

    ) -> Union[Union[Match, AnyStr], List[Union[Match, AnyStr]]]:

        patterns = to_list(pattern)

        res = []

        while patterns:

            try:

>               index = func(self, pattern, *args, **kwargs)  # noqa

 

C:\Users\user1\AppData\Local\Programs\Python\Python310\lib\site-packages\pytest_embedded\dut.py:67:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

 

self = <pytest_embedded_serial.dut.SerialDut object at 0x00000168C441F1F0>, pattern = 'aaa', kwargs = {}

 

    @_pexpect_func  # noqa

    def expect(self, pattern, **kwargs) -> Match:  # noqa

        """

        Expect from `pexpect_proc`. All the arguments would pass to `pexpect.expect()`.

 

        Returns:

            AnyStr: if you're matching pexpect.EOF or pexpect.TIMEOUT to get all the current buffers.

 

        Returns:

            re.Match: if matched given string.

        """

>       return self.pexpect_proc.expect(pattern, **kwargs)

 

C:\Users\user1\AppData\Local\Programs\Python\Python310\lib\site-packages\pytest_embedded\dut.py:109:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

 

self = <pytest_embedded.log.PexpectProcess object at 0x00000168C441C550>, pattern = 'aaa', timeout = -1

searchwindowsize = -1, async_ = False, kw = {}, compiled_pattern_list = [re.compile(b'aaa', re.DOTALL)]

 

    def expect(self, pattern, timeout=-1, searchwindowsize=-1, async_=False, **kw):

        '''This seeks through the stream until a pattern is matched. The

        pattern is overloaded and may take several types. The pattern can be a

        StringType, EOF, a compiled re, or a list of any of those types.

        Strings will be compiled to re types. This returns the index into the

        pattern list. If the pattern was not a list this returns index 0 on a

        successful match. This may raise exceptions for EOF or TIMEOUT. To

        avoid the EOF or TIMEOUT exceptions add EOF or TIMEOUT to the pattern

        list. That will cause expect to match an EOF or TIMEOUT condition

        instead of raising an exception.

 

        If you pass a list of patterns and more than one matches, the first

        match in the stream is chosen. If more than one pattern matches at that

        point, the leftmost in the pattern list is chosen. For example::

 

            # the input is 'foobar'

            index = p.expect(['bar', 'foo', 'foobar'])

            # returns 1('foo') even though 'foobar' is a "better" match

 

        Please note, however, that buffering can affect this behavior, since

        input arrives in unpredictable chunks. For example::

 

            # the input is 'foobar'

            index = p.expect(['foobar', 'foo'])

            # returns 0('foobar') if all input is available at once,

            # but returns 1('foo') if parts of the final 'bar' arrive late

 

        When a match is found for the given pattern, the class instance

        attribute *match* becomes an re.MatchObject result.  Should an EOF

        or TIMEOUT pattern match, then the match attribute will be an instance

        of that exception class.  The pairing before and after class

        instance attributes are views of the data preceding and following

        the matching pattern.  On general exception, class attribute

        *before* is all data received up to the exception, while *match* and

        *after* attributes are value None.

 

        When the keyword argument timeout is -1 (default), then TIMEOUT will

        raise after the default value specified by the class timeout

        attribute. When None, TIMEOUT will not be raised and may block

        indefinitely until match.

 

        When the keyword argument searchwindowsize is -1 (default), then the

        value specified by the class maxread attribute is used.

 

        A list entry may be EOF or TIMEOUT instead of a string. This will

        catch these exceptions and return the index of the list entry instead

        of raising the exception. The attribute 'after' will be set to the

        exception type. The attribute 'match' will be None. This allows you to

        write code like this::

 

                index = p.expect(['good', 'bad', pexpect.EOF, pexpect.TIMEOUT])

                if index == 0:

                    do_something()

                elif index == 1:

                    do_something_else()

                elif index == 2:

                    do_some_other_thing()

                elif index == 3:

                    do_something_completely_different()

 

        instead of code like this::

 

                try:

                    index = p.expect(['good', 'bad'])

                    if index == 0:

                        do_something()

                    elif index == 1:

                        do_something_else()

                except EOF:

                    do_some_other_thing()

                except TIMEOUT:

                    do_something_completely_different()

 

        These two forms are equivalent. It all depends on what you want. You

        can also just expect the EOF if you are waiting for all output of a

        child to finish. For example::

 

                p = pexpect.spawn('/bin/ls')

                p.expect(pexpect.EOF)

                print p.before

 

        If you are trying to optimize for speed then see expect_list().

 

        On Python 3.4, or Python 3.3 with asyncio installed, passing

        ``async_=True``  will make this return an :mod:`asyncio` coroutine,

        which you can yield from to get the same result that this method would

        normally give directly. So, inside a coroutine, you can replace this code::

 

            index = p.expect(patterns)

 

        With this non-blocking form::

 

            index = yield from p.expect(patterns, async_=True)

        '''

        if 'async' in kw:

            async_ = kw.pop('async')

        if kw:

            raise TypeError("Unknown keyword arguments: {}".format(kw))

 

        compiled_pattern_list = self.compile_pattern_list(pattern)

>       return self.expect_list(compiled_pattern_list,

                timeout, searchwindowsize, async_)

 

C:\Users\user1\AppData\Local\Programs\Python\Python310\lib\site-packages\pexpect\spawnbase.py:343:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

 

self = <pytest_embedded.log.PexpectProcess object at 0x00000168C441C550>, pattern_list = [re.compile(b'aaa', re.DOTALL)]

timeout = 30, searchwindowsize = -1, async_ = False, kw = {}

exp = <pexpect.expect.Expecter object at 0x00000168C441F760>

 

    def expect_list(self, pattern_list, timeout=-1, searchwindowsize=-1,

                    async_=False, **kw):

        '''This takes a list of compiled regular expressions and returns the

        index into the pattern_list that matched the child output. The list may

        also contain EOF or TIMEOUT(which are not compiled regular

        expressions). This method is similar to the expect() method except that

        expect_list() does not recompile the pattern list on every call. This

        may help if you are trying to optimize for speed, otherwise just use

        the expect() method.  This is called by expect().

 

        Like :meth:`expect`, passing ``async_=True`` will make this return an

        asyncio coroutine.

        '''

        if timeout == -1:

            timeout = self.timeout

        if 'async' in kw:

            async_ = kw.pop('async')

        if kw:

            raise TypeError("Unknown keyword arguments: {}".format(kw))

 

        exp = Expecter(self, searcher_re(pattern_list), searchwindowsize)

        if async_:

            from ._async import expect_async

            return expect_async(exp, timeout)

        else:

>           return exp.expect_loop(timeout)

 

C:\Users\user1\AppData\Local\Programs\Python\Python310\lib\site-packages\pexpect\spawnbase.py:372:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

 

self = <pexpect.expect.Expecter object at 0x00000168C441F760>, timeout = -0.00274658203125

 

    def expect_loop(self, timeout=-1):

        """Blocking expect"""

        spawn = self.spawn

 

        if timeout is not None:

            end_time = time.time() + timeout

 

        try:

            idx = self.existing_data()

            if idx is not None:

                return idx

            while True:

                # No match at this point

                if (timeout is not None) and (timeout < 0):

                    return self.timeout()

                # Still have time left, so read more data

                incoming = spawn.read_nonblocking(spawn.maxread, timeout)

                if self.spawn.delayafterread is not None:

                    time.sleep(self.spawn.delayafterread)

                idx = self.new_data(incoming)

                # Keep reading until exception or return.

                if idx is not None:

                    return idx

                if timeout is not None:

                    timeout = end_time - time.time()

        except EOF as e:

            return self.eof(e)

        except TIMEOUT as e:

>           return self.timeout(e)

 

C:\Users\user1\AppData\Local\Programs\Python\Python310\lib\site-packages\pexpect\expect.py:181:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

 

self = <pexpect.expect.Expecter object at 0x00000168C441F760>

err = TIMEOUT("<pytest_embedded.log.PexpectProcess object at 0x00000168C441C550>\nsearcher: searcher_re:\n    0: re.compile(b'aaa')")

 

    def timeout(self, err=None):

        spawn = self.spawn

 

        spawn.before = spawn._before.getvalue()

        spawn.after = TIMEOUT

        index = self.searcher.timeout_index

        if index >= 0:

            spawn.match = TIMEOUT

            spawn.match_index = index

            return index

        else:

            spawn.match = None

            spawn.match_index = None

            msg = str(spawn)

            msg += '\nsearcher: %s' % self.searcher

            if err is not None:

                msg = str(err) + '\n' + msg

 

            exc = TIMEOUT(msg)

            exc.__cause__ = None    # in Python 3.x we can use "raise exc from None"

>           raise exc

E           pexpect.exceptions.TIMEOUT: <pytest_embedded.log.PexpectProcess object at 0x00000168C441C550>

E           searcher: searcher_re:

E               0: re.compile(b'aaa')

E           <pytest_embedded.log.PexpectProcess object at 0x00000168C441C550>

E           searcher: searcher_re:

E               0: re.compile(b'aaa')

 

C:\Users\user1\AppData\Local\Programs\Python\Python310\lib\site-packages\pexpect\expect.py:144: TIMEOUT

 

The above exception was the direct cause of the following exception:

 

dut = <pytest_embedded_serial.dut.SerialDut object at 0x00000168C441F1F0>

 

    @pytest.mark.parametrize(

        'port',

        ['COM12'],

        indirect=True,

    )

    def test_echo_tcp(dut):

        dut.write(b'aaa')

>       dut.expect('aaa')

 

test_echo.py:10:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

 

self = <pytest_embedded_serial.dut.SerialDut object at 0x00000168C441F1F0>, pattern = 'aaa', expect_all = False

args = (), kwargs = {}, patterns = ['aaa'], res = [], wrapped_buffer_bytes = ''

debug_str = 'Not found "aaa"\nBytes in current buffer: \nPlease check the full log here: C:\\Users\\user1\\AppData\\Local\\Temp\\pytest-embedded\\2022-06-24_06-52-31\\test_echo_tcp[COM12]\\dut.log'

 

    @functools.wraps(func)  # noqa

    def wrapper(

        self, pattern, *args, expect_all: bool = False, **kwargs

    ) -> Union[Union[Match, AnyStr], List[Union[Match, AnyStr]]]:

        patterns = to_list(pattern)

        res = []

        while patterns:

            try:

                index = func(self, pattern, *args, **kwargs)  # noqa

            except (pexpect.EOF, pexpect.TIMEOUT) as e:

                wrapped_buffer_bytes = textwrap.shorten(

                    to_str(self.pexpect_proc.buffer),

                    width=200,

                    placeholder=f'... (total {len(self.pexpect_proc.buffer)} bytes)',

                )

                debug_str = (

                    f'Not found "{str(pattern)}"\n'

                    f'Bytes in current buffer: {wrapped_buffer_bytes}\n'

                    f'Please check the full log here: {self.logfile}'

                )

>               raise e.__class__(debug_str) from e

E               pexpect.exceptions.TIMEOUT: Not found "aaa"

E               Bytes in current buffer:

E               Please check the full log here: C:\Users\user1\AppData\Local\Temp\pytest-embedded\2022-06-24_06-52-31\test_echo_tcp[COM12]\dut.log

 

C:\Users\user1\AppData\Local\Programs\Python\Python310\lib\site-packages\pytest_embedded\dut.py:79: TIMEOUT

------------------------------------------------- Captured log setup --------------------------------------------------

INFO     root:dut.py:34 Logs recorded under folder: C:\Users\user1\AppData\Local\Temp\pytest-embedded\2022-06-24_06-52-31\test_echo_tcp[COM12]

=============================================== short test summary info ===============================================

FAILED test_echo.py::test_echo_tcp[COM12] - pexpect.exceptions.TIMEOUT: Not found "aaa"

================================================= 1 failed in 33.57s ==================================================

2022-06-24 08:52:31 aaa

Support expect from different source (RDT-311)

Now we have a few different sources, like serial, openocd, gdb, ..., etc. Each source is a single process, and would have their own output and input. if we combine all the output from different sources into one single pexpect process, then it would sometimes be messed up.

Success Criteria:

  • Record the output of each process into a single file, and has its own pexpect process.
  • modify the original PexpectProcess class, to make it as the parent class to hold all the processes created by different sources. The users would still use the same API to expect the patterns from different sources.
  • add keyword source of expect functions to indicate which source to expect from. default set to serial.
  • add keyword source of redirect fixture
  • add attr source to all derived fixture classes

and one reminder:

  • don't make breaking changes. it's not necessary.

Wrong Boot Mode When Multi-Dut Tests (RDT-108)

As many github actions jobs are failing, multi-dut tests would fail with the error "wrong boot mode ..." when detecting the port.
and this error often happens on the 2nd dut.

Not 100% reproducible, but the failure ratio is higher with earlier python versions.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.