Embedded testing about wg HOT 9 CLOSED

Lately, I've been exploring running Cortex-M programs in QEMU using two different approaches.

• Using QEMU user emulation (qemu-arm). This doesn't fully emulate a Cortex-M core; instead it works more like a translation layer that gives access to the host kernel to the Cortex-M program. Because it doesn't emulate a Cortex-M core you can't run cortex-m-rt programs or use instructions like WFI or access registers like BASEPRI (all these ops crash QEMU). However, you can execute pure (no I/O) Cortex-M machine code and use the host stdin, stdout and stderr. For more details see qemu-arm-rt.

• Using QEMU system emulation (qemu-system-arm). With this approach QEMU fully emulates a Cortex-M core. You can run cortex-m-rt programs in this mode, instructions like WFI work (or at least don't crash QEMU) and registers like BASEPRI are properly emulated. Because the emulated program doesn't have access to the host it seems harder to script this for testing purposes. However, you can hook GDB to the emulated core and work from there your way up using Python, but I haven't really explored this angle. For more details see lm3s6965evb.

It may be possible to leverage these approaches for testing Cortex-M programs w/o hardware but this will need more work and I won't have time to explore this further any time soon. So I'm sharing the info here hoping that someone else will continue to explore this area.

from wg.

qemu-system-arm has a good support for semihosting, you can open, close, read, and write files on your host system, with it. You can even terminate a qemu session from the inside (which is great for automating tests), I've added a call for that in cortex-m-semihosting some year ago: https://docs.rs/cortex-m-semihosting/0.3.0/cortex_m_semihosting/debug/index.html

So it shouldn't be too hard to use qemu-system-arm for testing purposes. The only thing that is missing are FPU instructions, so you can only test M3 targets, but not M4.

from wg.

I didn't have a chance to say this before we ran out of time in the meeting, but I'm very interested in this topic, specifically HIL testing. So far I haven't looked into the details, and I have some other stuff to get out of the way before I can work on this, but I'm definitely interested in anything that's going on in this space.

from wg.

For bobbin-cli test I implemented a simple text-based protocol similar to TAP. From the README:

[start] Running tests for frdm-k64f
[pass] 0
[pass] 1
[pass] 2
[pass] 3
[pass] 4
[done] All tests passed

bobbin test recognizes [start], [pass] and [done] tags, exiting with return code 0. It also recognizes [fail], [exception], and [panic] tags, which will cause it to exit with return codes 1, 2 or 3. All other output is ignored.

The test runner will exit with return code 1 if there is a delay of more than 5 seconds between lines or 15 seconds to complete the entire test. In the future these timeouts will be configurable.

This has worked pretty well for me. It's simple to implement, you can capture panics and exceptions by having your handlers print a [panic] or [exception] line after any diagnostics, and it's easy to add additional output that the test runner will ignore just by using println!.

I chose the prefix tags specifically because they are easy to scan for visually and to grep for in logs. One can build a basic distributed CI system simply by piping the test output into syslog which is then dumped into a log aggregator.

from wg.

I am also interested in assisting with this topic. I don't have anything to offer (except the blog post linked), but I would be happy to advise and assist once there is a direction to run with.

from wg.

I'm working on adding support for the redpitaya logic analyzer to sigrok/pulsview (completed uio driver this morning / goal is to implement a tcp server over the weekend). I'm considering building an ice40 based test board that hooks up the la channels for the unit test and loads the unit tests via ftdi. Does this sound generally useful? Does this already exist?

#Done

from wg.

One thing I have done some research on, in the past, is unit testing on the target. The opinions differ on that one, because most of the times the effort outweighs the benefits, but often that's the only reason that stands against it. So, I think it would be a good idea to keep that in mind while working on testing tools. With the help of traits, macros and maybe also compiler plug-ins it should be much easier to provide an easy to use and generic test framework, which could be used on host and/or target without too much additional work.

from wg.

Due @japaric suggestion, I'd like to bring up to discuss the use of https://github.com/labgrid-project/labgrid for automated board test. It tries to control the board externally and has the capability to manage external resources required for some automated tests (e.g pressing buttons, power control, ...) of DUT. Obviously, it may require specific hardware for some type of tests but it seems worth taking a look for use or inspiration.

from wg.

There is also a wrapper tool qemu-system-gnuarmeclipse that I have come across, which is a wrapper written over qemu. It also has support for cortex m4 targets.

freertos_rs by @hashmismatch uses qemu-system-gnuarmeclipse for running unit test cases over cortex m4 without FPU.

from wg.