We use a jessie base image in tests.

Debian moved jessie apt repos to archive.debian.org.
https://lists.debian.org/debian-devel-announce/2019/03/msg00006.html

This breaks all our jenkins meta-balena PR builds.

• Note down the version of the image being tested.
• Check if the same version shows up through resin cli (resin device )

Resin CLI has the following command that can be used to trigger the identification LED

resin device identify

• Will need to abstract GPIO checking in Robot.
• Will need to figure out how to get GPIO access on a qemu machine.

Looking at #48 (and the way it now sprinkles device_uuid1, device_uuid2 and device_uuid in various places), I wonder if it'd be worth running the Qemu autohat tests on TravisCI? Or would they be too slow / too intensive?

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

⚠ Dependency Lookup Warnings ⚠

• Renovate failed to look up the following dependencies: Failed to look up docker package balenalib/${ARCH}-node, Failed to look up docker package balenalib/${ARCH}-python.

Files affected: Dockerfile

This repository currently has no open or pending branches.

Detected dependencies

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency balena-io/balena-cli to v19.0.11

Detected dependencies

Resin devices have a resin-vpn IP address that should not be sent to the dashboard. Verify that the resin-vpn ip address is not in the IP address field returned by the API.

Currently we use resin-cli@^5.2.0 in our Dockerfile which sometimes pulls in unstable versions of resin-cli - fixing the version will ensure stability for the tests.

This test case would allow us to see trends over time of I/O speeds. Since Robot framework automatically records the execution time of test cases - we could just have two cases one just writing from /dev/null to a 100MB file and one reading from the 100 MB file to /dev/null using dd.

1. Helps us in predicting a failure of the hardware rig.
2. Find regressions in kernel configurations causing SDcard I/O slow downs.
3. Allow us to have a rig to benchmark different SD card speeds.

FD

Currently the FROM image and the pip libraries are not locked to a version - we need to lock them to stable versions to avoid backward incompatible future releases from causing errors.

Test steps:

• For each of the update strategies:
  • Set a RESIN_SUPERVISOR_UPDATE_STRATEGY config var to the corresponding value
  • Push an update to a running app

Expected result:
The system logs show the corresponding steps (kill, download, start, delete) for the selected strategy are done in the correct order.

Write a test to check if the VPN is reflecting the Online/Offline state properly by switching on and off the device.

Related to resinos.robot#L138

We have a resin sync test. But resin sync has been deprecated since v11 in favour of balena push.

https://github.com/resin-os/resinos-tests/blob/master/tests/all/resinhup/fingerprints

Similar to how we do it for root partition: https://github.com/resin-io/autohat/blob/master/resources/resinos.robot#L22

The CLI can introduce breaking changes which could make the tests fail until they are rewritten. If we use a fixed version of the CLI instead then we can know the tests will continue to work whilst we upgrade them to the new version.

Keyword that is not checking correctly is: Check if service "${service}" is running using socket "${socket}" from resources/resinos.robot.

After a successful device provision (in any supervisor test case), expected result is that:

• config.json has registered_at, uuid and deviceId properly set
• The API reports that the device has a logs_channel properly set

For each of the supervisor's API endpoints (docs) hit the endpoint and check response and expected behavior. (Shutdown might be hard to test, but it should be okay if reboot works).

Please ping me for any questions about the expected behavior of the endpoints.

It took quite long time to understand, that resin command is not resinctl, but balena

Ensure the working of -> http://docs.resin.io/tools/cli/#ssh-uuid-

1. Run resin ssh
2. A shell prompt (e.g. bash-4.3#) should appear after a few seconds.
3. Running 'env | grep RESIN' should return a list of environment variables.

Valid for any board with Resin OS.
Test steps:

• Check that the device host OS has a [email protected] that is enabled.
• Set a RESIN_HOST_LOG_TO_DISPLAY=0 config variable
• Expected result: the device logs show the change is being applied, the device reboots and the resin-info service is disabled.
• Change RESIN_HOST_LOG_TO_DISPLAY=1.
• Expected result: the device logs show the change is being applied, the device reboots and the resin-info service is enabled again.

Test Steps:

1. Add an application-wide RESIN_SUPERVISOR_DELTA = 1 config var. (It is important that the value is 1).
2. Push an update to the application (e.g. changing what is outputted to the console).

Expected result:

1. When setting RESIN_SUPERVISOR_DELTA, the device should not restart the app.
2. The device should download the update correctly after pushing.
3. Fetching the devices' environment variables should show that "RESIN_SUPERVISOR_DELTA" environment variable is set to 1.

It would be really nice to get the journald logs from the device in order to try to work out why tests failed

Naming fingerprint file differs from 1.X to 2.X resin images versions.

On 2.X: RESIN_FINGERPRINT_EXT ?= "fingerprint"
RESIN_FINGERPRINT_FILENAME ?= "resinos"
Source: https://github.com/resin-os/meta-resin/blob/b9bba027aad797c7f9401f2e2d2952a328e900fb/meta-resin-common/classes/image_types_resin.bbclass#L89

On 1.X: echo -n '' > ${WORKDIR}/${RESIN_BOOT_FS_LABEL}.${FINGERPRINT_EXT}
Source: https://github.com/resin-os/meta-resin/blob/1.X/meta-resin-common/classes/image_types_resin.bbclass#L180

Add support to use the SD-MUX test rig to run test cases on Real hardware.

Ensure the working of -> http://docs.resin.io/tools/cli/#sync-uuid-

1. Adapt ondevice repo to work with resin sync
2. Edit server.py in https://github.com/resin-io/autohat-ondevice/blob/master/server.py
   to return "Hello Resin Sync!" when accessed over device URL.
3. The sync process should start with status messages appearing on each step ('Stopping application container...', 'Syncing..', 'Starting application container...').
4. At the end of resin sync a "resin sync completed successfully!" message should appear.
5. The device's Public Device URL should now show the new message, e.g. 'Hello Resin Sync!'

Get the host OS version of the ${image}

Add test to avoid balena-os/balena-supervisor#239 .

CC @pcarranzav

Test Steps:

Provision a device on an app that already has the RESIN_SUPERVISOR_DELTA=1 var and a pushed image.

Expected result:

1. The device should appear on the dashboard and correctly download the image for the app.
2. The devices env variables should show RESIN_SUPERVISOR_DELTA env var set to 1.

Description: Provisioning a device on an app that already has deltas pushed

Test steps Provision a device on the same app used for TC10 (that already has the RESIN_SUPERVISOR_DELTA=1 var and a pushed image).

Expected result The device should appear on the dashboard and correctly download the image for the app. Running a web terminal and running "env" should show the RESIN_SUPERVISOR_DELTA env var set to 1.

From Robot Framework docs

This works fine normally, but if there is a lot of output, the output buffers may get full and the program can hang.

This causes the git push to the builder to hang.

Check for I/O errors in the kernel dmesg at the end of the test suite. This allows us to ensure that our rig hardware does not have I/O errors.

To simulate a I/O error the following links are useful ->

1. https://lxadm.com/Using_fault_injection
2. https://www.kernel.org/doc/Documentation/fault-injection/fault-injection.txt

https://github.com/resin-os/resinos-tests/blob/master/tests/all/rootfs/backup_files

To be tested only on the Raspberry Pi family.
Test steps:

• Set a RESIN_HOST_CONFIG_foo variable
• Expected result: Logs show the variable is being applied. The device reboots and has a foo variable in config.txt correctly set.
• Change the variable to a different value.
• Expected result: same as before, now after the reboot config.txt will show the updated value.
• Delete the variable
• Expected result: after reboot, the device no longer shows the foo variable in config.txt

Description: A supervisor that is running without deltas starts using them.

Test steps

• Provision a device
• Add an application-wide RESIN_SUPERVISOR_DELTA = 1
• Push an update to the application (e.g. changing what is outputted to the console).

Expected result

• When setting RESIN_SUPERVISOR_DELTA, the device should not restart the app.
• The device should download the update correctly after pushing.
• The containers ENV should have RESIN_SUPERVISOR_DELTA=1