This project supports the execution of Mozilla's Android Hardware tests at Bitbar.

Bitbar Projects and Device Groups will be created or updated as needed to match the specified configuration.

Bitbar Frameworks must be managed via the Testdroid user interface.

# Clone the repository
git clone https://github.com/bclary/mozilla-bitbar-devicepool.git
cd mozilla-bitbar-devicepool

# if needed, install poetry
#   - see https://python-poetry.org/docs/ for more options and info
curl -sSL https://install.python-poetry.org | python3 -

# activate the virtual environment
poetry shell

# install dependencies (with dev deps)
poetry install --with=dev

# Update the repository
cd mozilla-bitbar-devicepool
git pull --rebase

# activate the virtual environment
poetry shell

# install any updated dependencies
poetry install

# Restart the service

Configuring mozilla-bitbar-devicepool requires creating environment variables and a yaml configuration file for Bitbar.

export TESTDROID_URL=https://mozilla.testdroid.com
export TESTDROID_APIKEY=<testdroid apikey>
export <taskcluster worker type>=<taskcluster access token for worker type>
...

where TESTDROID_URL is the url of the Mozilla instance of the Bitbar cloud service and TESTDROID_APIKEY is the access token for the account under which tests will be executed.

For each Taskcluster worker type defined for Bitbar tests, an environment variable whose name is constructed from the actual workerType by replacing any dashes - with underscores _ and whose value is set to the Taskcluster access token for the corresponding Taskcluster client.

The [Service Installation](#Service Installation) assumes these environment variables are stored in /etc/bitbar/bitbar.env.

The Bitbar configuration is specified as a YAML file with the following layout:

# The projects section contains definitions for each of the projects
# defined for mozilla-bitbar-devicepool. Each subsection of projects
# consists of a project name followed by the project properties.
projects:
  # defaults is a special project name which is used to define
  # properties which will be set on the other projects. This
  # allows common properties to be specified once for defaults
  # rather than having to specify them for each individual project.
  # If a project also defines the same property, it will override
  # the default value.
  defaults:
    os_type: ANDROID
    project_type: APPIUM_ANDROID_SERVER_SIDE
    # Files should have unique names which prevent collisions
    # with files from other users. These examples use a prefix
    # to distinguish them.
    application_file: bclary-Testdroid.apk
    test_file: bclary-empty-test.zip
    timeout: 0
    scheduler: SINGLE
    archivingStrategy: DAYS
    archivingItemCount: 365
    taskcluster_provisioner_id: proj-autophone
    # additional_parameters specify a set of environment
    # variables which will be passed to the Docker container
    # executing the test.
    additional_parameters:
      bitbar_cloud_url: https://mozilla.testdroid.com
      DOCKER_IMAGE_VERSION: 20190109T064727
      TC_WORKER_CONF: gecko-t-ap
  # Each Bitbar project has its own section.
  mozilla-unittest-p2:
    device_group_name: pixel2-unit
    device_model: pixel2
    framework_name: mozilla-usb
    description: Mozilla Unit tests for Pixel2
    additional_parameters:
      TASKCLUSTER_CLIENT_ID: project/autophone/bitbar-x-unit-p2
      TC_WORKER_TYPE: gecko-t-ap-unit-p2
  #...
# The device-groups section contains definitions for each of the
# device groups. Each subsection of device_groups contains sections
# which list the names of the devices assigned to the device group.
device_groups:
  motog4-docker-builder:
    Docker Builder:
  motog5-batt:
    motog5-08:
    motog5-15:
  motog5-perf:
  #...

The mbd script is used to perform tasks related to running the Mozilla Android Hardware tests. It is implemented using sub-command arguments.

$ mbd --help
usage: mbd [-h] [--files FILES]
               [--log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}]
               {download-testdroid-apk,empty-test-zip,start-test-run-manager,run-test}
               ...

Mozilla Android Hardware testing at Bitbar.

positional arguments:
  {download-testdroid-apk,empty-test-zip,start-test-run-manager,run-test}
                        Specify one of the positional arguments to select the
                        command to execute.
    download-testdroid-apk
                        Download Testdroid.apk from https://github.com/bitbar
                        /bitbar-samples/blob/master/apps/builds/Testdroid.apk
                        to files/ then exit.
    empty-test-zip      Generate empty test zip file in files/ directory then
                        exit.
    start-test-run-manager
                        Run the test run manager.
    run-test            Run test for a project then exit.

optional arguments:
  -h, --help            show this help message and exit
  --files FILES         Directory where downloaded files are saved. Defaults
                        to /home/bclary/mozilla/projects/android-hardware-
                        testing/github.com/bclary/mozilla-bitbar-
                        devicepool/files
  --log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}
                        Logging level. Defaults to INFO.

Environment Variables:

The following environment variables must be set prior to attempting to
start the test_run_manager.

TESTDROID_URL
TESTDROID_APIKEY

To get additional help for each positional sub-command, add
--help to the sub-command.

Controlling the test_run_manager via signals:

Stop Now
    kill -USR2 <pid>

    Stop process while leaving test containers running.

Terminate Now
    kill -TERM  <pid>

    Abort any running test containers and exit immediately.

The Bitbar tests require an Android application apk be specified even though it is not used in the current tests. The download-testdroid-apk sub-command downloads the Testdroid.apk sample application and saves it to the files/ directory. You can specify a unique name to prevent name collisions with files from other users.

$ mbd download-testdroid-apk --help
usage: mbd download-testdroid-apk [-h] [--filename FILENAME] [--force]

optional arguments:
  -h, --help           show this help message and exit
  --filename FILENAME  Specify filename to save Testdroid.apk to in the files
                       directory.
  --force              Overwrite existing file.

The Bitbar tests require a zip test file be specified. The only project which currently uses a non-empty test zip file is the mozilla-docker-build project. The empty-test-zip sub-command create an empty zip file in the files/ directory. You can specify a unique name to prevent name collisions with files from other users.

$ mbd empty-test-zip --help
usage: mbd empty-test-zip [-h] [--filename FILENAME]

optional arguments:
  -h, --help           show this help message and exit
  --filename FILENAME  Specify filename to save in the files directory.
                       Defaults to empty-test.zip.

The sub-command start-test-run-manager starts a long running process which polls the Taskcluster queues for the worker types specified in the configuration for pending tasks and starts a corresponding Bitbar test run to service the task.

$ mbd start-test-run-manager --help
usage: mbd start-test-run-manager [-h] [--bitbar-config BITBAR_CONFIG]
                                      [--wait WAIT] [--delete-bitbar-tests]

optional arguments:
  -h, --help            show this help message and exit
  --bitbar-config BITBAR_CONFIG
                        Path to Bitbar yaml configuration file.
  --wait WAIT           Seconds to wait between checks. Defaults to 60.
  --delete-bitbar-tests
                        Delete bitbar tests after finishing. Defaults to
                        False.

The sub-command run-test is used to start a single test at Bitbar for a project. The primary use for this command apart from one-off test runs, is to execute test runs for the mozilla-docker-build project which creates a new Docker image for use in further testing.

$ mbd run-test --help
usage: mbd run-test [-h] --project_name PROJECT_NAME

optional arguments:
  -h, --help            show this help message and exit
  --project_name PROJECT_NAME
                        Specify a project name for which to start a test.

These instructions assume that you will be setting up the mozilla-bitbar-devicepool to run as a systemd service.

Customize bitbar.service to fit the location where you will install mozilla-bitbar-devicepool. By default, it assumes it will be installed into the /home/bitbar directory.

It uses the start_android_hardware_testing.sh and stop_android_hardware_testing.sh scripts in the bin directory to start and stop the test run manager.

sudo cp service/bitbar.service /etc/systemd/system/
sudo chmod 664 /etc/systemd/system/bitbar.service
sudo systemctl daemon-reload

Start the bitbar.service

systemctl start bitbar.service

Enable the bitbar.service to start on boot.

systemctl enable bitbar.service

To follow the log output of the bitbar service

sudo journalctl _SYSTEMD_UNIT=bitbar.service --follow

pip install -r requirements-dev.txt  # install deps
pre-commit install  # install the pre-commit hook
# make changes
# commit
# test
# make a PR

You must install the development requirements first. See the "Development" section above.

pytest  # runs once
# or
pytest-watch  # monitors files for changes and reruns