artemnikitin / devicefarm-ci-tool Goto Github PK

View Code? Open in Web Editor NEW

15.0 4.0 5.0 4.34 MB

Tool that helps you to run tests in AWS Device Farm easily

License: MIT License

Go 98.36% Makefile 1.64%

golang go device-farm aws-devicefarm aws cli ci-cd continuous-integration ci mobile

devicefarm-ci-tool's Introduction

Tool that helps you to run your tests in AWS Device Farm easily

Description

Did you try to run your app in AWS Device Farm via CLI or API? It was easy, right? Right now you can probably say ARN of your project after wake up in the middle of the night :)

This tool helps you to run tests in AWS Device Farm. You don't need to know ARN of your project because it's for machines and not for humans.

It's not a replacement for existing AWS CLI tools. It was created for a very specific purpose, to run tests in CI without a lot of configuration that required for existing solutions. It based on an assumption that all setup is already done. It means, that if you will specify an unexisted project name, then the tool will not create a project for you.

AWS Credentials

Set environment variables

export AWS_ACCESS_KEY_ID=<key>    
export AWS_SECRET_ACCESS_KEY=<secret>

Download

Get it via

go get -u github.com/artemnikitin/devicefarm-ci-tool

Or download binaries for Windows, MacOS or Linux from latest release

Run

Required launch parameters:

project name of a project.
app path to an app.
Example:

devicefarm-ci-tool -project name -app /path/to/my/app.apk

By default, "BUILTIN_FUZZ" tests will be run for your app.

Optional parameters:

run name of test run. Overrides value of runName parameter from config.
Example: -run myName
test path to tests. Overrides value of testPackagePath parameter from config.
Example: -test /path/to/my/testapp.apk
devices name of device pool where app will be run. If not specified, then tests will be run in default pool selected by AWS.
Example: -devices my-device-pool
randomDevices will randomly select device pool from provided list
Example: -randomDevices aaa,bbb,ccc,xxx,yyy,zzz
useRandomDevicePool device pool will be selected randomly from existed pools which are not used in any active test run
Example: -useRandomDevicePool
config path to config in JSON format.
Example: -config /path/to/config.json
wait wait for an end of test run. Useful for CI. Disabled by default.
Example: -wait
checkEvery checks every X seconds for test run completion. Default value is 5 second.
Example: -checkEvery 15
ignoreUnavailableDevices allows to consider test runs as passed for runs where tests passes on several devices, but some of devices were unavailable. For example, if device pool consists of 3 devices and on 2 devices everything is ok, but third device was unavailable, then using this option will mark test run as successful.
Example: -ignoreUnavailableDevices
testType allows to specify test type via command line. Test type should be one of test types available on AWS Device Farm, see ScheduleRunTest. Overrides value of type parameter from config.
Example: -testType INSTRUMENTATION

Configuration file

All parameters in the configuration file are optional. Configuration file is based on a syntax of ScheduleRun request.

Config example:

{
   "name": "string",
   "projectArn": "string",
   "projectName": "string",
   "appArn": "string",
   "appPath": "string",
   "devicePoolArn": "string",
   "devicePoolPath": "string",
   "testPackagePath": "string",
   "extraDataPackagePath": "string",
   "auxiliaryAppsPath": [ "path/to/app-1", "path/to/app-2" ],
   "configuration": { 
      "auxiliaryApps": [ "string" ],
      "billingMethod": "METERED|UNMETERED",
      "customerArtifactPaths": {
         "androidPaths": [ "string", "string" ],
         "deviceHostPaths": [ "string", "string" ],
         "iosPaths": [ "string", "string" ]
      },
      "extraDataPackageArn": "string",
      "locale": "string",
      "location": { 
         "latitude": 11.11,
         "longitude": 22.22
      },
      "networkProfileArn": "string",
      "radios": { 
         "bluetooth": boolean,
         "gps": boolean,
         "nfc": boolean,
         "wifi": boolean
      },
      "vpceConfigurationArns": [ "string", "string" ]
   },
   "executionConfiguration": { 
      "accountsCleanup": boolean,
      "appPackagesCleanup": boolean,
      "jobTimeoutMinutes": 111,
      "skipAppResign": boolean,
      "videoCapture": boolean
   },
   "test": { 
      "filter": "string",
      "parameters": { 
         "string" : "string" 
      },
      "testPackageArn": "string",
      "type": "BUILTIN_FUZZ|BUILTIN_EXPLORER|APPIUM_JAVA_JUNIT|APPIUM_JAVA_TESTNG|APPIUM_PYTHON|APPIUM_WEB_JAVA_JUNIT|APPIUM_WEB_JAVA_TESTNG|APPIUM_WEB_PYTHON|CALABASH|INSTRUMENTATION|UIAUTOMATION|UIAUTOMATOR|XCTEST"
   }
}

License

devicefarm-ci-tool's People

Contributors

Stargazers

Watchers

Forkers

fossabot gritsay dnascimento nghieptruong

devicefarm-ci-tool's Issues

Increase test coverage

Setup CI

We need to setup CI for this repo

Output name of randomly selected device pool

Refactor main.go

The main goal is to make possible end2end tests with mocked AWS

auxiliaryApps parameter giving error

Hi,
Thank you for sharing this repo, it's very useful.

I am getting error for auxiliaryApps parameter. I am putting local path to APKs in auxiliaryApps but is giving following error

validator.go:8: Failed to run tests: ValidationException: 1 validation error detected: Value '[extra/app1.apk, extra/app2.apk]' at 'configuration.auxiliaryApps' failed to satisfy constraint: Member must satisfy constraint: [Member must have length less than or equal to 1011, Member must have length greater than or equal to 32]

Can you please help on the same.

Thanking you in advance,
Jignesh

Can I embed AWS config into app config?

Add flag to pass or fail the job if some of test devices were unavailable

Add mechanism for minimize waiting time

If several tests jobs are running in parallel on the same device pool, then long waiting time for devices may occur. This task should provide a mechanism to minimize this waiting time.

Support latest API from Device Farm

Update config

Support latest changes https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_ScheduleRun.html

Dependabot cannot be upgraded to GitHub-native version due to unsupported `dep` package manager

Dependabot Preview will be shut down on August 3rd, 2021.

It appears that this repository has configured updates for dep.

The GitHub-native version of Dependabot does not support dep, which was deprecated as of 2020.
Guidance from the dep maintainers is to upgrade to go mod, which Dependabot supports.

In order to keep getting updates for your Go dependencies, you'll need to migrate to go mod and add a Dependabot configuration for it to this repository.

Retry on fail?

Should we retry test run on fail?
What if only 1 or several devices are failing in run?

Add support for Android test sharding

Should be done via creation of several runs and using specific test filter on every run.
Should provide aggregated feedback.

Is it possible to check device status before run?

Find a proper way to transform model.RunConfig to devicefarm.ScheduleRunInput

How to send a parameter for a RunListener

To specify one or more RunListeners to observe the test run one could add an extra parameter like this: -e listener com.foo.Listener,com.foo.Listener2.

Full command would look like this:

adb shell am instrument -w -e debug true -e listener com.example.testrail.exampleapp.ExampleRunListener com.example.testrail.exampleapp.test/androidx.test.runner.AndroidJUnitRunner

How could I pass such extra parameters to the devicefarm-ci-tool? Anyone had an experience with this?

Dependabot couldn't find a Gopkg.toml for this project

Dependabot couldn't find a Gopkg.toml for this project.

Dependabot requires a Gopkg.toml to evaluate your project's current Go dependencies. It had expected to find one at the path: /Gopkg.toml.

If this isn't a Go project, or if it is a library, you may wish to disable updates for it from within Dependabot.

Add customizable behaviour for handling flaky tests

Like #15, but for more granular control with different options.
The potential list of options:

If at least on 1 device tests are green, but all other devices are unavailable
If at least on 1 device tests are green, but on some devices they are failing
If at least on 1 device tests are green and don't care about anything else

Minimize number of packages
Get rid of errors pkg
Move launching tool to cmd/