Giter VIP home page Giter VIP logo

aws-device-farm-jenkins-plugin's Introduction

AWS Device Farm Jenkins Plugin

AWS Device Farm integration with Jenkins CI

This plugin provides AWS Device Farm functionality from your own Jenkins CI server:

main-page

configuration

It also provides the device state details specification if the checkboxs are checked. Otherwise, the default settings will be used: device-state-specification

It also can pull down all of the test artifacts (logs, screenshots, etc.) locally:

build-artifacts

Usage

Building the plugin:

  1. Clone the GitHub repository.
  2. Import the Maven project into your favorite IDE (Eclipse, IntelliJ, etc.).
  3. Build the plugin using the Makefile (make clean compile).
  4. The plugin is created at target/aws-device-farm.hpi.

Installing the plugin:

Manual install:

  1. Copy the hpi file to your Jenkins build server and place it in the Jenkins plugin directory (usually /var/lib/jenkins/plugins).
  2. Ensure that the plugin is owned by the jenkins user.
  3. Restart Jenkins.

Web UI install:

  1. Log into your Jenkins web UI.
  2. On the left-hand side of the screen, click “Manage Jenkins”.
  3. Click “Manage Plugins”.
  4. Near the top of the screen, click on the “Advanced” tab.
  5. Under the “Upload Plugin”, click “Choose File” and select the AWS Device Farm Jenkins plugin that you previously downloaded.
  6. Click “Upload”.
  7. Check the “Restart Jenkins when installation is complete and no jobs are running” checkbox.
  8. Wait for Jenkins to restart.

Generating a proper IAM user:

Note : Follow these steps if you are using Secret and Access key to access Device Farm. If you want to access Device Farm using IAM Role, refer to "Generating a proper IAM role" section

  1. Log into your AWS web console UI.
  2. Click “Identity & Access Management”.
  3. On the left-hand side of the screen, click “Users”.
  4. Click “Create New Users”.
  5. Enter a user name of your choice.
  6. Leave the “Generate an access key for each user” checkbox checked.
  7. Click “Create”.
  8. View or optionally download the User security credentials that were created; you will them them later.
  9. Click “Close” to return to the IAM screen.
  10. Click your user name in the list.
  11. Under the Inline Policies header, click the “AWS IAM role ARN for your AWS Device Farm account” link to create a new inline policy.
  12. Select the “Custom Policy” radio button.
  13. Click “Select”.
  14. Give your policy a name under “Policy Name”.
  15. Copy/paste the following policy into “Policy Document”:
{
  "Version": "2012-10-17",
  "Statement": [{
      "Effect": "Allow",
      "Action": [
        "devicefarm:*",
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups",
        "ec2:CreateNetworkInterface"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "iam:CreateServiceLinkedRole",
      "Resource": "arn:aws:iam::*:role/aws-service-role/devicefarm.amazonaws.com/AWSServiceRoleForDeviceFarm",
      "Condition": {
        "StringLike": {
          "iam:AWSServiceName": "devicefarm.amazonaws.com"
        }
      }
    }
  ]
}
  1. Click “Apply Policy”.

Generating a proper IAM role

Note : Only applies when you want to access Device Farm through an IAM Role.

You need to create one Device Farm access role to access the Device Farm Resource and one User or Role depending upon how you are running Jenkins.

  • Creating Device Farm Role.
  1. Log into your AWS web console UI.
  2. Click “Identity & Access Management”.
  3. On the left-hand side of the screen, click “Roles”.
  4. Click “Create Role”.
  5. In the AWS service section select Ec2 and click on Next.
  6. Under the Inline Policies header, search and click the “AWSDeviceFarmFullAccess ” policy.This will give complete device farm access.
  7. Go the the summary page of the role and click on edit next to "Maximum CLI/API session duration" and select 8hrs. This sets the expiration of the session associated with the IAM role to 8 hrs.
  8. Now you want to give the access to the role/user to assume this role Open the new role you have created in console and then click on "Trust Relationships" tab. Click on "Edit Trust Relationship" and enter the following policy :
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "To be replaced by Arn of user/role running on the Jenkins machine",
        "Service": "devicefarm.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
  • Setting up Role/User running Jenkins
  1. We need to also give access
  2. Go to the user or role that you want to add policy to.
  3. In the permission tab click on Create Policy.
  4. Select STS in the service name, search for assume role in actions and select it.
  5. In the Resources section enter the arn for the device farm role.
  6. Click on review policy and then enter name and description and click on Create policy.

First-time configuration instructions:

  1. Log into your Jenkins web UI.
  2. On the left-hand side of the screen, click “Manage Jenkins”
  3. Click “Configure System”.
  4. Scroll down to the “AWS Device Farm” header.
  5. If you are using IAM Role to configure the Jenkins Plugin, add the ARN for the role here.
  6. If not, Copy/paste your AKID and SKID you created previously into their respective boxes.
  7. Click “Save”.

Using the plugin in Jenkins job:

  1. Log into your Jenkins web UI.
  2. Click on the job you wish to edit.
  3. On the left-hand side of the screen, click “Configure”.
  4. Scroll down to the “Post-build Actions” header.
  5. Click “Add post-build action” and select “Run Tests on AWS Device Farm”.
  6. Select the project you would like to use.
  7. Select the device pool you would like to use.
  8. Select if you'd like to have the test artifacts (such as the logs and screenshots) archived locally.
  9. In “Application”, fill in the path to your compiled application for testing native or hybrid app. Check "It is a web application." for testing web app.
  10. Optional: If you are using VPC, then confirm your VPC Settings are correct by a.) ensuring you have the desired VPC ENI configuration [read only] or b.) Enabling VPCE
  11. Choose the test framework, provide the path to your test package location and other relavent details.
  12. Configure device state parameters like radio details, extra data and device locations.
  13. Configure the maximum execution timeout. The default execution timeout is 60 minutes.
  14. Set the execution configuration parameters: video recording and app performance monitoring.
  15. Click “Save”.

Using the plugin in Jenkins Pipeline

  1. Go to Job > Pipeline Syntax > Snippet Generator
  2. Select "devicefarm" sample step or "step: General Build Step" > "Run Tests on AWS Device Farm"
  3. Input the Device Farm Run Configuration
  4. Click "Generate Pipeline Script"

TroubleShooting

"Invalid Credentials" Error while validating Credentials

We validate two things: 1) If the credentials are valid, 2) and if they have access to AWS Device Farm

  1. Log in to your Jenkins host.
  2. If using access and secret key, Run "aws devicefarm list-projects" on the host running Jenkins using the AWS CLI. If this fails, first check you credentials. If they are correct, refer to the "Generating a proper IAM user" section to ensure that you have given Device Farm necessary credentials.
  3. If you are using role ARN Run "aws sts assume-role --role-arn "EnterRoleArnHere --duration-seconds 28800" using the AWS CLI. If this fails, refer to the "Generating a proper IAM role" section to verify if the role has correct assume role permissions. Run "aws devicefarm list-projects" using the AWS CLI. If this fails, your role doesn't have access to device farm. Please re-verify the steps above.

Project section not being populated with the latest data

  1. Verify that the Project you are looking for is visible by logging in to the Device Farm console.
  2. If yes, go to the Jenkins -> Manage Jenkins -> Configure System -> AWS Device Farm section and click on Validate.

Dependencies

  • AWS SDK 1.11.126 or later.

aws-device-farm-jenkins-plugin's People

Contributors

ahawker avatar anurag1408 avatar appwiz avatar bad-hambo avatar benkershner avatar bolzon2000 avatar bradley-curran avatar danewrye avatar dependabot[bot] avatar guneeshp avatar jlleitschuh avatar jmp-aws avatar kupchinskiyvasiliy avatar mraghu-athena avatar nikhil-dabhade avatar oleg-nenashev avatar oonemo avatar patcadelina avatar piyushag-aws avatar snbuback avatar stefanverhoeff avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aws-device-farm-jenkins-plugin's Issues

Jenkin job does not allow to configure webview without any apk

Suppose if I like to test webapp amazon.com or walmart.com in the browser using mobile or device with appium and java, I am not able to configure the job without any apk. Shall I need to provide a fake apk to test webapp?

I just only want to upload my test framework and test script without any apk and run the test only in browser. How should I configure the jenkins job and what packaging I need to do?

Debug messages for failures in uploading build_dependencies.zip

In case of zip packaging issue or any we don't get proper debugging statements on the console, so its good to have debugging enabled for any failure occurred while processing and uploading the zip.

We get error message from UI perspective on device farm I hope we can print the same if any thing fails from jenkins resource upload.

NPE with 1.20

Hi,

I'm still experiencing an NPE with 1.20 (following #78 )

java.lang.NullPointerException
	at org.jenkinsci.plugins.awsdevicefarm.AWSDeviceFarmRecorder.perform(AWSDeviceFarmRecorder.java:592)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50)
	at hudson.security.ACL.impersonate(ACL.java:290)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

Pass the value for 'application' dynamically [Feature Request ?]

💾 Use Case:

I have an APK file that is generated dynamically and stored on S3. Check this document- developerguide/continuous-integration-jenkins-plugin

jenkins_1

I know a part name of the APK file, the remaining name of the file I can get from the environment variable at runtime. I need to pass the dynamic name to this 'application' using '%ENVIRONMENT%', however, there is no such provision in the plug-in.

jenkins_2

✳️ Feature Request:

I guess this would be a feature request then if it's not possible through UI as of now

🔮 If there is some way to handle this scenario do share the same

Store test results locally option NOT working

Plugin works fine unless the 'Store test results locally' option is selected. The job fails with the below error message. Looks like there was an issue submitted around this feature in the recent past.

Is this feature still working?

Jenkins 2.19
aws-device-farm Plugin version: 1.13

Console Output Snippet:

[AWSDeviceFarm] Test run is complete.
[AWSDeviceFarm] Storing AWS Device Farm results in directory C:\Jenkins\jobs\Calc3-Android-UI_Tests\builds\6\archive\AWS Device Farm Results
ERROR: Build step failed with exception
java.lang.NullPointerException
    at hudson.FilePath.isAbsolute(FilePath.java:281)
    at hudson.FilePath.resolvePathIfRelative(FilePath.java:266)
    at hudson.FilePath.<init>(FilePath.java:262)
    at org.jenkinsci.plugins.awsdevicefarm.AWSDeviceFarmRecorder.getSuites(AWSDeviceFarmRecorder.java:427)
    at org.jenkinsci.plugins.awsdevicefarm.AWSDeviceFarmRecorder.perform(AWSDeviceFarmRecorder.java:356)
    at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:720)
    at hudson.model.Build$BuildExecution.post2(Build.java:185)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:665)
    at hudson.model.Run.execute(Run.java:1745)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:404)
Build step 'Run Tests on AWS Device Farm' marked build as failure
Finished: FAILURE

Set IAM Role ARN

I noticed the global.jelly file for the AWSDeviceFarmRecorder is commented out.

<f:entry title="IAM Role ARN" field="roleArn" description="AWS IAM Role ARN."> <f:textbox /> </f:entry>

Is there a particular reason why? I'm in need of this feature.
Alternatively, the EC2 plugin allows you to specify that the plugin should just run with the role given to the box it's running on. This would be another solution.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (display-info) on project aws-device-farm: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed.

I've just downloaded the repo from the master branch and tried to build it using the specified make command, but unfortunately, it failed with the following error log:

["INFO] Ignoring requireUpperBoundDeps in com.google.guava:guava
[INFO] Ignoring requireUpperBoundDeps in com.google.code.findbugs:jsr305
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireMavenVersion failed with message:
3.1.0 required by frontend-maven-plugin at least.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:13.155s
[INFO] Finished at: Fri Mar 05 11:54:12 UTC 2021
[INFO] Final Memory: 30M/1038M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (display-info) on project aws-device-farm: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
make: *** [compile] Error 1

The compilation process was attempted on the actual Jenkins Server, an Amazon Linux 2 EC2 instance running in us-east-1.

java -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-b10)
OpenJDK 64-Bit Server VM (build 25.272-b10, mixed mode)

Email integration

After the results are performed on devices, an email regarding results should be sent to the email address specified.

Running a single UI Automator test

To whom it may concern.
I have encountered a problem using aws-device-farm-jenkins-plaugin.
I try to run a single Android UI Automator test via jenkins using aws-device-farm-jenkins-plugin.
I point out in the field 'Filter' of 'com.uiautomator.test.BaseTestRunner#test' but the entire test class (bn.ereader.BaseTestRunner) runs.
Perhaps, you have a solution.
Thanks.
Best wishes, Nikolai

support for Appium 1.7.1 ?

Hey there awslabs!

I noticed in the AWS console, there's an option to use Appium version 1.7.1 (with Appium Python type.)

But I don't have this option in your Jenkins plugin, v1.15.
Any plans to support this? My tests are erroring on Appium version 1.6.5.

Currently I have to run my tests via command line or from the device farm console since in order to select the 1.7.1 version.

Thanks in advance,

Setting the Device Pool from the variable.

Hi,
I'm using your plugin with Multi-job plugin. I would like to create several different multi-jobs that run tests on different device pools. I wanted to create new environment variable on the start of each multi-job that contains name of the device pool I would like to use, but I can't use this variable because device pool choose field is a combobox. Could you change this field to be able to use variables? I have like 20 jobs for single tests and 3 device pools and I don't want to create another 40 jobs just to change the device pool.

[solved] Project list is not updated

Some time ago we were playing with deviceFarm. and had 1 project on our AWS Account.
this project IS seen inside Jenkins.

Today, we've created another Project on AWS side. with a Different name. but for now we don't see it in our list of available projects.

Is this feature working? do we have to wait for something?

I've pressed "Refresh" button many times, but nothing had changed. I see only 1 project and don't see the second one

Calling API via http proxy server

Jenkins and many plug-ins support invoking API via http proxy server. But this plug-in doesn't support it.

We can specify proxy server information by JVM parameter on launching jenkins.
Here are an example script:

/usr/bin/java -Dfile.encoding=UTF-8 -XX:PermSize=256m -XX:MaxPermSize=512m -Xms256m -Xmx512m 
-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=8888 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=8888 
-Djava.io.tmpdir=/Users/Shared/Jenkins/tmp -jar /Applications/Jenkins/jenkins.war --httpPort=14001

Enable IAM role to be used with role chaining

With Role chaining, max session duration is 1 hour.
Currently Device Farm plugin asks for 8 hours of assume role session, so IAM roles couldn't be used with device farm when role chaining is involved.

Modify Device State

Is it possible or will it ever be possible to modify the device state through the plugin? I'd like to upload dependencies, but I haven't found any way.

results trends and graphs not shown when using plugin with a multibranch pipeline project

I am using this plugin as part of a multibranch pipeline project in jenkins. Here's my Jenkinsfile stage for it:

stage('Start AWS Device Cloud Build') {
            devicefarm(appArtifact: 'apks/*staging-debug.apk', appiumJavaJUnitTest: '', appiumJavaTestNGTest: '', appiumNodeTest: '', appiumPythonTest: 'test_bundle.zip', appiumRubyTest: '', appiumVersionJunit: '1.4.16', appiumVersionPython: '1.7.2', appiumVersionTestng: '1.4.16', calabashFeatures: '', calabashProfile: '', calabashTags: '', deviceLatitude: 47.6204, deviceLocation: false, deviceLongitude: -122.3941, devicePoolName: 'nexus', environmentToRun: 'CustomEnvironment', eventCount: '', eventThrottle: '', extraData: false, extraDataArtifact: '', ifAppPerformanceMonitoring: true, ifBluetooth: true, ifGPS: true, ifNfc: true, ifSkipAppResigning: false, ifVideoRecording: true, ifVpce: false, ifWebApp: false, ifWifi: true, ignoreRunError: false, isRunUnmetered: false, jobTimeoutMinutes: 10, junitArtifact: '', junitFilter: '', password: '', projectName: 'CardioSignal', radioDetails: false, runName: '${BUILD_TAG}', seed: '', storeResults: true, testSpecName: 'Default TestSpec for Android Appium Python', testToRun: 'APPIUM_PYTHON', uiautomationArtifact: '', uiautomatorArtifact: '', uiautomatorFilter: '', username: '', vpceServiceName: '', xctestArtifact: '', xctestFilter: '', xctestUiArtifact: '', xctestUiFilter: '')
        }

My tests run successfully and I can see the results saved as artifacts, but the results trends and graphs, as well as the nice colorful results per device are not shown.

The graphs, trends and results by device show up just fine if I use a "normal" Jenkins project, where I configure the plugin from the Jenkins UI rather than from the Jenkinsfile in the repository.

Cannot perform multiple runs simultaneously from one Jenkins

When I have two Device Farm builds running at the same time on the same Jenkins, the second one to start blocks with the following message:

Run Tests on AWS Device Farm is waiting for a checkpoint on <project name and build number>

The second job then continues after the first one ends. I understand that this is a mechanism Jenkins provides to prevent simultaneous access to certain shared resources, but I do not see why this should apply to simultaneous Device Farm runs. I seem to be able to execute multiple Device Farm runs simultaneously via the AWS web console.

This is significant because it prevents me from running iOS and Android tests simultaneously on the same CI server, or from fragmenting my (large) test suite into several pieces and executing them simultaneously.

Can not schedule "APPIUM_JAVA_JUNIT"

AWS Device Farm Plugin Ver. 1.9
I' facing a issue when I create job to run "APPIUM_JAVA_JUNIT".
When I execute job, I got error below:

[AWSDeviceFarm] Using App 'target/app/build/outputs/apk/app-debug.apk'
[AWSDeviceFarm] Archiving artifact 'app-debug.apk'
[AWSDeviceFarm] Uploading app-debug.apk to S3
[AWSDeviceFarm] Waiting for upload app-debug.apk to be ready (current status: INITIALIZED)
[AWSDeviceFarm] Getting test to schedule.
[AWSDeviceFarm] Archiving artifact 'zip-with-dependencies.zip'
[AWSDeviceFarm] Uploading zip-with-dependencies.zip to S3
[AWSDeviceFarm] Waiting for upload zip-with-dependencies.zip to be ready (current status: PROCESSING)
[AWSDeviceFarm] Scheduling 'APPIUM_JAVA_JUNIT' run 'app-debug.apk (Jenkins)'
ERROR: Build step failed with exception
com.amazonaws.services.devicefarm.model.ArgumentException: Missing or unprocessed resources. (Service: AWSDeviceFarm; Status Code: 400; Error Code: ArgumentException; Request ID: f3f7c36d-5d17-11e5-ab63-e3978af19c81)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
    at com.amazonaws.services.devicefarm.AWSDeviceFarmClient.invoke(AWSDeviceFarmClient.java:1524)
    at com.amazonaws.services.devicefarm.AWSDeviceFarmClient.scheduleRun(AWSDeviceFarmClient.java:1460)
    at org.jenkinsci.plugins.awsdevicefarm.AWSDeviceFarm.scheduleRun(AWSDeviceFarm.java:419)
    at org.jenkinsci.plugins.awsdevicefarm.AWSDeviceFarmRecorder.perform(AWSDeviceFarmRecorder.java:283)
    at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:761)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:721)
    at hudson.model.Build$BuildExecution.post2(Build.java:183)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:670)
    at hudson.model.Run.execute(Run.java:1766)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:374)
Build step 'Run Tests on AWS Device Farm' marked build as failure
Finished: FAILURE

Can you look at this?
Thanks.

Does `Storing AWS Device Farm results in directory` feature still work well?

Hi, this great plugin almost works well, but Storing AWS Device Farm result feature always fails in my environment, as causing Status Code: 500; Error Code: InternalFailure. Does anyone have same issue ?

device-farm plugin version: 1.9
Jenkins version: 1.580.2

Error messages below;

[AWSDeviceFarm] Test run is complete.
[AWSDeviceFarm] Storing AWS Device Farm results in directory /var/jenkins_home/jobs/***mask***/builds/30/archive/AWS Device Farm Results
ERROR: Build step failed with exception
com.amazonaws.AmazonServiceException: null (Service: AWSDeviceFarm; Status Code: 500; Error Code: InternalFailure; Request ID: f1d5a8bb-5ae5-11e5-bfaf-7549af5d7ebf)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
    at com.amazonaws.services.devicefarm.AWSDeviceFarmClient.invoke(AWSDeviceFarmClient.java:1524)
    at com.amazonaws.services.devicefarm.AWSDeviceFarmClient.listSuites(AWSDeviceFarmClient.java:1263)
    at org.jenkinsci.plugins.awsdevicefarm.AWSDeviceFarm.listSuites(AWSDeviceFarm.java:488)
    at org.jenkinsci.plugins.awsdevicefarm.AWSDeviceFarmRecorder.getSuites(AWSDeviceFarmRecorder.java:373)
    at org.jenkinsci.plugins.awsdevicefarm.AWSDeviceFarmRecorder.perform(AWSDeviceFarmRecorder.java:311)
    at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:776)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:723)
    at hudson.model.Build$BuildExecution.post2(Build.java:183)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:670)
    at hudson.model.Run.execute(Run.java:1763)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:381)
Build step 'Run Tests on AWS Device Farm' marked build as failure

Problem of archiving download hangs Jenkins job

I use Device Farm Jenkins Plugin in my Jenkins project as a post build action. However, from time to time, my Jenkins project is stuck in storing AWS Device Farm results in directory, with the error below.

[AWSDeviceFarm] Storing AWS Device Farm results in directory /Volumes/Data/Home/jobs/Android_Monitoring_Tests/builds/368/archive/AWS Device Farm Results
[AWSDeviceFarm] Downloading AWS Device Farm results archive...

It looks to me an API issue which fails to return CUSTOMER_ARTIFACT_URL.

It'll be good if some resiliency (retry) and exit condition is added to the artifact download part of this plugin so that Jenkins projects don't get stuck.

df_plugin_ticket_01

Configuring Credentials Programmatically

I'm trying to configure the Access Key / Secret Key for this plugin. Is there a way to do this programmatically without having to open the browser? A lot of other Jenkins plugins can be configured through groovy scripts, so does something similar exist for this plugin?

Appium 1.6.5 option

Hey,

I noticed that in 1.14 version I don't have the option for Appium 1.6.5. There is a bug in Appium 1.6.3 that blocks me and I cannot run my tests through jenkins.
There is an issue on WebDriverAgent:
facebookarchive/WebDriverAgent#575

This issue is fixed in Appium 1.6.5.
Currently I run my tests from the device farm console since there I can select the 1.6.5 version, but this is just a workaround.

Thanks

Import app(.apk/.ipa) from S3 bucket to Application path

Hi,
Is there any way to get .apk/.ipa from S3 bucket to Jenkins-Device farm-Application path

Scenario :
After Dev team triggers the Jenkins-job, It will store the app in S3 bucket with AppName-CXXX.XX.XX.apk
In post build actions it will trigger the Automation job (Remote trigger Jenkins job) and I need to use that .apk file path in Application.

Thanks,
Madhu

Error while uploading the APK file

I am getting this error while uploading the APK file.

[AWSDeviceFarm] Using Project 'New Test Cdvr' [AWSDeviceFarm] Using DevicePool 'Test' [AWSDeviceFarm] Using App '/var/lib/jenkins/workspace/ADF_Test/Scripts/*.apk ' [AWSDeviceFarm] Unknown app artifact to upload: $Scripts/*.apk

I am new to ADF. Please let me know If I am doing anything wrong here.

Customer Artifacts/Customer Artifacts Log are missing while running through Jenkins plugin

Hi,

Customer Artifacts/Customer Artifacts Log are missing while running through Jenkins plugin. Could you please tell me how to enable through Jenkins plugin,

While running manually through AWS farm console, I am able to see the links:
image

While running through AWS Device Farm Jenkins Plugin, I am unable to see the links:
image

Here is the configuration snap of the Jenkins job:
image

NPE 1.20

Issue on 1.20

ERROR: Build step failed with exception
java.lang.NullPointerException
at org.jenkinsci.plugins.awsdevicefarm.AWSDeviceFarmRecorder.perform(AWSDeviceFarmRecorder.java:592)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.post2(MavenModuleSetBuild.java:1073)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
at hudson.model.Run.execute(Run.java:1823)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)

Refer to #78

"Ignore Device Farm errors in build result." option is not working

Hi,
I activated "Ignore Device Farm errors in build result." to ignore potential failed tests in DeviceFarm.
However, the option is not working, at the end of a failed test run, I got in Jenkins :
[AWSDeviceFarm] Test run is complete. Build step 'Run Tests on AWS Device Farm' changed build result to FAILURE

Is there a way to set Locale in the jenkins-plugin

In Aws Device Farm you can set the Device Locale for the test run.
But If you run your test with the Jenkins plugin there's no option to set the Locale.
If you are unlucky you might end up with another locale than what you expect, since it will be saved from the previous run on that device.
Is there a way to set the Device Locale in the plugin?

unable to select from iOS device pools

In the plugin configuration for a build I am able to select among Android device pools but I am not seeing any of the configured iOS device pools show up in the list.

Am I missing something? (Yes, I tried hitting refresh.)

Missing Option for XCTest UI

I noticed that Web Interface of AWS Device Farm has Test Filter for *-Runner.ipa
(screenshot attached)

Can you please make a same Filter for aws-device-farm-jenkins-plugin

That would be every much appreciated !!!

Thx!
test_filter

Upload for web App test always fail when picking the unmetered device

AWSDeviceFarmRecorder.java

Two places calling

if (isRunUnmetered != null && isRunUnmetered) {
String os = adf.getOs(appArtifact);

public FormValidation doCheckIsRunUnmetered(@QueryParameter Boolean isRunUnmetered, @QueryParameter String appArtifact) {
if (isRunUnmetered != null && isRunUnmetered) {
AWSDeviceFarm adf = getAWSDeviceFarm();
String os = null;
try {
os = adf.getOs(appArtifact);

Without checking whether it's a Web App test which does not have an appArtiffact.
It will always throw following error when testing Web App on an unmetered device.
" throw new AWSDeviceFarmException(String.format("Unknown app artifact to upload: %s", appArtifact));"

Device Pool not been listed for one project account

We have an aws device farm account with only one project. Trying to select the Device Pool items in a simple build job is not possible, since when you click in the project list unique item action "org.jenkinsci.plugins.awsdevicefarm.AWSDeviceFarmRecorder/fillProjectNameItems" is not called.

Plugin does not grab device pools from AWS

hi, I have a Device Farm project hooked up to the jenkins plugin with IAM user credentials. When we try to add the AWS Device Farm project to jenkins, the Project comes up however we do not see any of the Device Pools and are not able to type in the field. We do not even see the default 'Top Devices' pool. Any tips on how to get a device pool to show in the post build actions section when configuring AWS Device Farm to Jenkins?

Thanks!

Edit -

Trying to save without completing all the fields fixed the issue

Missing option for selecting Appium Version

Hey,

So in the jenkins plugin I don't have any way to select the appium version. I observed that will always run with Appium v .1.4.16 . Any chance to get this done and start Appium with version 1.6.3 ? In the aws console you have this option to select the Appium version

Thanks

1.14 version issues

Thanks for releasing a new version of the plugin.
I noticed some issues so far. Idk you guys aware about them, if so just close this ticket.

  1. Unable to increase Execution Timeout more than 60 mins.
    We bought a more time for our runs and currently we have 150 mins.
    executiontimeout

  2. Unable to Disable checkboxes on Execution Configuration.
    Every time i disabling them, saved, and re-enter the Job configuration they got enabled back by
    themselfs.
    execconfig

  3. Test Filter Field for XCTest UI is not saving a values that was given.
    But this problem is mentioned in Issue #53

Let me know if you need some additional info.

Regards,
Alex.

Enable @Suppress annotation?

Is there a plugin setting to make Device Farm respect the standard Android testrunner @Suppress annotation to exclude tests from being executed? The JUnit 4 annotations work, but we have a large test suite that is mostly JUnit 3.
Does the Jenkins plugin provide a config for this or is @Suppress support missing in Device Farm itself?
If support for that annotation is missing, will it be added or is there a know work-around?

NPE with 1.19

Hi,

I have the following error with the 1.19 version (I rollbacked to 1.17 and it's fine)
I'm using an unmetered device.

[Pipeline] devicefarm
[AWSDeviceFarm] Using Project 'xxx'
[AWSDeviceFarm] Using DevicePool 'devicepool-android-prod'
[AWSDeviceFarm] Using App 'app-test.apk'
[AWSDeviceFarm] Archiving artifact 'app-test.apk'
[AWSDeviceFarm] Uploading app-test.apk to S3
[AWSDeviceFarm] Waiting for upload app-test.apk to be ready (current status: INITIALIZED)
[AWSDeviceFarm] Upload app-test.apk succeeded
[AWSDeviceFarm] Getting test to schedule.
[AWSDeviceFarm] Archiving artifact 'app-test-autotest-build-27-v1.6.apk'
[AWSDeviceFarm] Uploading app-test-autotest-build-27-v1.6.apk to S3
[AWSDeviceFarm] Waiting for upload app-test-autotest-build-27-v1.6.apk to be ready (current status: PROCESSING)
[AWSDeviceFarm] Upload app-test-autotest-build-27-v1.6.apk succeeded
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline

GitHub has been notified of this commit’s build result

java.lang.NullPointerException
	at org.jenkinsci.plugins.awsdevicefarm.AWSDeviceFarmRecorder.getScheduleRunTest(AWSDeviceFarmRecorder.java:1011)
	at org.jenkinsci.plugins.awsdevicefarm.AWSDeviceFarmRecorder.perform(AWSDeviceFarmRecorder.java:557)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50)
	at hudson.security.ACL.impersonate(ACL.java:290)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

Leaving Fuzz parameters empty causes build errors

According to the description in the plugin, the Fuzz test parameters "Event Count", "Event Throttle" and "Seed" are optional. If I leave them empty, I get these python-errors in AWS:

stream() takes exactly 3 arguments (2 given)
'NoneType' object has no attribute 'filename'

As a work-around I used the same parameters in jenkins that are suggested when creating a Fuzz test on AWS directly:
Event Count: 6000
Event Throttle: 50
(The Seed can actually be left empty)

Not sure if this is a bug in the jenkins plugin or in AWS device farm itself.

Trying to upload an IPA file gives an error

Even though the help command reads "File pattern (local to workspace) where the newly built application (.apk, .ipa) exists.", specifying a path to an IPA-file results in error:

[AWSDeviceFarm] Using App 'build/Release-iphoneos/*.ipa'
[AWSDeviceFarm] Unknown app artifact to upload: build/Release-iphoneos/*.ipa

What is the test package location ?

Hi,
I'm using the aws-device-farm plugin in Jenkins.
When I choose Run Tests on AWS device Farm as post build action, there is a Choose test to run section.

I chose Appium Java JUnit to match my tests type.
Inside that, there is a field "Tests" which described as :

[Required] Pattern to find Java JUnit tests.

I'm using Gradle. How do I provide the zip file for the "Tests" field ?

Thanks !

Custom environment not listed

Hi there,

I use this plug in for android instrumentation test. But when I choose Choose your execution environment and look for my custom test spec in Select Test Specification file drop down menu, it doesn't show up there. Only Test Specs with category CURATED show up. I guess you only list built in test specs from Amazon?

I tried to upload the .yaml file using console and aws cli (https://docs.aws.amazon.com/devicefarm/latest/developerguide/how-to-create-test-run.html#how-to-create-test-run-cli-step5)

Add as a Build Step

We would like to integrate the AWS test into a Fastlane build process.
There is a way to separate the fastlane task to two different steps (build + upload store)
But in this situation we should execute the AWS test between this two 'build' steps.

Is any way to Add this plugin as a Build Step instead of Post Build Action?

Too many open files problem

I am using aws-device-farm plugin in jenkins for UI test.

Whenever the job is executed, open files are growing more and more.

To confirm this issue, I just run "lsof -p [jenkins pid]" on terminal.

The open files are almost report relative results.

I enlarged the limit to 65535 and when it comes to the limit, I should restart jenkins.

Could you tell me how to avoid it or confirm this issue?

NonProxyHosts are ignored

Hi there,

I am facing issue with plugin configuration. I am setting up environment variables(NO_PROXY, HTTP_PROXY, HTTPS_PROXY) in jenkins settings.
I also tried to add JAVA_ARGS to jenkins configuration file:
JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dhttp.proxyHost=xyz -Dhttp.proxyPort=yyyy -Dhttps.proxyHost=xyz -Dhttps.proxyPort=yyyy -Dhttp.nonProxyHosts='localhost|127.0.0.1|169.254.169.254|*.amazonaws.com|*.s3-us-west-2.amazonaws.com'"

I am pretty sure that NO_PROXY variable is correct as aws cli is working fine even from jenkins.

Here is log with exection:
amazon_df.log

Can someone advice what exactly is wrong?
Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.