Giter VIP home page Giter VIP logo

atf-headless-runner's Introduction

Headless Browser for Automated Test Framework

The Automated Testing Framework (ATF) enables customers to test their applications and instances, giving them the confidence when developing inside of the ServiceNow Platform that their changes both have the desired behavior, and don’t break other existing features. In the Orlando release of ServiceNow customers are able to automate the testing and deployment of their applications via the Continuous Integration and Continuous Delivery (CICD) API. This ability to automate common developer tasks is crucial to the advancement of ServiceNow as a full fledged PaaS offering. With that goal in mind, the following proposals put forth a plan for customers to automate the creation of browsers to process ATF User Interface (UI) tests. Right now, when a customer wants to test UI functionality, they must manually open a browser to the “Client Test Runner” which will process the UI tests in their local browser. However, when running UI tests via CICD, if a user does not have a Client Test Runner manually opened for testing, the tests will error out. This leads to users not being able to effectively automate their development processes, without putting forth unreasonable resources such as a dedicated computer + screen to process their UI tests.

This repo contains the code and dockerfile that allows any agent to create a headless web browser which logs into a specified instance, navigates to the client test runner page, and tears down the whole affair after.

Make sure you have Docker installed. Installation instructions for Linux/MacOS can be found here, and instructions for Windows Server can be found here

"Production" Usage

  1. Pull the image with the correct tag: docker pull ghcr.io/servicenow/atf-headless-runner:<lin | win>-<version>
  2. Follow instructions in docs/docker_host_setup.md to configure your server to be able to connect to your ServiceNow instance

Development

Running locally

  1. Install Docker
  2. Clone this repo https://github.com/ServiceNow/atf-headless-runner.git
  3. cd into the repo
  4. Build Image from dockerfile: $ docker build -t atf_headless_browser .
  5. $ docker swarm init this starts docker swarm which gives us access to the docker secrets module
  6. $ echo "ServiceNow password" | docker secret create sn_password - adding the instance password as a docker secret
  7. $ python3 start.py http://<ServiceNow Instance URL> <ServiceNow Username> (headlesschrome|headlessfirefox) starting a container from our image and having it connect to an already existing instance

Verify Success in instance (Rome and later)

  1. Start the docker container using the steps above
  2. Wait for the "Agent is online" message in the logs
  3. Go to the sys_atf_agent table
  4. Should see a record for the agent with correct OS type and correct browser type with status "online"

Instance integration

  1. Run docker run -d -v /var/run/docker.sock:/var/run/docker.sock --name socat -p 127.0.0.1:2375:2375 bobrik/socat TCP-LISTEN:2375,fork UNIX-CONNECT:/var/run/docker.sock to expose the docker API locally ono port 2375
  2. Verify the docker API is exposed by running curl http://localhost:2375/images/json and if you get a response then its successful
  3. In instance Create sys_connection record:
  • type: HTTP
  • Name: Anything (Docker ATF)
  • Connection Alias: the docker spoke
  • Connection URL: http://localhost:2375
  1. In ATF Properties page, set the following properties:
  • enabled = true
  • user account login = admin
  • secret id = run docker secret list and copy the ID of the sn_password secret
  • docker image = atf_headless_browser:latest
  1. Make sure no other Scheduled Client Test Runners are connected to the instance
  2. Create a new ATF Schedule Record,
  • Runs On: "On Demand"
  • Test Suite: "Child A"
  • Browser Name: Any (will default the headless to run in Chrome, can select Chrome or firefox)
  • OS Name: Any
  1. Click "Run Now"
  2. See the tests run in a headless environment, an ad hoc docker container will be created for each test run

CI Setup

The Continuous integration for this repo is setup in github actions defined in the .github/workflows directory. Upon every commit to a PR to master or commit on master the action will build the docker image an start a container that connects to https://atfheadlessrunner.service-now.com. Then it will use the Table API to verify that a sys_atf_agent record will all the correct values is present.

Releasing

  1. On the repo home page click "Releases"

Screen Shot 2021-10-05 at 3 54 59 PM

  1. Click "Draft a New Release"
  2. In the release title put the new version number
  3. Click publish release

Upon pubishing the release a github action will automatically be created which will build BOTH the windows and linux docker images from the repo, and will upload both of them to Github Containeer Registry. The tags of the images will be: lin-version (for linux), and win-version (for windows).

Notices

Support Model

ServiceNow built this integration with the intent to help customers get started faster in adopting Automated Testing and CI/CD APIs for DevOps workflows, but will not be providing formal support. This integration is therefore considered "use at your own risk", and will rely on the open-source community to help drive fixes and feature enhancements via Issues. Occasionally, ServiceNow may choose to contribute to the open-source project to help address the highest priority Issues, and will do our best to keep the integrations updated with the latest API changes shipped with family releases. This is a good opportunity for our customers and community developers to step up and help drive iteration and improvement on these open-source integrations for everyone's benefit.

Governance Model

Initially, ServiceNow product management and engineering representatives will own governance of these integrations to ensure consistency with roadmap direction. In the longer term, we hope that contributors from customers and our community developers will help to guide prioritization and maintenance of these integrations. At that point, this governance model can be updated to reflect a broader pool of contributors and maintainers.

Vulnerability Reporting

Please notify [email protected] regarding any vulnerability reports in addition to following current reporting procedure.

atf-headless-runner's People

Contributors

bmeeder22 avatar sn-gmcnamara avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

atf-headless-runner's Issues

Issue Running ATF Headless Browser Image on Windows

I'm encountering an error when trying to run the ATF Headless Browser image on Windows.
The error message is as follows:
2023-12-07 12:13:07 ==============================================================================
2023-12-07 12:13:07 Robot :: Script to startup a headless client test runner
2023-12-07 12:13:07 ==============================================================================
2023-12-07 12:13:07 Open browser | FAIL |
2023-12-07 12:13:07 The agent ID should be auto-generated from the instance
2023-12-07 12:13:07 ------------------------------------------------------------------------------
2023-12-07 12:13:07 Robot :: Script to startup a headless client test runner | FAIL |
2023-12-07 12:13:07 1 test, 0 passed, 1 failed
2023-12-07 12:13:07 ==============================================================================
2023-12-07 12:13:07 Output: C:\output.xml
2023-12-07 12:13:07 Log: C:\log.html
2023-12-07 12:13:07 Report: C:\report.html

I pulled the image using docker pull ghcr.io/servicenow/atf-headless-runner:win-1.0.0.
Kindly advise on any specific configurations or steps required to run the ATF Headless Browser image successfully on a Windows environment.
Thank you for your prompt attention to this matter. I look forward to your guidance and assistance in resolving this matter.

Regards,
Chandrakant

lin-1.0.1-alpha version issues

Wanted to try the alpha versions as i have some ATF that i cannot get to work with the Headless tester, but ran into more problems with it:

%Logging in user: atf_headless_tester
�>[ WARN ] Keyword 'BuiltIn.Run Keyword Unless' is deprecated.
�>[ WARN ] Keyword 'BuiltIn.Run Keyword Unless' is deprecated.
�d[ WARN ] Keyword 'Capture Page Screenshot' could not be run on failure: Message: invalid session id
� Stacktrace:
��#0 0x55d7dfdbef33
��#1 0x55d7dfb08faf
��#2 0x55d7dfb326ff
��#3 0x55d7dfb33985
��#4 0x55d7dfe034cd
��#5 0x55d7dfe075ec
��#6 0x55d7dfded71e
��#7 0x55d7dfe08238
��#8 0x55d7dfde2870
��#9 0x55d7dfe24608
��#10 0x55d7dfe24788
��#11 0x55d7dfe3ef1d
��#12 0x7f2b28db8ea7

| FAIL |
RWebDriverException: Message: unknown error: session deleted because of page crash
�from tab crashed
0 (Session info: headless chrome=102.0.5005.61)
Stacktrace:
�#0 0x55d7dfdbef33
#1 0x55d7dfb08faf
#2 0x55d7dfaf503d
#3 0x55d7dfaf4a55
#4 0x55d7dfaf3f15
#5 0x55d7dfaf2c0f
#6 0x55d7dfaf319c
#7 0x55d7dfaf30f4
#8 0x55d7dfafa625
#9 0x55d7dfaf464f
#10 0x55d7dfaf4cf4
#11 0x55d7dfaf4a55
#12 0x55d7dfaf3f15
#13 0x55d7dfaf3654
#14 0x55d7dfb0fdf1
#15 0x55d7dfb706e8
�#16 0x55d7dfb5ce63
�#17 0x55d7dfb3282a
�#18 0x55d7dfb33985
�#19 0x55d7dfe034cd
�#20 0x55d7dfe075ec
�#21 0x55d7dfded71e
�#22 0x55d7dfe08238
�#23 0x55d7dfde2870
�#24 0x55d7dfe24608
�#25 0x55d7dfe24788
�#26 0x55d7dfe3ef1d
�#27 0x7f2b28db8ea7
O------------------------------------------------------------------------------
ORobot :: Script to startup a headless client test runner | FAIL |
�1 test, 0 passed, 1 failed

Some Issues about `Running locally`

  • There's an issue that the document told me to use $ docker build -t atf_headless_browser . to build an image.
    However, in start.py, it sets the image name with IMAGE_NAME = 'atf_headless_runner'.
    So when I run the start.py, it can not create a container until I rename the image to make sure they are the same.

  • And the second issue is an Error during login. It can not find the file with sn_passwrod.
    There will be a "FileNotFoundError: [Errno 2] No such file or directory: '/var/servicenow/C:\\\\ProgramData\\\\docker\\\\secrets\\\\sn_password'"

Intermittent issue with TEST_RUNNER_BANNER_ID

More than half of my scheduled test runner executions attempts (sys_atf_schedule) fail with an error regarding the inability to find "test_runner_banner" string on the client test runner page.

When the scheduled execution starts, I confirm it connects successfully to the remote machine as the docker image runs on it, as confirmed with a docker ps showing that it's running the robot.robot script.

talbert@latrian:~$ docker ps
CONTAINER ID   IMAGE                                                    COMMAND                  CREATED         STATUS        PORTS     NAMES
6feb577f06f7   ghcr.io/servicenow/atf-headless-runner:lin-1.0.1-alpha   "/bin/sh -c 'robot r…"   3 seconds ago   Up 1 second             a14f25a1bd906d10798f6c924dde5aaa.1.9zf6pugadfykmd2x2o19rbad2

I can confirm the docker user has the correct roles (atf_test_designer, impersonator) and indeed sometimes the scheduled test runner works with the user as-is.

If I view the page source of atf_test_runner.do I can confirm it contains:

<div class="navbar-header" style="padding: 6px 5px;" id="test_runner_banner">

As you would expect from the validations here.

When it does work, it works immediately. When it fails, it cycles through
"Headless client test runners not started, attempts left: 9."

Configuration was completed as follows on the docs site. Port 2376 is open and as show above the PDI is connecting to the remote machine.

I'm kind of at a loss for what's happening?

Below is an example of the output from sn_atf_docker_service I get when a failure occurs.

(EDIT: can confirm sn_atf.schedule.enabled and sn_atf.runner.enabled are set to true on this instance).

O==============================================================================
ORobot :: Script to startup a headless client test runner                      
O==============================================================================
sOpen browser                                                          AGENT_ID is a94f25a1fd906d106c98272cfaddb5a9
�BROWSER is headlesschrome
2INSTANCE_URL is https://dev115395.service-now.com
�USERNAME is docker
&Secret Path: /run/secrets/sn_password
�TIMEOUT is 1440 minutes
�LOGIN_PAGE is login.do
nRUNNER_URL is atf_test_runner.do?sysparm_nostack=true&sysparm_scheduled_tests_only=true&sysparm_headless=true
NBROWSER_OPTIONS is add_argument('--no-sandbox');add_argument('--disable-gpu')
!LOGIN_BUTTON_ID is sysverb_login
�USER_FIELD_ID is user_name
#PASSWORD_FIELD_ID is user_password
9HEADLESS_VALIDATION_PAGE is atf_headless_validation_page
+VP_VALIDATION_ID is headless_vp_validation
'VP_HAS_ROLE_ID is headless_vp_has_role
%VP_SUCCESS_ID is headless_vp_success
,TEST_RUNNER_BANNER_ID is test_runner_banner
�HEARTBEAT_ENABLED is true
+HEARTBEAT_URI is /api/now/atf_agent/online
8Login URL is https://dev115395.service-now.com/login.do
�Logging in user: docker
�>[ WARN ] Keyword 'BuiltIn.Run Keyword Unless' is deprecated. 
�>[ WARN ] Keyword 'BuiltIn.Run Keyword Unless' is deprecated. 
�Clicked Login Button
OGoing to entry: https://dev115395.service-now.com/atf_headless_validation_page
�Going to runner: https://dev115395.service-now.com/atf_test_runner.do?sysparm_nostack=true&sysparm_scheduled_tests_only=true&sysparm_headless=true&sys_atf_agent=a94f25a1fd906d106c98272cfaddb5a9
	| FAIL |
tThe client test runner page could not load, Property sn_atf.schedule.enabled and sn_atf.runner.enabled must be true
O------------------------------------------------------------------------------
ORobot :: Script to startup a headless client test runner              | FAIL |
�1 test, 0 passed, 1 failed
O==============================================================================
$Output:  /var/servicenow/output.xml
"Log:     /var/servicenow/log.html
%Report:  /var/servicenow/report.html
O==============================================================================
ORobot :: Script to startup a headless client test runner                      
O==============================================================================
sOpen browser                                                          AGENT_ID is a94f25a1fd906d106c98272cfaddb5a9
�BROWSER is headlesschrome
2INSTANCE_URL is https://dev115395.service-now.com
�USERNAME is docker
&Secret Path: /run/secrets/sn_password
�TIMEOUT is 1440 minutes
�LOGIN_PAGE is login.do
nRUNNER_URL is atf_test_runner.do?sysparm_nostack=true&sysparm_scheduled_tests_only=true&sysparm_headless=true
NBROWSER_OPTIONS is add_argument('--no-sandbox');add_argument('--disable-gpu')
!LOGIN_BUTTON_ID is sysverb_login
�USER_FIELD_ID is user_name
#PASSWORD_FIELD_ID is user_password
9HEADLESS_VALIDATION_PAGE is atf_headless_validation_page
+VP_VALIDATION_ID is headless_vp_validation
'VP_HAS_ROLE_ID is headless_vp_has_role
%VP_SUCCESS_ID is headless_vp_success
,TEST_RUNNER_BANNER_ID is test_runner_banner
�HEARTBEAT_ENABLED is true
+HEARTBEAT_URI is /api/now/atf_agent/online
8Login URL is https://dev115395.service-now.com/login.do
�Logging in user: docker
�>[ WARN ] Keyword 'BuiltIn.Run Keyword Unless' is deprecated. 
�>[ WARN ] Keyword 'BuiltIn.Run Keyword Unless' is deprecated. 
�Clicked Login Button
OGoing to entry: https://dev115395.service-now.com/atf_headless_validation_page
�Going to runner: https://dev115395.service-now.com/atf_test_runner.do?sysparm_nostack=true&sysparm_scheduled_tests_only=true&sysparm_headless=true&sys_atf_agent=a94f25a1fd906d106c98272cfaddb5a9
�d[ WARN ] Keyword 'Capture Page Screenshot' could not be run on failure: Message: invalid session id
�Stacktrace:
��#0 0x55916cadef33 <unknown>
��#1 0x55916c828faf <unknown>
��#2 0x55916c8526ff <unknown>
��#3 0x55916c853985 <unknown>
��#4 0x55916cb234cd <unknown>
��#5 0x55916cb275ec <unknown>
��#6 0x55916cb0d71e <unknown>
��#7 0x55916cb28238 <unknown>
��#8 0x55916cb02870 <unknown>
��#9 0x55916cb44608 <unknown>
��#10 0x55916cb44788 <unknown>
��#11 0x55916cb5ef1d <unknown>
��#12 0x7f769d1f4ea7 <unknown>
�
	| FAIL |
RWebDriverException: Message: unknown error: session deleted because of page crash
4from unknown error: cannot determine loading status
�from tab crashed
0  (Session info: headless chrome=102.0.5005.61)
Stacktrace:
�#0 0x55916cadef33 <unknown>
�#1 0x55916c828faf <unknown>
�#2 0x55916c81503d <unknown>
�#3 0x55916c814a55 <unknown>
�#4 0x55916c813f15 <unknown>
�#5 0x55916c812c0f <unknown>
�#6 0x55916c81319c <unknown>
�#7 0x55916c8215ff <unknown>
�#8 0x55916c822162 <unknown>
�#9 0x55916c83024d <unknown>
�#10 0x55916c83366a <unknown>
�#11 0x55916c8135c6 <unknown>
�#12 0x55916c82fdf1 <unknown>
�#13 0x55916c8906e8 <unknown>
�#14 0x55916c87ce63 <unknown>
�#15 0x55916c85282a <unknown>
�#16 0x55916c853985 <unknown>
�#17 0x55916cb234cd <unknown>
�#18 0x55916cb275ec <unknown>
�#19 0x55916cb0d71e <unknown>
�#20 0x55916cb28238 <unknown>
�#21 0x55916cb02870 <unknown>
�#22 0x55916cb44608 <unknown>
�#23 0x55916cb44788 <unknown>
�#24 0x55916cb5ef1d <unknown>
�#25 0x7f769d1f4ea7 <unknown>
O------------------------------------------------------------------------------
ORobot :: Script to startup a headless client test runner              | FAIL |
�1 test, 0 passed, 1 failed
O==============================================================================
$Output:  /var/servicenow/output.xml
"Log:     /var/servicenow/log.html
%Report:  /var/servicenow/report.html

Output from from the rarer case of a successful execution:

O==============================================================================
ORobot :: Script to startup a headless client test runner                      
O==============================================================================
sOpen browser                                                          AGENT_ID is 5cb9216d42506d10db9f2db79903c20d
�BROWSER is headlesschrome
2INSTANCE_URL is https://dev115395.service-now.com
�USERNAME is docker
&Secret Path: /run/secrets/sn_password
�TIMEOUT is 1440 minutes
�LOGIN_PAGE is login.do
nRUNNER_URL is atf_test_runner.do?sysparm_nostack=true&sysparm_scheduled_tests_only=true&sysparm_headless=true
NBROWSER_OPTIONS is add_argument('--no-sandbox');add_argument('--disable-gpu')
!LOGIN_BUTTON_ID is sysverb_login
�USER_FIELD_ID is user_name
#PASSWORD_FIELD_ID is user_password
9HEADLESS_VALIDATION_PAGE is atf_headless_validation_page
+VP_VALIDATION_ID is headless_vp_validation
'VP_HAS_ROLE_ID is headless_vp_has_role
%VP_SUCCESS_ID is headless_vp_success
,TEST_RUNNER_BANNER_ID is test_runner_banner
�HEARTBEAT_ENABLED is true
+HEARTBEAT_URI is /api/now/atf_agent/online
8Login URL is https://dev115395.service-now.com/login.do
�Logging in user: docker
�>[ WARN ] Keyword 'BuiltIn.Run Keyword Unless' is deprecated. 
�>[ WARN ] Keyword 'BuiltIn.Run Keyword Unless' is deprecated. 
�Clicked Login Button
OGoing to entry: https://dev115395.service-now.com/atf_headless_validation_page
�Going to runner: https://dev115395.service-now.com/atf_test_runner.do?sysparm_nostack=true&sysparm_scheduled_tests_only=true&sysparm_headless=true&sys_atf_agent=5cb9216d42506d10db9f2db79903c20d
!Waiting for agent to come online
I2023-01-09 21:53:53 | Heartbeat Response: {'result': {'online': 'true'}}
�Agent is online
I2023-01-09 21:53:54 | Heartbeat Response: {'result': {'online': 'true'}}

EDITS: better success and failure examples

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.