Giter VIP home page Giter VIP logo

aio-e2e-tests's Introduction

Adobe I/O E2E Tests

E2e tests for Adobe I/O SDKs and tools.

Requirements

Each tested repository has its own requirements, mostly environment variables to be set.

Run

npm run all

Tests

Add a new e2e test

  • expose a npm run e2e command in the repo under test.
  • set the repo, branch, requiredEnv and doNotLog fields in repositories.json.
  • add to the above section a reference to a markdown describing what the test does.

Contributing

Contributions are welcomed! Read the Contributing Guide for more information.

Licensing

This project is licensed under the Apache V2 License. See LICENSE for more information.

aio-e2e-tests's People

Contributors

adobe-bot avatar amulyakashyap09 avatar dependabot[bot] avatar florind12 avatar himavanth avatar meryllblanchet avatar michaelgoberling avatar moritzraho avatar purplecabbage avatar rajarju avatar sandeep-paliwal avatar sangeetha5491 avatar shazron avatar snyk-bot avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aio-e2e-tests's Issues

daily smoke tests are failing (asset-compute)

Failing run: https://github.com/adobe/aio-e2e-tests/commit/8eb358608780d9c381aea3c5a2e9100d9f92c2ec/checks

Running the asset-compute-integration-tests repo tests locally, they pass (docker required, no aio-cli must be installed).

It is failing here as part of the step asset compute smoke integration test:

grep "Running tests in test/asset-compute/worker" consoleoutput.txt
grep "All tests were successful." consoleoutput.txt

One difference is, the repo tests use npm ci instead of npm i to initialize the repo.

Change Analytics e2e tests to use JWT token

Expected Behaviour

Right now analytics e2e tests are using oauth based token and this requires manual steps to generate tokens.
We can change Analytics e2e tests to use JWT based token which is already automated.

Actual Behaviour

Reproduce Scenario (including but not limited to)

Steps to Reproduce

Platform and Version

Sample Code that illustrates the problem

Logs taken while reproducing problem

break out asset-compute daily smoke tests into their own workflow

See

- name: Setup Asset Compute
uses: actions/checkout@master
with:
repository: adobe/asset-compute-integration-tests
- id: create-asset-compute
name: asset compute smoke integration test
run: |
npm i
./node_modules/mocha/bin/mocha test/index.test.js > consoleoutput.txt
grep "App initialization finished!" consoleoutput.txt
grep "Successful deployment" consoleoutput.txt
grep "Running tests in test/asset-compute/worker" consoleoutput.txt
grep "All tests were successful." consoleoutput.txt

Add test for asset-compute workers

Expected Behaviour

Add a simple asset compute custom worker test to the smoke tests suite (see #46). The goal is to add a test for a basic asset-compute custom worker: https://github.com/adobe/asset-compute-example-workers/tree/master/projects/worker-basic.

This will help detect breakages down the pipeline in user code, as Asset Compute uses aio for custom workers.

What need to be tested:

  • a basic worker can be created using aio tooling;
  • a basic worker can be built and deployed using aio tooling;
  • a basic worker can be tested using aio tooling;

Actual Behaviour

N/A, new feature.

Reproduce Scenario (including but not limited to)

N/A, new feature.

Steps to Reproduce

N/A, new feature.

Platform and Version

N/A, new feature.

Sample Code that illustrates the problem

N/A, new feature.

Logs taken while reproducing problem

N/A, new feature.

`npm pkg set` not available on node-14

Run: https://github.com/adobe/aio-e2e-tests/actions/runs/4333561587/jobs/7566785048

Run npm i @adobe/aio-cli-plugin-app-templates
npm WARN deprecated @oclif/[email protected]: Deprecated in favor of @oclif/core
npm WARN deprecated [email protected]: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated @npmcli/[email protected]: This functionality has been moved to @npmcli/fs
npm WARN deprecated [email protected]: This functionality has been moved to @npmcli/fs
npm WARN deprecated @npmcli/[email protected]: This functionality has been moved to @npmcli/fs

> [email protected] postinstall /home/runner/work/aio-e2e-tests/aio-e2e-tests/node_modules/core-js-pure
> node -e "try{require('./postinstall')}catch(e){}"

npm WARN [email protected] requires a peer of inquirer@^8.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of [email protected] - 8 but none is installed. You must install peer dependencies yourself.
npm WARN @adobe/[email protected] requires a peer of @adobe/aio-lib-ims@^6.0.0 but none is installed. You must install peer dependencies yourself.

+ @adobe/[email protected]
added 604 packages from 333 contributors and audited 604 packages in 32.766s

63 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities


Usage: npm <command>

where <command> is one of:
    access, adduser, audit, bin, bugs, c, cache, ci, cit,
    clean-install, clean-install-test, completion, config,
    create, ddp, dedupe, deprecate, dist-tag, docs, doctor,
    edit, explore, fund, get, help, help-search, hook, i, init,
    install, install-ci-test, install-test, it, link, list, ln,
    login, logout, ls, org, outdated, owner, pack, ping, prefix,
    profile, prune, publish, rb, rebuild, repo, restart, root,
    run, run-script, s, se, search, set, shrinkwrap, star,
    stars, start, stop, t, team, test, token, tst, un,
    uninstall, unpublish, unstar, up, update, v, version, view,
    whoami

npm <command> -h  quick help on <command>
npm -l            display full usage info
npm help <term>   search for help on <term>
npm help npm      involved overview

Specify configs in the ini-formatted file:
    /home/runner/.npmrc
or on the command line via: npm <command> --key value
Config info can be viewed via: npm help config

[email protected] /opt/hostedtoolcache/node/14.[21](https://github.com/adobe/aio-e2e-tests/commit/81aa5d70021648ba5673ac77801c45af2d534726/checks#step:6:22).3/x[64](https://github.com/adobe/aio-e2e-tests/commit/81aa5d70021648ba5673ac77801c45af2d534726/checks#step:6:65)/lib/node_modules/npm

e2e test for Analytics - getUsageLogs is failing

This is a problem with our e2e instance, it doesn't occur with other instances. The e2e test has been commented out with a reason in the comments for now.

Log:

FAIL e2e/e2e.js (17.198s)
  ✓ sdk init test (41ms)
  ✓ test getCollections and getCollectionById (2152ms)
  ✕ test getUsageLogs (877ms)
  ✓ test getMetrics and getMetricById (1233ms)
  ✓ test getCalculatedMetrics and getCalculatedMetricById (973ms)
  ✓ test getDateRanges and getDateRangeById (788ms)
  ✓ test getDimensions and getDimensionById (8168ms)
  ✓ test getSegments and validateSegment (461ms)
  ✓ test getUsers and getCurrentUser (1414ms)

  ● test getUsageLogs

    AnalyticsSDKError: [AnalyticsSDK:ERROR_GET_USAGE_LOGS] Error: Internal Server Error

      466 |         })
      467 |         .catch(err => {
    > 468 |           reject(new codes.ERROR_GET_USAGE_LOGS({ sdkDetails, messageValues: err }))
          |                  ^
      469 |         })
      470 |     })
      471 |   }

      at http (node_modules/swagger-client/lib/http.js:106:19)
      at processTicksAndRejections (internal/process/task_queues.js:97:5) {
        status: 500,
        statusCode: 500,
        response: [Object]
      }
      at new <anonymous> (node_modules/@adobe/aio-lib-core-errors/src/AioCoreSDKErrorWrapper.js:33:9)
      at src/index.js:468:18

Test Suites: 1 failed, 1 total
Tests:       1 failed, 8 passed, 9 total
Snapshots:   0 total
Time:        17.734s

Get Audience Manager env value

AUDIENCE_MANAGER_ID is missing for the tests (this is the "profile viewer id"). Right now the test is disabled in repositories.json

Convert travis.yml to Github Actions

Use Github Secrets for the encryption key used, the conversion should be straightforward. The only thing that needs research is the usage of openssl to decrypt the .env, find out if we have access to openssl in GHA, or an equivalent.

e2e tests for Events is failing

This worked previously.
Run log: https://github.com/adobe/aio-e2e-tests/pull/32/checks?check_run_id=2202968463

Might be related to integration/workspace changes.

2021-03-26T15:19:53.7074715Z     EventsSDKError: [EventsSDK:ERROR_DELETE_REGISTRATION] Error: 404 - Not Found (https://api.adobe.io/events/organizations/***/integrations/***/registrations/undefined)
2021-03-26T15:19:53.7076057Z 
2021-03-26T15:19:53.7076844Z       539 |         .catch(err => {
2021-03-26T15:19:53.7077816Z       540 |           this.__maskAuthData(sdkDetails)
2021-03-26T15:19:53.7079065Z     > 541 |           reject(new ErrorCode({ sdkDetails, messageValues: err }))
2021-03-26T15:19:53.7080119Z           |                  ^
2021-03-26T15:19:53.7080851Z       542 |         })
2021-03-26T15:19:53.7081586Z       543 |     })
2021-03-26T15:19:53.7082276Z       544 |   }
2021-03-26T15:19:53.7082700Z 
2021-03-26T15:19:53.7084324Z       at new <anonymous> (../node_modules/@adobe/aio-lib-core-errors/src/AioCoreSDKErrorWrapper.js:33:9)
2021-03-26T15:19:53.7086549Z       at fetchRetryClient.exponentialBackoff.then.catch.err (../src/index.js:541:18)
2021-03-26T15:19:53.7087846Z 
2021-03-26T15:19:53.7088691Z Test Suites: 1 failed, 1 total
2021-03-26T15:19:53.7091364Z Tests:       0 total
2021-03-26T15:19:53.7092193Z Snapshots:   0 total
2021-03-26T15:19:53.7093001Z Time:        66.359 s

remove pre-release step in app-smoke-test.yml

Error

› Error: loadConfig is not a function at create app with no extensions step https://github.com/adobe/aio-e2e-tests/actions/runs/5514397331/jobs/10053607346

Source:

https://github.com/adobe/aio-e2e-tests/blob/master/.github/workflows/app-smoke-test.yml
This test workflow uses the master branch of the app plugin.

Investigation

This is because of the pre-release step which may pull in breaking updates:

- name: Update package.json with an npm pre-release version
id: pre-release-version
uses: adobe/[email protected]
with:
pre-release-tag: 'pre'
dependencies-to-update: '@adobe/aio-cli-lib-app-config,@adobe/aio-cli-lib-console,@adobe/aio-lib-core-config,@adobe/aio-lib-core-logging,@adobe/aio-lib-core-networking,@adobe/aio-lib-env,@adobe/aio-lib-ims,@adobe/aio-lib-runtime,@adobe/aio-lib-web,@adobe/generator-aio-app'
dependencies-to-update-version-tag: 'next'

Fix:

Remove the Update package.json with an npm pre-release version step.

daily is logging only on success

When the daily ghaction runs it is supposed to post its result to logs/run.md
a failure in the test is preventing this job from running, so run.md will only ever contain success entries.

1 failing e2e test for Analytics

FAIL e2e/e2e.js (11.632s)
  ✓ sdk init test (46ms)
  ✓ test getCollections and getCollectionById (2537ms)
  ✓ test getMetrics and getMetricById (1726ms)
  ✓ test getCalculatedMetrics and getCalculatedMetricById (837ms)
  ✕ test getDateRanges and getDateRangeById (891ms)
  ✓ test getDimensions and getDimensionById (1671ms)
  ✓ test getSegments and validateSegment (862ms)
  ✓ test getUsers and getCurrentUser (2119ms)
  ● test getDateRanges and getDateRangeById
    AnalyticsSDKError: [AnalyticsSDK:ERROR_GET_DATE_RANGES] Error: Not Found
      215 |         })
      216 |         .catch(err => {
    > 217 |           reject(new codes.ERROR_GET_DATE_RANGES({ sdkDetails, messageValues: err }))
          |                  ^
      218 |         })
      219 |     })
      220 |   }
      at http (node_modules/swagger-client/lib/http.js:106:19)
      at processTicksAndRejections (internal/process/task_queues.js:97:5) {
        status: 404,
        statusCode: 404,
        response: [Object]
      }
      at new <anonymous> (node_modules/@adobe/aio-lib-core-errors/src/AioCoreSDKErrorWrapper.js:33:9)
      at src/index.js:217:18
Test Suites: 1 failed, 1 total
Tests:       1 failed, 7 passed, 8 total
Snapshots:   0 total
Time:        12.006s
Ran all test suites.

e2e tests failure: adobe/aio-lib-campaign-standard - Error: Service Unavailable: Back-end server is at capacity

- running tests..
FAIL e2e/e2e.js (5.734 s)
  ✓ HTTPS_PROXY must be set if E2E_USE_PROXY is set (1 ms)
  ✓ sdk init test (1 ms)
  ✓ test bad access token (284 ms)
  ✓ test bad api key ([77](https://github.com/adobe/aio-e2e-tests/actions/runs/7563493369/job/20596007142#step:7:78) ms)
  ✓ test bad tenant id (90 ms)
  ✕ test getAllProfiles API (395 ms)
  ✕ test getAllServices API (332 ms)
  ✕ test getAllOrgUnits API (339 ms)
  ✕ test getMetadataForResource API (313 ms)
  ✕ test getCustomResources API (339 ms)
  ✓ test getAllCustomResources API (308 ms)
  ✓ test createCustomResource API (296 ms)
  ✕ test updateCustomResource API (356 ms)
  ✓ test deleteCustomResource API (347 ms)
  ✕ test createProfile, updateProfile, getProfile API (330 ms)

  ● test getAllProfiles API

    CampaignStandardSDKError: [CampaignStandardSDK:ERROR_GET_ALL_PROFILES] Error: Service Unavailable: Back-end server is at capacity

      170 |         })
      171 |         .catch(err => {
    > 172 |           reject(new codes.ERROR_GET_ALL_PROFILES({ sdkDetails, messageValues: reduceError(err) }))
          |                  ^
      173 |         })
      174 |     })
      175 |   }

      at http (../node_modules/swagger-client/lib/http/index.js:90:19)
      at processTicksAndRejections (../node:internal/process/task_queues:95:5) {
        status: 503,
        statusCode: 503,
        response: [Object]
      }
      at new <anonymous> (../node_modules/@adobe/aio-lib-core-errors/src/AioCoreSDKErrorWrapper.js:33:9)
      at ../src/index.js:172:18

  ● test getAllServices API

    CampaignStandardSDKError: [CampaignStandardSDK:ERROR_GET_ALL_SERVICES] Error: Service Unavailable: Back-end server is at capacity

      254 |         })
      255 |         .catch(err => {
    > 256 |           reject(new codes.ERROR_GET_ALL_SERVICES({ sdkDetails, messageValues: reduceError(err) }))
          |                  ^
      257 |         })
      258 |     })
      259 |   }

      at http (../node_modules/swagger-client/lib/http/index.js:90:19)
      at processTicksAndRejections (../node:internal/process/task_queues:95:5) {
        status: 503,
        statusCode: 503,
        response: [Object]
      }
      at new <anonymous> (../node_modules/@adobe/aio-lib-core-errors/src/AioCoreSDKErrorWrapper.js:33:9)
      at ../src/index.js:256:18

  ● test getAllOrgUnits API

    CampaignStandardSDKError: [CampaignStandardSDK:ERROR_GET_ALL_ORG_UNITS] Error: Service Unavailable: Back-end server is at capacity

      608 |         })
      609 |         .catch(err => {
    > 610 |           reject(new codes.ERROR_GET_ALL_ORG_UNITS({ sdkDetails, messageValues: reduceError(err) }))
          |                  ^
      611 |         })
      612 |     })
      613 |   }

      at http (../node_modules/swagger-client/lib/http/index.js:90:19)
      at processTicksAndRejections (../node:internal/process/task_queues:95:5) {
        status: 503,
        statusCode: 503,
        response: [Object]
      }
      at new <anonymous> (../node_modules/@adobe/aio-lib-core-errors/src/AioCoreSDKErrorWrapper.js:33:9)
      at ../src/index.js:610:18

  ● test getMetadataForResource API

    CampaignStandardSDKError: [CampaignStandardSDK:ERROR_GET_METADATA_FOR_RESOURCE] Error: Service Unavailable: Back-end server is at capacity

      335 |         })
      336 |         .catch(err => {
    > 337 |           reject(new codes.ERROR_GET_METADATA_FOR_RESOURCE({ sdkDetails, messageValues: reduceError(err) }))
          |                  ^
      338 |         })
      339 |     })
      340 |   }

      at http (../node_modules/swagger-client/lib/http/index.js:90:19)
      at processTicksAndRejections (../node:internal/process/task_queues:95:5) {
        status: 503,
        statusCode: 503,
        response: [Object]
      }
      at new <anonymous> (../node_modules/@adobe/aio-lib-core-errors/src/AioCoreSDKErrorWrapper.js:33:9)
      at ../src/index.js:337:18

  ● test getCustomResources API

    CampaignStandardSDKError: [CampaignStandardSDK:ERROR_GET_CUSTOM_RESOURCES] Error: Service Unavailable: Back-end server is at capacity

      369 |         })
      370 |         .catch(err => {
    > 371 |           reject(new codes.ERROR_GET_CUSTOM_RESOURCES({ messageValues: reduceError(err) }))
          |                  ^
      372 |         })
      373 |     })
      374 |   }

      at http (../node_modules/swagger-client/lib/http/index.js:90:19)
      at processTicksAndRejections (../node:internal/process/task_queues:95:5) {
        status: 503,
        statusCode: 503,
        response: [Object]
      }
      at new <anonymous> (../node_modules/@adobe/aio-lib-core-errors/src/AioCoreSDKErrorWrapper.js:33:9)
      at ../src/index.js:371:18

  ● test updateCustomResource API

    expect(received).rejects.toThrow(expected)

    Expected substring: "'@gibberish' is not a valid constant"
    Received message:   "[CampaignStandardSDK:ERROR_UPDATE_CUSTOM_RESOURCE] Error: Service Unavailable: Back-end server is at capacity

      at http (../node_modules/swagger-client/lib/http/index.js:90:19)
      at processTicksAndRejections (../node:internal/process/task_queues:95:5) {
        status: 503,
        statusCode: 503,
        response: [Object]
      }"
            [81](https://github.com/adobe/aio-e2e-tests/actions/runs/7563493369/job/20596007142#step:7:82)6 |         })
            817 |         .catch(err => {
          > 818 |           reject(new codes.ERROR_UPDATE_CUSTOM_RESOURCE({ sdkDetails, messageValues: reduceError(err) }))
                |                  ^
            819 |         })
            [82](https://github.com/adobe/aio-e2e-tests/actions/runs/7563493369/job/20596007142#step:7:83)0 |     })
            821 |   }
      at http (../node_modules/swagger-client/lib/http/index.js:90:19)
      at processTicksAndRejections (../node:internal/process/task_queues:95:5) {
              status: 503,
              statusCode: 503,
              response: [Object]
            }
      at new <anonymous> (../node_modules/@adobe/aio-lib-core-errors/src/AioCoreSDKErrorWrapper.js:33:9)
      at ../src/index.js:818:18
      at Object.toThrow (../node_modules/expect/build/index.js:218:22)
      at Object.toThrow (e2e.js:152:34)

  ● test createProfile, updateProfile, getProfile API

    CampaignStandardSDKError: [CampaignStandardSDK:ERROR_CREATE_PROFILE] Error: Service Unavailable: Back-end server is at capacity

      189 |         })
      190 |         .catch(err => {
    > 191 |           reject(new codes.ERROR_CREATE_PROFILE({ sdkDetails, messageValues: reduceError(err) }))
          |                  ^
      192 |         })
      193 |     })
      194 |   }

      at http (../node_modules/swagger-client/lib/http/index.js:90:19)
      at processTicksAndRejections (../node:internal/process/task_queues:95:5) {
        status: 503,
        statusCode: 503,
        response: [Object]
      }
      at new <anonymous> (../node_modules/@adobe/aio-lib-core-errors/src/AioCoreSDKErrorWrapper.js:33:9)
      at ../src/index.js:191:18

Test Suites: 1 failed, 1 total
Tests:       7 failed, 8 passed, 15 total
Snapshots:   0 total
Time:        5.764 s
Ran all test suites.
Error: Command failed with exit code 1 (EPERM): npm run e2e
    at makeError (/home/runner/work/aio-e2e-tests/aio-e2e-tests/node_modules/execa/lib/error.js:59:11)
    at module.exports.sync (/home/runner/work/aio-e2e-tests/aio-e2e-tests/node_modules/execa/index.js:1[87](https://github.com/adobe/aio-e2e-tests/actions/runs/7563493369/job/20596007142#step:7:88):17)
    at runOne (/home/runner/work/aio-e2e-tests/aio-e2e-tests/src/run.js:61:9)
    at /home/runner/work/aio-e2e-tests/aio-e2e-tests/src/run.js:126:9
    at Array.forEach (<anonymous>)
    at runAll (/home/runner/work/aio-e2e-tests/aio-e2e-tests/src/run.js:120:33)
    at process.processTicksAndRejections (node:internal/process/task_queues:[95](https://github.com/adobe/aio-e2e-tests/actions/runs/7563493369/job/20596007142#step:7:96):5) {
  command: 'npm run e2e',
  exitCode: 1,
  exitCodeName: 'EPERM',
  stdout: '\n' +
    '> @adobe/[email protected] e2e\n' +
    '> jest --config e2e/jest.config.js\n',
  stderr: undefined,
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false,
  signal: undefined
}
!! e2e tests for aio-lib-campaign-standard failed !!

e2e tests error when logging to file

Log to File:

Run git config user.name github-actions
[master bde0557] generated
 1 file changed, 1 insertion(+)
remote: error: GH006: Protected branch update failed for refs/heads/master.        
remote: error: 3 of 3 required status checks are expected.        
To https://github.com/adobe/aio-e2e-tests
 ! [remote rejected] master -> master (protected branch hook declined)
error: failed to push some refs to 'https://github.com/adobe/aio-e2e-tests'
Error: Process completed with exit code 1.

This is because the master branch is protected now. Have to figure out a workaround like the npm publish Github Action, or disable master branch protection (which we use for PRs)

create unit tests for pull request tests

Often the pull requests are to update the dependencies in the test harness (this repo) that runs the e2e tests, and they are not dependencies in the e2e tests themselves.

Right now, for a PR, the full e2e test is run when it doesn't need to.

Create unit tests that test the test harness code, for pull requests to run against.

Motivation: all the dependabot PRs that update dependencies in this repo.

duplicate tests

Seems like we have 3 tests that do the same things but differ only on trigger

  • Daily runs tests once daily
  • nodejs runs tests on pull requests and commits to master
  • on demand runs tests ... well, ... on demand

There are slight differences, like do we update codecov? do we notify slack? what node versions do we cover 14.x or 14?

I think these should all be the same workflow file with a collection of triggers ... like we have in app-smoke-test.yml

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.