This stand-alone testing configurable application will help us execute different end-to-end tests and improve the Airbyte final product quality. You can find the development state and future roadmap in this epic issue.
- Automatic run before publishing a new connector version to the could - prevent versions without backward compatibility or braking change for one of the possible integrations between source and destination.
- Run scenario for a dev version in the PR.
- Run scenario locally during development.
The central concept of the tool is a universal testing platform that you can easily configure by providing scenario config and required credentials. The solution helps us run different test scenarios from the CI referring to already defined config files and cover all possible e2e test cases with minimal effort.
All credentials are independent of the selected scenario and can be used in different runs. In addition, we store credentials in the secrets storage as we do for the integration tests.
As the credential file is input data, you can find structure in the CredentialConfig.yaml file.
Source credential file example:
{
"credentialName": "test_source_creds",
"credentialType": "source_creds",
"instanceType": "Postgres",
"credentialJson": {
"database": "test_source_database",
"host": "put.your.host.here",
"password": "yourPassword",
"port": 5432,
"replication_method": {
"method": "Standard"
},
"schemas": [
"public"
],
"ssl": false,
"ssl_mode": {
"mode": "disable"
},
"tunnel_method": {
"tunnel_method": "NO_TUNNEL"
},
"username": "sourceuser"
}
}Airbyte credential file example:
{
"credentialName": "airbyte_local_creds",
"credentialType": "airbyte_creds",
"credentialJson": {
"apiHost": "your.airbyte.host",
"apiPort": "80",
"apiScheme": "https"
}
}The scenario config describes a list of actions, their sequence, and used instances. It doesn't contain any credentials for used instances. So, we can reuse the same scenario for different Airbyte instances, sources, and destinations. As there is no sensitive data, we can store it as part of the application.
You can find scenario config file structure in the ScenarioConfig.yaml file. (Sub files: ScenarioConfigAction.yaml , ScenarioConfigInstance.yaml)
Note! The instance name is a way to use the same instance in different actions
Scenario config example:
{
"scenarioName": "Simple sync scenario",
"usedInstances": [
{
"instanceName": "airbyte_1",
"instanceType": "AIRBYTE"
},
{
"instanceName": "source_1",
"instanceType": "SOURCE"
},
{
"instanceName": "destination_1",
"instanceType": "DESTINATION"
},
{
"instanceName": "connection_1",
"instanceType": "CONNECTION"
}
],
"preparationActions": [
{
"action": "CONNECT_AIRBYTE_API",
"resultInstance": "airbyte_1"
},
{
"action": "CREATE_SOURCE",
"requiredInstances": [
"airbyte_1"
],
"resultInstance": "source_1"
},
{
"action": "CREATE_DESTINATION",
"requiredInstances": [
"airbyte_1"
],
"resultInstance": "destination_1"
},
{
"action": "CREATE_CONNECTION",
"requiredInstances": [
"airbyte_1",
"source_1",
"destination_1"
],
"resultInstance": "connection_1"
}
],
"scenarioActions": [
{
"action": "SYNC_CONNECTION",
"requiredInstances": [
"airbyte_1",
"connection_1"
]
}
]
}You can put your comment with one of the testing tool commands to any PR in the repository to trigger the GitHub action. The action will run the testing tool using your comment message as input parameters for the main method. The GA publish the execution result in your original comment.
- Open some PR
- Put the comment
/run-scenario name="Simple sync scenario" airbyte_1=tt_airbyte_dev2 source_1=tt_postgres_source_aws_1 destination_1=tt_postgres_destination_aws_1 - Wait till GA updates your comment with the execution results.
The tool provides the possibility to run all scenarios locally. If your command requires credentials, you should provide them locally or request them from the secret service. You have two ways how to provide the required credentials to the tool. The main flow is to get credentials from the secret service, but also you can use your local files as a source for credentials. (See more about credentials below)
- Clone the repository
- Build the project
- Put the secret service credential file in the
secrets/service_account_credentials.jsonfolder in the project root (You can find the file structure in the ServiceAccountConfig.yaml) - Run the scenario by passing the required arguments to the method
TestingTool::mainwith the command/run-scenario
The result you can find in the execution log.
Note! The help text is formatted for GitHub. You can put it there to get a more readable view.
The tool has help command /list-scenarios. The command provides the full list of available scenarios and hints on how to get more details about a specific scenario.
Note! The command also runs logical validation for all scenarios. If some scenario is unfinished or invalid, you will see ❌ sing after its name.
Each scenario has its list of required credentials and their number. In addition, some scenarios might require parameter values in the command line.
To make this step easier, the tool has command /help name="<scenario name>"(How to get name see `How to get the list of scenarios). This command returns a prepared command line and a list of all credentials you should put into this line.
In the result, you can find such parts:
- Call examples - different command lines which can be useful for you. Note that you need to replace texts like
<put_credential_name>in this line before running.
- Instances in the scenario - List required instance credentials with their types. Use it to put the proper credential name into the command line.
- Parameters in the scenario - parameter is a simple text value required inside a scenario. The name and type of the parameter should be enough to understand which value is expected from you.
In the end, you should have a command line ready for execution.
For example, in this case we can get a line like this : /run-scenario name="Update destination version scenario" airbyte_1=tt_airbyte_docker source_1=tt_postgres_source_aws_1 destination_1=tt_postgres_destination_aws_1 old_version="0.3.22" new_version="0.3.23"
TBD
| Version | Description |
|---|---|
| 0.4.4 | Add common local instances for destinations + Postgres impl |
| 0.4.3 | Make normalization on by default |
| 0.4.2 | Add autonomous instances |
| 0.4.1 | Action structure improvement + add delete actions for conn/src/dest instances |
| 0.4.0 | Phase 4 and 5 complete |
| 0.3.0 | Add possibility to provide params by actions + new action ReadSourceVersion |
| 0.2.3 | Scenario validation result is now available in help and list commands |
| 0.2.2 | Scenario description |
| 0.2.1 | Add additional settings to CredentialConfig |
| 0.2.0 | Run scenario using secret service. !No default test run anymore! |
| 0.1.3 | Run scenario using local files |
| 0.1.2 | Action parameters. |
| 0.1.1 | Implement HelpService. Now you can get short description with examples for a scenario. |
| 0.1.0 | Stage 1. POC |
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent