Note: This target is derived from https://github.com/transferwise/pipelinewise-target-s3-csv. Some of the documentation below has not been completely updated yet.

Singer target that uploads loads data to AWS Athena in CSV format following the Singer spec.

The recommended method of running this target is to use it from PipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation at Target S3 CSV

If you want to run this Singer Target independently please read further.

First, make sure Python 3 is installed on your system or follow these installation instructions for Mac or Ubuntu.

It's recommended to use a virtualenv:

  python3 -m venv venv
  pip install git+https://github.com/MeltanoLabs/target-athena.git

or

  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install .

Like any other target that's following the singer specificiation:

some-singer-tap | target-athena --config [config.json]

It's reading incoming messages from STDIN and using the properites in config.json to upload data into Postgres.

Note: To avoid version conflicts run tap and targets in separate virtual environments.

Running the the target connector requires a config.json file. An example with the minimal settings:

{
  "s3_bucket": "my_bucket",
  "athena_database": "my_database"
}

Profile based authentication used by default using the default profile. To use another profile set aws_profile parameter in config.json or set the AWS_PROFILE environment variable.

For non-profile based authentication set aws_access_key_id , aws_secret_access_key and optionally the aws_session_token parameter in the config.json. Alternatively you can define them out of config.json by setting AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables.

Full list of options in config.json:

1. Define environment variables that requires running the tests

  export TARGET_ATHENA_ACCESS_KEY_ID=<s3-access-key-id>
  export TARGET_ATHENA_SECRET_ACCESS_KEY=<s3-secret-access-key>
  export TARGET_ATHENA_BUCKET=<s3-bucket>
  export TARGET_ATHENA_KEY_PREFIX=<s3-key-prefix>

2. Install python test dependencies in a virtual env and run nose unit and integration tests

  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install .[test]

3. To run unit tests:

  nosetests --where=tests/unit

4. To run integration tests:

  nosetests --where=tests/integration

1. Install python dependencies and run python linter

  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install .
  pip install pylint
  pylint target_athena -d C,W,unexpected-keyword-arg,duplicate-code

Apache License Version 2.0

See LICENSE to see the full text.