The project uses the library 'bashful' to execute the spark code from a 'workflow.yaml' YAML file. This file has the file run command and a tag to identify this command.
Link to bashful - https://github.com/wagoodman/bashful
Run the following within the code folder, since folder paths are relative to this folder.
- To run all tasks - bashful run workflow.yaml
- To run a specific task named 'build' - bashful run workflow.yaml --tags build
- data /temp-data /production-data
- code /temp-code /production-code
- output (FOR FILES)
- output-graphs (FOR FIGURES)
Execute the code in /code
Push the input and output data to Wasabi. The code and configuration information will be in Wasabi_data
aws s3 cp filename s3://BUCKET/filename --profile 'wasabi' --endpoint-url 'https://s3.wasabisys.com'