abdel-raouf / etl-project Goto Github PK

-  We can pass the path of the directory that contaning our dataset files  AS `command: ["scheduler 'your-directory-path'"]` (this command acts as the command line interface as it's a docker command that will path the parameter to the command line inside docker)
    - Ubuntu path example: `/home/your_username/input_folder/`
    - Windows path example: `d:/input_folder`
- We can pass the recipeName that we need to execute only AS `command: ["executor recipe-name"]` or we could pass nothing, then it will execute all the results.

- I Applied SOLID principles, decoupled each functionality and also decoupled technologies whenever is possible as using slf4j (logger interface -> Facade design pattern) on top of log4j to apply an abstraction layer between the app and the logger we will use to be able to replace the logger in the future without affecting our implementation of the logging which will still be the same for any logger we will use, as a result we can unit test with ease and might also don't need any integration tests between the components due to the granularity of each function, while expose them as a service which we might inject or replace it with a new implmentation without affecting the over-all design of the application (the system as a whole is open for any extenstions, closed for modifications), as a result we can scale easily.

- `SpringBoot`: from my view, it helped me to focus more on the Architecture as it handles implementation of many used services pretty well.
- `Apache Kafka`: an open-source distributed event streaming (will help us a lot when scalling enough to handle stream).
- `Docker`: easily pack, ship, and run any application code base.
- `Hibernate`: ORM 
- `Postgresql`: relational database that is powerful enough and flexiable to our demand also, as we can convert it to be a column oriented DB using this extension -> `cstore_fdw`
- `lombok`: manage boilerplate code as generating construcor, getter, setter, building objects.
- `Jackson`: Transcoder that manages Serialization and Deserialization pretty well.
- `Slf4j`: logger interface that can set on top of any logger
- `log4j`: logger to manage warning, error, debug, info inside our code base.

- using `AWS ECR` to hold our containers.
- using `Apache Airflow` to create a pipeline to extract, transform, and load JSON data from `Amazon S3` to `Amazon Redshift`(Data Warehouse) (Automating the process of ETL) .
- using `AWS lambda`  as our serverless computing service.
- we can also expose the `KafkaExecuteService` as a web service using `AWS API Gateway` to accept the filter that we will pass to it the recipe name.