This project has basic code functionality for ETL from MongoDB to PostgreSQL database.
Code structure:
upload_data.py
- responsible for first copy of available data to MongoDB.postgres.py
- connection to PostrgeSQL.bulk_queries.py
- provides functionality for bulk upserts into PostrgeSQL.etl.py
- where all ETL is orchestrated.application.py
- Flask based application to run as a web-service. Configured to run on Azure as an App Service.constrants.py
- all queries that are specific to database structure are found here.logger.py
- basic logging functionality.
This project was written as an interview take home assignment, and by no means represents a production ready code. The code is publicly shared as the interviewers did not forbid it, however, the interviewer company name is not mentioned for ethical reasons.