playground project to get an understanding how processing csv files in an event driven manner using docker-compose and specific frameworks without the use of cloud components.

1. Insert csv to root directory of solution

2. Filewatcher event will be triggered

3. Worker will be created via Rabbit MQ / Celery

   • Read CSV via Pandas
   • Store content in database

• docker
• docker-compose
• linux (pyinotify)

git clone https://github.com/pharindoko/csv-pipe-playground.git
cd csv-pipe-playground
make build

make run

you have 2 options to start the process

• Add a csv file into the root directory or a nested directory of the source folder (works recursively).
• This is limited to linux as I`m using Pyinotify which supports IN_CLOSE_WRITE (relies on linux kernel)

1. Open vue application (http://localhost:3000/)
2. Drag and drop csv file(s) or select file(s) via file dialog
3. Click on button "Upload"

data will be written into postgre db

Open Graphql Playground
http://localhost:5433/graphiql

new endpoints will be automagically added by postgraphile when a new csv has been added)

Open a DB Connection postgres://postgres:postgres@postgres:5432/postgres

recommendation is to use a db client like dbeaver

It might be a discussion point if celery and rabbitmq are a bit too much for this solution.

But I really enjoyed how easy it is to implement rabbitmq and celery and how stable this works.

Having IO operations and database connections I guess it`s not that bad to split it up into other processes and keep the event handler process lean.