Giter VIP home page Giter VIP logo

blitzkz23 / final-project-end-to-end-banking-campaign-pipeline Goto Github PK

View Code? Open in Web Editor NEW
8.0 2.0 2.0 248.28 MB

Final Project for IYKRA Data Fellowship 8 Program, creating an end-to-end banking campaign pipeline using lambda architecture (providing acess to batch and stream processing)

Shell 1.09% Python 78.12% Dockerfile 0.14% HTML 4.43% R 1.21% CSS 0.08% Batchfile 0.43% Java 5.84% Scala 8.35% Makefile 0.12% Jupyter Notebook 0.18% Less 0.02%
batch-processing data-engineering data-pipeline data-streaming

final-project-end-to-end-banking-campaign-pipeline's Introduction

Bank Marketing Campaign's Subscription End-to-End Data Pipeline

Bussiness Understanding

A certain banking institution on Portugal, conducted a campaigns in order to offer a their client a term deposit. Campaign are conducted mainly based on Telemarketing method such as direct phone calls while offering their client. If after all that effort the client agreed to place deposit it will result on target variable (y) marked 'yes', otherwise 'no'. The bank-side wanted to know what kind of feature or status that make the offer more successfull in order to be more cost and time efficient, so they created this dataset in order to analyze them about historical condition of their client such as job, marital status, loan, etc and whether they agreed to place deposit.

Problem Statements

From said back story, we can conclude that the bank want to increase the efficiency of their campaign by targeting client with higher chance of success based on the feature from the data.

Goals

In order solve the problems, as a Data Engineer our team are proposing some objective in order to pull off that goals, as follows:

  1. Design end-to-end data pipeline with Lambda Architecture, so the bussiness intellegence/analyst could choose between whether to use batched data or real time streamed data.
  2. Create analytics dashboard based on data from the analytics database (data warehouse) and derive some insight from it.

End-to-End Schema with Lambda Architecture

data-pipeline

Reproducibility

In order to reproduce the dependencies and overall environment used on this project refer to this guide README.MD

Data Visualization Dashboard

Access the dashboard here

Dashboard 1

Dashboard 2

Google Cloud Usage Billing Report

Data infrastructure we used in this project are entirely built on Google Cloud Platform with more or less 3 weeks of project duration, using this following services:

  • Google Cloud Storage (pay for what you use)
  • Google BigQuery (first terrabyte processed are free of charge)
  • Google Looker Studio (cost is based from number of Looker Blocks (data models and visualizations), users, and the number of queries processed per month)
  • Pub/sub (cost is based from amount of data exchanged)
  • Dataflow (cost is based on duration of processing job and number of resources utilized)

Total cost around 51$ out of 300$ free credits that GCP provided

Resources

Slides and Jobdesc

Kelompok 5 Kuli Data Warrior

  • Naufal Aldy Pradana
  • Yuditya Mulia Insani
  • Adrian Maulana Muhammad
  • Yuda Satria Syaifi

final-project-end-to-end-banking-campaign-pipeline's People

Contributors

adrn-mm avatar blitzkz23 avatar bulletsrip avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.