tigstep Goto Github PK

avro_sample_serde

A sample project that AVRO serializes and deserializes a record. Some logic from this will later be used in tigstep/KafkaConnectSpooldir_Avro_Kafka.

connect_transform

docker_kafka_cluster

This is the upgraded version of gten/docker-kafka-cluster (kafka version is changed from 0.9 to 2.1.0 and the start-kafka.sh is changed to work with kafka2.1.0 specific server.properties)

ecs_airflow

A project that sets up an AWS ECS infrastructure and hosts an Apachi Airflow running inside a docker container.

file_obfuscator_importer_pipeline

A simple file obfuscation/ingestion pipeline that runs on AWS with automated infrastructure deployment and execution.

kafka

Mirror of Apache Kafka

kafka_connect_spooldir_pipeline

A dockerized Confluent Kafka Cluster running on AWS EC2 instances that uses the Spooldir Connector(https://github.com/jcustenborder/kafka-connect-spooldir) to spool a directory, Avro serialize the data using the Schema Registry and publish to Kafka Brokers. This project also makes the Confluent Control Center available for visualization.

kafka_misc

This repo containes various modules/classes related to Apache Kafka development efforts

kafka_storm_pipeline

A real-time, dockerized Kafka event processor pipeline (utilizing an Apache Storm topology). The project is running in AWS with automated infrastructure deployment and execution, using Ansible.

loan_dafault_predictor

visitor_count

A sample Spark exercise to count the visitors based on given conditions.