Giter VIP home page Giter VIP logo

tf-splunk-fargate-efs's Introduction

tf_splunk_fargate_efs

Demo app of Splunk running on AWS ECS Fargate with EFS for storage. It shows:

  • Fargate usage on ECS
  • Fargate EFS integration
  • ECS Exec to log in to containers
  • ALB integration
  • Graviton Fargate usage
  • Splunk running in a container
  • Kinesis Firehose -> Splunk integration with HEC

This uses a custom docker image which builds on top of the official Splunk image only to pre-set up some indexes and HEC keys

It also uses CloudFront as a hack/trick to add HTTPS support without the need for you to have a public domain for a cert. Kinesis requires that the HEC endpoint use HTTPS.

Quick Start

  • terraform apply
  • wait for deployment
  • wait 10 minutes more
    • initially Kinesis will have logs like Failed to deliver data to Splunk or to receive acknowledgment. Make sure HEC endpoint is reachable from Firehose and it is healthy.
      • not sure exactly what timing issue is here
  • Query Splunk: index=ecs

TODO

  • SSM Session Manager for instance
  • shorter log buffering? fluent-bit config?

Debugging

ECS Exec

aws ecs execute-command --cluster tf-splunk-fargate-ecs --command /bin/sh --interactive --container httpbin --task TASK_ID

Console Access

  • log in to web console
  • default creds = admin/password
  • Settings -> Data Inputs -> HTTP Event Collector
    • 1111... requires HEC ACK
    • 2222... does not require ACK
      • Firehose seems to require this version

## TODO

- reduce delivery stream delay
- HTTPS only on console?
- fargate private IP

- clean up SGs

tf-splunk-fargate-efs's People

Contributors

ericdahl avatar

Watchers

James Cloos avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.