kinesis-tail's Introduction

kinesis-tai

Tool which provides tail for Kinesis, it allows you to use one of two processors for the data returned, firstly one which decompresses and parses CloudWatch Logs data, and secondly one which just returns the raw data.

background

This cloudwatch logs reader works with a pattern used at Versent for log distribution and storage.

For more information on the setup for cwlogs sub command to function it assumes the logs are gzipped batches of log JSON records in Kinesis see Real-time Processing of Log Data with Subscriptions

installation

You can download kinesis-tail from Releases or install it using npm.

usage

usage: kinesis-tail [<flags>] <command> [<args> ...]

Flags:
      --help           Show context-sensitive help (also try --help-long and --help-man).
  -t, --trace          Enable trace mode.
  -r, --region=REGION  Configure the aws region.
      --version        Show application version.

Commands:
  help [<command>...]
    Show help.


  cwlogs [<flags>] <stream>
    Process cloudwatch logs data from kinesis.

    --include=INCLUDE ...  Include anything in log group names which match the supplied string.
    --exclude=EXCLUDE ...  Exclude anything in log group names which match the supplied string.

  raw [<flags>] <stream>
    Process raw data from kinesis.

    --timeout=3600000  How long to capture raw data for before exiting in ms.
    --count=0          How many records to capture raw data for before exiting.

example

List the kinesis streams in your account.

aws kinesis list-streams

To tail one of these streams and exit once you have captured 20 records.

kinesis-tail raw dev-1-stream --count 20

To tail one of these streams and exit after 30 seconds, and write the data to a file.

kinesis-tail raw dev-1-stream --timeout 30000 | tee data.log

license

This code is released under MIT License.

kinesis-tail's People

Contributors

Stargazers

Watchers

kinesis-tail's Issues

count not applied in 1.1.6

Looks like the --count option is not being acknowledged in version 1.1.6, the specified number of records are received, yet the tail continues to wait until the timeout is reached.

AWS_REGION=us-west-2 AWS_PROFILE=saml kinesis-tail raw myKinesisStream --count 20 --debug  

DEBU[0000] built kinesis service                        
DEBU[0001] describe stream response                      respDesc="{\n  StreamDescription: {\n    EncryptionType: \"NONE\",\n    EnhancedMonitoring: [{\n        ShardLevelMetrics: []\n      }],\n    HasMoreShards: false,\n    RetentionPeriodHours: 24,\n    Shards: [{\n        HashKeyRange: {\n          EndingHashKey: \"ABC123\",\n          StartingHashKey: \"0\"\n        },\n        SequenceNumberRange: {\n          StartingSequenceNumber: \"ABC123\"\n        },\n        ShardId: \"shardId-000000000000\"\n      },{\n        HashKeyRange: {\n          EndingHashKey: \"ABC123\",\n          StartingHashKey: \"ABC123\"\n        },\n        SequenceNumberRange: {\n          StartingSequenceNumber: \"ABC123\"\n        },\n        ShardId: \"shardId-000000000001\"\n      }],\n    StreamARN: \"arn:aws:kinesis:us-west-2:123:stream/myKinesisStream\",\n    StreamCreationTimestamp: 2017-12-13 22:44:19 +0000 UTC,\n    StreamName: \"myKinesisStream\",\n    StreamStatus: \"ACTIVE\"\n  }\n}"
DEBU[0001] get shard iterator                            shard=shardId-000000000001
DEBU[0001] get shard iterator                            shard=shardId-000000000000
DEBU[0002] waiting for records                           count=20
DEBU[0007] received records                              count=0 shard=shardId-000000000000
DEBU[0007] received records                              count=0 shard=shardId-000000000001
DEBU[0012] received records                              count=0 shard=shardId-000000000001
DEBU[0012] received records                              count=0 shard=shardId-000000000000
DEBU[0017] received records                              count=0 shard=shardId-000000000000
DEBU[0017] received records                              count=0 shard=shardId-000000000001
DEBU[0022] received records                              count=0 shard=shardId-000000000001
DEBU[0022] received records                              count=0 shard=shardId-000000000000
DEBU[0027] received records                              count=11 shard=shardId-000000000001
DEBU[0027] received records                              count=9 shard=shardId-000000000000
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
{"example":"record"}
DEBU[0032] received records                              count=0 shard=shardId-000000000001
DEBU[0032] received records                              count=0 shard=shardId-000000000000
DEBU[0037] received records                              count=0 shard=shardId-000000000001
DEBU[0037] received records                              count=0 shard=shardId-000000000000
DEBU[0042] received records                              count=0 shard=shardId-000000000001
DEBU[0042] received records                              count=0 shard=shardId-000000000000
DEBU[0047] received records                              count=0 shard=shardId-000000000000
DEBU[0047] received records                              count=0 shard=shardId-000000000001
DEBU[0052] received records                              count=0 shard=shardId-000000000001
DEBU[0052] received records                              count=0 shard=shardId-000000000000
DEBU[0057] received records                              count=0 shard=shardId-000000000000
DEBU[0057] received records                              count=0 shard=shardId-000000000001

Add "bin" entry to package json

Hi!

There's a commit (a229795) that mentions this was removed because it was unused, but might not be completely the case.

When you run npm tools on a per-project basis, the first spot to look is under node_modules/.bin. This is the spot where npm symlinks all the executables present on the various node_modules.

kinesis-tail's node module appears to have an empty /bin, but a populated /cmd. If I interpreted the documentation correctly, that means our package.json would need to contain:

"bin": "./cmd/kinesis-tail"

But I've just realized that there's not even a package.json anymore 😢

Thanks for the awesome tool!

versent / kinesis-tail Goto Github PK

kinesis-tail's Introduction

kinesis-tai

background

installation

usage

example

license

kinesis-tail's People

Contributors

Stargazers

Watchers

Forkers

kinesis-tail's Issues

count not applied in 1.1.6

Add "bin" entry to package json

Doesn't start tailing records till all the shard iterators have started

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent