The s3_log from ess

S3Log

Naively log events to a file on S3

Gem Setup

gem install s3_log

# Gemfile
gem 's3_log'

Configuration

S3Log requires an AWS access key id, AWS secret access key, and a S3 bucket (preferably created in advance) name to function. It can be configured (in a Rails initializer or what have you) like so:

S3Log.configure(
  access_key_id: 'YOUR AWS ACCESS KEY ID',
  secret_access_key: 'YOUR AWS SECRET ACCESS KEY',
  bucket: 'YOUR S3 BUCKET'
)

Usage

Basically, logging an event is a matter of using S3Log.write to write content to a given file location on S3. The file does not have to exist prior to writing, and a write always appends to an existing file.

S3Log.write('path/to/file', 'Some content')

Caveats

There is no locking in this process, and an existing file will always be read and appended rather than blindly replaced. That being the case, near-simultaneous writes are won by the most recent write, and you would do well to do one of the following to avoid data loss:

Ensure that only one process writes to a given file at a time
Use a new file path for each write (ie models/user/1/TIMESTAMP.log rather than models/user/1.log)

Formal Documentation

The actual library docs can be read over on rubydoc.

Contributing

Do you use git-flow? I sure do. Please base anything you do off of the develop branch.

Fork it.
Perform some BDD magic. Seriously. Be testing.
Submit a pull request.

License

Make logging safe/fire-and-forget

At present, the following happens during write:

Read the file from S3
Append the new content to the end of the old content
Write the file to S3

This means that the write function currently can lose data: two writes happen near enough to each other that one writes while the other is trying to read/append means that one of the two writes will blow the other away.

Thoughts on getting around this:

Locking

There's no intrinsic file locking for S3, so this would be implemented via a sentinel file. The read/append/write cycle doesn't change, but it does get bookended by checking for a lock file, creating a lock file, and removing a lock file.

Pros:

Relatively easy to implement
A log file is still generally easy to read via either the AWS console, s3fs, what have you

Cons:

Still not totally safe
What is the behavior for .write() when there is a lock present?
How do we handle stale locks?

Log == Collection of Events

Essentially, each time .write() is called, rather than reading and appending, it simply creates a new file ("#{Time.now.to_f}-#{machine id}-#{process id}" or what have you), treating the given path as a directory rather than a file.

Pros:

LOADS safer than the current model
Relatively easy to implement

Cons:

Logs are no longer easy to read via standard tools (console, s3fs, so on)
Basically necessitates the creation of a utility for reading the logs
Are there number-of-file limitations in S3? If not a limitation, does the number of files present increase cost?
While much safer than current, still not totally safe (two processes writing at exactly the same time will cause a collision)

ess / s3_log Goto Github PK

s3_log's Introduction

S3Log

Gem Setup

Configuration

Usage

Caveats

Formal Documentation

Contributing

License

s3_log's People

Stargazers

Watchers

s3_log's Issues

Locking

Log == Collection of Events

Recommend Projects

Recommend Topics

Recommend Org