Giter VIP home page Giter VIP logo

node-redis-timeseries's Introduction

Node.js API for storing and querying time series in Redis

Build Status

Convenient module for storing and querying time series statistics in Redis using Node.js.

The design (and even parts of the implementation) were picked from the ApiAxle project.

You can find basic usage examples in examples. This module also powers a real-time dashboard written in Node.js. Check the sources out for more insight.

Dependencies

redis-timeseries has no dependencies, and will work along the redis module you'll install in your own project. redis@~0.9.0 versions are compatible.

Usage

	var TimeSeries = require('redis-timeseries'),
		redis = require('redis').createClient();
		
	// Create the TimeSeries client
	//
	// "stats" is the Redis namespace which will be used
	// for storing all the TimeSeries related keys
    //
	// "granularities" encodes the granularities at which
	// you want to store statistics. More on that in the next section
	//
	var ts = new TimeSeries(redis, "stats", granularities);
	
	// Recording hits
	//
	// This increments the counters for the
	// stats keys you provide
	//
	// "timestamp" defaults to the current time. If providing
	// a timestamp, it should be in unix timestamp format (seconds
	// since epoch).
    // "increment" defaults to 1
	//
	// .exec takes an optional callback with no arguments.
	ts.recordHit('your_stats_key')
	  .recordHit('another_stats_key', timestamp)
      .recordHit('another_stats_key', timestamp2, increment)
	  
	  .exec(callback);

    // Removing hits
    //
    // Decrements the hits for a specified point in time.
    ts.removeHit('your_stats_key', [timestamp]).exec();
    
    // Decrement defaults to 1, but can be specified explicitly (below).
    ts.removeHit('your_stats_key', [timestamp], 5).exec();
	  
	// Querying statistics
	//
	// Returns "count" chunks of counters at the precision described by
	// "granularity_label"
	// 
	ts.getHits('your_stats_key', granularity_label, count, function(err, data) {
		// data.length == count
		// data = [ [ts1, count1], [ts2, count2]... ]
	});

Defining custom statistics granularities

For each key, TimeSeries stores statistics at different granularities. For further information about this, please refer to the detailed blog post from the ApiAxle project.

The default granularities are:

{
    '1second'  : { ttl: this.minutes(5), duration: 1 },
    '1minute'  : { ttl: this.hours(1)  , duration: this.minutes(1) },
    '5minutes' : { ttl: this.days(1)   , duration: this.minutes(5) },
    '10minutes': { ttl: this.days(1)   , duration: this.minutes(10) },
    '1hour'    : { ttl: this.days(7)   , duration: this.hours(1) },
    '1day'     : { ttl: this.weeks(52) , duration: this.days(1) }
}

This means that the number of hits per second will be stored for 5 minutes, and the corresponding hashset will expire afterwards. Likewise, the number of hits per minute for a given key will be kept for an hour. Daily counters on the other hand are kept for a full year.

When querying for statistics, a granularity label is expected:

	// Give me the hits/second for the last 3 minutes
	ts.getHits('your_stats_key', '1second', ts.minutes(3), function(err, data){
		//process the data
	});
	
	// Give me the number of hits per day for the last 2 weeks
	ts.getHits('your_stats_key', '1day', 14, function(err, data){
		//process the data
	});
	
	// And so on

When creating the TimeSeries client, you can override the default granularities with your own.

Bitdeli Badge

node-redis-timeseries's People

Contributors

alph486 avatar bitdeli-chef avatar dangerdespain avatar martin308 avatar thanosa75 avatar tonyskn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

node-redis-timeseries's Issues

Sum of hits

Hey,

Is it possible to just have the number of hits (sum) for a timespan?
Example for the last 30 seconds:

ts.getsHits('this_one', ts.seconds(30), callback);

Thanks,
Philmod

Calculate time series duration starting from start of period unit

Not an issue more of a question really: there are some time ranges that really make sense only from the beginning of the unit, e.g. "month" range should calculate from 1st of month, "week" from Sunday, etc. How can this module be modified or what would be the approach to handle such cases?

Cheers,

Thanos

Concurrency issue with pendingMulti variable

I am writing a piece of code to increase/store counters and hitting it with average load. In some cases I get the strange behaviour that a specific counter is increased by an extra digit - 2-3 every 250. This is in a transactional multithreaded environment and node app is being hit by at least 3-10 concurrent hits.

I've checked MONITOR output and in all my cases, situation is similar to the below:

1390991605.597054 "MULTI"
1390991605.597119 "hget" "cmp:2_11223300050:w:1389830400" "1389830400"
1390991605.597149 "hget" "cmp:2_11223300050:w:1389830400" "1390435200"
1390991605.597158 "EXEC"
1390991605.597315 "MULTI"
1390991605.597383 "hincrby" "cmp:2_11223300055:d:1390867200" "1390953600" "1"
1390991605.597395 "expireat" "cmp:2_11223300055:d:1390867200" "1391212800"
1390991605.597405 "EXEC"
1390991605.597632 "MULTI"
1390991605.597739 "hincrby" "cmp:2_11223300055:d:1390867200" "1390953600" "1"
1390991605.597754 "expireat" "cmp:2_11223300055:d:1390867200" "1391212800"
1390991605.597766 "hincrby" "cmp:2_11223300053:d:1390867200" "1390953600" "1"
1390991605.597777 "expireat" "cmp:2_11223300053:d:1390867200" "1391212800"
1390991605.597789 "EXEC"

Problematic behaviour (additional +1) comes at the point marked with timestamps 1390991605.597383 and 1390991605.597739. It seems that the "pendingMulti" variable is picked up by the next event without it being cleared by the previous caller, due to high concurrency.

Code used to create the issue is part of an express() application, function to record the hit is as below:

// this POST request records 1 hit for (now)
// on all granularities defined
app.post('/hit/:rule/:type/:id', function(req,res) {
        var tsKey = req.params.id ;
        var dataMap = { ver : PROTO, tp: req.params.rule };
        ts = getTS(req.params.rule,req.params.type);
        if (ts == null) { // not found or invalid rule given
                dataMap['error'] = 'not a correct rule/type:'+req.params.rule+','+req.params.type;
                res.json(404,dataMap);
                return;
        }
        //console.log("selected = "+ts.keyBase);
        ts.recordHit(tsKey).exec(function(err,result) {
                if (err != null) {
                        console.log("ERROR writing redis "+err);
                        res.json(404, err);
                }
                //console.log("Recorded hit ", ++hitCounter, new Date());
                res.json(200, dataMap);
        });
});

add decrement functionality

Hi

would like for various reasons (e.g. rollback) to be able to decrement a counter or set of counters - can you please add this api call to your library?

thanks

Thanos

When creating the TimeSeries client, you can override the default granularities with your own.

When creating the TimeSeries client, you can override the default granularities with your own. How?

Since you are defining the timeseries granularities as '1second' : { ttl: this.minutes(5), duration: 1 }, if i define a new one... '10seconds' : { ttl: this.days(1), duration: 10 } will not work because days is not defined. is there a simple/clean way to do this? Currently I just added the 10seconds to the source file, I know its a horrible thing, but i could not find out a clean way to do it and still remain with the this.days() syntax.

please add timestamp in getHits() parameter

Hi.
I faced a difficult situation. I can record and remove a hit, because recordHit and removeHit functions require timestamp in function parameter. But there is no option in getHits(). getCurrentTime() return timestamp with UTC timezone always. But i need KST timezone. How can i get hits with other timezone??

Merging time-series.

Note an issue; but, a user question.

I will have two sources of data that publish at an arbitrary rate. I would like to merge these time series into a third one by taking a 1 minute slice of the other two time series. How do I do this with node-redis-timeseries ?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.