hotelde / node-cache-manager-fs Goto Github PK

View Code? Open in Web Editor NEW

32.0 3.0 18.0 139 KB

Node Cache Manager store for filesystem

JavaScript 100.00%

node-cache-manager-fs's Introduction

node-cache-manager-fs

Node Cache Manager store for Filesystem

The Filesystem store for the node-cache-manager module.

Installation

npm install cache-manager-fs --save

Features

Saves anything that is JSON.stringify-able to disk
limit maximum size on disk
refill cache on startup (in case of application restart)
"Callback"-interface style (no promises, no async)

Usage example

Here are examples that demonstrate how to implement the Filesystem cache store.

Single store

// node cachemanager
var cacheManager = require('cache-manager');
// storage for the cachemanager
var fsStore = require('cache-manager-fs');
// initialize caching on disk
const diskCache = cacheManager.caching({
    store: fsStore, 
    options: {
        path: 'diskcache',          //path for cached files
        ttl: 60 * 60,               //time to life in seconds
        maxsize: 1000*1000*1000,    // max size in bytes on disk
        zip: true,                  //zip files to save diskspace (default: false)        
        preventfill:true            
    }
});

(async () => {

    await diskCache.set('key', 'value');
    console.log(await diskCache.get('key')); //"value"
    console.log(await diskCache.ttl('key')); //3600 seconds
    await diskCache.del('key');
    console.log(await diskCache.get('key')); //undefined

    console.log(await getUserCached(5)); //{id: 5, name: '...'}
    console.log(await getUserCached(5)); //{id: 5, name: '...'}

    await diskCache.reset();

    function getUserCached(userId) {
        return diskCache.wrap(userId /* cache key */, function () {
            return getUser(userId);
        });
    }

    async function getUser(userId) {
        return {id: userId, name: '...'};
    }

})();

Options

options for store initialization

options.ttl = 60; // time to life in seconds
options.path = 'cache/'; // path for cached files
options.preventfill = false; // prevent filling of the cache with the files from the cache-directory
options.fillcallback = null; // callback fired after the initial cache filling is completed
options.zip = false; // if true the cached files will be zipped to save diskspace
options.reviveBuffers = true; // if true buffers are returned from cache as buffers, not objects

Installation

npm install cache-manager-fs

Tests

To run tests:

npm test

Code Coverage

To run Coverage:

npm run coverage

License

cache-manager-fs is licensed under the MIT license.

node-cache-manager-fs's People

Contributors

Stargazers

Watchers

Forkers

baggersio wcoppens lewisdawson akylas luckyllama amit777 sheershoff cherijs paracycle timdumol mar-pfa sframe mannharleen michaltajchertdev heysasha tryretool

node-cache-manager-fs's Issues

diskCache.set is not a function when using the example on readme..

Code from readme gives diskCache.set is not a funciton... is this library still valid ?

// node cachemanager
var cacheManager = require('cache-manager');
// storage for the cachemanager
var fsStore = require('cache-manager-fs');
// initialize caching on disk
const diskCache = cacheManager.caching({
    store: fsStore,
    options: {
        path: 'diskcache',          //path for cached files
        ttl: 60 * 60,               //time to life in seconds
        maxsize: 1000*1000*1000,    // max size in bytes on disk
        zip: true,                  //zip files to save diskspace (default: false)        
        preventfill:true
    }
});

(async () => {

    await diskCache.set('key', 'value');
    console.log(await diskCache.get('key')); //"value"
    console.log(await diskCache.ttl('key')); //3600 seconds
    await diskCache.del('key');
    console.log(await diskCache.get('key')); //undefined

    console.log(await getUserCached(5)); //{id: 5, name: '...'}
    console.log(await getUserCached(5)); //{id: 5, name: '...'}

    await diskCache.reset();

    function getUserCached(userId) {
        return diskCache.wrap(userId /* cache key */, function () {
            return getUser(userId);
        });
    }

    async function getUser(userId) {
        return {id: userId, name: '...'};
    }

})();

get method returning undefined

var cacheManager = require('cache-manager');
var fsStore = require('cache-manager-fs');

var diskCache = cacheManager.caching({ store: fsStore, options: {   ttl:86400, maxsize: 1000000} });
diskCache.set('test',"1");
console.log(diskCache.get('test')) ==>undefined;

cache key as filename

Hi, I was wondering if there was any particular reason to use the UUID method for generating the .dat files? I noticed an issue when I run in cluster mode, each cluster process seems to generate it's own file in the cache directory so there are duplicates.

I was going to create a pull request for an optional flag to use the cache key as the filename. Is there something obviously bad I'm missing about this idea? Very long cache keys would have a problem or if the keys contained special characters and such.

Thanks for creating this!

Can you release the latest version with reviveBuffers to NPM?

Doesn't reinitialize with zip option enabled

I've fixed it here sheershoff/node-cache-manager-fs-binary@5c94062

Since the tests are failing, it's meaningless to PR a separated fix until #7 is merged.

So, one can go fetch changes or I'll come in with new PR when #7 is merged.

Does not delete the old files(Expired cache file)

Hi,

I want to delete expired ttl cache's file from my disk. is it possible?

Module doesn't delete if ttl expired

Hi All,

node-cache-manager-fs doesn't delete the cache files if ttl has expired. Is there any way to remove/delete those files using this node-cache-manager-fs node module(method).

Breaking change introduced in 1.0.5

#3 is a breaking change, as calls which worked previously using .get(key, cb) will now never have the callback called.

cache-manager-redis does indeed support 3 params, but has this important fallback:

    if (typeof options === 'function') {
      cb = options;
    }

https://github.com/dial-once/node-cache-manager-redis/blob/master/index.js#L91-L93

Support of larger file size

Is there a limit of how big the size of the cache can be?

We are trying to cache a file of ~100MB, it crashed. Around 90MB would be OK.

when caching a Buffer, the cache returns json

I noticed that when I cache an image contained in a buffer, when I hit the cache, I'm returned a json that looks like:

{type: "Buffer", data: .... }

Should this return the image buffer?

How to clear file cache when ttl has expired

Hi All,

I am using multi cache

cacheManager.multiCaching([memoryCache, diskCache], { isCacheableValue: isCacheableValue });

i have to delete the physical files if ttl has expired. Please advise anyone how to process next?

Library throws an exception when trying to recover cached values which cache files has been deleted

I had some values cached. I wanted to "invalidate" this cached values before they've reached the end of their TTL values, so I went and deleted the content of the "cache" directory.

The result is that, when trying to get the values, an exception is being thrown indicating that the ".dat" file can't be loaded.

errno ENOENT syscall open path diskcache/cache_1f24a906-5b4c-4357-90e4-2dba0aaa7f1f.dat

Shouldn't the library considered that there's no value cached and attempt to retrieve a new one?
I've used other filesystem cache libraries and that is the standard behavior.

Delete operation does not support options arg

The delete operation does not support options arg, which results in incompatibility with several caching managers such as node-cache-manager. To fix this, the first few lines of the DiskStore delete operation need to be rewritten to the following.

DiskStore.prototype.del = function (key, options, cb) {

  if (typeof options === 'function') {
        cb = options;
        options = null;
  }
  cb = typeof cb === 'function' ? cb : noop;

Data is not cached

I am using this with express session 'express-session-cache-manager' . My current code is:
`
var cacheManager = require('cache-manager')
var CacheManagerStore = require('express-session-cache-manager').default

var fsStore = require('cache-manager-fs');

var diskCache = cacheManager.caching({
store: fsStore,
options:
{ttl: 606024/* seconds /,
maxsize: 100010001000 / max size in bytes on disk */,
path:'diskcache',
preventfill:false,
}});

const sessionMiddleware = session({
store: new CacheManagerStore(diskCache),
secret: APP_SECRET,
resave: true,
saveUninitialized: true,
cookie: { maxAge: 2419200000}
})

app.use(sessionMiddleware)
`
This creates multiple cache files on each call.
What am i missing here?

ENOENT: no such file or directory on the odd occasion when cache retrieves up around 50 per second

The following error was observed when diskCache.get() was called at a frequency of around 50 per second. At lower rates like 5 to 10 per second we never observed the error.

[Error: ENOENT: no such file or directory, open '/tmp/diskcache/cache_99e4ffff-6392-45b1-ba69-21786819c784.dat'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/tmp/diskcache/cache_99e4ffff-6392-45b1-ba69-21786819c784.dat'
}

Following cache-manager code used in an AWS Lambda.

"use strict";

const cacheManager = require("cache-manager");
const fsStore = require("cache-manager-fs");

const diskCache = cacheManager.caching({
   store: fsStore,
   options: {
      ttl: 300,
      maxsize: 10485760,
      path: "/tmp/diskcache",
      preventfill: true
   }
});

The following code was being used when the error above was encountered.

   diskCache.get("SOME KEY FOR CACHE", function (err, result) {
      if (err) {
         console.error({ error: err, result: result });
      } else if (result) {
         // do something based on result from cache
      } else {
         var args = {
            "Bucket": "SOME AWS S3 BUCKET",
            "Key": "SOME AWS S3 KEY"
         };

         s3.getObject(args, function (err, data) {
            if (err) {
               console.error({ error: err, data: data });
            } else {
               var json = data.Body.toString("ascii");
               diskCache.set("SOME KEY FOR CACHE", json);
               // do something based on result from cache
            }
         });
      }
   });

Changing to the following will address and work-around the odd occurrence of the error, however we figured we'd raise this incase we're not using cache-manager correctly.

   diskCache.get("SOME KEY FOR CACHE", function (err, result) {
      if (result) {
         // do something based on result from cache
      } else {
         if (err) {
            console.error({ error: err, result: result });
         }
         var args = {
            "Bucket": "SOME AWS S3 BUCKET",
            "Key": "SOME AWS S3 KEY"
         };

         s3.getObject(args, function (err, data) {
            if (err) {
               console.error({ error: err, data: data });
            } else {
               var json = data.Body.toString("ascii");
               diskCache.set("SOME KEY FOR CACHE", json);
               // do something based on result from cache
            }
         });
      }
   });

We cannot use client side rendering

Hi,

When we use this module in client side rendering. we are getting the below issue,

"Module not found: Error: Can't resolve 'fs'"

Is there any idea to resolve this issue?

Does not refill cache on startup

No matter if I have preventfill true or false, every time I restart my script, the library does not fetch values from the existing cache on the filesystem. Here is some example code:

var cacheManager = require('cache-manager');
var fsStore = require('cache-manager-fs');
var cache = cacheManager.caching({
    store: fsStore, options: {
      ttl: 60*60,
      maxsize: 1000*1000*1000,
      fillcallback: function(){
        console.log('done filling from cache');
      }
    }
  });

var ttl = 30;

function getUser(id, cb) {
    console.log("Returning user from slow database.");
    setTimeout(function () {
        cb(null, {id: id, name: 'Bob'});
    }, 5000);
}

var userId = 123;
var key = 'user_' + userId;

cache.wrap(key, function (cb) {
    getUser(userId, cb);
}, {ttl: ttl}, function (err, user) {
    console.log(user);

    cache.wrap(key, function (cb) {
        getUser(userId, cb);
    }, function (err, user) {
        console.log(user);
    });
});

Every time I run my script.js, the getUser() method is called (which takes 5 seconds). It never seems to repopulate the cache on startup.

Am I missing something really obvious here?

Location of cache files?

// node cachemanager
var cacheManager = require('cache-manager');
// storage for the cachemanager
var fsStore = require('cache-manager-fs');
// initialize caching on disk
var diskCache = cacheManager.caching({store: fsStore, options: {path:'diskcache', preventfill:true}});

Everything works fine, but I don't know the location of cache files.

I search 'diskcache' directory but nothing.

find / -type d -name 'diskcache'

Can you help me?

Missing parameter in get method

Hi,

I am using your llibrary and got some problems to make it work in my app. I noticed that the get method in your code has only 2 parameters, while otherwise it has 3 (see cache-manager-redis).

Alse when invoked in cache-manager wrap function 3 parameters are used

 self.store.get(key, options, function(err, result) { ... }

When options is null then cb is set to noop:

DiskStore.prototype.get = function (key, cb) {
cb = typeof cb === 'function' ? cb : noop;
    ...
}

Should I fix it and create a pull request or can you fix it yourself?

Regards,
Karel

hotelde / node-cache-manager-fs Goto Github PK

node-cache-manager-fs's Introduction

node-cache-manager-fs

Node Cache Manager store for Filesystem

Installation

Features

Usage example

Single store

Options

Installation

Tests

Code Coverage

License

node-cache-manager-fs's People

Contributors

Stargazers

Watchers

Forkers

node-cache-manager-fs's Issues

Recommend Projects

Recommend Topics

Recommend Org