Giter VIP home page Giter VIP logo

fs-write-stream-atomic's Introduction

fs-write-stream-atomic

Like fs.createWriteStream(...), but atomic.

Writes to a tmp file and does an atomic fs.rename to move it into place when it's done.

First rule of debugging: It's always a race condition.

USAGE

var fsWriteStreamAtomic = require('fs-write-stream-atomic')
// options are optional.
var write = fsWriteStreamAtomic('output.txt', options)
var read = fs.createReadStream('input.txt')
read.pipe(write)

// When the write stream emits a 'finish' or 'close' event,
// you can be sure that it is moved into place, and contains
// all the bytes that were written to it, even if something else
// was writing to `output.txt` at the same time.

fsWriteStreamAtomic(filename, [options])

  • filename {String} The file we want to write to
  • options {Object}
    • chown {Object} User and group to set ownership after write
      • uid {Number}
      • gid {Number}
    • encoding {String} default = 'utf8'
    • mode {Number} default = 0666
    • flags {String} default = 'w'

fs-write-stream-atomic's People

Contributors

codeimpossible avatar iarna avatar isaacs avatar linusu avatar othiym23 avatar stefanmb avatar thefourtheye avatar zkat avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fs-write-stream-atomic's Issues

1.0.6 and 1.0.7 breaks bower

bower users are seeing a variety of issues since bumping the dependency on fs-write-stream-atomic up from 1.0.5 to 1.0.7:

bower/bower#2118 (known and proven)
bower/bower#2101 (suspected)
https://ci.appveyor.com/project/bodil/pulp/build/1.0.67 (suspected)

This is an example of the sorts of errors that are being seen, although there are several others mentioned. Note that they occur seemingly randomly, perhaps due to timing issues or an improperly handled async operation:

bower install
...
bower paper-menu#^1.0.0                             extract archive.tar.gz
bower iron-validatable-behavior#^1.0.0              extract archive.tar.gz
bower platinum-https-redirect#^1.0.0                extract archive.tar.gz
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: ENOENT, rename 'C:\Users\Ken\AppData\Local\Temp\mypc-Ken\bower\cefb8f774d98384ed30dec610f03a72d-174564-wZPTqe\archive.tar.gz.207942742'
    at Error (native)

Downgrading to 1.0.5 resolves the issue (tested for bower/bower#2118)

Support append mode?

Would be useful if this module could support appends { flags: 'a' }. I'm not sure how this would be implemented though. I guess you'd have to copy the entire destination file each time?

Leak temp file when source emit error

When the readable stream source emit an error :

  • fs-write-stream-atomic doesn't remove the temp file.
  • fs-write-stream-atomic doesn't emit error.

This file leaks and cannot be removed.

I've write a test to illustrate this issue :

var should = require('should');
var fs = require('fs');
var fsWriteStreamAtomic = require('fs-write-stream-atomic');
var stream = require('stream');
var util = require('util');
var uuid = require('node-uuid');
var path = require('path');

function ReadableStream(options) {
  stream.Readable.call(this, options);
  this.options = options;
  this._total = options && options.total || 1000;
  this._index = 1;
}
util.inherits(ReadableStream, stream.Readable);

ReadableStream.prototype._emitError = function() {
  var self = this;
  process.nextTick(function () {
    self.emit('error', new Error('stoped stream'));
  });
};

ReadableStream.prototype._read = function() {
  var i = this._index++;
  if (i > this._total) {
    this.push(null);
  } else if(i > this.options.stopAt) {
    this._emitError();
  } else {
    var str = '' + i;
    var buf = new Buffer(str, 'ascii');
    this.push(buf);
  }
};

var DIR = path.join(__dirname, 'samples');
var FILE_TO_WRITE = path.join(DIR, 'MyTestFile.txt');

describe('Fs-write-stream-atomic: ', function() {
  it('works with no error', function(done){
    var rs = new ReadableStream({total: 1000}); 
    // A readable stream emit 1000 chars and finish with no error
    var ws = fsWriteStreamAtomic(FILE_TO_WRITE);
    rs.pipe(ws);
    ws.on('finish', function(){
      console.log('finished');
      fs.unlinkSync(FILE_TO_WRITE);
      fs.readdir(DIR, function(err, files){
        if(!err){
          for(var i=0; i< files.length; i++){
            should(files[i]).not.containEql('MyTestFile.txt');
          }
        }
        done();
      });
    });
  });

  it('leak temp file when source emit error', function(done){
    var rs = new ReadableStream({total: 1000, stopAt:600});
    // A readable stream emit error at 600 chars
    var ws = fsWriteStreamAtomic(FILE_TO_WRITE);
    rs.pipe(ws);
    ws.on('finish', function(){
      console.log('finished');
      fs.unlinkSync(FILE_TO_WRITE);
      done();
    });
    ws.on('error', function(err){
      console.log('error write stream' + err);
      done();
    });
    rs.on('error', function(err){
      console.log('error readable stream' + err);
      fs.readdir(DIR, function(err, files){
        if(!err){
          for(var i=0; i< files.length; i++){
            should(files[i]).not.containEql('MyTestFile.txt');
          }
        }
        done();
      });
    });
  });
});

When I start the test.

npm test

Result :

Fs-write-stream-atomic: 
finished
    โœ“ works with no error
error readable streamError: stoped stream
    1) leak temp file when source emit error


  1 passing (37ms)
  1 failing

  1) Fs-write-stream-atomic:  leak temp file when source emit error:
     Uncaught AssertionError: expected 'MyTestFile.txt.1272449043' not to contain 'MyTestFile.txt' (false negative fail)
      at Assertion.fail (node_modules/should/lib/assertion.js:92:17)
      at Assertion.Object.defineProperty.value [as containEql] (node_modules/should/lib/assertion.js:174:17)

fs-write-stream-atomic leave a file "MyTestFile.txt.1272449043" in the directory

Fs-write-stream-atomic-on-source-emit-error.zip

Not FIPS compliant; breaks npm when FIPS-compliance compiled

When NodeJS is compiled for FIPS-compliance, fs-write-stream-atomic actually breaks NPM.

When NPM attempts to write the local cache file .cache.json using WriteStream from the library, it throws the following:

fips_md.c(146): OpenSSL internal error, assertion failed: Digest update previous FIPS forbidden algorithm error ignored
Aborted

Instead of hardcoding md5 hash for the temporary name, consider moving to something higher like sha1 at a minimum or check for FIPs compliance and make it variable. I tested by moving the function createhash() call to sha1 and it proceeds fine.

option to specify where to write the temp file

It would be very useful to be able to set the location where the temp file is written to, could this be added as an option? And if it isn't specified it'll write to the default location where it currently writes.

Remove file when process is canceled

Currently, if you're doing stuff using fsWriteStreamAtomic() and cancels the process the temp file will remain in place. Optimally it'd clean up any temp files on exit.

Also, are there any reasons to not write the temp files to os.tmpdir()?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.