Giter VIP home page Giter VIP logo

vinyl's Introduction

vinyl

NPM version Downloads Build Status Coveralls Status

Virtual file format.

What is Vinyl?

Vinyl is a very simple metadata object that describes a file. When you think of a file, two attributes come to mind: path and contents. These are the main attributes on a Vinyl object. A file does not necessarily represent something on your computer’s file system. You have files on S3, FTP, Dropbox, Box, CloudThingly.io and other services. Vinyl can be used to describe files from all of these sources.

What is a Vinyl Adapter?

While Vinyl provides a clean way to describe a file, we also need a way to access these files. Each file source needs what I call a "Vinyl adapter". A Vinyl adapter simply exposes a src(globs) and a dest(folder) method. Each return a stream. The src stream produces Vinyl objects, and the dest stream consumes Vinyl objects. Vinyl adapters can expose extra methods that might be specific to their input/output medium, such as the symlink method vinyl-fs provides.

Usage

var Vinyl = require('vinyl');

var jsFile = new Vinyl({
  cwd: '/',
  base: '/test/',
  path: '/test/file.js',
  contents: Buffer.from('var x = 123'),
});

API

new Vinyl([options])

The constructor is used to create a new instance of Vinyl. Each instance represents a separate file, directory or symlink.

All internally managed paths (cwd, base, path, history) are normalized and have trailing separators removed. See Normalization and concatenation for more information.

Options may be passed upon instantiation to create a file with specific properties.

options

Options are not mutated by the constructor.

options.cwd

The current working directory of the file.

Type: String

Default: process.cwd()

options.base

Used for calculating the relative property. This is typically where a glob starts.

Type: String

Default: options.cwd

options.path

The full path to the file.

Type: String

Default: undefined

options.history

Stores the path history. If options.path and options.history are both passed, options.path is appended to options.history. All options.history paths are normalized by the file.path setter.

Type: Array

Default: [] (or [options.path] if options.path is passed)

options.stat

The result of an fs.stat call. This is how you mark the file as a directory or symbolic link. See isDirectory(), isSymbolic() and fs.Stats for more information.

Type: fs.Stats

Default: undefined

options.contents

The contents of the file. If options.contents is a ReadableStream, it is wrapped in a cloneable-readable stream.

Type: ReadableStream, Buffer, or null

Default: null

options.{custom}

Any other option properties will be directly assigned to the new Vinyl object.

var Vinyl = require('vinyl');

var file = new Vinyl({ foo: 'bar' });
file.foo === 'bar'; // true

Instance methods

Each Vinyl object will have instance methods. Every method will be available but may return differently based on what properties were set upon instantiation or modified since.

file.isBuffer()

Returns true if the file contents are a Buffer, otherwise false.

file.isStream()

Returns true if the file contents are a Stream, otherwise false.

file.isNull()

Returns true if the file contents are null, otherwise false.

file.isDirectory()

Returns true if the file represents a directory, otherwise false.

A file is considered a directory when:

  • file.isNull() is true
  • file.stat is an object
  • file.stat.isDirectory() returns true

When constructing a Vinyl object, pass in a valid fs.Stats object via options.stat. If you are mocking the fs.Stats object, you may need to stub the isDirectory() method.

file.isSymbolic()

Returns true if the file represents a symbolic link, otherwise false.

A file is considered symbolic when:

  • file.isNull() is true
  • file.stat is an object
  • file.stat.isSymbolicLink() returns true

When constructing a Vinyl object, pass in a valid fs.Stats object via options.stat. If you are mocking the fs.Stats object, you may need to stub the isSymbolicLink() method.

file.clone([options])

Returns a new Vinyl object with all attributes cloned.

By default custom attributes are cloned deeply.

If options or options.deep is false, custom attributes will not be cloned deeply.

If file.contents is a Buffer and options.contents is false, the Buffer reference will be reused instead of copied.

file.inspect()

Returns a formatted-string interpretation of the Vinyl object. Automatically called by node's console.log.

Instance properties

Each Vinyl object will have instance properties. Some may be unavailable based on what properties were set upon instantiation or modified since.

file.contents

Gets and sets the contents of the file. If set to a ReadableStream, it is wrapped in a cloneable-readable stream.

Throws when set to any value other than a ReadableStream, a Buffer or null.

Type: ReadableStream, Buffer, or null

file.cwd

Gets and sets current working directory. Will always be normalized and have trailing separators removed.

Throws when set to any value other than non-empty strings.

Type: String

file.base

Gets and sets base directory. Used for relative pathing (typically where a glob starts). When null or undefined, it simply proxies the file.cwd property. Will always be normalized and have trailing separators removed.

Throws when set to any value other than non-empty strings or null/undefined.

Type: String

file.path

Gets and sets the absolute pathname string or undefined. Setting to a different value appends the new path to file.history. If set to the same value as the current path, it is ignored. All new values are normalized and have trailing separators removed.

Throws when set to any value other than a string.

Type: String

file.history

Array of file.path values the Vinyl object has had, from file.history[0] (original) through file.history[file.history.length - 1] (current). file.history and its elements should normally be treated as read-only and only altered indirectly by setting file.path.

Type: Array

file.relative

Gets the result of path.relative(file.base, file.path).

Throws when set or when file.path is not set.

Type: String

Example:

var file = new File({
  cwd: '/',
  base: '/test/',
  path: '/test/file.js',
});

console.log(file.relative); // file.js

file.dirname

Gets and sets the dirname of file.path. Will always be normalized and have trailing separators removed.

Throws when file.path is not set.

Type: String

Example:

var file = new File({
  cwd: '/',
  base: '/test/',
  path: '/test/file.js',
});

console.log(file.dirname); // /test

file.dirname = '/specs';

console.log(file.dirname); // /specs
console.log(file.path); // /specs/file.js

file.basename

Gets and sets the basename of file.path.

Throws when file.path is not set.

Type: String

Example:

var file = new File({
  cwd: '/',
  base: '/test/',
  path: '/test/file.js',
});

console.log(file.basename); // file.js

file.basename = 'file.txt';

console.log(file.basename); // file.txt
console.log(file.path); // /test/file.txt

file.stem

Gets and sets stem (filename without suffix) of file.path.

Throws when file.path is not set.

Type: String

Example:

var file = new File({
  cwd: '/',
  base: '/test/',
  path: '/test/file.js',
});

console.log(file.stem); // file

file.stem = 'foo';

console.log(file.stem); // foo
console.log(file.path); // /test/foo.js

file.extname

Gets and sets extname of file.path.

Throws when file.path is not set.

Type: String

Example:

var file = new File({
  cwd: '/',
  base: '/test/',
  path: '/test/file.js',
});

console.log(file.extname); // .js

file.extname = '.txt';

console.log(file.extname); // .txt
console.log(file.path); // /test/file.txt

file.symlink

Gets and sets the path where the file points to if it's a symbolic link. Will always be normalized and have trailing separators removed.

Throws when set to any value other than a string.

Type: String

Vinyl.isVinyl(file)

Static method used for checking if an object is a Vinyl file. Use this method instead of instanceof.

Takes an object and returns true if it is a Vinyl file, otherwise returns false.

Note: This method uses an internal flag that some older versions of Vinyl didn't expose.

Example:

var Vinyl = require('vinyl');

var file = new Vinyl();
var notAFile = {};

Vinyl.isVinyl(file); // true
Vinyl.isVinyl(notAFile); // false

Vinyl.isCustomProp(property)

Static method used by Vinyl when setting values inside the constructor or when copying properties in file.clone().

Takes a string property and returns true if the property is not used internally, otherwise returns false.

This method is useful for inheritting from the Vinyl constructor. Read more in Extending Vinyl.

Example:

var Vinyl = require('vinyl');

Vinyl.isCustomProp('sourceMap'); // true
Vinyl.isCustomProp('path'); // false -> internal getter/setter

Normalization and concatenation

Since all properties are normalized in their setters, you can just concatenate with /, and normalization takes care of it properly on all platforms.

Example:

var file = new File();
file.path = '/' + 'test' + '/' + 'foo.bar';

console.log(file.path);
// posix => /test/foo.bar
// win32 => \\test\\foo.bar

But never concatenate with \, since that is a valid filename character on posix system.

Extending Vinyl

When extending Vinyl into your own class with extra features, you need to think about a few things.

When you have your own properties that are managed internally, you need to extend the static isCustomProp method to return false when one of these properties is queried.

var Vinyl = require('vinyl');

var builtInProps = ['foo', '_foo'];

class SuperFile extends Vinyl {
  constructor(options) {
    super(options);
    this._foo = 'example internal read-only value';
  }

  get foo() {
    return this._foo;
  }

  static isCustomProp(name) {
    return super.isCustomProp(name) && builtInProps.indexOf(name) === -1;
  }
}

// `foo` won't be assigned to the object below
new SuperFile({ foo: 'something' });

This makes properties foo and _foo skipped when passed in options to constructor(options) so they don't get assigned to the new object and override your custom implementation. They also won't be copied when cloning. Note: The _foo and foo properties will still exist on the created/cloned object because you are assigning _foo in the constructor and foo is defined on the prototype.

Same goes for clone(). If you have your own internal stuff that needs special handling during cloning, you should extend it to do so.

License

MIT

vinyl's People

Contributors

amilajack avatar blaimi avatar bnjmnt4n avatar bobintornado avatar bridgear avatar danielhusar avatar darsain avatar edwardbetts avatar erikkemperman avatar felixrabe avatar hughsk avatar jeremyruppel avatar jmm avatar laurelnaiad avatar lpinca avatar mcollina avatar nfroidure avatar pascalduez avatar pdehaan avatar pgilad avatar phated avatar popomore avatar shinnn avatar shuhei avatar soslan avatar sttk avatar t1st3 avatar vvakame avatar whyrusleeping avatar yocontra avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

vinyl's Issues

Normalize paths upon construction without stat object

As I brought up in @darsain's PR, I'm not sure we should normalize the history of an object upon construction if no stat object is passed. If the history represents a directory but no stat object is passed, will the paths be normalized incorrectly?

I'd like to get some feedback on this.

Possibly unhandled Error: path should be string

I'm getting the error below, and I've got no idea what in my gulpfile is causing it. Any ideas as to what it could be?

Possibly unhandled Error: path should be string
    at File.Object.defineProperty.set (/Users/benjaminrh/appem/node_modules/gulp/node_modules/vinyl-fs/node_modules/vinyl/index.js:166:41)
    at Function.assign (/Users/benjaminrh/appem/node_modules/gulp-cache/node_modules/lodash-node/modern/objects/assign.js:63:21)
    at /Users/benjaminrh/appem/node_modules/gulp-cache/lib/TaskProxy.js:20:19
    at tryCatch1 (/Users/benjaminrh/appem/node_modules/gulp-cache/node_modules/bluebird/js/main/util.js:64:19)
    at Promise$_callHandler [as _callHandler] (/Users/benjaminrh/appem/node_modules/gulp-cache/node_modules/bluebird/js/main/promise.js:708:13)
    at Promise$_settlePromiseFromHandler [as _settlePromiseFromHandler] (/Users/benjaminrh/appem/node_modules/gulp-cache/node_modules/bluebird/js/main/promise.js:724:18)
    at Promise$_settlePromiseAt [as _settlePromiseAt] (/Users/benjaminrh/appem/node_modules/gulp-cache/node_modules/bluebird/js/main/promise.js:896:14)
    at Promise$_fulfillPromises [as _fulfillPromises] (/Users/benjaminrh/appem/node_modules/gulp-cache/node_modules/bluebird/js/main/promise.js:1041:14)
    at Async$_consumeFunctionBuffer [as _consumeFunctionBuffer] (/Users/benjaminrh/appem/node_modules/gulp-cache/node_modules/bluebird/js/main/async.js:64:12)
    at Async$consumeFunctionBuffer (/Users/benjaminrh/appem/node_modules/gulp-cache/node_modules/bluebird/js/main/async.js:37:14)

Improve the API for File objects

It took me way too long to find out how to get the contents of a file as a string (.contents.toString('utf8')). It would be really helpful to have this in the readme.

Thanks!

Virtual directories

Working on path normalization I've noticed that currently, file.isDirectory() check looks like this:

File.prototype.isDirectory = function() {
  return this.isNull() && this.stat && this.stat.isDirectory();
};

This makes creating purely virtual directories quite awkward, since you basically have to mock the fs.stat object. How about we extend it with something like:

File.prototype.isDirectory = function() {
  return this._isDirectory || this.isNull() && this.stat && this.stat.isDirectory();
};

Where this._isDirectory will be a flag configurable only on creation via constructor options:

new Vinyl({isDirectory: true});

Readable only, since there is no meaningful folder to file conversion (and the other way around).

Reading a file into a vinyl

What's the best way of instantiating a vinyl based on a file on disk. I looked at vinyl-fs but I don't think it's what I'm looking for. Maybe I'm being stupid / missing something but in some tests for my gulp plugins, I do stuff like:

new Vinyl({
//...
    contents: fs.readFileSync('./fixtures/test.xml');
});

What I'm for is something that I can pass a path to and get a new Vinyl in return.

Changelog?

I'm seeing some behavior changes between vinyl 0.5 and 1.0, and between vinyl-fs 1 and 2. Neither project has a changelog or release notes - could there be a little more information that lists what breaking changes occur?

"Error: Cannot find module './lib/cloneBuffer'" when vinyl is a dependency of gulp-util

Here is part of the console output log of a Jenkins build of my project. I have inserted two find commands in the build script just before invoking "gulp default" to show why vinyl's lib/cloneBuffer.js cannot be found. There are lots of copies of the file, just none on that subtree of node_modules.
...

  • find -iname 'clonebuffer'
    ./node_modules/gulp-jasmine/node_modules/gulp-util/node_modules/vinyl/coverage/lcov-report/vinyl/lib/cloneBuffer.js.html
    ./node_modules/gulp-jasmine/node_modules/gulp-util/node_modules/vinyl/lib/cloneBuffer.js
    ./node_modules/gulp-jasmine/node_modules/gulp-util/node_modules/vinyl/test/cloneBuffer.js
    ./node_modules/gulp-util/node_modules/vinyl/lib/cloneBuffer.js
    ./node_modules/gulp-util/node_modules/vinyl/test/cloneBuffer.js
    ./node_modules/gulp/node_modules/vinyl-fs/node_modules/vinyl/coverage/lcov-report/vinyl/lib/cloneBuffer.js.html
    ./node_modules/gulp/node_modules/vinyl-fs/node_modules/vinyl/lib/cloneBuffer.js
    ./node_modules/gulp/node_modules/vinyl-fs/node_modules/vinyl/test/cloneBuffer.js
    ./node_modules/gulp-karma/node_modules/gulp-util/node_modules/vinyl/lib/cloneBuffer.js
    ./node_modules/gulp-karma/node_modules/gulp-util/node_modules/vinyl/test/cloneBuffer.js
    ./node_modules/gulp-uglify/node_modules/vinyl/lib/cloneBuffer.js
    ./node_modules/gulp-uglify/node_modules/vinyl/test/cloneBuffer.js
    ./node_modules/gulp-ng-annotate/node_modules/gulp-util/node_modules/vinyl/lib/cloneBuffer.js
    ./node_modules/gulp-ng-annotate/node_modules/gulp-util/node_modules/vinyl/test/cloneBuffer.js
    ./node_modules/gulp-ng-html2js/node_modules/gulp-util/node_modules/vinyl/coverage/lcov-report/vinyl/lib/cloneBuffer.js.html
    ./node_modules/gulp-ng-html2js/node_modules/gulp-util/node_modules/vinyl/lib/cloneBuffer.js
    ./node_modules/gulp-ng-html2js/node_modules/gulp-util/node_modules/vinyl/test/cloneBuffer.js
    ./node_modules/gulp-minify-css/node_modules/gulp-util/node_modules/vinyl/coverage/lcov-report/vinyl/lib/cloneBuffer.js.html
    ./node_modules/gulp-minify-css/node_modules/gulp-util/node_modules/vinyl/lib/cloneBuffer.js
    ./node_modules/gulp-minify-css/node_modules/gulp-util/node_modules/vinyl/test/cloneBuffer.js
  • find /usr/local/lib/node_modules -iname 'clonebuffer'
    /usr/local/lib/node_modules/gulp/node_modules/vinyl-fs/node_modules/vinyl/coverage/lcov-report/vinyl/lib/cloneBuffer.js.html
    /usr/local/lib/node_modules/gulp/node_modules/vinyl-fs/node_modules/vinyl/lib/cloneBuffer.js
    /usr/local/lib/node_modules/gulp/node_modules/vinyl-fs/node_modules/vinyl/test/cloneBuffer.js
    /usr/local/lib/node_modules/gulp/node_modules/gulp-util/node_modules/vinyl/coverage/lcov-report/vinyl/lib/cloneBuffer.js.html
    /usr/local/lib/node_modules/gulp/node_modules/gulp-util/node_modules/vinyl/lib/cloneBuffer.js
    /usr/local/lib/node_modules/gulp/node_modules/gulp-util/node_modules/vinyl/test/cloneBuffer.js
  • gulp default

Error: Cannot find module './lib/cloneBuffer'
at Function.Module._resolveFilename (module.js:338:15)
at Function.Module._load (module.js:280:25)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object. (/var/lib/jenkins/workspace/****/node_modules/gulp/node_modules/gulp-util/node_modules/vinyl/index.js:4:19)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)

normalize paths

In testing on Windows, I found that the slashes in the path history were incorrect. It seems these are coming from node-glob but we aren't doing any translation.

Not sure where we should do this, but in the meantime, I am normalizing in vinyl-fs before we construct the Vinyl object because it requires the least releases.

cc @contra

node-task

Would you be willing to migrate this to the node-task org? I think this specification currently fits the needs for all task runners and I don't want to duplicate the effort. We can call it vinyl, or record, whichever you prefer.

file.pipe: options and emitting an "end" for buffered/null contents [Feature]

When using file.pipe, the writable stream will be closed where file.contents is a stream; however, neither buffer or null contents will reach an end event. Since it appears that file.pipe is a bridge for streams, could it mimic a readable stream by ending the writable with data (buffer) or without (null)? Further, this could be conditional to an options.end to match node's readable.pipe(destination, [options]) api.

File.prototype.pipe = function (stream, options) {
    if (this.isStream()) {
        return this.contents.pipe(stream, options);
    }
    if (this.isBuffer()) {
        stream.write(this.contents);
    }
    if (!(options && options.end === false) && stream !== process.stdout && stream !== process.stderr) {
        stream.end();
    }
    return stream;
}

This would allow for abstract handling of file.pipe in plugins, such as:

// Some generic, async plugin
module.exports = es.map(function (file, callback) {
    file.pipe(es.wait(function (err, contents) {
        // async handling here...
        callback(err, file);
    }));
});

Unfortunately, this now leaves an exception for stream contents as the only case that is unable to pipe repeatedly. If even necessary, this could be remedied by utilizing a stream.PassThrough to "reset" the flowing contents stream. With this approach, File would act as a stream provider, in much the same way that fs.createReadStream provides streams.

Cloned stream file content is not drained

Steps to reproduce:

  1. Create file large enough to exceed highWatermark for file streams. In my case it’s 131KB file.
  2. This code works fine:
var vfs = require('vinyl-fs');

vfs.src('./large-file.txt', {buffer: false})
.pipe(vfs.dest('./dest'))
.on('end', function() {
    console.log('done!');
});

but this doesn’t, there’s no done! message and file isn’t written:

var vfs = require('vinyl-fs');
var through = require('through2');

vfs.src('./large-file.txt', {buffer: false})
.pipe(through.obj(function(file, enc, next) {
    next(null, file.clone());
}))
.pipe(vfs.dest('./dest'))
.on('end', function() {
    console.log('done!');
});

Tested on node v0.10.38 and iojs v.2.3.1

After debugging, I found out that file contents reading stops after second 64KB chunk, looks like writing stream is not able to properly drain and resume reading.

Playing with File.clone(), I’ve commented this line https://github.com/wearefractal/vinyl/blob/master/index.js#L68 and this test case worked fine.

Do you know if there are side-effects of this solution?

Error: Cannot find module './lib/cloneBuffer'

When I run this locally on OSX, I don't encounter this problem, but inside of Docker when this runs, I get:

NPM: 2.14.4
Node: 4.1.1
Gulp: 3.9.0

18:32:49 + ./node_modules/gulp/bin/gulp.js default
18:32:49 module.js:338
18:32:49     throw err;
18:32:49     ^
18:32:49 
18:32:49 Error: Cannot find module './lib/cloneBuffer'
18:32:49     at Function.Module._resolveFilename (module.js:336:15)
18:32:49     at Function.Module._load (module.js:286:25)
18:32:49     at Module.require (module.js:365:17)
18:32:49     at require (module.js:384:17)
18:32:49     at Object.<anonymous> (/dashboard/node_modules/gulp/node_modules/gulp-util/node_modules/vinyl/index.js:4:19)
18:32:49     at Module._compile (module.js:434:26)
18:32:49     at Object.Module._extensions..js (module.js:452:10)
18:32:49     at Module.load (module.js:355:32)
18:32:49     at Function.Module._load (module.js:310:12)
18:32:49     at Module.require (module.js:365:17)

Any thoughts on what might be causing this?

Potential memory leak

In nodejs/readable-stream#202, @mcollina brought up the fact that we might have memory leaks when a Vinyl object with streaming content is cloned. The suggested solution was to delay creation until the streams were needed.

I'm opening this issue so we can discuss further.

Enhanced stat object

Opening this as the solution to a bunch of other issues.

This to achieve:

  • *nix-like behavior for atime/mtime/etc properties (#72)
  • Virtual directories (#100)
  • User-friendly mode property (gulpjs/vinyl-fs#112)
  • Construct with always-valid properties (new fs.Stats doesn't do this)
  • Flag remote files (isRemote) and avoid path normalization or do URI normalization (#127)
  • More?

path.extname does not honor extensions with more than one dot

path.extname of foo.min.css becomes .css instead of the expected .min.css

I use https://github.com/MaKleSoft/gulp-style-modules where I had to use the following configuration to make it work:

.pipe(stylemod({
  filename: function(file) {
    var fixedPath = file.path.replace('.min', '');
    var extension = path.extname(file.path);
    return path.basename(fixedPath, extension) + "-styles";
  }
}))

with input app/bower_components/fullcalendar/dist/fullcalendar.min.css

Community information on vinyl ecosystem and vinyl adapters

Hello.

Very nice project. Had something very similar to this a few years back that was unfortunately shut-down / not open-sourced.

Is there a central location or wiki to list / discover new adapters for Vinyl? Is there a development plan / timeline / wishlist for new adapters? Is there a contribution doc for creating adapters?

Searching npm I have found:

https://github.com/wearefractal/vinyl-fs
https://github.com/nkostelnik/gulp-s3
https://github.com/jden/node-vinyl-github
https://github.com/floatdrop/vinyl-git
https://github.com/doowb/vinyl-dat

Are there more adapters?

Creating an Adapter

I'm curious if a vinyl file adapter might be a good way to create a bundling system. So, in place of gulp.dest after transforming files, instead myBundle.dest(). Is this a legitimate use case? If so, can you point me to any resources that could be used in implementing the adapter? Thanks!

What is Vinyl for?

Stumbled across this while looking into Gulp. Could I get an ELI5 on Vinyl and its intended use?

[question] vinyl virtual adapter

Admittedly a question: is there a module that works with vinyl that would allow piping files from a stream of vinyl objects into a virtual filesystem? I'm working on a live reloading server for which it'd be useful to never save the results of a stream to disk, but to keep them in memory so that they can be served over HTTP.

Test refactor

Need to bring the tests inline with other repos, switch from should to expect.

Re-hydrating a cloned file from disk

It looks like #24 significantly changed the way path is stored on a File instance. Ever since this change, you can't re-"hydrate" a File object from a clone because the clone result doesn't include the path property, but the File constructor requires a path and won't respect the passed in history.

I'm left with a sort of hacky

// Storing a copy of the file to disk for later "hydration"
var copy = _.clone(file);

if (_.has(file, 'path') && !_.has(copy, 'path')) {
  copy.path = file.path
}

// .. write the file

I think you should check for the passed in history and use it if present, otherwise fall back to your current this.history = file.path ? [file.path] : []; line.

Great Thanks and Proposal

I very love the KISS of Gulp 4. I am too lazy to re-invent the wheel. But I have not reused the codes of vinyl when I'm making a project base on Gulp 4. So I had to create the abstract-file.
I am tired to type useless/unreadable punctuation(eg, braces), so I use coffee-script too.

  • abstract property-manager to manage the file attributes.
  • abstract the load(stats, contents) supports
  • abstract folder/directory supports: It's the array of file object and read-dir-stream
  • abstract path.js
  • abstract fs: It should apply via AbstractFile.fs = fs
  • abstract cwd: It should apply via fs.cwd = process.cwd

The usage of abstract-file is here: https://github.com/snowyu/custom-file.js

I hope we can work together, if you do not hate coffee-script.
Anyway, great thanks.

Constructor behavior when path and history are provided

I noticed in the PR (#94) that history is reset if a vinyl object is constructed with both a path and history property; e.g. new Vinyl({ path: '/pets/cat', history: ['/pets/dog'] }) has ['/pets/cat'] as the history.

Shouldn't the constructor append the new path to the history if we specify both? In my example, it would result in ['/pets/dog', '/pets/cat'] as the history on the new object.

/cc @contra

`vinyl.toJSON` method?

Hello.

Was wondering if there is any support for turning a vinyl File into a JSON object? I couldn't seem to find anything.

Obviously I can write a simple serialization method, but I was wondering if it exists anywhere else in the tool-chain. If not, would you want to add File.toJSON to core?

Next release

Is there a time frame for the next release? I'm interested in using the new dirname, basename, extname features.

Could someone explain what this does in layman's terms?

I've read the docs and that article twice, but I still don't really understand what this does and how it does it. I get the fact that it somehow adds information about the file and the path to the stream, but that's about everything I know, and it seems like it does more than that

vinyl-diff / vinyl-compare

Are there currently any tools for comparing two sets of vinyl files and returning a diff between the two based on checksum?

I have found https://github.com/sindresorhus/gulp-changed/ , but this seems to be coupled directly to file-system and I seek to perform diff of vinyl files ( i.e. compare local files to remote files )

Any suggestions?

isSymlink property

In vinyl-fs, we look at a symlink property that doesn't seem to be part of vinyl at all, so I guess we just attach it in vinyl-fs. I think this should be bubbled up to vinyl and we could add a isSymlink or isSymbolic method like isNull, etc.

@contra thoughts?

Document/implement setting arbitrary keys on Vinyl objects

Vinyl objects should have some way of setting arbitrary data keys in addition to the builtin ones, like path, stem, etc. This would be incredibly useful in certain cases: for example, the ability to parse out YAML Front Matter into a Vinyl object that could then be consumed by downstream Gulp plugins would be super powerful.

I feel like there should be some way to do this already, but I've tried a bunch of different ways with no luck. So, this process should be documented or, if it's not actually possible, it should be implemented.

vinyl-http - send and receive vinyl over http

Hello again.

I had the need to easily push vinyl between two nodes over HTTP, so I created a basic module vinyl-http

Not sure if something like this exists in the ecosystem yet. This could be comparable to gulp-sftp, except files are moved over HTTP instead of SSH.

Module is 0.0.0 right now. I am using this as part of a larger project and plan to add more features as I go. Since the vinyl-http remote endpoint can be implemented as middleware, existing HTTP tools can be used ( like basicAuth );

Feedback would be appreciated.

Example Usage

Start the server

var server = require('vinyl-http').createServer();
server.listen(8888);

Sending files

var map = require('map-stream');
var fs = require('vinyl-fs');
var vhttp = require('vinyl-http');

var log = function(file, cb) {
  console.log(file.path);
  cb(null, file);
};

fs.src('test/fixtures/**') 
.pipe(vhttp.dest('/upload', {
  uri: "http://localhost:8888/"
}))
.pipe(map(log))

Retrieving files

var map = require('map-stream');
var fs = require('vinyl-fs');

var log = function(file, cb) {
  console.log(file.path);
  cb(null, file);
};

vhttp.src('test/fixtures/**', {
  uri: "http://localhost:8888/"
})
.pipe(map(log))
.pipe(vfs.dest('./download'));

Constructor of File and contents/stats

I really do not understand, why File is just a object with fields. Is there any reason, that it does not fills contents and stats? Seems that src at start filling content of emitting File object, so lazy loading isn't a reason.

node v0.8 support

is the only reason this doesn't support node v0.8.x because of the streams? if so couldn't readable-stream be used to polyfill until v0.12 has been released and out for a bit.

file type deserves its own property

Hello,

It seems to me that there is a missing concept in the vinyl spec, that of the current file type of the "virtual file". Clearly this concept is important. Glup plugins can, and often do, change the type of a file, for example from .scss to .css. Right now the way this change is "recorded" is by changing the path property of the vinyl object. However the path then contains the path of a file that does not (and likely never will) exist, and there is no way to map the vinyl "virtual file" back to its original source file.

Although in the gulp use case this does not seem to cause problems, for a more generalized use case it seems quite brittle to merge what are semantically two different concepts, path and fileType, into the same property. For example, perhaps something needs to be done with vinyl objects after they have been transformed by plugins based on the original source file to which they correspond. What are people's thoughts on adding a separate file type property so that these two concepts do not collide?

Having a buffer option in vinyl too

It is part of this PR #4 but since it's pending more input i think we should consider to add it to current vinyl version.

When a vinyl object have no content (isNull() === true) some plugins will lack informations on using buffers or streams to create file.contents.

By example, for plugins like gulp.watch that create contents from null object, the lack of it's information will lead to this kind of repetitive code:

gulp.src('**', {buffer: false}).pipe(gulp.watch({buffer: false});

Adding a buffer option to vinyl and filing her at the vinyl-fs level could avoid this issue.

On my side, i think that isBuffer() and isStream() could be mapped with this option and the file.contents setter could be more consistant by refusing to set a buffer to file.contents in streaming mode and a stream in buffer mode.

file.relative for base='/' returns non-leading slash prefix

Currently file.relative returns path.relative(file.base, file.path).

This works great, but when you want a plugin to use file.relative and supply base: '/', you get file paths that are non-absolute, and are incorrect (but are correct relative to root):

So you would get:

file.path === '/Users/pgilad/repos/personal/gulp-todo/lib/reporter.js'
file.relative === 'Users/pgilad/repos/personal/gulp-todo/lib/reporter.js'

This problem occurs when you use file.relative in your plugin to get file information, and expect users to set the base option in order to control if the path will be absolute or relative.

Any ideas on how to work around this without adding an option to the plugin?

vinyl is fetching too many dependencies

Vinyl have lodash.clonedeep in dependencies, which is very complicated inside. Since vinyl should be base for all plugins, that emitting Vinyl Files it is not very robust to install 45 packages to just deep clone object. Could it be replaced by another similar package (like node-clone)?

└─┬ [email protected]
  ├── [email protected]
  └─┬ [email protected]
    ├─┬ [email protected]
    │ ├─┬ [email protected]
    │ │ └── [email protected]
    │ ├─┬ [email protected]
    │ │ ├── [email protected]
    │ │ └── [email protected]
    │ ├── [email protected]
    │ ├─┬ [email protected]
    │ │ ├── [email protected]
    │ │ └─┬ [email protected]
    │ │   ├── [email protected]
    │ │   └── [email protected]
    │ ├── [email protected]
    │ ├─┬ [email protected]
    │ │ ├── [email protected]
    │ │ └─┬ [email protected]
    │ │   ├── [email protected]
    │ │   └── [email protected]
    │ ├─┬ [email protected]
    │ │ └── [email protected]
    │ └─┬ [email protected]
    │   └── [email protected]
    └─┬ [email protected]
      ├─┬ [email protected]
      │ ├── [email protected]
      │ └── [email protected]
      ├─┬ [email protected]
      │ ├─┬ [email protected]
      │ │ ├─┬ [email protected]
      │ │ │ ├─┬ [email protected]
      │ │ │ │ ├── [email protected]
      │ │ │ │ └── [email protected]
      │ │ │ └─┬ [email protected]
      │ │ │   └── [email protected]
      │ │ ├─┬ [email protected]
      │ │ │ ├─┬ [email protected]
      │ │ │ │ ├── [email protected]
      │ │ │ │ └── [email protected]
      │ │ │ └─┬ [email protected]
      │ │ │   └── [email protected]
      │ │ └── [email protected]
      │ └── [email protected]
      ├── [email protected]
      └─┬ [email protected]
        └── [email protected]

async processing via new gutil.File() ?

I'm experiencing some Error: stream.push() after EOF errors when I run my gulp plugin:

        var embedder = new ResourceEmbedder(filepath);

        embedder.get(function (markup) {
            var f = new gutil.File({
                path: path.basename(filepath),
                contents: new Buffer(markup)
            });
            this.push(f);
            cb();
        }.bind(this));

but the codes below run successfully with no error:

        var embedder = new ResourceEmbedder(filepath);

        embedder.get(function (markup) {
            var f = new gutil.File({
                path: path.basename(filepath),
                contents: new Buffer(markup)
            });

            setTimeout(function(){
                this.push(f);
                cb();
            }.bind(this),3000)

        }.bind(this));

so I wonder if new gutil.File() was an async function and how to solve it?
p.s, embedder refer to resource-embedder

Abstracting Streams/Buffer

Related to: gulpjs/gulp#74

To achieve this, we have to provide an asynchronous API when writing a buffer to a File object and when reading a buffer in a file object.

When reading a Buffer while the file contents is a stream, since a stream cannot be consumed twice, we will have to choose between:

  • caching the Buffer (i think we shouldn't)
  • testing if the stream ended and return an error to avoid reading the buffer twice.

A simple API proposal could be:

  • for streams:
// Reading
File.pipe(stream);
// Writing
stream.pipe(File);
  • for buffers:
// Reading
File.getBuffer(function callback(err, buf) {
  // playing with buf
});
// Writing
File.setBuffer(buf, function callback(err) {
  // My buffer is set
});

We should keep a way to know if we are in buffer mode or in stream mode in order to be able to make optimizations if needed. To me the option.buffer value can do the job:

var file = new File({ buffer: false});
// inside a plugin
if(file.buffer) {
  // Playing with buffers
} else {
  // Playing with streams
}

File constructor should also expose the complete stream API to achieve this. Inheriting of Stream.PassTrough can do the job.

I'm volunteer to help coding those features, just let me know if you're ok with it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.