Support interaction with Buffers about node-scalable-blob-store HOT 5 CLOSED

Hi @oliversalzburg,

Way back when I first started sbs I had support for strings and buffers. I decided to follow KISS and make sbs a low level component and just support streams. That said, if we kept the API the same and just extended it with buffer and even string / json support, that would work. The original API had "saveBlob" or something like that and would determine the data type dynamically. I didn't like that and it made it difficult to support the stream events.

So if buffers / string / json support was added, we would need to determine the public api and also support events (EventEmitter) which it doesn't support.

That's a fair bit of work for something I am quite happy with.

If you want to add support into it, I'm happy to work with you.

from node-scalable-blob-store.

@oliversalzburg
I just had a closer look at stream-buffers. It might be easier that I thought.
Expose a "createWriteBuffer" and return the stream-buffer object or something like that.

from node-scalable-blob-store.

I really like your approach with this library and, for now, I just wrapped it in another module that supports our Buffer interaction. This module turned out to be very little code indeed.

The relevant code looks something like this:

const storageRoot = configuration.get( "storageLocation" );
let blobStore     = null;

function getStorage() {
    if( blobStore === null ) {
        blobStore = sbsFactory.create( {
            blobStoreRoot : storageRoot,
            idType        : "uuid"
        } );
    }

    return blobStore;
}

function insertBuffer( buffer ) {
    return getStorage()
        .createWriteStream()
        .then( blob => {
            return new Promise( ( resolve, reject ) => {
                blob.writeStream.on( "error", reject );
                blob.writeStream.write( buffer, () => resolve( blob.blobPath ) );
            } );
        } );
}

function readBuffer( storageLocation ) {
    let stream = getStorage().createReadStream( storageLocation );

    const buffer = new streamBuffers.WritableStreamBuffer( {
        initialSize     : 100 * 1024,
        incrementAmount : 10 * 1024
    } );

    return new Promise( ( resolve, reject ) => {
        stream.on( "error", reject );
        stream.on( "end", () => resolve( buffer.getContents() ) );

        stream.pipe( buffer );
    } );
}

So, we don't really need the change any more and your original idea, to keep the module to the bare minimum, might actually be the preferable solution.

from node-scalable-blob-store.

Nice work @oliversalzburg! This could be made as a new module with sbs as a dependent.

Thanks for the good comments also. Apart from stars, this is the first feedback I've had.

I never though of using sbs with shared storage as I believe you are. It would work well in that situation also. I was originally thinking about a replicated file system.

Thanks again.

from node-scalable-blob-store.

This could be made as a new module with sbs as a dependent.

Awesome idea! I hope I'll find time to implement this :)

The module works really well for us. We're in exactly the spot you outline in the README. We don't feel comfortable pushing into S3 and would just prefer to use local storage. We recently made our backend scalable and transitioned to multiple workers, so, naturally we ran into conflicts.

We're currently in staging and will deploy to production next week. This module was a huge help in allowing us to stay on schedule, because we did not anticipate the file storage issue.

So, thanks :) 👍

from node-scalable-blob-store.