Giter VIP home page Giter VIP logo

turistforeningen / node-s3-uploader Goto Github PK

View Code? Open in Web Editor NEW
242.0 12.0 54.0 23.65 MB

Flexible and efficient resize, rename, and upload images to Amazon S3 disk storage. Uses the official AWS Node SDK for transfer, and ImageMagick for image processing. Support for multiple image versions targets.

Home Page: https://www.npmjs.com/package/s3-uploader

License: MIT License

JavaScript 100.00%
nodejs npm-package image-processing imagemagick thumbnails aws-s3 aws s3

node-s3-uploader's Introduction

AWS S3 Image Uploader

Documentation for [email protected] can be found here.

Build status Codacy grade Codacy coverage NPM downloads NPM version Node version Dependency status Join the chat

Flexible and efficient image resize, rename, and upload to Amazon S3 disk storage. Uses the official AWS Node SDK, and im-resize and im-metadata for image processing.

Overview of image upload to AWS S3

Changelog

All changes are documentated on the releases page. Changes for latest release can be found here.

Install

npm install s3-uploader --save

Requirements

  • Node.JS >= v4.0.0
  • ImageMagic >= v6.8

API

var Upload = require('s3-uploader');

new Upload(string awsBucketName, object opts)

  • string awsBucketName - name of Amazon S3 bucket
  • object opts - global upload options
    • object cleanup

      • boolean original - remove original image after successful upload (default: false)
      • boolean versions - remove thumbnail versions after sucessful upload (default: false)
    • boolean returnExif - return exif data for original image (default false)

    • string url - custom public url (default build from region and awsBucketName)

    • object aws - see note

      • string region - region for you bucket (default us-east-1)
      • string path - path within your bucket (default "")
      • string acl - default ACL for uploaded images (default private)
      • string accessKeyId - AWS access key ID override
      • string secretAccessKey - AWS secret access key override
    • object resize

      • string path - local directory for resized images (default: same as original image)
      • string prefix - local file name prefix for resized images (default: "")
      • integer quality - default quality for resized images (default: 70)
    • object[] versions

      • string suffix - image file name suffix (default "")
      • number quality - image resize quality
      • string format - force output image file format (default format of original image)
      • number maxWidth - max width for resized image
      • number maxHeight - max height for resized image
      • string aspect - force aspect ratio for resized image (example: 4:3
      • string background - set background for transparent images (example: red)
      • boolean flatten - flatten backgrund for transparent images
      • string awsImageAcl - access control for AWS S3 upload (example: private)
      • number awsImageExpires - add Expires header to image version
      • number awsImageMaxAge - add Cache-Control: max-age header to image version
    • object original

      • string awsImageAcl - access control for AWS S3 upload (example: private)
      • number awsImageExpires - add Expires header to image version
      • number awsImageMaxAge - add Cache-Control: max-age header to image version
    • function randomPath - custom random path function

AWS note

The aws object is passed directly to aws-sdk. You can add any of these options in order to fine tune the connection โ€“ if you know what you are doing.

Example

var client = new Upload('my_s3_bucket', {
  aws: {
    path: 'images/',
    region: 'us-east-1',
    acl: 'public-read'
  },

  cleanup: {
    versions: true,
    original: false
  },

  original: {
    awsImageAcl: 'private'
  },

  versions: [{
    maxHeight: 1040,
    maxWidth: 1040,
    format: 'jpg',
    suffix: '-large',
    quality: 80,
    awsImageExpires: 31536000,
    awsImageMaxAge: 31536000
  },{
    maxWidth: 780,
    aspect: '3:2!h',
    suffix: '-medium'
  },{
    maxWidth: 320,
    aspect: '16:9!h',
    suffix: '-small'
  },{
    maxHeight: 100,
    aspect: '1:1',
    format: 'png',
    suffix: '-thumb1'
  },{
    maxHeight: 250,
    maxWidth: 250,
    aspect: '1:1',
    suffix: '-thumb2'
  }]
});

#upload(string src, object opts, function cb)

  • string src - path to the image you want to upload

  • object opts

    • string awsPath - override the path on AWS set through opts.aws.path
    • string path - set absolute path for uploaded image (disables random path)
  • function cb - callback function (Error err, object[] versions, object meta)

    • Error err - null if everything went fine
    • object[] versions - original and resized images with path/location
    • object meta - metadata for original image

Example

client.upload('/some/image.jpg', {}, function(err, versions, meta) {
  if (err) { throw err; }

  versions.forEach(function(image) {
    console.log(image.width, image.height, image.url);
    // 1024 760 https://my-bucket.s3.amazonaws.com/path/110ec58a-a0f2-4ac4-8393-c866d813b8d1.jpg
  });
});

Algorithm

A
+-- B
    `-- C
    `-- D
    `-- E

Where A is the original image uploaded by the user. An mpc image is created, B,
which is used to crate the thumbnails C, D, and E.

Collaborators

Individuals making significant and valuable contributions are made Collaborators and given commit-access to the project. These individuals are identified by the existing Collaborators and their addition as Collaborators is discussed as a pull request to this project's README.md.

Note: If you make a significant contribution and are not considered for commit-access log an issue or contact one of the Collaborators directly.

node-s3-uploader's People

Contributors

anthonyringoet avatar codemanki avatar gitter-badger avatar greenkeeperio-bot avatar lfreneda avatar starefossen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

node-s3-uploader's Issues

alternative for the resize an image

it is possible to make the resize an image that is stored on aws. I have this code that run from another application

var im = require('imagemagick');
var Upload = require('s3-uploader

--config
var client = new Upload('bucket', {

aws: {
region: 'us-west-1',
path: 'imgnormal/',
acl: 'public-read',
accessKeyId: 'KeyID',
secretAccessKey: 'AK',
endpoint: 'https://s3-us-west-1.amazonaws.com',
},cleanup: {
versions: false,
original: false
},
original: {
awsImageAcl: 'public'
},
resize: {
quality: 100
},
versions: [{
maxWidth: 780,
aspect: '3:2!h',
suffix: '-medium'
},{
maxWidth: 320,
aspect: '16:9!h',
suffix: '-small'
}]

});

app.post('/resizeImg',function(req,res){

client.upload('https://s3-us-west-1.amazonaws.com/bucket/imgnormal/house.jpeg', {}, function(err, versions, meta) {

if (err) {
  console.log('----Nope---------', err.message);
  return res.status(400).send('NO');
}
console.log('----Success----', versions, data);
return res.status(200).send('YES');

});

});

when I executo the petition I have the following error

convert.im6: unable to open image `https:/s3-us-west-1.amazonaws.com/bucket/imgnormal/house.jpeg': No such file or directory @ error/blob.c/OpenBlob/2638

what am I doing wrong?

Force type of image, but not work

I set in object opts, but create just image with type .jpg.

My code:

versions: [{
    maxHeight: 270,
    maxWidth: 310,
    suffix: '-small',
    quality: 100,
    format: 'png'
  }]

result:

{ quality: 100,
    format: 'jpeg',
    src: '/tmp/gm-e7cf903cd727e32e1097dfd5ca9a55eee94fc8cf-small.jpeg',
    width: 310,
    height: 270,
    etag: 'adcc36a20b17c0053ba7ef987b6c37c5',
    path: 'track_cover/hb/yF/tY-small.jpeg',
    url: 'https://s3.amazonaws.com/welearnfs/track_cover/hb/yF/tY-small.jpeg' }

Renaming of file - what happens?

Apologies if this is mentioned somewhere, I have looked and cannot see it.
When "image.jpg" is uploaded is the name retained in the object but with the suffix appended? The example show is being returned as a long string but I cannot see where this has come from.

AWS credentials as global options

Add the possibility to pass AWS credentials directly to the Upload class constructor.

client = new Upload('my-bucket', {
  awsAccessKeyId: String,
  awsAccessSecretKey: String
});

Drop support for Node.js v0.10

I think we should discontinue support for Node.js v0.10, as mentioned below this version of Node.js is officially discontinued after October 1st 2016.

From nodejs/lts repo:

  1. nodejs/node v4.2.0 becomes the current LTS Release
  2. joyent/node v0.10 continues in Maintenance only mode until October 1st, 2016
  3. joyent/node v0.12 continues as LTS until April 1st, 2016, after which it moves into Maintenance only mode until April 1st, 2017.
  4. On or around October 1st, 2016, the second LTS Release from the converged is cut.
  5. LTS for v4.2.0 continues until April 1st, 2017, after which it moves to Maintenance mode until around April 1st, 2017.
  6. On or around October 1st, 2017, the third LTS Release from the converged is cut.

Idea: post-resize hook / event

Just want to pitch this idea here.
I'd like to optimize the resized images before they are uploaded.
What do you think about adding the possibility to let the user do stuff (could be anything) after the images are resized?

Please help me on this.

OS: Windows 7, 32-bit
Node: 0.12.0
Imagemagick: 6.9.0

client.upload('/uploads/site-logo.png', {}, function(err, versions, meta) {
                  if (err) { throw err; }

                  versions.forEach(function(image) {
                    console.log(image.width, image.height, image.url);
                  });
                });

Please help me on this.

Use greenkeeper.io for keeping dependencies up to date

Let's automate the boring part. ๐Ÿ˜„
Agree/disagree?

http://greenkeeper.io

It knows when your dependencies got updated and simply sends you a pull request with the new package.json.

Should a new version inside your existing version range cause your build to fail, weโ€™ll let you know immediately.

If youโ€™ve got continuous integration tests, theyโ€™ll run, and you can keep your software up to date with a single click on GitHubโ€™s merge button.

Added support for pngs

Pngs resizes to a black transparency as jpeg is the only format it resizes to, It would be nice to be able to detect the png and output pngs rather than jpeg to preserve the image type and properties (such as transparency)

upload options override path

Is it possible to have the 'path' config option to be overridden when an upload takes place?
ie:
client.upload(file, {path: 'imagesJanuary/'}, function(err, images, meta) {...
client.upload(file, {path: 'imagesFebruary/'}, function(err, images, meta) {...

thanks
Jason

Cleaning original image that wasn't uploaded

I would have liked to be able to clean the original image even if it wasn't uploaded. Currently, the removeVersions function only cleans images that were uploaded.

Also, I would have liked to configure if the original should be uploaded or not without overriding any configuration. For the moment, you need to override an option, such as the awsImageAcl, awsImageExpires, awsImageMaxAge.

Image resize error

Node.js v0.10.32
mageMagick v6.9.0-10
Ubuntu v14.04.1 LTS

the code for the uploader:
var uuid = require('node-uuid');
var config = require('../config');
var _ = require('underscore');
var Upload = require('s3-uploader');

module.exports = {

connect: function() {
    var awsConfig = config.get('s3config');
    console.log(awsConfig);

    try {
        return new Upload(awsConfig.bucket, {
            url: awsConfig.url,
            aws: {
                region: awsConfig.region,
                acl: 'public-read',
                accessKeyId: awsConfig.accessKey,
                secretAccessKey: awsConfig.secretAccessKey,
                endpoint: awsConfig.url,
                s3BucketEndpoint: true
            },
            versions: [{
                original: false
            }, {
                suffix: '-large',
                quality: 80,
                maxHeight: 1040,
                maxWidth: 1040,
            }, {
                suffix: '-medium',
                maxHeight: 780,
                maxWidth: 780
            }, {
                suffix: '-small',
                maxHeight: 320,
                maxWidth: 320
            }]
        });
    } catch (e) {
        console.log('error from uploader connect', e);
    }
},
upload: function(file, client) {
    try {
        client.upload(file, {}, function(err, images, meta) {
            if (err) {
                console.error('Upload Error', err);
            } else {
                for (var i = 0; i < images.length; i++) {
                    console.log('Thumbnail with width %i, height %i, at %s', images[i].width, images[i].height, images[i].url);
                }
            }
        });
    } catch (e) {
        console.log('error from upload', e);
    }
}

}

In the middleware, we call execute the following:
uploader.upload(req.files.uploadedFiles.path, req.uploader);
req.uploader gets set when the application starts and it creates the connection and returns the client.

the error we see is:
{ [Error: Command failed: convert: invalid argument for option `-resize': -auto-orient @ error/convert.c/ConvertImageCommand/2555.
] code: 1, signal: null }

any thoughts?

Error code:4

Hi,
I have the following error:

 { [Error: Command failed: Parametro non valido - server\uploads\86caf0d05da8d7fa
 c0fc964368a44939.jpg.mpc
 ] code: 4, signal: null }

In folder server\uploads\ there is the file: 86caf0d05da8d7fac0fc964368a44939.jpg

In myBucket no file is uploaded.

my configuration is:

var client = new Upload('myBucket', {
aws:{
  region: 'eu-west-1',
  acl: 'public-read',
  accessKeyId:'XXXXXXXXXXXXXX',
  secretAccessKey:'XXXXXXXXXXXXXXXXXXX'
},
versions: [{
  original: true
},{
  suffix: '-large',
  quality: 80,
  maxHeight: 1040,
  maxWidth: 1040,
},{
  suffix: '-medium',
  maxHeight: 780,
  maxWidth: 780
 },{
  suffix: '-small',
  maxHeight: 320,
  maxWidth: 320
 }]
});

var filePath='server\uploads\346edfc4433be42e5b85e9927f41b0cf.jpg';

client.upload(filePath, {}, function(err, images, meta) {
  if (err) {
    console.error(err);
  } else {
    for (var i = 0; i < images.length; i++) {
      console.log('Thumbnail with width %i, height %i, at %s', images[i].width, images[i].height, images[i].url);
    }
  }
});

Example usage

Some more examples on how to use the s3-uploader would be nice. Maybe in an example folder, that way we can more easily integrate them with the test suite making sure no example gets broken between releases.

Optionally clean up temp files

Provide option to clean up temp files after upload has finished. This should not be necessary when using system default temp directory as this should be clean up automatically. But for custom temp directories this must be done manually.

Return actual image height and width

Currently we just return the maxHeight and maxWidth version properties without checking the actual resized image (because this takes time). If someone needs it, they can submit a PR and make it optional for those who needs it.

Custom random path generator function

Provide a constructor option to define a custom random path generator function.

Example:

var uuid = require('uuid');
var Upload = require('s3-uploader');

var client = new Upload('myBucket', {
  randomPathGenerator: uuid.v4
}

Force aspect ratio orientation

Force aspect ratio orientation for resized image; landscape or horisontal.

{
  versions: [{
    maxHeight: 500
    maxWidth: 500
    aspect: '4:3!'
ย ย ]}
}

Node.js single thread with image processing question

I have use this lib in my project and it works fine in staging. But Image resizing is a cpu intense task, and I can see my staging server cpu usage almost 100% when uploading and resizing a image. Although the staging server is just EC2 t2.micro, I am still afraid that this will lead to server being blocked once pushed to production. (I have tried to upload using two device at the same time and seems to be fine, but still not sure what will happen in production, and the http response takes almost 3000ms!)

Could you tell me how it is handled when multiple users uploading images at the same time and if there is a possibility that the event loop would be blocked which is a big concern in node.js.

My question is also here:

http://stackoverflow.com/questions/34018487/node-js-with-imagemagick-why-server-is-not-blocked

cleanup does nothing

The option to cleanup both original and resized images does not work. In fact there's no reference to it in the code. Am I missing something ?

'starefossen/rand-path' is not in the npm registry

npm WARN package.json [email protected] cluster is also the name of a node core module.
npm ERR! 404 Not Found
npm ERR! 404
npm ERR! 404 'starefossen/rand-path' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it
npm ERR! 404 It was specified as a dependency of 's3-uploader'
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, or http url, or git url.

npm ERR! System Linux 3.13.0-48-generic
npm ERR! command "/usr/local/bin/node" "/usr/local/bin/npm" "install" "s3-uploader"
npm ERR! cwd /home/ubuntu/web-service-api-v2
npm ERR! node -v v0.10.33
npm ERR! npm -v 1.4.28
npm ERR! code E404
npm ERR! not ok code 0

node version 0.10.33
npm version 1.4.28

Handle image paths without extension when uploading

Some files does not have an extension (latest version of expressjs/multer for instance) which crates a problem for s3-uploader. We should add the ability to pass inn one of the following:

a) the original file name
b) file extension
b) image mime type

client.upload('/some/file', { mime: "image/jpeg" }, function(err, versions, meta) {
  ...
});

Emit S3 upload result as soon an image version was uploaded

In some case we need result from on version is enough, so we don't need to wait the entire upload process completed.

For example: I save the original image in database, and sub version deduce from original URL, so the original URL is what I need.

Nope Command failed: /bin/sh -c identify -format

I'm trying to upload some photos as I receive them from a POST request, here is my code:

var multer  = require('multer');
var Upload = require('s3-uploader');
var upload = multer({ dest: 'uploads/' });
var photoRouter = require('express').Router();

// Same as Example in README
var client = new Upload('my-bucket-name', {
  aws: {
    path: 'images/',
    region: 'eu-west-1',
    accessKeyId: 'my-key-id',
    secretAccessKey: 'my-secret-acess-key',
    acl: 'public-read'
  },
  cleanup: {
    versions: true,
    original: false
  },
  original: {
    awsImageAcl: 'private'
  },
  versions: [{
    maxHeight: 1040,
    maxWidth: 1040,
    format: 'jpg',
    suffix: '-large',
    quality: 80,
    awsImageExpires: 31536000,
    awsImageMaxAge: 31536000
  },{
    maxWidth: 780,
    aspect: '3:2!h',
    suffix: '-medium'
  },{
    maxWidth: 320,
    aspect: '16:9!h',
    suffix: '-small'
  },{
    maxHeight: 100,
    aspect: '1:1',
    format: 'png',
    suffix: '-thumb1'
  },{
    maxHeight: 250,
    maxWidth: 250,
    aspect: '1:1',
    suffix: '-thumb2'
  }]
});

photoRouter.post('/', upload.array('photos', 12), postPhoto);

function postPhoto(req, res, next) {

  console.log(req.files);
  // I'm ignoring the posted files, that are correctly saved, and trying to upload the photo I have in the folder
  return client.upload('uploads/profile-picture.jpg', {}, function(err, versions, data) {
    if (err) {
      console.log('Nope', err.message);
      return res.status(400).send('KO');
    }

    console.log('Success', versions, data);
    return res.status(200).send('OK');
  });
}

And here is the error I'm getting:

Nope Command failed: /bin/sh -c identify -format "name=
size=%[size]
format=%m
colorspace=%[colorspace]
height=%[height]
width=%[width]
orientation=%[orientation]
" uploads/profile-picture.jpg
/bin/sh: identify: command not found

I'm using node 4.2.4 on OSX 10.11.2 with s3-uploader 1.1.0

Command failed: /bin/sh -c identify -format "name=size=%[size] etc...

hello community have the following doubt'm trying to do the resize an image that I have in the following path of aws /imgnormal/quiko.jpeg

almost copy the code replacing keys aws

 var express = require('express');
 var bodyParser = require('body-parser');
 var im = require('imagemagick'); 
 var Upload = require('s3-uploader');

 var client = new Upload('my-buckets',{

  aws : {
    region : 'us-west-2',
    path : 'images/',
    acl: 'public-read',
    accessKeyId : 'aaaa',
    secretAccessKey  : 'bbbb'
  },
  cleanup: {
    versions: true,
    original: false
  },original: {
    awsImageAcl: 'private'
  },
  versions: [{
    maxHeight: 1040,
    maxWidth: 1040,
    format: 'jpg',
    suffix: '-large',
    quality: 80,
    awsImageExpires: 31536000,
    awsImageMaxAge: 31536000
  },{
    maxWidth: 780,
    aspect: '3:2!h',
    suffix: '-medium'
  },{
    maxWidth: 320,
    aspect: '16:9!h',
    suffix: '-small'
  },{
    maxHeight: 100,
    aspect: '1:1',
    format: 'png',
    suffix: '-thumb1'
  },{
    maxHeight: 250,
    maxWidth: 250,
    aspect: '1:1',
    suffix: '-thumb2'
  }]

});

create this route for test

app.post('/Test', function(req, res) {

  client.upload('imgnormal/quiko.jpeg', {}, function(err, versions, meta) {
      if (err) {
        console.log('----Nope-------', err.message);
        return res.status(400).send('KO');
      }
      console.log('----Success----', versions, data);
      return res.status(200).send('OK');
  });

};

to make the request to the resize the image I have this error

 ----Nope------- Command failed: /bin/sh -c identify -format "name=
size=%[size]
format=%m
colorspace=%[colorspace]
height=%[height]
width=%[width]
orientation=%[orientation]
identify.im6: unable to open image `imgnormal/quiko.jpeg': No such file or directory @ error/blob.c/OpenBlob/2638.

was researching the problem and here's the solution did not have installed imagemagick #73
put the var im = require ( 'imagemagick'); and yet I have this error
in my package.json i have this dependencies

"dependencies": {
    "ejs": "2.3.3",
    "express": "~4.2.x",
    "imagemagick": "^0.1.3",
    "s3-uploader": "^1.1.0"
  },

path the imagemagick https://github.com/rsms/node-imagemagick
what i do wrong in the code?

thanks for your time.

Error Code 1

I seem to be getting this error
{ [Error: Command failed: ] code: 1, signal: null }
running my dev environment on Mac OS X 10.10.3
Node.js v0.12.2

my code:

exports.uploadProductImg = function(path, sku){

    var client = new Upload('cdn-hickies', {
        "aws": {region : 'us-east-1',
            path: "/products/" + sku + "/",
            acl: 'public-read'},

        versions: [{
            original: true
        },{
            suffix: '-large',
            quality: 80,
            maxHeight: 1280,
            maxWidth: 1280
        },{
            suffix: '-medium',
            maxHeight: 640,
            maxWidth: 640
        },{
            suffix: '-small',
            maxHeight: 320,
            maxWidth: 320
        },{
            suffix: '-xsmall',
            maxHeight: 150,
            maxWidth: 150
        }
        ]
    });
    client.upload(path, {}, function(err, images, meta) {
        if (err) {
            console.error(err);
        } else {
            for (var i = 0; i < images.length; i++) {
                console.log('Thumbnail with width %i, height %i, at %s', images[i].width, images[i].height, images[i].url);
            }
            require('./inventory-manager').addImages(sku, images);
        }
    });

};

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.