nitaigao / gulp-s3 Goto Github PK

View Code? Open in Web Editor NEW

100.0 2.0 83.0 36 KB

License: MIT License

JavaScript 100.00%

gulp-s3's Introduction

gulp-s3

s3 plugin for gulp

Usage

First, install gulp-s3 as a development dependency:

npm install --save-dev gulp-s3

Then, use it in your gulpfile.js:

var s3   = require('gulp-s3')
var gulp = require('gulp')

var AWS = {
  "key":    process.env.AWS_ACCESS_KEY_ID,
  "secret": process.env.AWS_SECRET_ACCESS_KEY,
  "bucket": "dev.example.com",
  "region": "eu-west-1"
}

gulp.task('default', () => {
  gulp.src('./dist/**').pipe(s3(AWS));
});

API

options.uploadPath

Type: String
Default: ``

Set the remote folder on the S3 bucket

var options = { uploadPath: 'remote-folder' } // It will upload the 'src' into '/remote-folder'
gulp.src('./dist/**', {read: false})
    .pipe(s3(AWS, options));

options.headers

Type: Object
Default: {}

Headers to set to each file uploaded to S3

var options = { 
  headers: {
    'Cache-Control': 'max-age=315360000, no-transform, public',
    'x-amz-acl': 'private'
  } 
};
gulp.src('./dist/**', {read: false})
    .pipe(s3(AWS, options));

options.gzippedOnly

Type: Boolean
Default: false

Only upload files with .gz extension, additionally it will remove the .gz suffix on destination filename and set appropriate Content-Type and Content-Encoding headers.

var options = { gzippedOnly: true };
gulp.src('./dist/**').pipe(gzip())
    .pipe(s3(AWS, options));

options.failOnError

Type: Boolean Default: false

Throw error if upload to s3 fails.

var options = { failOnError: true };
gulp.src('./dist/**').pipe(gzip())
    .pipe(s3(AWS, options));

License

MIT License

gulp-s3's People

Contributors

Stargazers

Watchers

gulp-s3's Issues

Signature v4 Support

Newly added regions only supports v4 signatures. What is sufficient to support this needs to be added to gulp-s3 since many will be using new regions in future. The lack of this feature may be one reason to get 400 FAILED responses.

Is there anyway to upload files only if the file has changed?

Thank you.

Download or Copy from S3

On a project that I am on we have a need to download some files from S3 prior to uploading our files (using gulp). I could create a separate project to do this; however, I think it would be better to include this in an existing project. I wanted to see if there would be any interest in adding download functionality to the gulp-s3 project.

Thanks

Shannon

Uploading to a subdirectory

Allow for user to define a subdirectory for files to be uploaded to.

aws = {
      "key": gutil.env.aws_key,
      "secret": gutil.env.aws_secret,
      "bucket": gutil.env.aws_bucket,
      "region": gutil.env.aws_region,
      "subdirectory": "aSubDirectory/"
    };

    gulp.src('./dist/**')
        .pipe(s3(aws));

this will upload everything in dist to aSubDirectory on S3

Typo in headers options for docs.

Hey @nkostelnik ,

Just trying to convert over a gruntfile to gulp and one of the grunt tasks was using grunt-s3 so I'm giving your gulp equivalent a go.

I did however notice that here

var options = { headers: ['Cache-Control': 'max-age=315360000, no-transform, public'] }
gulp.src('./dist/**', {read: false})
    .pipe(s3(aws, options));

options is actually an object and not an object array I think according to the source so options should be defined with

var options = {headers: {'Cache-Control': 'max-age=315360000, no-transform, public' } }

Please do correct me if I'm wrong but I tried to run the example code not really looking at it and it of course threw an error.

Hope that helps.

Best

@jh3y

Specify charset for HTML files

Is there a way to set the charset to UTF-8 only on files with a Content-Type: text/html?

These are my settings currently:

gulp.task('deploy', function() {
  var awsConfig = require('./lib/config/aws.js');
  var s3Options = {
    headers: {
      'Cache-Control': 'max-age=3600, no-transform, public'
    }
  };
  gulp.src('./public/**')
    .pipe(gzip())
    .pipe(gulpS3(awsConfig, s3Options));
});

Event 'end' emit

Can you please have the EventSteam emit the "end" event? So when binding to the standard event, my gulp tasks can execute the 'done' callback and can continue synchronously as I'm codependent on this completion to cleanup.

I've forked the repo and may try myself if you're unavailable, do you actually manage pull requests?

question: what AWS IAM policies are required?

Thanks for the package, works great!

I'm trying to figure out what permissions are required. Right now if I attach the S3 full access policy, everything works fine, but if I try to make a more restricted policy that only works on a particular bucket, I get a 403 error.

Any advice on this?

Here is my policy JSON (sensitive fields changed):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1494895366000",
      "Effect": "Allow",
      "Action": [
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::foobar-bucket"
      ]
    }
  ]
}

Versions missing for dependencies

Is there a particular reason why dependencies async and knox do not have a version explicitly set in package.json?

https://github.com/nitaigao/gulp-s3/blob/master/package.json#L26
https://github.com/nitaigao/gulp-s3/blob/master/package.json#L29

On npm install these are being bypassed and not installed, and thus bringing up errors.
As far as I understand, leaving the version empty is not accounted for in semantic versioning: https://docs.npmjs.com/getting-started/semantic-versioning#semver-for-consumers

How do we set Acl permissions to the Uploaded Folders in gulp s3

I am uploading various folders into a s3 bucket and now i need to be able to have full control for everyone accessing the files . How is this possible using gulp s3 . How to the set the ACL's

Random FAILED assets

Been using this plugin for quite sometime and never had issues. Recently I have been getting a lot of random assets failing. Anyone know what would cause this? Any help would be great.

Delete files on sync

Acording to AWS docs

http://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html

You can use --delete parameter when sync with s3 and that will delete all files in bucket that are not presented in sync.

This is very useful feature to keep things neat.

Possible to implement?

Obsolete Package

This package no longer works with aws v4 signatures. All PUT requests to S3 buckets will return a 400 HTTP code.

If you are looking for an alternative, use gulp-awspublish.

Errors are undetectable

Hey, I noticed in the code that errors appear to be logged but then nothing is done about them:

      client.putBuffer(file.contents, uploadPath, headers, function(err, res) {
        if (err || res.statusCode !== 200) {
          gutil.log(gutil.colors.red('[FAILED]', file.path + " -> " + uploadPath));
        } else {
          gutil.log(gutil.colors.green('[SUCCESS]', file.path + " -> " + uploadPath));
          res.resume();
        }
      });

It would be nice if an error event or something was emitted, so that gulp tasks could raise errors instead of having the process exit with status code 0 even though things went amiss.

uploadPath on Windows

There is a problem with uploadPath on Windows.
'' should be replaced correctly.

Syncing

Is it possible to sync with this plugin? (Add new files & remove old ones.) Is it a planned feature?

New Repo - gulp-s3-ls

Hi everyone,
So it's been quite some time since this repo has been active and I think there are some great pull requests that are still open! So I decided to fork the repo and publish it under a new name "gulp-s3-ls". Check it out, and resubmit your pull requests to it. Ill review them all and accept them if they're worthy with expediency!

Always getting FAILED (StatusCode 400)

I'm always getting an FAILED error. If I take a look into the index.js I can see an status code of 400.
Anyone else having this issue and/or a solution?

npm package is out of date

npm package is on version 0.11.0 but code has been merged since that version was published. Please publish a new version with the updated code.

Thanks,
Cyrus

gulp-s3 removes .gz suffix

I was surprised that gulp-s3 now removes .gz extensions from gzipped files. It wasn't the case with 0.2.x. Is there any reason why? It is pretty annoying and broke my deployments when upgrading.

if (regexGzip.test(file.path)) {
      // Set proper encoding for gzipped files, remove .gz suffix
      headers['Content-Encoding'] = 'gzip';
      uploadPath = uploadPath.substring(0, uploadPath.length - 3);
  }

Setting the header is good but why removing the extension?

Question: How can I delete the previous version files from the s3 bucket?

Hey guys,

I am using gulp-s3 to deploy a web app to S3.

Since the .js and .css files are versioned on each deploy I would like to delete the previous versions of these files (or even rotate it) from the S3 bucket.

I thought about deploying the files to a versioned directory inside the bucket, but I didn't find an option on the code to pass the name of the directory to upload the files.

Can you please give me any suggestion about how to delete or rotate the previous version?

Thank you.

*.txt files have incorrect content type

uploadPath with file name

i have a requirement to deploy files to S3 with different name than local file name. right now uploadPath concates the actual file name with this.
gulp-rename plugin also doesn;t work .. anythoughts..

e.g) src_file_name >> SOURCE_GEN_1.js
gulp.src(src_file_name)
.pipe(s3(aws,{'uploadPath':'/SOURCE_GEN.js','encoding':'UTF-8'})) );

currently s3 plugin creates file like
/SOURCE_GEN.jsSOURCE_GEN_1.js

File is always null

Hi I'm not sure what I'm doing wrong here. The file is showing null.
var s3_creds = {
"key": "xxxxxxxxx",
"secret": "xxxxxxxxxxx",
"bucket": "sample",
"region": "ireland"
};

/* upload the dist files */
gulp.task('deploy_s3', function(){
var options = { gzippedOnly: true };

return gulp.src('app/builds/index.html', {read: false})
.pipe(gzip({append: true}))
.pipe(s3(s3_creds, options));
});

Allow configuration via environment variables

Like the grunt-s3 plugin allows.

`uploadPath` fails without tailing slash

var aws = JSON.parse(fs.readFileSync('./aws.json')),
    options = {
        uploadPath: '/site/js'
    };
gulp.src('./dist/file.js').pipe(s3(aws, options));

Results in the file being uploaded as: /site/jsfile.js instead of /site/js/file.js.

[email protected] has security vulnerability

https://npmjs.com/advisories/535
Update to version 2.0.3 or later

semver tags: npm doesn't have the latest version

The 0.1.0 version from npm has the old error handling output that is closer to:
c8bbd5c

requesting 'latest' from npm still doesn't pull the latest version. semver tags should clear this up.

Setting the `Expires` header option.

Could this be added?

For example, grunt-s3 accepts an object, something like

headers: {
    // Two Year cache policy (1000 * 60 * 60 * 24 * 730)
    "Cache-Control": "max-age=630720000, public",
    "Expires": new Date(Date.now() + 63072000000).toUTCString()
}