Giter VIP home page Giter VIP logo

serverless-s3-sync's Introduction

Serverless S3 Sync npm

A plugin to sync local directories and S3 prefixes for Serverless Framework ⚡ .

Use Case

  • Static Website ( serverless-s3-sync ) & Contact form backend ( serverless ) .
  • SPA ( serverless ) & assets ( serverless-s3-sync ) .

Install

Run npm install in your Serverless project.

$ npm install --save serverless-s3-sync

Add the plugin to your serverless.yml file

plugins:
  - serverless-s3-sync

Compatibility with Serverless Framework

Version 2.0.0 is compatible with Serverless Framework v3, but it uses the legacy logging interface. Version 3.0.0 and later uses the new logging interface.

serverless-s3-sync Serverless Framework
v1.x v1.x, v2.x
v2.0.0 v1.x, v2.x, v3.x
≥ v3.0.0 v3.x

Setup

custom:
  s3Sync:
    # A simple configuration for copying static assets
    - bucketName: my-static-site-assets # required
      bucketPrefix: assets/ # optional
      localDir: dist/assets # required

    # An example of possible configuration options
    - bucketName: my-other-site
      localDir: path/to/other-site
      deleteRemoved: true # optional, indicates whether sync deletes files no longer present in localDir. Defaults to 'true'
      acl: public-read # optional
      followSymlinks: true # optional
      defaultContentType: text/html # optional
      params: # optional
        - index.html:
            CacheControl: 'no-cache'
        - "*.js":
            CacheControl: 'public, max-age=31536000'
      bucketTags: # optional, these are appended to existing S3 bucket tags (overwriting tags with the same key)
        tagKey1: tagValue1
        tagKey2: tagValue2

    # This references bucket name from the output of the current stack
    - bucketNameKey: AnotherBucketNameOutputKey
      localDir: path/to/another

    # ... but can also reference it from the output of another stack,
    # see https://www.serverless.com/framework/docs/providers/aws/guide/variables#reference-cloudformation-outputs
    - bucketName: ${cf:another-cf-stack-name.ExternalBucketOutputKey}
      localDir: path

    # Setting the optional enabled field to false will disable this rule.
    # Referencing other variables allows this rule to become conditional
    - bucketName: DisabledSync
      localDir: path
      enabled: false

resources:
  Resources:
    AssetsBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: my-static-site-assets
    OtherSiteBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: my-other-site
        AccessControl: PublicRead
        WebsiteConfiguration:
          IndexDocument: index.html
          ErrorDocument: error.html
    AnotherBucket:
      Type: AWS::S3::Bucket
  Outputs:
    AnotherBucketNameOutputKey:
      Value: !Ref AnotherBucket

Usage

Run sls deploy, local directories and S3 prefixes are synced.

Run sls remove, S3 objects in S3 prefixes are removed.

Run sls deploy --nos3sync, deploy your serverless stack without syncing local directories and S3 prefixes.

Run sls remove --nos3sync, remove your serverless stack without removing S3 objects from the target S3 buckets.

sls s3sync

Sync local directories and S3 prefixes.

Offline usage

If also using the plugins serverless-offline and serverless-s3-local, sync can be supported during development by placing the bucket configuration(s) into the buckets object and specifying the alterate endpoint (see below).

custom:
  s3Sync:
    # an alternate s3 endpoint
    endpoint: http://localhost:4569
    buckets:
    # A simple configuration for copying static assets
    - bucketName: my-static-site-assets # required
      bucketPrefix: assets/ # optional
      localDir: dist/assets # required
# ...

As per serverless-s3-local's instructions, once a local credentials profile is configured, run sls offline start --aws-profile s3local to sync to the local s3 bucket instead of Amazon AWS S3

bucketNameKey will not work in offline mode and can only be used in conjunction with valid AWS credentials, use bucketName instead.

run sls deploy for normal deployment

Always disable auto sync

custom:
  s3Sync:
    # Disable sync when sls deploy and sls remove
    noSync: true
    buckets:
    # A simple configuration for copying static assets
    - bucketName: my-static-site-assets # required
      bucketPrefix: assets/ # optional
      localDir: dist/assets # required
# ...

Sync on other hooks

custom:
  s3Sync:
    hooks:
      # This hook will run after the deploy:finalize hook
      - after:deploy:finalize
    buckets:
    # A simple configuration for copying static assets
    - bucketName: my-static-site-assets # required
      bucketPrefix: assets/ # optional
      localDir: dist/assets # required
# ...

serverless-s3-sync's People

Contributors

aklitzke avatar alexandrusavin avatar brettdh avatar chichicuervo avatar chrisschneider avatar coyoteecd avatar dependabot[bot] avatar frozenbonito avatar georgehb avatar guidos avatar honzajerabek avatar horike37 avatar jaffa avatar k1low avatar kking-biometrica avatar lepirlouit avatar lvass74 avatar marchant avatar maxmode avatar mdlima avatar mihaerzen avatar mpvosseller avatar mtobola-amz avatar nibynool avatar patrickbussmann avatar rcambrj avatar svdgraaf avatar trilom avatar williamsandowaracle avatar ziktar avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

serverless-s3-sync's Issues

No Such Tag Set error

Hi there, I was trying to tag my bucket using "bucketTags", but I got error below:

S3 Sync: Syncing directories and S3 prefixes...
.
S3 Sync: Synced.
S3 Sync: Syncing metadata...
.
S3 Sync: Synced metadata.
S3 Sync: Updating bucket tags...

No Such Tag Set -----------------------------------------

NoSuchTagSet: The TagSet does not exist
at Request.extractError (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/services/s3.js:714:35)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at callNextListener (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
at IncomingMessage.onEnd (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:336:13)
at IncomingMessage.emit (events.js:412:35)
at IncomingMessage.emit (domain.js:470:12)
at endReadableNT (internal/streams/readable.js:1317:12)
at processTicksAndRejections (internal/process/task_queues.js:82:21)

 For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com

Your Environment Information ---------------------------
Operating System: linux
Node Version: 14.17.6
Framework Version: 2.60.0
Plugin Version: 5.4.4
SDK Version: 4.3.0
Components Version: 3.17.1

NoSuchKey during deployment

Since version 1.10.1 deployments fail with a NoSuchKey error

S3 Sync: Synced metadata.
Serverless: Invoke aws:deploy:finalize
Serverless: [AWS s3 404 0.314s 0 retries] copyObject({
  CopySource: '<redacted>//index.html',
  Key: '/index.html',
  Bucket: '<redacted>',
  Metadata: { CacheControl: 'no-cache' },
  ACL: 'private',
  MetadataDirective: 'REPLACE'
})

  No Such Key --------------------------------------------

  NoSuchKey: The specified key does not exist.
      at Request.extractError (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/services/s3.js:816:35)
      at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
      at Request.transition (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
      at AcceptorStateMachine.runTo (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
      at /usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
      at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
      at Request.transition (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
      at AcceptorStateMachine.runTo (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
      at /usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
      at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
      at callNextListener (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
      at IncomingMessage.onEnd (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:307:13)
      at IncomingMessage.emit (events.js:215:7)
      at IncomingMessage.EventEmitter.emit (domain.js:476:20)
      at endReadableNT (_stream_readable.js:1183:12)
      at processTicksAndRejections (internal/process/task_queues.js:80:21)

Serverless Deprecation warning

Hi,
with the latest version of serverless framework a warning message appears with the text:

Serverless: Deprecation warning: CLI options definitions were upgraded with "type" property (which could be one of "string", "boolean", "multiple"). Below listed plugins do not predefine type for introdu
ced options:
- ServerlessS3Sync for "bucket"
Please report this issue in plugin issue tracker.
Starting with next major release, this will be communicated with a thrown error.
More Info: https://www.serverless.com/framework/docs/deprecations/#CLI_OPTIONS_SCHEMA

Is there a serverless-s3-sync update planned to fix this problem?
Thanks in advance.

Use Serverless App AWS credentials

It would appear that the sync is done via local user AWS credentials, whereas the rest of the serverless deploy/remove chain uses the credentials I have stored against my account in the serverless dashboard.

Is there something I'm missing that prevents s3sync using these credentials?

double sync

I have just a single budket/prefix defined, but end up the the files from build in both the root AND the bucketPrefix and have the following log in serverless deploy

S3 Sync: Syncing directories and S3 prefixes...
........
S3 Sync: Synced.
S3 Sync: Syncing metadata...
.
S3 Sync: Synced metadata.

Here's the relevant config:

  s3Sync:
    - bucketName: ${self:custom.siteName}
      bucketPrefix: shell/
      localDir: build
      params:
        - 'index.html':
            CacheControl: 'max-age=0,no-cache,no-store,must-revalidate'
        - 'sw.js':
            CacheControl: 'max-age=0,no-cache,no-store,must-revalidate'

This deployed via bitbucket pipelines but I don't think that matters.

Multipart uploads aren't successful

I have seen this issue around Error: ETag does not match MD5 checksum being discussed on some PRs.
The above does not seem to be an issue when the object is uploaded as a single request.

But when an object size is large enough to require a multi-part upload, the process errors out with the above error in between uploading parts and hence the file isn't synced to S3 bucket.

For my instance, the file posing a problem is about 17.9 MB in size and fails to upload with the error:

S3 Sync: Syncing directories and S3 prefixes...
. 
 Error ---------------------------------------------------
 
  Error: ETag does not match MD5 checksum
      at /app/qb-serverless-python/services/smart-builder/node_modules/@auth0/s3/lib/index.js:415:16
      at Pend.wait (/app/qb-serverless-python/services/smart-builder/node_modules/pend/index.js:21:5)
      at Response.<anonymous> (/app/qb-serverless-python/services/smart-builder/node_modules/@auth0/s3/lib/index.js:412:14)
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:367:18)
      at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:686:14)
      at Request.transition (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
      at AcceptorStateMachine.runTo (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
      at /usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:688:12)
      at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:686:14)
      at Request.transition (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
      at AcceptorStateMachine.runTo (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
      at /usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:688:12)
      at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:686:14)
      at Request.transition (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
      at AcceptorStateMachine.runTo (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
      at /usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:688:12)
      at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:686:14)
      at Request.transition (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
      at AcceptorStateMachine.runTo (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
      at /usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:688:12)
      at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
      at callNextListener (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
      at IncomingMessage.onEnd (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:335:13)
      at IncomingMessage.emit (events.js:412:35)
      at IncomingMessage.emit (domain.js:475:12)
      at endReadableNT (internal/streams/readable.js:1334:12)
      at processTicksAndRejections (internal/process/task_queues.js:82:21)

TypeError: Cannot read property 'length' of undefined

Getting this error when use the s3Sync with python. Followiing is my yaml file.

`provider:
name: aws
runtime: python3.7
region: ${opt:region, 'us-east-1'}

custom:
s3Sync:
- bucketName: abc
localDir: localfiles

functions:
hello:
handler: handler.hello

plugins:

  • serverless-python-requirements
  • serverless-s3-sync`

.DS_Store files uploaded

My config inside serverless.yml

  s3Sync:
    - bucketNameKey: BucketKey
      bucketPrefix: myFolder/
      localDir: myFolder

End up with each folder having .DS_Store files

Error when renaming buckets "The bucket you tried to delete is not empty"

If I rename my bucket in my serverless config, it fails to delete the previous bucket causing the deploy to fail.

An error occurred: AssetsBucket - The bucket you tried to delete is not empty (Service: Amazon S3; Status Code: 409; Error Code: BucketNotEmpty;

For more context, I am creating the bucket in the serverless config as well.


Is this behavior intended? Is there anyway I can work around this? Thanks for any guidance!

Error: ETag does not match MD5 checksum

Error: ETag does not match MD5 checksum

I am seeing the following error when using this module. I see that the files sync to s3 just fine, however this error still occurs while the 'S3 Sync: Syncing directories and S3 prefixes' is occurring. The error seems to be random when it occurs. 85% of the time it is occurring. It seems like an error that doesn't mean anything related the syncing, because the syncing still happens and looks fine. This happens on both encrypted and unencrypted buckets.

/<mypath>/node_modules/pend/index.js:21
    cb(this.error);
    ^
Error: ETag does not match MD5 checksum
    at /<mypath>/node_modules/@monolambda/s3/lib/index.js:415:16
    at Pend.wait (<mypath>/node_modules/pend/index.js:21:5)
    at Response.<anonymous> (<mypath>/node_modules/@monolambda/s3/lib/index.js:412:14)
    at Request.<anonymous> (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:364:18)
    at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at callNextListener (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:95:12)
    at IncomingMessage.onEnd (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:294:13)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)

s3Sync operation cause IllegalLocationConstraintException

Hello,
I am using serverless s3Sync to copy local file to aws s3 in HongKong(ap-east-1) region, I got below error:

IllegalLocationConstraintException: The ap-east-1 location constraint is incompatible for the region specific endpoint this request was sent to.
at Request.extractError (/home/ec2-user/environment/gcr-solutions/service-workbench-on-aws/node_modules/.pnpm/[email protected]/node_modules/aws-sdk/lib/services/s3.js:837:35)

Does serverless s3Sync support AWS HongKong region?
If yes, how can I config serverless.yml to fix this issue?

Thanks
Mingtong

Files removed from S3 even with noSync: true

Hi, in my serverless.yml file I've included the following:

plugins:
  - serverless-s3-sync
custom:
  s3Sync:
    buckets:
    - bucketName: my-bucket
      bucketPrefix: folder/
      localDir: config/folder
      deleteRemoved: false
      noSync: true # Prevent removal of objects with `sls remove`

However, when I run sls remove -s stage, the objects are still getting removed from the bucket.

npm ERR! notarget No matching version found for [email protected].

We are using [email protected] in our application. We do deploy our code through CD which was working correct previously but now suddenly its getting failed with the below error. We checked the xml2js version which was upgraded recently in
[email protected].

npm ERR! code ETARGET
15:49:22 npm ERR! notarget No matching version found for [email protected].
15:49:22 npm ERR! notarget In most cases you or one of your dependencies are requesting
15:49:22 npm ERR! notarget a package version that doesn't exist.

We tried to use version 3.1.0 but getting below error after executing below command.
serverless plugin install --name [email protected] --stage dev --param=appVersion=5.0.6 --param=gitHash=7489b9c839

npm ERR! code ETARGET
17:39:22 npm ERR! notarget No matching version found for json-cycle@^1.4.0.
17:39:22 npm ERR! notarget In most cases you or one of your dependencies are requesting
17:39:22 npm ERR! notarget a package version that doesn't exist.
17:39:22
17:39:22 npm ERR! A complete log of this run can be found in:
17:39:22 npm ERR! /opt/jenkins/npm-repo-v2/_logs/2023-04-12T16_39_15_632Z-debug.log
17:39:22
17:39:22 Environment: linux, node 16.16.0, framework 3.25.0 (standalone), plugin 6.2.2, SDK 4.3.2
17:39:22 Docs: docs.serverless.com
17:39:22 Support: forum.serverless.com
17:39:22 Bugs: github.com/serverless/serverless/issues
17:39:22
17:39:22 Error:
17:39:22 Error: node /snapshot/serverless/node_modules/npm/bin/npm-cli.js install --save-dev [email protected] Exited with code 1
17:39:22 at ChildProcess. (/snapshot/serverless/node_modules/child-process-ext/spawn.js:38:8)
17:39:22 at ChildProcess.emit (node:events:527:28)
17:39:22 at ChildProcess.emit (node:domain:475:12)
17:39:22 at maybeClose (node:internal/child_process:1092:16)
17:39:22 at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)

Crash! callback called twice

eg.

 s3Sync:
    - bucketName: bucket1
      bucketPrefix: prefix1
      localDir: dir1

    - bucketName: bucket1
      bucketPrefix: prefix2
      localDir: dir2

    - bucketName: bucket1
      bucketPrefix: prefix3
      localDir: dir3

`S3 Sync: Syncing directories and S3 prefixes...
..../usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:31
throw err;
^

Error: callback called twice
at onCb (/Users/xxxxxx/xxxxxxxxxxx/node_modules/pend/index.js:36:23)
at Response. (/Users/xxxxxx/xxxxxxxxxxx/node_modules/@monolambda/s3/lib/index.js:406:9)
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:364:18)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
at callNextListener (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:95:12)
at Request.RESET_RETRY_STATE (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:442:9)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:101:18)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
at callNextListener (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:95:12)
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:296:11)
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
at error (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:295:22)
at ClientRequest. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/http/node.js:83:7)
at Object.onceWrapper (events.js:313:30)
at emitNone (events.js:106:13)
at ClientRequest.emit (events.js:208:7)
at TLSSocket.emitTimeout (_http_client.js:708:34)
at Object.onceWrapper (events.js:313:30)
at emitNone (events.js:106:13)
at TLSSocket.emit (events.js:208:7)
at TLSSocket.Socket._onTimeout (net.js:407:8)
at ontimeout (timers.js:475:11)
at tryOnTimeout (timers.js:310:5)
at Timer.listOnTimeout (timers.js:270:5)`

Setting ContentType and ContentEncryption to file on s3

Hello everyone,
i'm trying to set metadata to some files using this plugin.

      defaultContentEncoding: gzip
      params: # optional
        - "*.js":
            ContentType: 'text/javascript'
        - "*.gz":
            ContentType: 'text/javascript'
            ContentEncoding: 'gzip'
        - "*.js.gz":
            ContentType: 'text/javascript'
            ContentEncoding: 'gzip'

unfortunately this is not working, the files still have the default metadata.

Folder structure in the bucket is
root -> public -> _FILES_

Cache-Control headers not updated if files have not changed.

If you first sync a folder via s3sync, then edit the metadata via params, the changes are not reflected in S3. Solution is to delete file manually from S3 and it will get the proper settings on next run.

On first run I had:

  hostName: <REDACTED>
  s3Sync:
    - bucketName: ${self:custom.hostName}
      localDir: static
      defaultContentType: text/html

Then to disable cache, I updated to:

  hostName: <REDACTED>
  s3Sync:
    - bucketName: ${self:custom.hostName}
      localDir: static
      defaultContentType: text/html
      params:
        - 'index.html':
          CacheControl: 'no-cache'

EDIT: This seems to also be the behavior of calling aws s3 sync... and passing a cache-control parameter. Unless file is modified, the cache is not reflected in s3.

Support syncing single file

Feature request - Support syncing single file

Need

I would like to use this plugin to sync a single local file to an S3 bucket created by Serverless. However, this plugin only supports syncing directories, as stated in the description.

Would you be open to extending this plugin so it can also sync simple files?

Contribution

I've already modified it on my side to make it work and am wondering if I could contribute it back.

Implementation

The main question is regarding the breaking change possibility.

While it is possible to just handle single file with its path passed through the current localDir parameter, it isn't really logical for users to path a file to something called localDir imho.

To avoid that, it would be possible to rename this field into localPath but it would then be a breaking change. It's also possible to introduce a new parameter called localFile which would be mutually exclusive with localDir, but that adds more conditional logic to the code.

What do you think about all of this? I'm ready to send a PR with whatever will be needed.

Lifecycle

Using hooks, I run a bash script to update a file with an endpoint address. I want to run s3-sync after updating the file. However s3-sync runs at first uploading the old file. How can I control the sequence of running?

The specified bucket does not exist error message

Hello,

I'd like to try this plugin but after initial yml setup I receive following error message:
The specified bucket does not exist
Does this package handle the creation/update of a folder somehow? I guess some BucketPolicy is missing but the README does not mention it. I'm new to serverless would be great to find some resource here

Serverless Framework v3: new design & how to integrate

Hi 👋 , Serverless Framework core team here!

We're working on the v3 beta version right now. The v3 will include a redesigned CLI output:

image

You can try the v3 beta with: npm i serverless@pre-3

To help plugins integrate with that new design (and build a great user experience), we've added new APIs for plugins:

We've identified 2 options for plugins:

  • either support both v2 and v3 in the current version: if the 3rd argument is passed to the plugin's constructor, you can switch logging to the new design and use the new CLI output APIs
  • or create a new version of the plugin that only supports v3

Any feedback is appreciated! If you want to update the plugin to use these new APIs, feel free to ping me if you have any questions!

If you don't have time to update the plugin, let me know, we'll see how we can help.

IAM Permissions needed to deploy and sync stack

I'm trying to identify which IAM permissions are needed to deploy this stack. I can deploy with full admin rights but not with limited.

I have the correct permissions to deploy the stack to cloud formation and all the resources, but it consistently fails when trying to sync the resources and I've burnt too many hours now trying to identify which is needed. (AWS doesn't seem to help with identifying what's required and neither does the verbose logging!)

I've tried and get AccessDenied: Access Denied.

{
  Effect: 'Allow',
  Action: ['*'],
  Resource: ['arn:aws:s3:::*'],
}

Audit vulnerability through dependency

I just installed serverless-s3-sync and was prompted with an audit warning through a dependency. Just wanted to mention it. Maybe it's an easy fix?

➜  server git:(next) ✗ npm audit
                                                                                
                       === npm audit security report ===                        
                                                                                
┌──────────────────────────────────────────────────────────────────────────────┐
│                                Manual Review                                 │
│            Some vulnerabilities require your attention to resolve            │
│                                                                              │
│         Visit https://go.npm.me/audit-guide for additional guidance          │
└──────────────────────────────────────────────────────────────────────────────┘
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Moderate      │ Regular Expression Denial of Service                         │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ mime                                                         │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Patched in    │ >= 1.4.1 < 2.0.0 || >= 2.0.3                                 │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ serverless-s3-sync                                           │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ serverless-s3-sync > @monolambda/s3 > mime                   │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://nodesecurity.io/advisories/535                       │
└───────────────┴──────────────────────────────────────────────────────────────┘
found 1 moderate severity vulnerability in 1007 scanned packages
  1 vulnerability requires manual review. See the full report for details.

add support of Fn::ImportValue

i proposed to add support of cloudformation Fn::ImportValue.

eg.

s3Sync:
    - bucketName: 
         Fn::ImportValue: otherStackS3Bucket
      bucketPrefix: assests/
      localDir: resources/feature/folder

No Such Key error

Hi i encounter this error on the sync. And i discover that the probleme not arrived when i flush all assets in the s3.

S3 Sync: Syncing directories and S3 prefixes...
............
S3 Sync: Synced.
S3 Sync: Syncing metadata...

No Such Key --------------------------------------------

NoSuchKey: The specified key does not exist.
at Request.extractError (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/services/s3.js:816:35)
at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at callNextListener (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
at IncomingMessage.onEnd (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:307:13)
at IncomingMessage.emit (events.js:228:7)
at IncomingMessage.EventEmitter.emit (domain.js:475:20)
at endReadableNT (_stream_readable.js:1185:12)
at processTicksAndRejections (internal/process/task_queues.js:81:21)

 For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com

Your Environment Information ---------------------------
Operating System: linux
Node Version: 12.15.0
Framework Version: 1.67.0
Plugin Version: 3.6.0
SDK Version: 2.3.0
Components Version: 2.22.3

Support S3 bucket tag updates

Use case:

  • main repo deploys CloudFormation stack, which includes S3 bucket necessary for SPA web app
  • web app repo uses a minimal serverless.yml with serverless-s3-sync plugin to build the SPA web app and publish the files to the existing S3 bucket created by main repo

I'd like to write/update a tag on the S3 bucket, containing information from the web app repo at the time of copying files. For example, write the git commit SHA and maybe branch name.

Configuration might look like this:

s3Sync:
    - bucketName: my-static-site # required
      localDir: dist # required
      bucketTags:
         tagKey1: tagValue1
         tagKey2: tagValue2

After sync, the plugin would use putBucketTagging to write the tags - see docs.

Writing tags is not a "tag sync". This means the tags configured are written as-is, therefore:

  • an existing tagKey will be overwritten if it exists
  • renaming a tagKey will NOT delete the previously written tagKey with the old value
  • removing a previously written tagKey from the bucketTags entry will NOT remove it from the bucket (on the next sync)
    The reason for this behavior is to make sure tags added by other tools (e.g. Serverless deployment that created the bucket) are left untouched when running s3sync.

@k1LoW Does the above sound good? I can fork and submit a pull request with the implementation.

sync wont work

sls s3sync command does not seem to work for me..Help appreciated!!

sls version 1.26.1
python 3.6
"serverless-s3-sync": "^1.4.0"

Config :
custom:
pythonRequirements:
dockerizePip: true
defaultStage: dev
currentStage: ${opt:stage, self:custom.defaultStage}
s3Sync:
- bucketName: ${file(configs/config.${self:custom.currentStage}.json):syncBucketName}
localDir: required-artifacts/${self:custom.currentStage} # required
sse: AES256

Error:
$ sls s3sync
S3 Sync: Syncing directories and S3 prefixes...
~/node_modules/aws-sdk/lib/request.js:31
throw err;
^

Error: connect ECONNREFUSED 52.216.226.48:443

I am operating behind a corp proxy and have the proxy settings exported in env variables
also have

HTTPS_PROXY=<>
HTTP_PROXY=<>
NPM_CONFIG_PROXY=<>

S3 sync fails if provider.region is a variable

This means we have to duplicate variables in a mono-repo because the references to shared variables break s3-sync. It appears that s3-sync does not attempt to resolve the variable at all.

AWS provider.region not set

in index.js on line 77 we are not setting the region, we need to update it like this below so that we can get the region:

   } else {
      //in this else condition, region is never set, so setting it and assigning to correct region issue
      region = provider.getRegion()
      awsCredentials = provider.getCredentials().credentials
    }

The S3 option deleteRemoved is not documented in the readme

Hi there!

I was having an issue with this plugin removing js files from a create-react-app deploy.

Users would have these files cached and they would then 404 on after a deploy because serverless-s3-sync defaults to deleteRemoved: true.

https://github.com/k1LoW/serverless-s3-sync/blob/master/index.js#L80

Are you open to a PR that outlines this option in the readme? It took me some source diving to discover that it was possible not to delete removed files.

I'm more than happy to write the documentation.

Thanks!

Move hook to before:deploy:finalize rather than after:deploy:deploy

I'm attempting to have a single source project containing both React app and serverless spec, and have sls deploy:

  1. Deploy backend resources to AWS
  2. Use serverless-stack-output to update React app with details from CloudFormation
  3. Run npm build
  4. Use serverless-s3-sync to copy the React build/ directory up to S3.

Currently both serverless-stack-output and serverless-s3-sync run off after:deploy:deploy asynchronously. I'm not sure there's a way of defining the ordering or dependency between them without writing my own plugin, but it might be sufficient to move S3 sync later (and stack output earlier).

I think this could be achieved by changing after:deploy:deploy to before:deploy:finalize in index.js.

(Alternatively is there a way of defining the {{nos3sync}} for S3 deploy, and I can have the stack-output-handler maybe invoke S3 sync directly?)

Cache-control headers not getting set

We're using serverless-s3-sync along with gatsby-plugin-s3 to deploy a site to S3 and Cloudfront.

In serverless.yml:

custom:
  frontendBucketName: ${file(./.cache/s3.config.json):bucketName, env:AWS_ORIGIN_S3_BUCKET}
  videoBucketName: ${self:service}-video-${self:provider.stage}
  s3Sync:
  - bucketName: ${self:custom.frontendBucketName}
    localDir: public
    acl: public-read
    defaultContentType: text/html
    params: ${file(./.cache/s3.params.json)}
  certificates:
    ...

If I look in .cache/s3.params.json it all looks as expected: (reformatted)

{
  "**/**.html": {
    "CacheControl": "public, max-age=0, must-revalidate"
  },
  "static/**": {
    "CacheControl": "public, max-age=31536000, immutable"
  },
  "**/**/!(sw).js": {
    "CacheControl": "public, max-age=31536000, immutable"
  },
  "**/**.css": {
    "CacheControl": "public, max-age=31536000, immutable"
  },
  "sw.js": {
    "CacheControl": "public, max-age=0, must-revalidate"
  }
}

Yet the files which end up on S3, which certainly match those paths (such as static/1-52e5afcd825a509c9e73.js) only have one metadata attached, the Content-Type.

Feels like there's something obvious we must be missing.

Using [email protected]

Public Access

Hi! It's a great plugin! Thanks so much!
Just one trouble.
I have Access Denied when trying sync data. I have full Admin Permission in IAM. Do you have any ideas how to fix it? If I add public write/read access everything works fine.

Also, probably you know how to set Public Access to the bucket with serverless.yml? I tried Canned ACL but it doesn't work for me. http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html
My yml file is here https://github.com/BestMood-Tech/media-shop-service/blob/refactoring/serverless.yml
Best regards,
Andrew

Documentation error

I think this line

$ npm install --save serverless-s3-sync

should be

$ npm install --save-dev serverless-s3-sync

Files uploaded twice with bucketPrefix

Hi guys,

Im a new user of s3sync with serverless but I noticed my files are uploaded 2 times if I use bucketPrefix.
My files are always uploaded in the root AND in the bucketPrefix/folder.

custom:
  s3Sync:
    - bucketNameKey: AssetBucketRef
      bucketPrefix: assets
      localDir: ./dist/
      deleteRemoved: true
      acl: public-read
      params:
        - "*.*":
            CacheControl: 'public, max-age=31536000'

This results in a /assets/xxx.json file AND a /xxx.json file.
If I remove files and deleteRemoved is enabled, only the files in the bucketPrefix folder are removed, the files in the root are not removed.

Can someone check this please?

Thanks

Access Denied

My deploy account has full aws admin access and yet when it gets to the s3 sync part it fails with access denied?
https://www.dropbox.com/s/1o4w2g0ohvdhfwn/Screenshot%202018-10-15%2017.06.37.png?dl=0

My yaml is very simple - here's the relevant bits:

custom:
  s3Sync:
    - bucketName: mybucketname
      localDir: emails/assets
      acl: public-read

resources:
  Resources:
    EmailAssetsBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: mybucketname
        AccessControl: PublicRead

The s3 bucket is created as part of the standard serverless deploy and I can see it in aws - I can't see what s3-sync is doing which is failing?

SPA with react + react-router

Is there a way to configure this for use with a Single Page Application. Currently using react-router and if you refresh a route or link to a route other than root it sends the page as a 404. It appears to work for the user because ErrorDocument also points to index.html, however if you open up the console you can see it is serving it as a 404 page

TypeError: Cannot read property 'hasOwnProperty' of undefined at serverlessS3Sync.sync

Hi, i am trying to deploy my react app frontend application using serverless and the serverlessS3Sync plugin and upon running the command sls deploy -v, everything works fine up until the end and it throws the error

Type Error ---------------------------------------------- TypeError: Cannot read property 'hasOwnProperty' of undefined at ServerlessS3Sync.sync (/Users/dami/dev/capitack/capitack-main-frontend/node_modules/serverless-s3-sync/index.js:112:15) at ServerlessS3Sync.tryCatcher (/Users/dami/dev/capitack/capitack-main-frontend/node_modules/bluebird/js/release/util.js:16:23) at Promise._settlePromiseFromHandler (/Users/dami/dev/capitack/capitack-main-frontend/node_modules/bluebird/js/release/promise.js:547:31) at Promise._settlePromise (/Users/dami/dev/capitack/capitack-main-frontend/node_modules/bluebird/js/release/promise.js:604:18) at Promise._settlePromiseCtx (/Users/dami/dev/capitack/capitack-main-frontend/node_modules/bluebird/js/release/promise.js:641:10) at _drainQueueStep (/Users/dami/dev/capitack/capitack-main-frontend/node_modules/bluebird/js/release/async.js:97:12) at _drainQueue (/Users/dami/dev/capitack/capitack-main-frontend/node_modules/bluebird/js/release/async.js:86:9) at Async._drainQueues (/Users/dami/dev/capitack/capitack-main-frontend/node_modules/bluebird/js/release/async.js:102:5) at Immediate.Async.drainQueues [as _onImmediate] (/Users/dami/dev/capitack/capitack-main-frontend/node_modules/bluebird/js/release/async.js:15:14) at processImmediate (internal/timers.js:462:21)

I have checked the plugin files in the node_modules folder and the issue is origination from these lines:
sync() { let s3Sync = this.serverless.service.custom.s3Sync; if(s3Sync.hasOwnProperty('buckets')) { s3Sync = s3Sync.buckets; } .......

Please is there any fix for this? please i need this ASAP as i am tasked with deploying the company's frontend on aws. Thanks and i await your responses!

Using Ref as bucketName

It seems like whenever we use something like {Ref: WebSite} for the bucketname we get an error like....

Expected params.Bucket to be a string

Exclude files

Is it possible to exclude files by pattern\directory? For example i dont want to upload source map

Bucket not found

In the latest version,
The bucket name isnt being read unless globally installed. When rolled back to v1.7.2 works just fine.

It directly throws error msg stating,

S3 Sync: Synced metadata.
No Such Bucket -----------------------------------------

Please keep the backward support on.
@k1LoW @nibynool

CacheControl header and other additional header turned into User-defined x-amz-meta-* metadata?

Hi, the use case is to set the System metadata for objects uploaded on S3 buckets. However, the result turned into User-defined metadata with prefix x-amz-meta- which is not expected.

e.g.

- bucketName: ${self:custom.bucketName}
      defaultContentType: "text/javascript"
      params: # optional
        - "*.js":
            CacheControl: "public, max-age=31536000"
            ContentType: "text/javascript"

Turned into:

Screen Shot 2020-02-06 at 11 47 13 AM

While the existing system metadata Content-type value remains intact. i.e. not synced
Also, the defaultContentType does not work either, instead of setting it to text/javascript, it always fall back to application/octet-stream

Question

Is there a support to set or override System metadata for objects uploaded on S3 buckets?
Many thanks!

Validation errors when specifying cache-control

I'm trying to specify cache-control headers with this syntax:

  s3Sync:
    - bucketName: ${self:custom.s3Bucket} 
      localDir: .dir/sub_dir
      bucketPrefix: _dir/sub_dir 
      params:
        - '**/*.js':
          CacheControl: 'public, max-age=31536000, immutable'
        - '**/*.css':
          CacheControl: 'public, max-age=31536000, immutable'
        - '**/*.woff':
          CacheControl: 'public, max-age=31536000, immutable'

But am getting this error when trying to run sls deploy

S3 Sync: Syncing directories and S3 prefixes...
.... 
  Multiple Validation Errors -----------------------------
 
  MultipleValidationErrors: There were 35 validation errors:
  * UnexpectedParameter: Unexpected key '0' found in params
  * UnexpectedParameter: Unexpected key '1' found in params
  * UnexpectedParameter: Unexpected key '2' found in params
  * UnexpectedParameter: Unexpected key '3' found in params
  * UnexpectedParameter: Unexpected key '4' found in params
.
.
.

Not sure quite what I'm doing wrong here

Cannot read property 'defaultEncoding'

I had installed the plugin by following command in serverless 2.53.1.

npm install --save-dev serverless-s3-sync

After installing I had imported the plugin like this in serverless.yml

plugins:

  • serverless-plugin-common-excludes
  • serverless-plugin-include-dependencies
  • serverless-python-requirements
  • serverless-s3-sync

But (serverless deploy) is giving the following error. Screenshot of error is also attached. (This error only happens when I import this plugin of serverless-s3-sync in serverless.yml file).
eror

ERROR DETAILS:
TypeError: Cannot read property 'defaultEncoding' of undefined
at _write (node:internal/streams/writable:291:24)
at WriteStream.Writable.write (node:internal/streams/writable:334:10)
at NodejsStreamOutputAdapter.ondata (C:\serverless_v2\node_modules\readable-stream\lib_stream_readable.js:619:20) at NodejsStreamOutputAdapter.emit (node:events:394:28)

bucketPrefix uploads one file too much (not in localDir)

Very small serverless.yml:

service: example

provider:
    name: aws
    stage: ${opt:stage, 'dev'}
    region: eu-central-1
    runtime: provided.al2

custom:
    s3Sync:
        -   bucketNameKey: UpdateBucketName
            bucketPrefix: app/
            localDir: update
            params:
                -   "*.json":
                        CacheControl: 'no-cache'

plugins:
    - serverless-s3-sync

resources:
    Resources:
        UpdateBucket:
            Type: AWS::S3::Bucket
            Properties:
                AccessControl: PublicRead
        UpdateBucketPolicy:
            Type: AWS::S3::BucketPolicy
            Properties:
                Bucket: !Ref UpdateBucket
                PolicyDocument:
                    Statement:
                        -   Effect: Allow
                            Principal: '*'
                            Action: 's3:GetObject'
                            Resource: !Join [ '/', [ !GetAtt UpdateBucket.Arn, '*' ] ]

    Outputs:
        UpdateBucketName:
            Value: !Ref UpdateBucket

Deployment works successful.

S3 Sync: Syncing directories and S3 prefixes...
...........
S3 Sync: Synced.
S3 Sync: Syncing metadata...
.
S3 Sync: Synced metadata.
S3 Sync: Updating bucket tags...
.
S3 Sync: Updated bucket tags.
Serverless: Removing old service artifacts from S3...

And then I have one file too much in my bucket.

manifest.json
app/manifest.json
app/update.zip.001
app/update.zip.002
app/update.zip.003
app/update.zip.004
app/update.zip.005

Everything fine except the manifest.json which is in root directory.
But I used the bucketPrefix so that this shouldn't happen.

The timestamp is like the others. When I remove the file manually and run sls deploy again the file is again there.

Maybe there is an logical issue with the bucketPrefix?

deleteRemoved: false is not working always

Hello,

i figured out that there is an issue, if one directory has the option deletedRemoved: true and all other are using deletedRemoved: false.

s3Sync:
    - bucketName: belayus-${self:service}-${opt:stage, self:provider.stage}
      bucketPrefix: ${opt:stage, self:provider.stage}/_next/static-pages
      localDir: .serverless_nextjs/assets/static-pages/
      acl: private
      deleteRemoved: false
      
    - bucketName: belayus-${self:service}-${opt:stage, self:provider.stage}
      bucketPrefix: ${opt:stage, self:provider.stage}/_next/static/
      localDir: .serverless_nextjs/assets/_next/static
      acl: private
      deleteRemoved: false

    - bucketName: belayus-${self:service}-${opt:stage, self:provider.stage}
      bucketPrefix: ${opt:stage, self:provider.stage}/_next/data/
      localDir: .serverless_nextjs/assets/_next/data
      acl: private
      deleteRemoved: false

    - bucketName: belayus-${self:service}-${opt:stage, self:provider.stage}
      bucketPrefix: ${opt:stage, self:provider.stage}
      localDir: env-public-files/${opt:stage, self:provider.stage}/
      acl: private
      deleteRemoved: true

some files of the other configurations will be deleted even if they are configured with deleteRemoved: false.

After i set the last one to false. it works like expected.

I was trying to dig deeper but didn't found the exact issue in your code.

Thanks for checking.

timeout on sync folder with folder around 300mb

following exception i have no idea what the problem is:

Error: callback called twice
at onCb (/Users/dev/Desktop/tmp/..../node_modules/pend/index.js:36:23)
at Response. (/Users/dev/...../node_modules/@auth0/s3/lib/index.js:308:11)
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:369:18)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at callNextListener (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
at Request.RESET_RETRY_STATE (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:510:9)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:102:18)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at callNextListener (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
at IncomingMessage.onEnd (/usr/local/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:336:13)
at IncomingMessage.emit (events.js:327:22)
at IncomingMessage.EventEmitter.emit (domain.js:467:12)
at endReadableNT (internal/streams/readable.js:1327:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.