Giter VIP home page Giter VIP logo

webpack-google-cloud-storage-plugin's Introduction

webpack-google-cloud-storage-plugin

npm version

A Webpack plugin to upload assets in Google Cloud Storage.

Installation

npm install --save webpack-google-cloud-storage-plugin

Usage

// In your webpack.config.js
import WebpackGoogleCloudStoragePlugin from 'webpack-google-cloud-storage-plugin';

module.exports = {
  ...
  plugins: [
    new WebpackGoogleCloudStoragePlugin({
      directory: './src',
      // NOTE: Array of regexes matching filenames to include in the uploading process
      include: ['app.js'],
      // NOTE: Array of regexes matching filenames to exclude in the uploading process
      exclude: ['cats.js'],
      // NOTE: Options passed directly to Google cloud Node Storage client. This is 
      // mostly for authentication-wise. For more information:
      // https://github.com/GoogleCloudPlatform/google-cloud-node/tree/master/packages/storage#authentication
      storageOptions: {
        // projectId is optional if you specify a keyFilename
        projectId: 'grape-spaceship-123',
        // keyFilename: '/path/to/keyfile.json'
        // keyFilename: './examples/my-credentials.json', // note: Filename, not FileName
        // key: 'mykey',
        // credentials: require('/path/to/credentials.json'),
      },
      // NOTE: Options used by
      // WebpackGoogleCloudStoragePlugin
      // regarding uploading.
      uploadOptions: {
        // Where to upload files
        bucketName: 'my-bucket-name',
        // NOTE: Prefix to add in the bucket file path.
        // E.g: app.js => assets/v1/app.js,
        // file is an object with { name:, path: }.
        destinationNameFn: file =>
           path.join('assets', file.path)
        ,
        // Available properties in returned object are defined here;
        // https://cloud.google.com/storage/docs/json_api/v1/objects/insert#request_properties_JSON
        // It will be passed into 'upload' here:
        // https://github.com/googleapis/nodejs-storage/blob/master/samples/files.js#L116
        metadataFn: file => ({
          // cache it a little longer!
          cacheControl: 'public, max-age=31536000',
        }),
        // Make gzip compressed (default: false)
        gzip: true,
        // Make file public (default: false)
        makePublic: true,
        // Resumable upload (default: true)
        // https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload
        resumable: true,
        // Concurrency ratio when uploading files (default: 10)
        concurrency: 5,
      },
    }),
  ],
};

Examples

Check out the examples folder for a working demo.

Add your credentials to storageOptions and set uploadOptions.

Then you can run the demo webpack using npm run example.

webpack-google-cloud-storage-plugin's People

Contributors

alexgorbatchev avatar bw avatar haf avatar hiramhuang avatar pythonicninja avatar syndbg avatar yocontra avatar youanden avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

webpack-google-cloud-storage-plugin's Issues

upload distribution directory directly to bucket

Hi, I have a React.JS project I'm trying to deploy to a domain-named bucket.
the example code has no 'dist' directory, but 'dist/' is where the post-compilation code is dumped, i tried using 'destinationNameFn' as mentioned below to change the output directory path (as below) but that doesn't work.
it just creates a symlink to $BUCKET/dist/.. and then leaves $BUCKET/ empty
:|

        new WebpackGoogleCloudStoragePlugin({
              directory: './dist',
              // NOTE: Array of filenames to include in the uploading process
              include: ['.js', '.html', '.png', '.eot', '.ttf', '.woff', '.woff2', '.svg'],
              // NOTE: Array of filenames to exclude in the uploading process
              exclude: ['.js.map'],
              // NOTE: Options passed directly to
              // Google cloud Node Storage client.
              // This is mostly authentication-wise.
              // For more information:
              // https://github.com/GoogleCloudPlatform/google-cloud-node/tree/master/packages/storage#authentication
              storageOptions: {
                  projectId: 'my-project-id-12345',
                  keyFilename: './config/keyfile.json',
              },
              // NOTE: Options used by
              uploadOptions: {
                  // Where to upload files
                  bucketName: 'example.domain.com',
                  // NOTE: Currently not tested and most likely not working.
                  forceCreateBucket: false,
                  destinationNameFn: file => path.join('../', file.path),
              },
        })

Avoiding overwriting exactly the same content

This plugin looks great, possibly just what I was looking for to move my websites static files over to Cloud Storage.

Upon looking at the configuration I see there is no option for "don't overwrite" or similar. But understandably, you do want to overwrite changed files even when the file name has not changed.

A way to possibly solve this could be a file hash of sorts for all the files - which is then saved to a separate file in Google Cloud Storage, hashes.json or something, which is first downloaded and compared before running the upload on only those files who's hash does not exist in the hashes.json file - and then overwriting the hashes.json file with all the new hashes. At least that sounds kind of reasonable in my head. What do you think? Or is uploading the same files over again not too much of a concern from your side?

I could look into implementing it at some point. My internet isn't great and static folders tend to get big sometimes, to re-upload every file on every build might be too much.

Issues uploading for large projects

It seems that the current strategy to upload will cause a bottleneck for client / server for larger projects. We have some projects with many files due to locale support so we see socket hang up and ECONNRESET due to the fire hose of attempted uploads to GCS. It would be great to have controls on concurrency and retries. And I haven't looked yet, but it would be great to make sure files that already exist aren't re-uploaded or overwritten.

ERROR in WebpackGoogleCloudStoragePlugin: Error: The uploaded data did not match the data from the server. As a precaution, the file has been deleted. To be sure the content is the same, you should try uploading the file again.

Looks related to #14

Error: connect EHOSTDOWN 169.254.169.254:80 - Local

@syndbg @bw @youanden @chown9835
Error: connect EHOSTDOWN 169.254.169.254:80 - Local

my webpack.config.js:

new WebpackGoogleCloudStoragePlugin({
      directory: './dist',
      // NOTE: Array of filenames to include in the uploading process
      include: ['.js', '.html', '.css', '.svg', '.png', '.jpg'],
      // NOTE: Array of filenames to exclude in the uploading process
      // exclude: ['cats.js'],
      // NOTE: Options passed directly to
      // Google cloud Node Storage client.
      // This is mostly authentication-wise.
      // For more information:
      // https://github.com/GoogleCloudPlatform/google-cloud-node/tree/master/packages/storage#authentication
      storageOptions: {
        projectId: 'project_id',
      },
      // NOTE: Options used by
      // WebpackGoogleCloudStoragePlugin
      // regarding uploading.
      uploadOptions: {
        // Where to upload files
        bucketName: 'bucket-with-my-domain.com',
        destinationNameFn: file => file.path.replace('dist/', ''),
        forceCreateBucket: true,
        gzip: true,
        makePublic: true,
      },
    }),

traceback:

events.js:163
      throw er; // Unhandled 'error' event
      ^

Error: connect EHOSTDOWN 169.254.169.254:80 - Local (192.168.1.143:51072)
    at Object.exports._errnoException (util.js:1050:11)
    at exports._exceptionWithHostPort (util.js:1073:20)
    at internalConnect (net.js:889:16)
    at lookupAndConnect (net.js:977:5)
    at Socket.realConnect (net.js:945:5)
    at Agent.connect [as createConnection] (net.js:77:22)
    at Agent.createSocket (_http_agent.js:195:26)
    at Agent.addRequest (_http_agent.js:157:10)
    at new ClientRequest (_http_client.js:212:16)
    at Object.request (http.js:26:10)
    at Request.start (/node_modules/google-auto-auth/node_modules/request/request.js:740:32)
    at Request.end (/node_modules/google-auto-auth/node_modules/request/request.js:1394:10)
    at end (/node_modules/google-auto-auth/node_modules/request/request.js:567:14)
    at Immediate.<anonymous> (/node_modules/google-auto-auth/node_modules/request/request.js:581:7)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)

deprecation error

(node:13210) DeprecationWarning: Tapable.plugin is deprecated. Use new API on `.hooks` instead
    at WebpackGoogleCloudStoragePlugin.apply (/Users/contra/Projects/staeco/node_modules/webpack-google-cloud-storage-plugin/dist/webpack-google-cloud-storage-plugin.js:218:17)
    at webpack (/Users/contra/Projects/staeco/node_modules/webpack/lib/webpack.js:37:12)
    at Object.<anonymous> (/Users/contra/Projects/staeco/bin/build/cli.js:17:18)
    at Module._compile (internal/modules/cjs/loader.js:702:30)
    at loader (/Users/contra/Projects/staeco/node_modules/babel-register/lib/node.js:144:5)
    at Object.require.extensions.(anonymous function) [as .js] (/Users/contra/Projects/staeco/node_modules/babel-register/lib/node.js:154:7)
    at Module.load (internal/modules/cjs/loader.js:612:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
    at Function.Module._load (internal/modules/cjs/loader.js:543:3)
    at Module.require (internal/modules/cjs/loader.js:650:17)
    at require (internal/modules/cjs/helpers.js:20:18)
    at Object.<anonymous> (/Users/contra/Projects/staeco/bin/build/index.js:2:18)
    at Module._compile (internal/modules/cjs/loader.js:702:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
    at Module.load (internal/modules/cjs/loader.js:612:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
    at Function.Module._load (internal/modules/cjs/loader.js:543:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:744:10)
    at startup (internal/bootstrap/node.js:240:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:564:3)

just a heads up ^ looks like the latest webpack just started doing this today

How to make file public

I need upload and make public the webpack bundle, but i did not see any options in your plugin.

Incompatible `exclude` filter types, glob style and regexp

It seems at version 0.8.2 there was an incompatible change rolled-out: cc89418

Notably, the exclude option was changed, but was not updated for the recursive-readdir package https://github.com/syndbg/webpack-google-cloud-storage-plugin/blob/master/src/index.js#L163.

Comparing, the two patterns are incompatible, and will not compile if they are used https://github.com/syndbg/webpack-google-cloud-storage-plugin/blob/master/src/index.js#L139

Error output (poor visibility for debugging)

node_modules/minimatch/minimatch.js:116
    throw new TypeError('glob pattern string required')
    ^

when the webpack for exclude uses RegExp style patterns.

GCS Resumable Upload Error

I got this error twice while trying to upload file with webpack. Can anyone help?

node_modules/gcs-resumable-upload/index.js:326
  this.configStore.del(this.file)

TypeError: this.configStore.del is not a function
    at Upload.deleteConfig (/node_modules/gcs-resumable-upload/index.js:326:20)
    at Request.<anonymous> (/node_modules/gcs-resumable-upload/index.js:172:12)
    at emitTwo (events.js:111:20)
    at Request.emit (events.js:191:7)
    at Request.<anonymous> (/node_modules/request/request.js:1171:10)
    at emitOne (events.js:101:20)
    at Request.emit (events.js:188:7)
    at IncomingMessage.<anonymous> (/node_modules/request/request.js:1091:12)
    at IncomingMessage.g (events.js:291:16)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)

Update vulnerable packages

There's a bunch of vulnerable packages that would need an update:

GHSA-pw2r-vq6v-hr8c|webpack-google-cloud-storage-plugin>@google-cloud/storage>@google-cloud/common>google-auto-auth>gcp-metadata>axios>follow-redirects
GHSA-pw2r-vq6v-hr8c|webpack-google-cloud-storage-plugin>@google-cloud/storage>@google-cloud/common>google-auto-auth>google-auth-library>gcp-metadata>axios>follow-redirects
GHSA-74fj-2j2h-c42q|webpack-google-cloud-storage-plugin>@google-cloud/storage>@google-cloud/common>google-auto-auth>gcp-metadata>axios>follow-redirects
GHSA-74fj-2j2h-c42q|webpack-google-cloud-storage-plugin>@google-cloud/storage>@google-cloud/common>google-auto-auth>google-auth-library>gcp-metadata>axios>follow-redirects
GHSA-8fr3-hfg3-gpgp|webpack-google-cloud-storage-plugin>@google-cloud/storage>@google-cloud/common>google-auto-auth>google-auth-library>gtoken>google-p12-pem>node-forge
GHSA-2r2c-g63r-vccr|webpack-google-cloud-storage-plugin>@google-cloud/storage>@google-cloud/common>google-auto-auth>google-auth-library>gtoken>google-p12-pem>node-forge
GHSA-x4jg-mjrx-434g|webpack-google-cloud-storage-plugin>@google-cloud/storage>@google-cloud/common>google-auto-auth>google-auth-library>gtoken>google-p12-pem>node-forge
GHSA-cfm4-qjh2-4765|webpack-google-cloud-storage-plugin>@google-cloud/storage>@google-cloud/common>google-auto-auth>google-auth-library>gtoken>google-p12-pem>node-forge
GHSA-4w2v-q235-vp99|webpack-google-cloud-storage-plugin>@google-cloud/storage>@google-cloud/common>google-auto-auth>gcp-metadata>axios
GHSA-4w2v-q235-vp99|webpack-google-cloud-storage-plugin>@google-cloud/storage>@google-cloud/common>google-auto-auth>google-auth-library>gcp-metadata>axios
GHSA-cph5-m8f7-6c5x|webpack-google-cloud-storage-plugin>@google-cloud/storage>@google-cloud/common>google-auto-auth>gcp-metadata>axios
GHSA-cph5-m8f7-6c5x|webpack-google-cloud-storage-plugin>@google-cloud/storage>@google-cloud/common>google-auto-auth>google-auth-library>gcp-metadata>axios

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.