floydspace / serverless-esbuild Goto Github PK
View Code? Open in Web Editor NEW๐จ A Serverless framework plugin to bundle JavaScript and TypeScript with extremely fast esbuild
License: MIT License
๐จ A Serverless framework plugin to bundle JavaScript and TypeScript with extremely fast esbuild
License: MIT License
When using serverless-offline and making changes to files the memory usage of the esbuild process seems to grow exponentially.
I start out with about 1 GB of ram usage and after 10 changes the esbuild process is using 8 GB of ram.
#87 - this PR fixed the individual function packaging, but I'm noticing sourcemaps are still generated for the other functions (not the js files, just the js.map extensions).
To repro just have:
esbuild:
sourcemap: true
package:
individually: true
ps. thanks @olup for fixing the individual packaging bug!
Hey! Not sure if I'm doing something wrong here, but I see the sourcemap next to my files and it's not being parsed by my log viewer... Anyone else using source maps?
Hey, really awesome initiative, got our build process from ~15 minutes to ~30 seconds!
I noticed one issue with the resolving of the config variables when using e.g. the ${file
import. Those are not resolved during the constructor call yet, e.g.
esbuild:
packager: npm
bundle: true
define: ${file(./serverless.js):define}
will show up as
{
packager: npm,
bundle: true,
define: ${file(./serverless.js):define},
}
rather than
{
packager: npm,
bundle: true,
define: {
some: 'resolved',
variables: 123
}
I put up a PR #132 moving the config generation outside of the constructor which fixes that behaviour. It does now regenerate the config at every request to this.buildOptions
but it seemed to me that this might be just as well since there is some deletion going on in the bundle
step and this is likely super infrequent. If you mind that, I can cache the config file.
Hello!
First of all i have to say that i am not sure if it's a bug really. Probably i missing something of configuration. The problem occurs when i build a project with apollo-server-lambda
and in this project includes some .graphql
files. Theses files aren't included in .build folder. So the error is the following:
Unable to find any GraphQL type definitions for the following pointers
/.esbuild/.build/src/graphql/schema.graphql
these .graphql
isn't in that build folder.
How can i bundle these kind of files?
Thanks!
After looking for lifecycleEvents
in the source code, I didn't find any hookable events I could use to customize the serverless-esbuild
plugin behavior.
However it is common best practice to expose such events when you write a plugin (See here).
For example, serverless-offline
exposes init
, ready
and end
event that serverless-esbuild
actually uses to build the module before launching serverless-offline (See here).
I would need this hook to copy some static assets before bundling the package with esbuild
.
Let me know if you want me to open the PR.
serverless.yml
custom:
esbuild:
target: es2019
command
$ sls package
output:
Error --------------------------------------------------
Error: Invalid target: "e,s,2,0,1,7,9" (valid: esnext, es6, es2015, es2016, es2017, es2018, es2019, es2020)
Online Reproducibility Demo is Here
The README says
All files from package/include will be included in the final build file. See Exclude/Include
But package.include/exclude has been deprecated and replaced by package.patterns.
However, if I do something like
myFn:
package:
patterns:
- resources/cert.cnf
that file is not included in the zip.
If I do
myFn:
package:
patterns:
- '!**'
that doesn't have any effect either. It seems to just be ignored.
After upgrading serverless to the just-released version 2.36.0, started receiving the following error:
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
at validateString (internal/validators.js:124:11)
at Object.join (path.js:1039:7)
at new EsbuildPlugin (<project dir>/node_modules/serverless-esbuild/dist/index.js:48:33)
The problem appears to be dependence on the serverless.config.servicePath
prop here:
serverless-esbuild/src/index.ts
Line 75 in 753e4ae
This appears to have been renamed in 2.36.0:
https://github.com/serverless/serverless/releases/tag/v2.36.0
Currently trying to package knex
with pg
. This was also a problem serverless-webpack
faced but by force including the required packages it worked in the end. This is the only issue that I could find related to knex and esbuild but doesn't seem to solve my problem.
This is the error I get when running sls package
:
# ...
Serverless: Packing external modules: knex@^0.95.4, mysql, mysql2, pg@^8.6.0, pg-native, sqlite3, tedious
Error ---------------------------------------------------
Error: npm install failed with code 1
at ChildProcess.<anonymous> (/home/arpad/workspace/sls-webpack-v-esbuild/esbuild/node_modules/serverless-esbuild/dist/utils.js:48:24)
at ChildProcess.emit (events.js:314:20)
at ChildProcess.EventEmitter.emit (domain.js:483:12)
at maybeClose (internal/child_process.js:1022:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:287:5)
# ...
To note that it also tries to include mysql
, mysql2
, pg-native
, sqlite3
and tedious
and I'm not sure how I can ignore them.
And this is my config:
// ...
custom: {
esbuild: {
bundle: true,
minify: false,
external: ['knex', 'pg'],
},
},
package: {
individually: true,
},
// ...
I could not get this plugin to auto compile on save with serverless-offline
plugin.
Does the plugin support this function or I am missing something?
I am running the plugin with the default configuration.
Thanks
Hi,
it would be nice to pass the whole serverless instance as we need to configure esbuild plugins according to our serverless settings.
Something like this maybe?
custom:
esbuild:
plugins: plugins.js
and
const getSecretConfigEnvs = (slsw) => {
const { custom } = slsw.lib.serverless.service
const secretConfigEnvs =
custom && custom.secretConfigEnvs ? custom.secretConfigEnvs : {}
return {
'process.env.__SECRETS__': JSON.stringify(secretConfigEnvs)
}
}
module.exports = (serverless) => {
const definePlugin = {
name: 'auto-node-env',
setup(build) {
const options = build.initialOptions
options.define = options.define || {}
options.define = {
...options.define,
...getSecretConfigEnvs(serverless)
}
}
}
return [
definePlugin
]
}
What do you think?
I'm using serverless 2.39.0, also tried 2.38.0. Both package and deploy commands get stuck at the end and fail to exit the process without any clear error. Outputs are below and are with verbose+debug enabled.
Tried different configs, but it keeps on hanging on that moveToArtifactsToPackage step. Deploying still does work though, but the process will need to be cancelled with ctrl+c.
If I exchange serverless-esbuild
with serverless-bundler
, the process exits correctly.
Logs (package)
....
Serverless: Packaging service...
Serverless: Installing dependencies for custom CloudFormation resources...
Serverless: Invoke aws:package:finalize
Serverless: Invoke aws:common:moveArtifactsToPackage
Logs (deploy)
....
Serverless: Invoke aws:deploy:finalize
Serverless: Removing old service artifacts from S3...
Serverless: Publishing service to the Serverless Dashboard...
Serverless: Successfully published your service to the Serverless Dashboard: xxxx
Config
custom:
esbuild:
packager: npm
bundle: true
minify: false
sourcemap: true
target: node14
watch:
pattern: ['./index.ts', 'src/**/*.ts'] # default .
ignore: ['.serverless/**/*', '.build'] # default ['.build', 'dist', 'node_modules']
Any idea what could be wrong?
Besides the 'aws-sdk' hard-coded in this line, there are other libraries, such as the one provided by Lambda Layers (often for faster builds).
Hi!
First off, thanks for this plugin! I have been trying it and it looks awesome! A decent alternative to Webpack, indeed.
I am having compatibility issues with another plugin though (specifically with serverless-appsync-simulator). It is an offline solution for AppSync and it relies on this.serverless.config.servicePath
in order to build some config objects.
Since the servicePath is being suffixed with /.build
, it cannot find some files it requires.
I was wondering what was the purpose of this override? My understanding is that it is to make serverless-offline work as it is looking for the handlers in that path.
Am I missing something else?
I think it is probably a bad idea to change serverless config like such as there is always a risk to affect other plugins or the serverless implementation itself.
I'd be happy to help to find solutions and alternatives to this, but I'd like to have a better understanding of the motivations and consequences this could have.
Thank you
Hello!
This plugin works great, thank you for sharing it!
I wanted to test, if it works for our use case and I've found some problems.
We are using Yarn as package manager with workspaces enabled. In the stack that I'm testing, we are having 4 lambda functions. When I was using Webpack, each lambda had different package size, where the smallest was 1.5 MB and the biggest was 3 MB.
However when I was experimenting with this plugin, it looks like, package individually option doesn't work. The package size is 6 MB for each lambda, which equals the size of package, that is created when package individually is disabled.
I was looking a little bit into the source code and it looks like, package individually option is handled by this plugin. Is this expected behaviour of esbuild? This is the config that I'm using:
custom:
esbuild:
packager: yarn
bundle: true
minify: true
sourcemap: true
keepNames: true
Versions:
"serverless-esbuild": "1.5.1"
"serverless": "2.3.0"
Node.js v12.19.1
I'm trying to deploy 2 functions: one in typescript and one in python.
I didn't find anything on the documentation of how to disable/enable per function, so I wonder if there's a configuration for that. Also, I think python functions should be automatically ignored.
If there's no such feature, I can create a PR for either toggling the plugin per function or for ignoring functions with python set as the runtime.
Kudos to the team which is behind this plugin and the packaging speeds it comes with! ๐
I have a question (feature-request?) for you. When the below code is packaged, the final deployment does not include the externals (node modules). I think it is because the externals are populated by a plugin in this case esbuild-node-externals
.
const { nodeExternalsPlugin } = require('esbuild-node-externals');
// default export should be an array of plugins
module.exports = [nodeExternalsPlugin()];
esbuild:
plugins : ./plugins.js
packager: npm
Sample repo: https://github.com/vamche/sls-esbuild-node-externals
Is there a way to include node_modules
when externals are populated via plugins?
I've enabled source maps, but all errors stacktrace still points to bundled minified file which is not readable
i've tried sourcemap: inline / enabled /true
tried adding --enable-source-maps
to NODE_OPTIONS env in serverless config and tried passing it directly before launching sls offline
maybe i'm missing something ?
(sorry guys i know i'm creating too many issues, just trying to get it working correctly)
Hello.
On NPM v7+, my package.json requires npm install --legacy-peer-deps
command to be executed, instead of plain npm install
.
However, it seems that serverless-esbuild currently does not support passing options to npm install
:
serverless-esbuild/src/packagers/npm.ts
Lines 104 to 109 in 753e4ae
sls deploy
fails due to npm installation failure.
It would be helpful if there is an option to passing arguments to npm install
something like:
custom:
esbuild:
installExtraArgs:
- "--legacy-peer-deps"
$ sls deploy
Serverless: Compiling with esbuild...
Serverless: Compiling completed.
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: aws-appsync@^4.0.3, @react-native-community/async-storage, @react-native-community/netinfo, graphql-tag@^2.11.0, graphql
Error ---------------------------------------------------
Error: npm install failed with code 1
at ChildProcess.<anonymous> (/Users/cw-tatsuno/IdeaProjects/sagrada-appsync-poc/backend/issue/node_modules/serverless-esbuild/dist/utils.js:48:24)
at ChildProcess.emit (events.js:315:20)
at ChildProcess.EventEmitter.emit (domain.js:467:12)
at maybeClose (internal/child_process.js:1048:16)
at Socket.<anonymous> (internal/child_process.js:439:11)
at Socket.emit (events.js:315:20)
at Socket.EventEmitter.emit (domain.js:467:12)
at Pipe.<anonymous> (net.js:673:12)
For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com
Your Environment Information ---------------------------
Operating System: darwin
Node Version: 14.16.0
Framework Version: 2.35.0 (local)
Plugin Version: 4.5.3
SDK Version: 4.2.2
Components Version: 3.8.3
npm install --legacy-peer-deps
sls deploy
Files
serverless.yml
service: issue-repro
plugins:
- serverless-esbuild
custom:
esbuild:
external:
- 'aws-appsync'
- 'graphql-tag'
provider:
name: aws
region: ap-northeast-1
runtime: nodejs14.x
memorySize: 256
versionFunctions: false
lambdaHashingVersion: 20201221
functions:
helloworld:
handler: src/index.helloworld
events: []
package.json
{
"name": "issue-repro",
"version": "0.1.0",
"devDependencies": {
"@types/aws-lambda": "^8.10.75",
"@types/node": "^14.14.41",
"serverless": "^2.35.0",
"serverless-esbuild": "^1.10.2",
"ts-node": "^9.1.1",
"typescript": "~4.2.4"
},
"dependencies": {
"aws-appsync": "^4.0.3",
"graphql-tag": "^2.11.0"
},
"scripts": {}
}
index.ts
import type {MSKHandler} from 'aws-lambda';
import gql from 'graphql-tag';
import AWSAppSyncClient from 'aws-appsync';
const appSyncClient = new AWSAppSyncClient({
auth: {...},
region: "...",
url: "..."
});
const helloworld: MSKHandler = async (event) => {
const query = gql(`mutation Test($message: String!){
create(message: $message){
id
message
}
}`);
const params = {
"message": "TEST"
}
try {
await appSyncClient.mutate({
variables: params,
mutation: query
});
console.log("Success.");
} catch (err) {
console.error(JSON.stringify(err));
throw new Error("Mutation failed !!")
}
};
export {
helloworld
};
I have two functions in a test serverless config:
functions: { hello, bye },
And I have this in my esbuild plugin code:
build.onEnd(args => {
console.log('### onEnd args:', args);
})
When I deploy, I see this output from my plugin:
### onEnd args: {
errors: [],
warnings: [],
outputFiles: [
{
path: '/Users/samh/foo/.esbuild/.build/src/functions/hello/handler.js.map',
contents: [Uint8Array],
text: [Getter]
},
{
path: '/Users/samh/foo/.esbuild/.build/src/functions/hello/handler.js',
contents: [Uint8Array],
text: [Getter]
}
]
}
### onEnd args: {
errors: [],
warnings: [],
outputFiles: [
{
path: '/Users/samh/foo/.esbuild/.build/src/functions/bye/handler.js.map',
contents: [Uint8Array],
text: [Getter]
},
{
path: '/Users/samh/foo/.esbuild/.build/src/functions/bye/handler.js',
contents: [Uint8Array],
text: [Getter]
}
]
}
From the name onEnd
, I'd assume this would just be executed once when the entire build process is done, with a complete list of files. Is there some way to achieve this?
I've been trying to use your plugin but my setup involves Instana (a log analyzer vendor). The @instana/serverless
module works this way :
layer
that provides the librariesSo the lambda looks like this :
Login:
handler: instana-aws-lambda-auto-wrap.handler
environment:
LAMBDA_HANDLER: src/handlers/Login.handler
layers:
- arn:aws:lambda:eu-west-1:410797082306:layer:instana-nodejs:38
When using serverless-esbuild
this breaks handler resolution. I've been able to make it work by forking the module and replacing this line :
serverless-esbuild/src/helper.ts
Lines 36 to 37 in 65b7431
with
return Object.values(functions)
.map(fn => (fn.environment && fn.environment.LAMBDA_HANDLER) || fn.handler)
Of course, this is very specific, so I wonder if you could add a way of providing this in the configuration or overriding the extractFileNames
function, as you already provide a special behavior for the google provider.
If you're ok or have an idea about how it should be implemented, I may be able to submit a PR.
Thanks
A nice to have feature request.
Docs say
The default JavaScript syntax target is set to ES2017
Plugin could use provider.runtime
to change the target accordingly
In my case I ended up with
provider:
runtime: nodejs14.x
custom:
esbuild:
target: es2020
And thank you for the super fast compiler!
Is there a way to include graphql files?
There's a esbuild-graphql plugin, but i'm not sure how to use it here
I have uncovered some odd behaviour when bundling external dependencies. In our case it would always include dev dependencies into the build. I have dug a bit and found:
https://github.com/floydspace/serverless-esbuild/blob/master/src/pack-externals.ts#L78
which in our case would use the package.json of serverless-esbuild instead of the projects package.json. I am wondering if this should not actually be
const packageJson = require(rootPackageJsonPath);
Because if
if (!packageJson.devDependencies || !packageJson.devDependencies[externalModule.external]) {
prodModules.push(externalModule.external);
} else {
refers to the package.json of serverless-esbuild it would add all dev dependencies, at least it does in our case.
Our setup looks like this:
node_modules
src
- api
-- serverless.yml
- api2
-- serverless.yml
- api3
-- serverless.yml
package.json
package-lock.json
Hi,
First of all, thank you for the great plugin. It speeds up deployments for me a lot.
Recently I ran into an issue odd issue after trying to upgrade to a later version of it. What happens it the following:
Up to 1.10.0 the zip files in the .serverless folder are fine and just contain a single bundled file for each function as expected.
Working zip example:
Starting with 1.10.1 the zips contain the original (unbundled) source file in addition to the bundled version. This seems to overwrite the bundled version during the cloud formation deployment for me.
Broken zip example:
Configuration Info
serverless.yaml
package:
individually: true
plugins:
- serverless-esbuild
- serverless-offline
- serverless-offline-direct-lambda
- serverless-reqvalidator-plugin
- serverless-aws-documentation
- serverless-pseudo-parameters
custom:
esbuild:
bundle: true
minify: true
packager: npm
exclude: ['pg-native']
...
package.json
"devDependencies": {
"dotenv": "^8.2.0",
"eslint": "6",
"eslint-config-airbnb-base": "^14.0.0",
"eslint-config-node": "^4.0.0",
"eslint-config-prettier": "^6.10.0",
"eslint-plugin-import": "^2.20.1",
"eslint-plugin-node": "^10.0.0",
"jest": "^24.9.0",
"prettier": "^1.19.1",
"serverless": "^2.46.0",
"serverless-aws-documentation": "^1.1.0",
"serverless-esbuild": "^1.10.0",
"serverless-offline": "^6.8.0",
"serverless-offline-direct-lambda": "0.0.1",
"serverless-pseudo-parameters": "^2.5.0",
"serverless-reqvalidator-plugin": "^1.0.3",
"typescript": "^3.6.3"
},
"dependencies": {
"aws-sdk": "^2.656.0",
"node-fetch": "^2.6.0",
"node-gyp": "^7.1.2",
"pg": "^8.3.0",
"pg-copy-streams": "^5.1.1",
"pump": "^3.0.0",
"request": "^2",
"rxjs": "^6.5.5",
"s3-download-stream": "^1.1.1",
"winston": "https://gitlab.com/mm123/winston-kf/-/archive/3.2.2/winston-kf-3.2.2.tar.gz",
"xmlhttprequest": "^1.8.0"
}
I tried several later versions (1.10.1 - the 1st version with the issue, 1.10.8, 1.11.0, 1.12.0) and all of those have the same problem for me on [email protected].
As soon as I downgrade to e.g. 1.10.0 or 1.9.1 via npm npm i [email protected]
and run sls deploy
the issue is gone and all works as expected .
After that I upgrade and deploy again and the file duplication reappears for me.
The esbuild version npm installs is consistent (0.12.8) and the only thing that varies is the serverless-esbuild version.
I would try to illustrate the issue with a sample repo. When I try to build the below repo, I see that the final package doesn't contain all the required dependencies.
Repo: https://github.com/vamche/sls-esbuild-node-externals/tree/missing-module-dependecies-for-externals
Branch: missing-module-dependecies-for-externals
So, here the externals declared are lodash
and openpgp
https://github.com/vamche/sls-esbuild-node-externals/blob/3eb11fe11c8d8088d9fe4502a3b46a195cfb6d82/serverless.yml#L17-L19
external:
- lodash
- openpgp
Dependency graph for lodash
and openpgp
Though the tmp build directory (.esbuild\.build
) contains all the modules, the final package includes only the declared modules but not all the required dependencies. In this case the openpgp
's dependencies are missing.
I think the easiest way to fix this is to pass a valid depth here (may be 5?), right now the depth is missing.
serverless-esbuild/src/pack.ts
Lines 103 to 105 in 2fc9bb4
Another option I could think of is
prune
.I would like to know your thoughts before I proceed to create a PR with one of the approaches. Please feel free to suggest if there is any better approach to resolve this.
We need to include marked as external modules to the deployment package, we cannot just use serverless include them because they can depend on other modules. so the idea like serverless-webpack
do, to make temporary package.json
with the modules, install them and put in deployment artifacts folder
It seems #110 is back.
serverless.yml:
functions:
myFn:
package:
patterns:
- resources/some.txt
Running serverless package
with serverless-esbuild v1.10.6 works as expected.
$ cd .serverless
$ unzip myFn.zip
Archive: myFnl.zip
inflating: resources/some.txt
inflating: myFn.js
But with v1.10.7 and onwards it's no longer working, the extra file is missing in the zip.
$ cd .serverless
$ unzip myFn.zip
Archive: myFn.zip
inflating: myFn.js
Using serverless-esbuild >= v1.7.0
with google
provider cause an error
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
at validateString (internal/validators.js:120:11)
at Object.dirname (path.js:1128:5)
at EsbuildPlugin.<anonymous> (/home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless-esbuild/dist/index.js:141:223)
at Generator.next (<anonymous>)
at /home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless-esbuild/dist/index.js:8:71
at new Promise (<anonymous>)
at __awaiter (/home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless-esbuild/dist/index.js:4:12)
at /home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless-esbuild/dist/index.js:140:76
at Array.map (<anonymous>)
at EsbuildPlugin.<anonymous> (/home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless-esbuild/dist/index.js:140:51)
at Generator.next (<anonymous>)
at /home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless-esbuild/dist/index.js:8:71
at new Promise (<anonymous>)
at __awaiter (/home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless-esbuild/dist/index.js:4:12)
at EsbuildPlugin.bundle (/home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless-esbuild/dist/index.js:137:16)
at EsbuildPlugin.<anonymous> (/home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless-esbuild/dist/index.js:72:28)
at Generator.next (<anonymous>)
at /home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless-esbuild/dist/index.js:8:71
at new Promise (<anonymous>)
at __awaiter (/home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless-esbuild/dist/index.js:4:12)
at Object.before:package:createDeploymentArtifacts [as hook] (/home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless-esbuild/dist/index.js:71:63)
at PluginManager.invoke (/home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless/lib/classes/PluginManager.js:551:20)
at PluginManager.spawn (/home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless/lib/classes/PluginManager.js:573:5)
at Object.before:deploy:deploy [as hook] (/home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless/lib/plugins/deploy.js:53:13)
at PluginManager.invoke (/home/runner/work/monorepo/monorepo/serverless/set-order-datastore/node_modules/serverless/lib/classes/PluginManager.js:551:9)
The commit 7325a78 changes the return type of src/helper.ts:extractFileNames
from string[]
to { entry: string; func: any }[]
for all provider except google
.
If the provider is google
, the return type is still string[]
and src/index/ts:bundle
fails while iterating on the array: rootFileNames.map(async ({ entry, func }) => {
downgrade serverless-esbuild
to v1.6.0
Hello.
I've added a watch pattern in esbuild config
watch:
pattern: [ 'src/**/*.ts']
it's recompiling code after changes, but those changes are not working
like if i add console.log('hello')
it will recompile but i won't see that console.log() anywhere
same for other changes
Hi - when saying "without imports", do you mean imports in the .graphql file or importing .graphql files? Trying to see if I can load a .graphql schema into a function call: #102
Hi, I just started using this plugin today, and I'm super impressed! With serverless-webpack, my API (37 Lambda functions) took 12 minutes to build & deploy. Now, with serverless-esbuild, it takes about 2 minutes. ๐
The only thing missing for me, is that I was relying on the Sentry Webpack plugin to automatically push my source maps to Sentry. With esbuild, is there some way I can have it dump my .map
files to a folder, separately from the CloudFormation deployment, and then upload those to Sentry? Even if I have to use their CLI tool to upload the source maps, that's fine. I just need some way to hook into the build process and execute a process to do this.
Thanks!
I have an external dependency 'A' which is needed by other external dependency 'B'.
'A' is not used in my code so when function got bundled it doesn't adds 'A' into node_modules.
Is there a workaround for that ? Or maybe i can create a PR with a fix for that ?
support for esbuild with for wait
Before using serverless-esbuild the installation of private packages from a private GitHub npm registry was possible without any errors.
Only when trying to use [email protected]
(while using [email protected]
) with an external package that resides in a private npm registry the step Packing external modules: @myregistry/mymodule
results in
Error: npm install failed with code 1
--
96 | at ChildProcess.<anonymous> (.../node_modules/serverless-esbuild/dist/utils.js:48:24)
97 | at ChildProcess.emit (events.js:311:20)
98 | at ChildProcess.EventEmitter.emit (domain.js:482:12)
99 | at maybeClose (internal/child_process.js:1021:16)
100 | at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
Any clue on how to specify which exact .npmrc
it is using or whow to get deeper insights regarding the error?
On a second thought (and please excuse me for not being that deep into the topic yet): Is it ultimately needed to temporarily "npm install" the external packages again (into folder .esbuild/.build
) or can't they just be reused from the project root's node_modules
folder?
I tried adding metafile: meta.json
but got an error from esbuild that metafile must be a boolean. So I changed it to metafile: true
. The build ran successfully but I was never able to find any output. Has anyone managed to analyze their esbuild bundle sizes?
When using the latest version of the plugin with the latest of esbuild, the following error is encountered:
$ sls deploy -v
Serverless: Auto instrumenting functions with Datadog
Serverless: Adding Lambda Layers to functions
Serverless: Compiling with esbuild...
Error --------------------------------------------------
Error: Invalid option: "exclude"
at checkForInvalidFlags (/home/bryantbiggs/Documents/dogged-params/node_modules/esbuild/lib/main.js:226:13)
at flagsForBuildOptions (/home/bryantbiggs/Documents/dogged-params/node_modules/esbuild/lib/main.js:312:3)
at Object.build (/home/bryantbiggs/Documents/dogged-params/node_modules/esbuild/lib/main.js:469:49)
at /home/bryantbiggs/Documents/dogged-params/node_modules/esbuild/lib/main.js:703:67
at new Promise (<anonymous>)
at Object.build (/home/bryantbiggs/Documents/dogged-params/node_modules/esbuild/lib/main.js:703:26)
at /home/bryantbiggs/Documents/dogged-params/node_modules/esbuild/lib/main.js:629:27
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async Promise.all (index 0)
For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
This is resolved by downgrading to v1.4.0 or by setting the esbuild version to be ~0.6.0
(0.6.34). It looks like somewhere in v0.7.x of esbuild this plugin stopped working with the excludes option
I have a repro here
Hey,
I have these options in my SLS config:
package:
exclude:
- 'cache-manager'
- 'class-transformer'
- 'class-validator'
- '@nestjs/microservices'
- '@nestjs/websockets'
individually: true
custom:
esbuild:
bundle: true
minify: true
keepNames: true
sourcemap: true
external:
- 'cache-manager'
- 'class-transformer'
- 'class-validator'
- '@nestjs/microservices'
- '@nestjs/websockets'
plugins: 'esbuild.plugins.js'
However, once it bundles, it runs this:
Packing external modules: cache-manager, class-transformer, class-validator, @nestjs/microservices, @nestjs/websockets
I want to completely skip it as these packages are opt-in in NestJS. Is there any way to achieve it? I see from the code that this plugin packages externals. Is there any way to completely skip them?
If externals get over a certain number, on macOS 11.5 I'm getting this error when packaging:
Serverless: Compiling to node14 bundle with esbuild...
Serverless: Compiling completed.
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: < ... list of external modules>
Error ---------------------------------------------------
Error: spawn E2BIG
at ChildProcess.spawn (internal/child_process.js:403:11)
at spawn (child_process.js:580:9)
at Object.spawnWithSignal [as spawn] (child_process.js:717:17)
at /Users/.../node_modules/bestzip/lib/bestzip.js:74:29
at /Users/.../node_modules/bestzip/node_modules/async/dist/async.js:1802:20
at /Users/.../node_modules/bestzip/node_modules/async/dist/async.js:248:13
at wrapper (/Users/.../node_modules/bestzip/node_modules/async/dist/async.js:268:20)
at iterateeCallback (/Users/.../node_modules/bestzip/node_modules/async/dist/async.js:421:28)
at /Users/.../node_modules/bestzip/node_modules/async/dist/async.js:321:20
at /Users/.../node_modules/bestzip/node_modules/async/dist/async.js:245:17
at /Users/.../node_modules/bestzip/node_modules/async/dist/async.js:1792:24
at /Users/.../node_modules/bestzip/lib/bestzip.js:49:7
at processTicksAndRejections (internal/process/task_queues.js:77:11)
For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com
Your Environment Information ---------------------------
Operating System: darwin
Node Version: 14.17.3
Framework Version: 2.52.1 (local)
Plugin Version: 5.4.3
SDK Version: 4.2.5
Components Version: 3.14.0
Same error using node version 16.5.0
.
On a Linux distro it's running fine though.
Any help would be appreciated.
Hi,
this is a nice lib.
but I try this lib with the serverless framework, I face some problem:
serverless.yml
package:
individually: true
I didn't set anything in custom
..inside the .serverless
folder I don't see any node_modules
+ package.json
file.
If I deploy this lambda func to aws, I guess will fail?
How can I make node_modules
+ package.json
file included in each lambda func zip?
Thanks!
Excuse my ignorance, as this could very well be a problem with my use of the package - but I'm encountering an issue when trying to bundle my Externals with serverless-esbuild
. Also apologies for the long winded explanation - the examples kept cropping up as I went...
Relevant scope of serverless.ts
:
esbuild: {
bundle: true,
minify: false,
plugins: 'esbuild-plugins.ts' // exports esbuild-node-externals
packager: 'npm',
packagePath: path.join(__dirname, 'package.json')
}
The error I receive when running serverless offline
or serverless deploy
is as follows:
Error: ENOENT: no such file or directory, open '/Users/atyrrell/package.json'
Another relevant aspect of the stack trace is:
EsbuildPlugin.<anonymous> (/Users/atyrrell/my-project/node_modules/serverless-esbuild/dist/pack-externals.js:156:55)
Upon doing some debugging with that file - I'm wary of the way findProjectRoot
looks for the directory - specifically with the findUp
function:
The main issue being - I believe that it doesn't recover well after not finding a yarn.lock
file. I've tried just using Yarn instead, but issues with grabbing Packages from a private registry opens a different can of worms.
FWIW - here's a my local Project structure, and a little flow of the findProjectRoot
execution:
Users/
atyrrell/
Projects/
sub-project/
project_x/
serverless.ts
package.json
package-lock.json
<other project files...>
I believe the trigger is here: https://github.com/floydspace/serverless-esbuild/blob/master/src/pack-externals.ts#L214
however I'm looking at the /dist
files...
Looking for the rootPackageJsonPath
results in looking for the following (filename, directory) being hit in findUp
:
yarn.lock /Users/atyrrell/Projects/sub-project/project_x
yarn.lock /Users/atyrrell/Projects/sub_project
yarn.lock /Users/atyrrell/Projects
yarn.lock /Users/atyrrell
Then it goes quiet and complains about not being able to open /Users/atyrrell/package.json
. It almost seems like the Directory path isn't being reset back down when it starts looking again?
Ironically this line: https://github.com/floydspace/serverless-esbuild/blob/master/src/pack-externals.ts#L217
doesn't seem to be accurate, as packageJsonPath actually DOES contain the value I'd expect of /Users/atyrrell/Projects/sub_project/project_x/package.json
, but rootPackageJsonPath DOESN'T.
A bit longwinded, and hopefully not too scatter brained - I appreciate any input or info, if this is a confirmed issue I'm happy to throw up a PR. Just wanted to confirm it's not User Error!
https://www.npmjs.com/package/bestzip should make deployment quicker and use less memory. serverless-webpack has decided to use it too: https://github.com/serverless-heaven/serverless-webpack/pull/596/files
While investigating #103 I saw that pg-native
gets always bundled with pg
which might not be needed by the code (if it doesn't actually require native pg from pg). This is listed as peerDependencies
, but set optional with :
"peerDependenciesMeta": {
"pg-native": {
"optional": true
}
}
We might not want to auto bundle optional peer deps in the externals @floydspace ?
Hi there,
I was testing this plugin out and ran into some issue. We have a monorepo setup where the individual services do not have their own package.json
. I noticed that this plugin uses the code from serverless-webpack. Any plans to add the ability to set your own path for the package.json
? Or alternatively, climbing up the directories to look for a package.json
file?
https://github.com/serverless-heaven/serverless-webpack/blob/31662103851933ac277b02119912b593c2914ece/lib/packExternalModules.js#L242 => https://github.com/floydspace/serverless-esbuild/blob/master/src/packExternalModules.ts#L151
Upgraded to 1.10.3 to fix #117 which now works with serverless >= 2.36.0. However, it appears to have broken invoke local
. I get a "cannot find module" error and it appears to be looking in src/...
rather than .esbuild/.build/src/...
:
Error: Cannot find module '/Users/schmidtt/w/video-tracking/src/list'
Require stack:
- /Users/schmidtt/w/video-tracking/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js
- /Users/schmidtt/w/video-tracking/node_modules/serverless/lib/plugins/index.js
- /Users/schmidtt/w/video-tracking/node_modules/serverless/lib/classes/PluginManager.js
- /Users/schmidtt/w/video-tracking/node_modules/serverless/lib/Serverless.js
- /Users/schmidtt/w/video-tracking/node_modules/serverless/scripts/serverless.js
- /Users/schmidtt/w/video-tracking/node_modules/serverless/bin/serverless.js
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
at Function.Module._load (internal/modules/cjs/loader.js:725:27)
at Module.require (internal/modules/cjs/loader.js:952:19)
at require (internal/modules/cjs/helpers.js:88:18)
at AwsInvokeLocal.invokeLocalNodeJs (/Users/schmidtt/w/video-tracking/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:743:33)
at AwsInvokeLocal.invokeLocal (/Users/schmidtt/w/video-tracking/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:222:19)
at Object.invoke:local:invoke [as hook] (/Users/schmidtt/w/video-tracking/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:49:47)
at PluginManager.invoke (/Users/schmidtt/w/video-tracking/node_modules/serverless/lib/classes/PluginManager.js:551:20)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
Hello,
When I do sls deploy
, the bundle is correctly built however the handler is set to s_function.handler
for a function located in let's say src/function.js
. It works if I manually rename it to src/function.handler
in the AWS console but it gets overwritten after every deploy.
I can reproduce that with any of the esbuild starter I have found. Even with https://github.com/floydspace/serverless-esbuild/tree/master/examples/minimal
Note that when doing sls package
or sls deploy
, I can see the file s_function.handler
being created then deleted on my root dir.
Hi Team
Firstly this is an awesome plugin.
This is an issue/feature request
When I try package my serverless project I get the following issue Error: ENOENT: no such file or directory, open 'C:\Users****\repos\databus-serverless.esbuild.serverless\r-#{AWS::AccountId}-npr-ds-databus-hello.zip'
The issue comes about because I am trying to add a prefix to my lambda name using the serverless-pseudo-parameters plugin.
I have pasted my serverless config below
If I remove the name field from the lambda config then your library defaults to the object property in the serverless config.
{
service: "databus-serverless",
frameworkVersion: "2.35",
provider: {
name: "aws",
tags: globalTags,
profile: "developerrole",
region: "eu-west-1",
runtime: "nodejs14.x",
lambdaHashingVersion: 20201221,
deploymentBucket: {
name: "${self:custom.prefix}-global-auto-scaling-s3",
},
environment: {
NODE_PATH: "./:/opt/node_modules",
},
},
custom: {
accountId: "#{AWS::AccountId}",
account: "${opt:account, self:custom.accountId}",
esbuild:{
bundle:true,
minify:false,
sourcemap:true
},
region: "${opt:region, self:provider.region}",
stage: '${opt:stage, "npr"}',
prefix: "r-${self:custom.account}-${self:custom.stage}-ds-databus",
},
package: {
excludeDevDependencies: true,
individually: true
},
plugins: [
"serverless-esbuild",
"serverless-pseudo-parameters",
"serverless-deployment-bucket",
"serverless-offline",
],
functions: {
Hello: {
name: "${self:custom.prefix}-hello",
handler: "functions/hello/index.handler",
role: "arn:aws:iam::${self:custom.account}:role/${self:custom.prefix}-iam-lambda-bifrost-role",
layers: [
"arn:aws:lambda:${self:custom.region}:${self:custom.account}:layer:modules:1",
]
}
}
}
Is there or could there be a flag to enable this naming functionality. For example by default the build folders are named using the object property rather than the name field in function definition. In order to use the lambda name for the build folders there would be an options as follows:
esbuild:{
useLambdaName:true
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.