Giter VIP home page Giter VIP logo

serverless-toolkit's Introduction

Seagull with a French Fry*

Serverless Toolkit

Part of Twilio Labs Banner

What is the Serverless Toolkit?

The Serverless Toolkit is CLI tooling to help you develop locally and deploy to the Twilio Functions & Assets.

There are two ways you can use the toolkit. If you are already using the Twilio CLI, you can install it via a plugin. Alternatively, you can use the toolkit as a standalone using twilio-run via npm or another Node.js package manager.

Throughout the docs, you can switch in the code snippets between Twilio-CLI and Bash Session to get the commands for both versions.

Let's work together

Everything in this toolkit is released under Twilio Labs and fully open-source. If you find any problems with this, please file an issue or even create a pull request to work together with us on the toolkit. We would love to hear your ideas and feedback!

Project Structure & Contributing

This project is a monorepo, meaning it contains multiple packages in one repository. It consists out of the following packages:

Also part of the Serverless toolkit, but in another repository:

To understand more about the structure and the design of the Toolkit check out the design documentation.

Setup & Development

This project uses npm workspaces as a tool to manage the monorepo. If you are unfamiliar with the tool, start by checking out the the npm docs and make sure you use at least npm version 8 or newer (npm install -g npm@8).

git clone [email protected]:twilio-labs/serverless-toolkit.git
cd serverless-toolkit
npm install
npm run build

License

MIT

Disclaimer

Unofficial logo. Not a Twilio logo.

serverless-toolkit's People

Contributors

badsketch avatar bharatr21 avatar bobtfish avatar butuzov avatar colossal-toby avatar dabockster avatar daniel-simpson avatar david-amores-anz avatar dependabot[bot] avatar dkundel avatar github-actions[bot] avatar imnotjames avatar makserik avatar mbeldartwilio avatar michizhou avatar nihonjinrxs avatar philnash avatar pranjalv9 avatar pthirumurthi avatar saadismail avatar shelbyz avatar shrutiburman avatar snyk-bot avatar stefanjudis avatar stern-shawn avatar stevennic-twilio avatar thinkingserious avatar twilio-product-security avatar vernig avatar victoray avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

serverless-toolkit's Issues

Better error message for errors with the GitHub API

I just encountered an error with the new command where it would throw a random unhandled rejection error saying this:

DEBUG=twilio-run:new:template-data twilio serverless:new hello --template=blank                                                           โฌข 8.10.0
  twilio-run:new:template-data Response code 403 (Forbidden) +0ms
(node:66826) UnhandledPromiseRejectionWarning: Error: Invalid template
    at Object.<anonymous> (/Users/dkundel/.twilio-cli/node_modules/twilio-run/dist/templating/data.js:71:19)
    at Generator.throw (<anonymous>)
    at rejected (/Users/dkundel/.twilio-cli/node_modules/twilio-run/dist/templating/data.js:5:65)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
(node:66826) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function with
out a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:66826) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will t
erminate the Node.js process with a non-zero exit code.

After some investigation I realized it was because the GitHub API was returning a 403 - Forbidden response because the IP I was on, was Rate Limited. We should probably handle errors like that better and just forward the error message in the body by parsing the response body.

Here's an example payload from GitHub:

{
  "message": "API rate limit exceeded for 19.253.99.123. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
  "documentation_url": "https://developer.github.com/v3/#rate-limiting"
}

husky > prepare-commit-msg hook failed

Committing from Windows machine appears to fail on the husky commands (with no real help on what path/file was not found):

image

The prepare-commit-msg command will not work for win32 as is:

exec < /dev/tty && git-cz --hook

Detect User-Agent to return Error HTML

Right now we return a pretty Error page regardless of who requests it.
This is great for the browser but not great for the Twilio Debugger. We should return a text/plain response in that case.

The easiest way for this would be to detect the browser through a User-Agent and otherwise return text/plain.

Protected functions should be protected locally

When starting to test the support for protected functions, I expected that I would receive a 401 response when calling my protected function locally without a signature.

I understand that when deploying protected functions will be protected, but to make that more obvious when developing, I think that protected functions should still be protected. This will give confidence to developers when testing functions with webhooks and ngrok.

To make for easier testing, we could add a flag that unprotects protected functions locally.

TypeError: Runtime.getFunctions is not a function

Hi ๐Ÿ‘‹ !

First, thank you for creating this! I'm excited to see it in action! ๐Ÿ˜„

However, I'm not sure what I'm doing wrong. I'm getting the following error when running twilio-run in the CLI. I also tried installing twilio-run as a global dependency but that didn't change anything.

/Users/joeprevite/Dev/learning/twilio-run-test/node_modules/twilio-run/src/cli/route-info.js:10
  const functions = Runtime.getFunctions();
                            ^

TypeError: Runtime.getFunctions is not a function
    at getRouteInfo (/Users/joeprevite/Dev/learning/twilio-run-test/node_modules/twilio-run/src/cli/route-info.js:10:29)
    at Server.app.listen (/Users/joeprevite/Dev/learning/twilio-run-test/node_modules/twilio-run/src/cli.js:30:20)
    at Object.onceWrapper (events.js:273:13)
    at Server.emit (events.js:182:13)
    at emitListeningNT (net.js:1370:10)
    at process._tickCallback (internal/process/next_tick.js:63:19)
    at Function.Module.runMain (internal/modules/cjs/loader.js:745:11)
    at startup (internal/bootstrap/node.js:266:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:596:3)

Expected behavior

Based on the documentation, when I run twilio-run or more specifically node_modules/.bin/twilio-run while inside the root directory, it should look for a sub-directory titled functions then find all my functions in there and run them.

Actual behavior

It throws an error. ^See code block above.

Steps to reproduce the behavior

  1. Clone this repository
  2. Run npm install
  3. Run node_modules/.bin/twilio-run
  4. See error

For some reason, it seems like getFunctions() is not being returned on the Runtime object in /twilio-run/src/cli/route-info.js. ๐Ÿค”

Happy to help try and solve this (with some guidance) if you'd like.

Add config support

It would be nice to have config support especially for deployment configuration or to override dependencies you want to install.

Watch mode

Local dev mode should have a watch flag that automatically adds newly created files to the files that are being served. It could either just restart the entire server or just update the Routes cache and print a message to the terminal that a new Function or Asset has become available.

Feature Request: Total Execution time for a function run

With a Twilio Function we have a limited resource time and need to try and work within a 5 second time window. Turning up logging via logLevel/detailedLogs does not give an 'at-a-glance' total execution time for a ran function:

image

Looking at the Function exuction line we can get an estimate of the execution time since the last log statement.

We could make use of console.time(...) in routes.ts like:

const FUNCTION_LABEL = 'Function Execution';

try {
    console.time(FUNCTION_LABEL);
    fn(context, event, callback);
    console.timeEnd(FUNCTION_LABEL);
} catch (err) {
    console.timeEnd(FUNCTION_LABEL);
    callback(err);
}

There might be an easier way to achieve this with the current logger, but I hadn't dived down into it.

Fix slashes at getFunctions() result

Apparently the result of getAssets() and getFunctions() differs in the cloud. getAssets() has a slash in front of each path while getFunctions results don't.

Show message from failed build

When the deployment failed we should show a more usable message. The API returns a good message that we should forward

Replace `projectName`

Right now we are using projectName as a way to configure the name of your deployment. This is confusing especially with the Twilio CLI having a different concept of projects. Let's move to serviceName instead.

We should support projectName as a fallback with a warning.

Function creation in the same namespace throws

Currently, it is not possible to create multiple function templates using the same namespace via twilio serverless:new or twilio-run new.

$ twilio-run new "forward-everything"[13:28:12]
? Select a template Blank Template - Barebones template to get started

โ”‚ ERROR Error
โ”‚
โ”‚ Template with name "forward-everything" already exists in "/private/tmp/burner/functions"

My use case was that I wanted to create functions for sms and call forwarding under the same namespace forward-everything.

I tracked down where the exception is thrown and the template creation expects the namespace dir to not exist?

That surprised me and I think it's a bug, because I was expecting that I can create several functions under the same namespace. :)

Support template projects `init`

This should probably sit in twilio-labs/create-twilio-function but I'll leave it here for now to keep an overview of what needs to be done.

I wonder if something like npm.im/degit would be useful for this

@philnash

`start` should support `--cwd`

There's a different behavior between start and deploy at the moment with respect to how to specify the directory. start should also support the --cwd flag.

Handle API Key and API Secret properly

Right now when a project is set-up using the Twilio CLI we are actually setting the API Key and API Secret that the Twilio CLI is using as ACCOUNT_SID and AUTH_TOKEN in the .env file. That works for deployment and interacting with the Twilio Serverless API because it doesn't require the knowledge of the Account SID.

However, it causes two problems.

  1. If you use context.getTwilioClient() to get a Twilio client this is going to fail for some APIs that require the knowledge of the Account SID (SMS, Calls etc.)
  2. We cannot verify locally the Twilio Signature for protected webhooks because we need the auth token for that which we don't have in that situation.

Why do we use the Auth Token in the first place?
In the legacy Functions environment we have ACCOUNT_SID and AUTH_TOKEN available on the context object. This enabled the use of context.getTwilioClient() but people could also access these variables directly. Therefore we are setting them upon project creation in your .env file.

By default these are also read for other interactions with the Serverless API for consistency. For deploy and other commands we introduced two flags --auth-token and --account-sid to override the ones that are read from the .env file.

When the Serverless Plugin was built, this introduced another complexity. The logic that is being used right now is the following:

  1. When serverless:init is called we'll pass username and password of the Twilio Client in the CLI to create-twilio-function. This will write them as ACCOUNT_SID and AUTH_TOKEN into .env.
  2. When you run serverless:start both ACCOUNT_SID and AUTH_TOKEN are being read and made available on context argument inside Function executions
  3. For deployment and other API interactions we follow the following logic:
    i) if a specific project has been passed with -p or --project we will use that username and password to interact with the API.
    ii) if the user passed in --account-sid and --auth-token we'll use those credentials
    iii) if the user as ACCOUNT_SID and AUTH_TOKEN in the .env file we'll use those
    iv) otherwise we'll fallback to the default credentials of the Twilio CLI that are under username and password.

Right now my train of thought here is:

  1. Rename --account-sid and --auth-token for API-related commands to --username and --password for consistency and deprecate the other ones
  2. Still keep ACCOUNT_SID and AUTH_TOKEN and set them to username and password essentially. Meaning these could be API Key and API Secret
  3. Detect if the pair is actually API Key and API Secret and display a warning to the user for the start command that client.getTwilioClient() and protected routes might not operate as expected and that if they want to make this more realistic, they have to change to Account SID and Auth Token.

Alternatively we can fix client.getTwilioClient() by detecting in create-twilio-function already that we are setting an API Secret not an Auth Token and instead set ACCOUNT_SID to the actual account SID and introduce TWILIO_API_KEY and TWILIO_API_SECRET instead and use that in creating the getTwilioClient() locally but then we will have the issue that context.AUTH_TOKEN will be available in the deployment but not locally. And since that's a sensitive piece of information, I wouldn't want to surprise the user that it exists.

Would love to have some input from you folks on how to solve this problem.
@philnash @dprothero

Fix error when canceling out of `new`

If you cancel out of new you'll get a confusing error message. Instead of a clean exit.

twilio serverless:new
โœ– Select a template โ€บ Hunt / Find Me - Will call a list of configured numbers until one answers

^C
TypeError: Cannot read property 'replace' of undefined
    at handler (~/.twilio-cli/node_modules/twilio-run/src/commands/new.js:99:39)

Fix help message for deploying existing services with the plugin

Right now the output has parts that are undefined when using deploy via the github.com/twilio-labs/plugin-serverless. Example

โœ– Project with name "twilio-example" already exists with SID "ZSxxxxxxxxxxxxxxxxxxxxxxx".
Here are a few ways to solve this problem:
- Rename your project in the package.json "name" property
- Pass an explicit name to your deployment
  > undefined deploy -n my-new-service-name
- Deploy to the existing service with the name "undefined"
  > undefined deploy --override-existing-project
- Run deployment in force mode
  > undefined deploy --force

Add support for `open` method on Assets

There's an undocumented method on assets returned by Runtime.getAssets() that returns the content of an asset as pointed out by @ShelbyZ. This should at least be supported in the new version of the CLI.

twilio cli inside a docker container - issues with npm pkg window-size

Hello!

I'm running the Twilio-cli inside a docker container, running "twilio server less:start" works perfectly if I run it manually from inside the running container but it fails when run using the docker CMD in the Dockerfile or from the entrypoint file. I didn't dig too much but it looks like the window-size npm pkg doesn't like to run without an actual console.

Here's the error I get:

twilio    | > twilio serverless:start
twilio    |
twilio    | TypeError: Cannot read property 'width' of undefined
twilio    |     at wrapText (~/.twilio-cli/node_modules/twilio-run/dist/printers/utils.js:32:76)
twilio    |     at importantMessage (~/.twilio-cli/node_modules/twilio-run/dist/printers/utils.js:35:13)
twilio    |     at Object.warningMessage (~/.twilio-cli/node_modules/twilio-run/dist/printers/utils.js:41:12)
twilio    |     at Logger.warn (~/.twilio-cli/node_modules/twilio-run/dist/utils/logger.js:49:23)
twilio    |     at printVersionWarning (~/.twilio-cli/node_modules/twilio-run/dist/checks/nodejs-version.js:16:21)
twilio    |     at Object.checkNodejsVersion [as default] (~/.twilio-cli/node_modules/twilio-run/dist/checks/nodejs-version.js:22:9)
twilio    |     at ~/.twilio-cli/node_modules/twilio-run/dist/commands/start.js:27:33
twilio    |     at Generator.next (<anonymous>)
twilio    |     at ~/.twilio-cli/node_modules/twilio-run/dist/commands/start.js:7:71
twilio    |     at new Promise (<anonymous>)
twilio    | npm ERR! code ELIFECYCLE
twilio    | npm ERR! errno 1
twilio    | npm ERR! [email protected] serverless-start: `twilio serverless:start`
~/.twilio-cli/node_modules/twilio-run/dist/printers/utils.js:32:76
---

const window_size_1 = __importDefault(require("window-size"));
..

###### I would perform a quick check on "window_size_1" and maybe use a default value ? ######
const wrapText = (text) => wrap_ansi_1.default(text, window_size_1.default.width - 5, { trim: false });
function importantMessage(label, color, title, body) {
    label = chalk_1.default.keyword(color)(label);
    title = wrapText(chalk_1.default.bold.underline(`${label} ${chalk_1.default.bold(title)}`));
    body = wrapText(body);
    return '\n' + borderLeft(`${title}\n\n${chalk_1.default.dim(body)}`, color) + '\n';
}

Implement machine readable output format

I'm currently writing an automation shell script that buys a number, creates a sync service, and deploys a function. These deployed function should be used by the just bought number. Unfortunately, I didn't see a way to get the command response in an easy machine readable format.

The twilio CLI provides the command option -o=json for several other commands. This option allows to get the command response in JSON and thus in a machine readable format. It makes it possible to re-use values and to write advanced scripts. It would be great to have this option in the Twilio Serverless CLI plugin, too.

As the CLI is only a wrapper around twilio-run I thought that it's best to open the issue here first. :)

What I'm looking for is something as follows:

# in the Twilio CLI
$ twilio serverless.deploy -o=json

# and in twilio-run
$ twilio-run deploy --output=json

Is this something you'd be up for?

Missing concept of Private Asset(s)

We have the concept of functions and assets but not private assets which means calls like:

Runtime.getAssets()['config.json'].open();

Fails with open is undefined or not a function. Possibly approaches:

  • Check for special comment in asset files and change their load process
  • Create/Check a new folder private_assets and load them in a way that provides open()
  • add to an existing (of forthcoming) configuration file to mark some assets as private

Don't allow project names characters not allowed in subdomains

I have a project name_with_underscores, but when I do serverless:deploy it (correctly) fails to deploy with that name b/c it's not valid. Maybe disallow project names that won't be valid later? Or catch the error with instructions on how to resolve/rename it.

Replace all `console.` calls with a centralized logger

This way we can implement log levels for info, warn and error better and centralize the output style. Debug logs should remain separate in the files and keep relying on the debug module. Instead if the log level is set to debug, the appropriate environment flags (DEBUG=twilio*) should be enabled.

Improved Output / Error Messages

  • Improve message when build fails #24
  • Show more help when commands are being executed in the wrong directory
  • Update message for wrong Node.js version #22
  • Fix existing service message for plugins #16
  • Space out pretty list output
  • Improve error message in activate for missing serviceSid #30

Runtime un-initialized on first request

When the local development server is running, on first run the Runtime object seems to be undefined.

  twilio-run:server Load & route to function at "/Users/user/Projects/xxxxxxx/functions/example.js" +4s
  twilio-run:server Uncached loading of /Users/user/Projects/xxxxxxxxx/functions/example.js +0ms
  twilio-run:server Failed to retrieve function. ReferenceError: Runtime is not defined
  twilio-run:server     at Object.<anonymous> (/Users/user/Projects/xxxxxxxxx/functions/example.js:1:84)
  twilio-run:server     at Module._compile (module.js:652:30)
  twilio-run:server     at Object.Module._extensions..js (module.js:663:10)
  twilio-run:server     at Module.load (module.js:565:32)
  twilio-run:server     at tryModuleLoad (module.js:505:12)
  twilio-run:server     at Function.Module._load (module.js:497:3)
  twilio-run:server     at Module.require (module.js:596:17)
  twilio-run:server     at require (internal/module.js:11:18)
  twilio-run:server     at requireUncached (/Users/user/.twilio-cli/node_modules/twilio-run/dist/runtime/server.js:26:12)
  twilio-run:server     at loadTwilioFunction (/Users/yser/.twilio-cli/node_modules/twilio-run/dist/runtime/server.js:31:16) +1ms

Improve messaging for missing service SID in list command

Right now when you run twilio serverless:list outside of a deployed project project it fails with:

It should instead:

  1. tell people to find existing services through list services
  2. listing services should be the default
twilio serverless:list

โ”‚ ERROR Could not find Service SID
โ”‚
โ”‚ We could not find a Twilio Serverless Service SID to perform this action.
โ”‚
โ”‚ You can either pass the Service SID via the "--service-sid" flag or by storing it in a ".twilio-functions" file
โ”‚ inside your project like this:
โ”‚
โ”‚   {
โ”‚     "serviceSid": "ZSf9dec7e059e0695f4c8axxxxxxxxxxxx"
โ”‚   }

Deploy to production

The Serverless API allows to have a "production" environment by not specifying any Domain Suffix. Right now the CLI doesn't support this yet. We should find the right way to do this. Maybe it's just passing an empty environment flag.

Improve error message for invalid JS files

Right now it returns a 404 when a file could not be parsed because we try to require it without checking if it exists. If it fails because of invalid JS but if the file is present, we should return a 500 with an appropriate error message.

Confusing warning message about "Twilio Runtime"

You are currently running 10.16.0 but the Twilio Runtime is runnning version 8.10. You might encounter differences between local development and production.

It's not clear what this error means, specifically that "Twilio Runtime" is a server side environment for a product which you may or may not be using. At first I thought it was the version of node packaged with my CLI vs installed on my system.

Maybe something more like You are currently running Node.js 10.16.0 on this local machine. The environment for Twilio Functions, Twilio's serverless production environment, is currently 8.10. When you deploy to Twilio Functions, you may encounter differences between local development and production.

Also note "runnning" typo.

--logLevel=debug overwrites context variable AUTH_TOKEN

Using the run option --logLevel=debug will redact AUTH_TOKEN if part of the context object.

image

This becomes a problem when a function uses the context object to interact with the twilio library via context.getTwilioClient() as it will use ACCOUNT_SID and AUTH_TOKEN to authenticate. Printing the values stored in context results in below:

image

It looks like the redaction code is permanently changing the context variable and it seems like it should not.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.