Giter VIP home page Giter VIP logo

nextjs-serverless-demo's Introduction

Next.js Serverless Demo

Maintenance Status

Deploy Next.js to AWS Lambda using the Serverless Application Framework.

Project notes

This demo uses the following tools:

and is based on the following projects:

Goals

The main goals of this demo project are as follows:

  1. Slim down a Next.js Lambda deployment: The Next.js target: "serverless" Node.js outputs are huge. Like really, really big because each page contains all the dependencies. This project aims to use target: "server" Node.js outputs to achieve a smaller package.

    Here's our starting point with serverless target:

    $ yarn clean && yarn build && yarn lambda:sls package --report
    $ du -sh .serverless/blog.zip && zipinfo -1 .serverless/blog.zip | wc -l
    4.0M	.serverless/blog.zip
    290
    $ du -sh .next/serverless/pages/index.js
    2.7M	.next/serverless/pages/index.js

    Here's with server target:

    $ yarn clean && yarn build && yarn lambda:sls package --report
    $ du -sh .serverless/blog.zip && zipinfo -1 .serverless/blog.zip | wc -l
    3.0M	.serverless/blog.zip
    1291
    $ du -sh .next/server/pages/index.js
    12K	.next/server/pages/index.js

    While the package sizes at 2 pages are comparable for the overall zip, the server (12K) vs serverless (2.7M) per page cost of pages/index.js, and each additional page, becomes apparent.

  2. Single Lambda/APIGW proxy: The Next.js target: "serverless" requires you to either manually create a routing solution based on Next.js generated metadata files or use something like next-routes. However, target: "server" contains a router itself for one endpoint. Thus, by using the server target we can avoid one of the biggest pains of deploying to a single Lambda target for an entire Next.js application.

Implementation

Runtime

We use the production-only Node server found in next/dist/server/next-server.js instead of the development augmented core server found in next/dist/server/next.js. This has a few extra constraints, but ends up being a good choice for the following reasons:

  • Both next-server.js and next.js get to use the built-in Next.js router that is unavailable when using serverless target.
  • The traced file bundle for next-server.js is much slimmer as tracing can easily skip build dependencies like webpack, babel, etc. that come in with next.js
  • Next.js itself now follows this exact model for their experimental tracing support and we can see a similar server configuration here.

Packaging

We package only the individual files needed at runtime in our Lambda using the Serverless Application Framework with the serverless-jetpack plugin. The Jetpack plugin examines all the application entry points and then traces all imports and then creates a zip bundle of only the files that will be needed at runtime.

For those doing their own Lambda deployments (say with Terraform), we provide a standalone CLI, trace-pkg, to produce traced zip bundles from entry points.

Part of the underlying bundle size problem is that the next package ships with a ton of build-time and development-only dependencies that artificially inflate the size of a bundle suitable for application deployment. By using the next-server.js runtime and file tracing, we get the smallest possible package for cloud deployment that is still correct.

To read more about file tracing and integration with your applications, see

Caveats

Some caveats:

  1. Static files: To make this demo a whole lot easier to develop/deploy, we handle serve static assets from the Lambda. This is not what you should do for a real application. Typically, you'll want to stick those assets in an S3 bucket behind a CDN or something. Look for the TODO(STATIC) comments variously throughout this repository to see all the shortcuts you should unwind to then reconfigure for static assets "the right way".
  2. Deployment URL base path: We have the Next.js blog up at sub-path /blog. A consumer app may go instead for root and that would simplify some of the code we have in this repo to make all the dev + prod experience work the same.
  3. Lambda SSR + CDN: Our React SSR hasn't been tuned at all yet for caching in the CDN like a real world app would want to do.

Local development

Start with:

$ yarn install

Then we provide a lot of different ways to develop the server.

Command URL
dev http://127.0.0.1:3000/blog/
http://127.0.0.1:3000/blog/posts/ssg-ssr
start http://127.0.0.1:4000/blog/
http://127.0.0.1:4000/blog/posts/ssg-ssr
lambda:localdev http://127.0.0.1:5000/blog/
http://127.0.0.1:5000/blog/posts/ssg-ssr
deployed https://nextjs-sls-sandbox.formidable.dev/blog/
https://nextjs-sls-sandbox.formidable.dev/blog/posts/ssg-ssr

Next.js Development server (3000)

The built-in Next.js dev server, compilation and all.

$ yarn dev

and visit: http://127.0.0.1:3000/blog/

Node.js production server (4000)

We have a Node.js custom express server that uses almost all of the Lambda code, which is sometimes an easier development experience that serverless-offline. This also could theoretically serve as a real production server on a bare metal or containerized compute instance outside of Lambda.

$ yarn clean && yarn build
$ yarn start

and visit: http://127.0.0.1:4000/blog/

Lambda development server (5000)

This uses serverless-offline to simulate the application running on Lambda.

$ yarn clean && yarn build
$ yarn lambda:localdev

and visit: http://127.0.0.1:5000/blog/

Deployment

We target AWS via a simple command line deploy using the serverless CLI. For a real world application, you'd want to have this deployment come from your CI/CD pipeline with things like per-PR deployments, etc. However, this demo is just here to validate Next.js running on Lambda, so get yer laptop running and fire away!

Names, groups, etc.

Environment:

Defaults:

  • SERVICE_NAME=nextjs-serverless: Name of our service.
  • AWS_REGION=us-east-1: Region
  • STAGE=localdev: Default for local development on your machine.

For deployment, switch the following variables:

  • STAGE=sandbox: Our cloud sandbox. We will assume you're deploying here for the rest of this guide.

Prepare tools

Get AWS vault

This allows us to never have decrypted credentials on disk.

$ brew install aws-vault

We will assume you have an AWS_USER configured that has privileges to do the rest of the cloud provisioning needed for the Serverless application deployment.

Deploy to Lambda

We will use serverless to deploy to AWS Lambda.

Deploy the Lambda app.

# Build for production.
$ yarn clean && yarn build

# Deploy
$ STAGE=sandbox aws-vault exec AWS_USER -- \
  yarn lambda:deploy

# Check on app and endpoints.
$ STAGE=sandbox aws-vault exec AWS_USER -- \
  yarn lambda:info

See the aws-lambda-serverless-reference docs for additional Serverless/Lambda (yarn lambda:*) tasks you can run.

As a useful helper we've separately hooked up a custom domain for STAGE=sandbox at:

https://nextjs-sls-sandbox.formidable.dev/blog/

ℹ️ Note: We set BASE_PATH to /blog and not /${STAGE}/blog like API Gateway does for internal endpoints for our references to other static assets. It's kind of a moot point because frontend assets shouldn't be served via Lambda/APIGW like we do for this demo, but just worth noting that the internal endpoints will have incorrect asset paths.

Maintenance Status

Active: Formidable is actively working on this project, and we expect to continue for work for the foreseeable future. Bug reports, feature requests and pull requests are welcome.

nextjs-serverless-demo's People

Contributors

jpdriver avatar paulmarsicloud avatar ryan-roemer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nextjs-serverless-demo's Issues

Chore: Upgrade to next@12.

Presumably lots of various changes to update to most current Next.js 12.

Research & Issues

Next has one dist file import a non-existent file in the package

https://unpkg.com/browse/[email protected]/dist/compiled/webpack/bundle5.js at line 141344 has in relevant part:

"use strict";
module.exports = require("next/dist/build/webpack/plugins/terser-webpack-plugin");
 
/***/ }),

However, https://unpkg.com/browse/[email protected]/dist/build/webpack/plugins/terser-webpack-plugin is just a directory with a sub src directory in it.

Experiment: Break up next deployment into two lambdas@edge, behind cloudfront

Architecture

image

Notes

Use the lambda-at-edge library to break up next into two lambdas:

  • default-lambda that contains all the pages
  • api-lambda that contains just the api routes

Pre-rendered and other static assets are extracted from the .next build and are uploaded to S3.

Cloudfront cache behaviours are then configured to send the requests along to either these lambdas or to fetch static assets that were uploaded to S3 (no lambda hit necessary).

Pros

  • AWS deployment is nearly identical to local server (with the proviso that cloudfront forwards on expected cookies or other original request headers e.g. referrer)
  • Fast to deploy
  • Global deployment on lambda@edge that promises lowest latency to end users
  • Cloudfront cache plays well with webpack "chunks" generated by next build

Cons

  • The deployment as configured in terraform is complex (900+ lines of terraform)
  • Requires deeper knowledge of next internals (page routing, dynamic page routing, static generation, api routes, webpack chunks etc.). These details could easily change when upgrading next
  • lambda@edge puts cloudwatch logs in the AZ closest to the user that made the request. So additional work might be needed to aggregate logs
  • Cold starts seem very slow (5-10 seconds), but I've not looked into this much yet

Feature: Inflate configuration like Next.js does internally.

Background

Use case is that we want to get Next.js configured for the server-side as follows:

  • Ideally mimic something like assignDefaults in https://unpkg.com/browse/[email protected]/dist/server/config.js wherein we pass a config object from reading next.config.js and get it all fixed up per that function.
  • Unfortuately, we can't do this because (1) assignDefaults is not exported, and (2) we don't want to require anything that has Next's webpack stuff involved (for loadConfigs webpack hooks).

For our current code, we have a pure object (with object values but no arrays), which we can get away with a naive Object.assign() on, but this lacks:

  1. Handling arrays, functions, etc.
  2. Validation, checking, etc.

Like the real assignDefaults and loadConfig do in Next.js proper.

Task

  • Find some way to get all the Next.js configuration goodness and parity.

Ideas

  • Find a third party library
  • Build and monkey patch dist/server/config.js to a new file that exports assignDefaults
  • At build time inflate the server object using assignDefaults and write it out to disk that gets included in the zip bundle.

Chore: Remove terraform module

The TF IAM permissions module isn't needed for this demo. Let's remove all the terraform.

  • Remove policies in formidops
  • Destroy the TF stack
  • Destroy the AWS bootstrap
  • Remove the TF and AWS code
  • Remove script commands in package.json
  • Update README

chore(server): Various application clean up.

  • Move application source to src to clarify files. Leave server as-is
  • Add a new logo/profile
  • Search on other TODOs in app and either fix or ticket.
  • Memoize retrieving the post data and post content? (Doesn't seem to be a big deal)
  • We're not doing any routing in express, with the Next.js router doing it all. Try out removing express and using http built-in module instead. (Fix or ticket) (Not doing because we use express.static() in root handler.

Deps: Figure out ESM-only updates

remark             13.0.0  13.0.0 14.0.2 dependencies    https://remark.js.org                         
remark-html        13.0.2  13.0.2 15.0.1 dependencies    https://github.com/remarkjs/remark-html#readme

bug(next): [email protected] webpack hook issues

It looks like a monkey-patch for node internals for webpack to decide between 4 and 5 around this code: https://www.runpkg.com/[email protected]/dist/next-server/server/config.js#397 (after prettifying)

await (0, _configUtils.loadWebpackHook)(phase, dir);

produces this error because of missing files, where awkwardly it looks like we have to include webpack even though we don't want to use it at runtime:

"Error: Cannot find module 'next/dist/compiled/webpack/webpack-lib'",
        "Require stack:",
        "- /var/task/node_modules/next/dist/build/webpack/require-hook.js",
        "- /var/task/node_modules/next/dist/next-server/server/config-utils.js",
        "- /var/task/node_modules/next/dist/next-server/server/config.js",
        "- /var/task/node_modules/next/dist/next-server/server/next-server.js",
        "- /var/task/node_modules/next/dist/server/next.js",
        "- /var/task/server/index.js",
        "- /var/runtime/UserFunction.js",
        "- /var/runtime/index.js",
        "    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:815:15)",
        "    at Function.resolve (internal/modules/cjs/helpers.js:80:19)",
        "    at /var/task/node_modules/next/dist/build/webpack/require-hook.js:4:1520",
        "    at Array.map (<anonymous>)",
        "    at Object.<anonymous> (/var/task/node_modules/next/dist/build/webpack/require-hook.js:4:1474)",
        "    at Module._compile (internal/modules/cjs/loader.js:999:30)",
        "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)",
        "    at Module.load (internal/modules/cjs/loader.js:863:32)",
        "    at Function.Module._load (internal/modules/cjs/loader.js:708:14)",
        "    at Module.require (internal/modules/cjs/loader.js:887:19)"

We could just trace and package those files, but it would be nice to not bloat the package.

We've bandaided this with a tracing configuration like this: afc7d2e#diff-8cf16232f3b7094ea431234555d38ad71e6e62764c3e29a5841e25172aea5049R127-R130

Task:

  • Figure out if we can avoid this...

Experiment: Next.js via `serverless` target.

  • Have a target: "serverless" build.
  • Bridge routing with next-aws-lambda or something similar to target one base Lambda
  • Assess package size
    • Check if there are duplicates across page bundles and see if we can tweak Next.js webpack config to split common chunks out.
  • Verify correctness.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.