Doesn't work with next.js Edge runtime about node-postgres HOT 23 OPEN

Sorry for the incorrect/misleading post previously!

absolutely no problem! I'll continue to noodle on this....might take a bit of time to do a similar retrofit that the CF stuff did. I'm also about to go out of town for a bit so...hang tight - i'll get there eventually!

from node-postgres.

This patch fixes for me. Specifically: prevents the error Error: The edge runtime does not support Node.js 'crypto' module. when using pg under the Next.js Edge runtime locally.

pg+8.11.5.patch:

diff --git a/node_modules/pg/lib/crypto/utils-webcrypto.js b/node_modules/pg/lib/crypto/utils-webcrypto.js
index 0433f01..588066e 100644
--- a/node_modules/pg/lib/crypto/utils-webcrypto.js
+++ b/node_modules/pg/lib/crypto/utils-webcrypto.js
@@ -13,7 +13,10 @@ module.exports = {
  * The Web Crypto API - grabbed from the Node.js library or the global
  * @type Crypto
  */
-const webCrypto = nodeCrypto.webcrypto || globalThis.crypto
+
+// PATCH: Don't break under Next.js Edge runtime production build
+const webCrypto = ("webcrypto" in nodeCrypto ? nodeCrypto.webcrypto : undefined) ?? globalThis.crypto
+
 /**
  * The SubtleCrypto API for low level crypto operations.
  * @type SubtleCrypto

(I opted to use runtime-agnostic Feature Detection for webcrypto on the crypto import rather than try detect the Next.js Edge runtime specifically via typeof EdgeRuntime === "string", which I assume could be a valid alternative - haven't verified)

from node-postgres.

@andyjy when i applied your patch to the crypto file I now get this:

⨯ Error: The edge runtime does not support Node.js 'net' module.
Learn More: https://nextjs.org/docs/messages/node-module-in-edge-runtime
    at Object.construct (webpack-internal:///(middleware)/./node_modules/next/dist/esm/server/web/globals.js:37:19)
    at getStream (webpack-internal:///(middleware)/./node_modules/pg/lib/stream.js:8:12)
    at new Connection (webpack-internal:///(middleware)/./node_modules/pg/lib/connection.js:19:36)
    at new Client (webpack-internal:///(middleware)/./node_modules/pg/lib/client.js:48:7)
    at handler (webpack-internal:///(middleware)/./src/pages/api/hello.ts:13:16)
    at eval (webpack-internal:///(middleware)/./node_modules/next/dist/esm/server/api-utils/index.js:32:16)
    at eval (webpack-internal:///(middleware)/./node_modules/next/dist/esm/server/lib/trace/tracer.js:115:36)
    at NoopContextManager.with (webpack-internal:///(middleware)/./node_modules/next/dist/compiled/@opentelemetry/api/index.js:2:7062)
    at ContextAPI.with (webpack-internal:///(middleware)/./node_modules/next/dist/compiled/@opentelemetry/api/index.js:2:518)
    at NoopTracer.startActiveSpan (webpack-internal:///(middleware)/./node_modules/next/dist/compiled/@opentelemetry/api/index.js:2:18108)
 ✓ Compiled in 107ms (263 modules)

from node-postgres.

@janpio here's the reproduction repo deployed directly to cloudflare via @cloudflare/next-on-pages

.. it works(!) - connecting to a Neon database using node-postgres:

• Pages router API route: https://44d82908.mre-pg-nextjs-edge.pages.dev/api/hello
• App router: https://44d82908.mre-pg-nextjs-edge.pages.dev/api/hello-app
• Edge middleware (you contributed): https://44d82908.mre-pg-nextjs-edge.pages.dev/api/foo

The (currently super ugly..) patches to pg I mentioned above are what fixes cloudflare/next-on-pages#605

🎉

from node-postgres.

@janpio ..and here it is deployed to Cloudflare with Prisma added (via @prisma/adapter-pg):

Pages router API route: https://d2ff1a42.mre-pg-nextjs-edge.pages.dev/api/hello-app
App router: https://d2ff1a42.mre-pg-nextjs-edge.pages.dev/api/hello-app
Edge middleware: https://d2ff1a42.mre-pg-nextjs-edge.pages.dev/api/foo

from node-postgres.

Do I understand correctly that you seem a way to make pg work on Cloudflare Pages via next-on-pages via some changes

Exactly! And specifically those changes are:

1. Patch pg to compile/bundle successfully under Next.js' Webpack config. This requires a bunch of patching.. and additional next.config.js config.

I believe this fixes cloudflare/next-on-pages#605

(The patches linked above are still a bit of a messy proof-of-concept - in particular there are some "unhappy paths" in pg that execute during handling of e.g. connection errors/exceptions that use other unsupported Node APIs e.g. process.nextTick(), that I believe will crash currently, but could probably(?) be switched out for the cross-platform queueMicrotask()(?).

The changes in my patches would also need a bit more love before they can be merged into pg in a clean way and without breaking anything for non-Edge runtimes and non-Next.js bundler configs. Should be very doable but I expect quite a chore unfortunately..)

Re: Vercel:

But deploying to Vercel is still completely out of the picture, correct

Yes - at least, I observe that when pushing bundled code to Vercel that includes require("cloudflare:sockets"), Vercel's deployment process objects with error The Edge Function "...." is referencing unsupported modules: cloudflare:sockets

(It's the same error if I push any code referencing e.g. require("this_module_does_not_exist"))

As I understand it - Vercel's Edge runtime product specifically supports their Edge Runtime API that is designed to be runnable under various V8-powered environments - AFAIK they don't particualrly publicise the fact it's Cloudflare Workers under the hood (and could host their Vercel Edge functions elsewhere under non-Cloudflare at any time - for all I know they may do already!).

So at this point while it may be worth asking a friendly contact at Vercel about the possibility of bypassing the unsupported module: cloudflare:sockets error and running this code on Vercel, even if there is some way to make it work today I still wouldn't be too optimistic about it not breaking in future. The only practical way I can imagine would be if Vercel add support for a TCP socket library to their official Edge Runtime spec.

from node-postgres.

Quick note from @prisma that this affects some of our users and is being reported to us: prisma/prisma#24430

(This issue has a reproduction, but that includes Prisma, Next.js and next-auth - that is probably a bit more than what you are asking for @brianc. I am pretty sure just trying to use pg in a Next.js middleware should have the same result though.)

We would love to see this fixed and released, so that Prisma users wanting to use pg in Next.js projects with middleware can do so. Let us know if we can help in any way.

from node-postgres.

@brianc here you go - reproduction for pages router:
https://github.com/andyjy/mre-pg-nextjs-edge

from node-postgres.

Ok wow, that is pretty wild @andyjy - amazing!

Do I understand correctly that you seem a way to make pg work on Cloudflare Pages via next-on-pages via some changes?
But deploying to Vercel is still completely out of the picture, correct?

from node-postgres.

Yeah it would probably...how does one reproduce this issue locally? If running in node then pg should be using the node pieces.

from node-postgres.

Set up a basic next.js app, create an API endpoint, set its' runtime to edge by specifying:
export const runtime = 'edge';

And then assume to use node-postgres from within that route.

CloudFlare Pages requires a next.js app to specify that all backend endpoints will be using the edge runtime.

There's a further complication that NodeEdge runtime isn't the same as CloudFlare workers. I suppose next.js' NodeEdge is emulating what Vercel's edge runtime would look like. But one important detail is that it won't have the ability to run cloudflare:sockets

from node-postgres.

I am trying to use Prisma, PrismaPg and Pg.

Error: The edge runtime does not support Node.js 'crypto' module.

Call Stack

Next.js
eval
node_modules\.pnpm\[email protected]\node_modules\pg\lib\crypto\utils-webcrypto.js (21:1)
(middleware)/./node_modules/.pnpm/[email protected]/node_modules/pg/lib/crypto/utils-webcrypto.js
file:///C:/Code/React/nextplate/.next/server/middleware.js (1292:1)

Next.js
eval
node_modules\.pnpm\[email protected]\node_modules\pg\lib\crypto\utils.js (8:3)
(middleware)/./node_modules/.pnpm/[email protected]/node_modules/pg/lib/crypto/utils.js
file:///C:/Code/React/nextplate/.next/server/middleware.js (1303:1)

Next.js
eval
node_modules\.pnpm\[email protected]\node_modules\pg\lib\crypto\sasl.js (2:16)

https://vercel.com/docs/functions/runtimes/edge-runtime#unsupported-apis

I know it's not really an issue related to Pg but I wanted to inform here anyway.

from node-postgres.

I am using Next.js and Drizzle, and get the same: Error: The edge runtime does not support Node.js 'crypto' module. Learn More: https://nextjs.org/docs/messages/node-module-in-edge-runtime

from node-postgres.

k i'll get this looked at today if I have time!

from node-postgres.

@janpio @andyjy @etodanik would one of y'all be able to provide me w/ more concrete steps to follow to set up a local reproduction? I did npx create-next-app@latest and did NOT use the API router (so the legacy or older form of nextjs). It created a hello api route:

So i tested it at http://localhost:3000/api/hello and it worked. great. then I went into the route file and added

export const runtime = 'edge'

and now its erroring when I hit the route:

So I'm doing something wrong, but not sure what.

This is the full contents of my hello.ts api route file:

// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import type { NextApiRequest, NextApiResponse } from "next";

type Data = {
  name: string;
};

export const runtime = 'edge'

export default function handler(
  req: NextApiRequest,
  res: NextApiResponse<Data>,
) {
  res.status(200).json({ name: "John Doe" });
}

from node-postgres.

update: I tried a different next-js app using the app router instead of the old pages/* approach

I just did same steps as above except said "yes" when the setup asked me to use an app router.

Setup looks like this:

my route.ts file:

import * as pg from 'pg'

export const runtime = 'edge'
const pool = new pg.Pool()

export async function GET() {
  const { rows } = await pool.query('SELECT 1 as one')

  return Response.json({ rows })
}

Immediately when I run 'yarn next dev' to start the project it errors out with being unable to import pg-native which is annoying....I've never messed w/ the edge runtime before - it's quite a bit different...so any thoughts on this?

Import trace for requested module:
./node_modules/pg/lib/connection-parameters.js
./node_modules/pg/lib/client.js
./node_modules/pg/lib/index.js
./src/app/api/route.ts
./node_modules/next/dist/build/webpack/loaders/next-edge-app-route-loader/index.js?absolutePagePath=%2FUsers%2Fbmc%2Fsrc%2Fpg-edge-test%2Fsrc%2Fapp%2Fapi%2Froute.ts&page=%2Fapi%2Froute&appDirLoader=bmV4dC1hcHAtbG9hZGVyP25hbWU9YXBwJTJGYXBpJTJGcm91dGUmcGFnZT0lMkZhcGklMkZyb3V0ZSZhcHBQYXRocz0mcGFnZVBhdGg9cHJpdmF0ZS1uZXh0LWFwcC1kaXIlMkZhcGklMkZyb3V0ZS50cyZhcHBEaXI9JTJGVXNlcnMlMkZibWMlMkZzcmMlMkZwZy1lZGdlLXRlc3QlMkZzcmMlMkZhcHAmcGFnZUV4dGVuc2lvbnM9dHN4JnBhZ2VFeHRlbnNpb25zPXRzJnBhZ2VFeHRlbnNpb25zPWpzeCZwYWdlRXh0ZW5zaW9ucz1qcyZyb290RGlyPSUyRlVzZXJzJTJGYm1jJTJGc3JjJTJGcGctZWRnZS10ZXN0JmlzRGV2PXRydWUmdHNjb25maWdQYXRoPXRzY29uZmlnLmpzb24mYmFzZVBhdGg9JmFzc2V0UHJlZml4PSZuZXh0Q29uZmlnT3V0cHV0PSZwcmVmZXJyZWRSZWdpb249Jm1pZGRsZXdhcmVDb25maWc9ZTMwJTNEIQ%3D%3D&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D!
 ○ Compiling /api ...
 ⚠ ./node_modules/pg/lib/native/client.js
Module not found: Can't resolve 'pg-native' in '/Users/bmc/src/pg-edge-test/node_modules/pg/lib/native'

I know its trying to webpack the server...which is fun.

from node-postgres.

here you go - reproduction for pages router:

amazing, ty

from node-postgres.

@andyjy when i applied your patch to the crypto file I now get this:

ah ok, turns out I am using import { Pool } from "pg" in my app, which the patch above solves for - whereas in the reproduction repo I happened to write import { Client } from "pg" which exposes the new net incompatibility.

Update: 🤦‍♂️ sorry, I realised that while the patch I shared above indeed prevents the Next.js local edge runtime compilation error, I'm not actually executing the pg Client under the Edge runtime locally. So the patch only fixes import, not usage.. (yes I know, facepalm..)

(I remembered I'm actually serving my API routes that use pg under the Node runtime, although my code that references pg is imported - albeit never executed - within some of my Edge runtime routes, which is why I produced the patch.)

Sorry for the incorrect/misleading post previously!

Thinking ahead to whether/how it's possible to get things actually working under the Next.js Edge runtime locally - it would probably(?) require breaking out of Next.js' local Edge Runtime "sandbox" to access node:net locally (i.e. via the Next.js Webpack config). So if this is the case, some tweaks to pg may be necessary but not sufficient alone.

If I get an actually-working prototype then I'll share back.

from node-postgres.

Tiny PR to @andyjy's repro to show the same problem with a Next.js middleware, the common case where Prisma users run into this: andyjy/mre-pg-nextjs-edge#1

Thinking ahead to whether/how it's possible to get things actually working under the Next.js Edge runtime locally - it would probably(?) require breaking out of Next.js' local Edge Runtime "sandbox" to access node:net locally (i.e. via the Next.js Webpack config). So if this is the case, some tweaks to pg may be necessary but not sufficient alone.

Semi informed comment from the sidelines: Afaik the Cloudflare Workers support for pg had similar challenges, that parts of the internal pg code had to be disabled or skipped to not use APIs that are not available in the CfW environment.

from node-postgres.

🤔
Can node-postgres even work in the edge runtime by Vercel and Next.js? Isn't it missing TCP socket support, and Vercel does not offer a connect() like Cloudflare does, that could be used instead. That would be a limitation of Next.js functionality that uses the edge runtime (middleware, app routes) to not be able to support pg at all.

from node-postgres.

🤔 Can node-postgres even work in the edge runtime by Vercel and Next.js? Isn't it missing TCP socket support, and Vercel does not offer a connect() like Cloudflare does, that could be used instead. That would be a limitation of Next.js functionality that uses the edge runtime (middleware, app routes) to not be able to support pg at all.

Yes - having looked into it more, I also realised you're essentially correct - that lack of TCP socket support is a blocker for the Next.js/Vercel Edge Runtime.

Despite running on Cloudflare Workers behind the scenes (still ?) - when deploying to Vercel, Edge runtime code that references require("cloudflare:sockets") fails to deploy, with error The Edge Function "middleware" is referencing unsupported modules: cloudflare:sockets.

What I assume is possible is deploying a Next.js app using the Edge runtime directly to Cloudflare. With no Vercel / Edge Runtime wrapper, the bundled code should just work on Cloudflare Workers.

And if someone wishes to do that, it is technically possible to:

1. Patch pg to compile/bundle successfully under Next.js' Webpack config. This requires a bunch of patching.. and additional next.config.js config.

2. Patch Next.js to successfully pg under the Edge runtime (middleware, Pages API routes + App router) under local development (next dev), by breaking through the local Edge sandbox to expose net.Socket(!). Requires a patch and additional next.config.js config.

Potentially handy if anyone really wants to develop using Next.js + pg under Edge before deploying to Cloudflare.. but I can't imagine this ever being supported by Next.js "officially" unless the Edge runtime is somehow updated to incorporate TCP socket support...(!?)

The reproduction repo is now updated with a proof of concept for both (1) and (2). I haven't deployed to Cloudflare directly but see no reason why this wouldn't work (potentially with a bit more tweaking to get the output bundle correct)

/nerd-snipe over

from node-postgres.

Thanks for the validation of my thought, I was not sure if I was really mixing things up now.

Note, that will probably make this a lot more confusing instead of helping, as it makes it more complex: For Cloudflare Pages there is an additional package next-on-pages so one can deploy Next.js to Cloudflare Pages, and that has its own problem with pg: cloudflare/next-on-pages#605 Not sure if the "Edge Runtime" (from Next.js and Vercel) is even involved there anymore though.

Probably means no easy way to use pg in Next.js middlewares though 😢

from node-postgres.

Shared that workaround over at next-on-pages: cloudflare/next-on-pages#605 (comment)

from node-postgres.