We should implement DO style i/o gates so that the storage functions behave transactionally/consistently as expected (like it already does on the platform).

Have an example showing how to transmit and consume error codes. Bonus points if you can do this for the initial handshake connection too

Umbrella issue tracking all the things to do before we can open the repository to the public:

• write better docs for all packages, including all the flags/commands
• #67

nevermind:

• tests for dev
• figure out licensing story?
• #55
• publish all npm packages to @latest

We currently just use "test" lol. mustfix asap.

import partykit from 'partykit'

partykit.dev(...)

https://tiptap.dev/hocuspocus/
https://github.com/ueberdosis/hocuspocus

y-partykit provider, via y-websocket, has its own logic for resilience etc. We could consolidate it with PartySocket which does much of the same (and better).

Maybe we won't do this because the original need was for y-partykit's onCommand, which might go away with #36

diff --git a/packages/y-partykit/src/provider.ts b/packages/y-partykit/src/provider.ts
index 334929c..de0cb6f 100644
--- a/packages/y-partykit/src/provider.ts
+++ b/packages/y-partykit/src/provider.ts
@@ -1,5 +1,8 @@
 import { WebsocketProvider } from "y-websocket";
-import type * as Y from "yjs";
+import PartySocket from "partysocket";
+import * as Y from "yjs";
+
+console.log(Y);
 
 export default class YPartyKitProvider extends WebsocketProvider {
   constructor(
@@ -20,6 +23,11 @@ export default class YPartyKitProvider extends WebsocketProvider {
     } else {
       options.params._pk = crypto.randomUUID();
     }
-    super(serverUrl, room, doc, options);
+    super(serverUrl, room, doc, {
+      // TODO: fix this
+      // @ts-expect-error We know that this is a valid polyfill
+      WebSocketPolyfill: PartySocket,
+      ...options,
+    });
   }
 }

depends on automerge/automerge#554

right now onConnect takes ws: WebSocket, room: PartyKitRoom. This is annoyong for a couple of reasons

• folks attach handlers directly on the websocket, but we may want to introduce alternate transports in the future
• the "initial"data for a connection is under room.connections.[id].initial, but you don't even have accessto the socket's id

I think we want a better api/abstraction. Like so:

interface PartyKitConnection extends EventTarget {
  transport: 'websocket', // might add 'polling' later 
  ws: WebSocket, 
  id: string, 
  room: PartyKitRoom,
  initial: unknown // or parameterise 
}

// so you'd use it as 
onConnect(conn: PartyKitConnection){
  // ...
}

beta releases aren't being published since I moved the repo to this org, will fix first thing tomorrow

to delete a whole bag of vars at once

bug

Summary

We (@jevakallio @aweary) are using the load API for onConnect to fetch an initial document from a remote database. We noticed that when the document is first loaded, the client doesn't sync correctly. This appears to be because the await load() call happens after the WebSocket connection is open, but before we register any listeners.

Specifics

When the document is not cached and load exists, we hit this path.

This is within getYDoc which is also awaited here

but the message listener on this connect isn't setup until after getYDoc resolves

In our case, await load takes long enough that the client has already sent a SyncStep1 message and received no response. I don't know why exactly yet but this causes the messageSync handler to not send a reply message on the next SyncStep1 message it receives.

That encoder length is just 1 so no reply is sent. I assume missing the state vector in the client's initial SyncStep1 and then eagerly sending a SyncStep1 message from the server puts something in a bad state? In any case, the crux of the problem is that await load can take an arbitrary amount of time, and every sync message received while we were waiting on that gets dropped.

Solutions

We worked around this initially by doing the data-fetching in onBeforeConnect and then passing that to onConnect through unstable_initial. Our load hook is still async but it's fast enough that it completes before any incoming message is received.

I think this is a better paradigm for initial data fetching in general for y-partykit: doing it in onBeforeConnect means you know that the data fetching has completed before the connection is opened so you're not dropping messages.

If that's the best solution here, then I think load should probably not be async since you probably never want to do a bunch of network/IO stuff that would cause messages to be missed and having it be async means people probably think they can await whatever there.

Alternative (aka worse) solutions we explored:

• Add a message queue to WSSharedDoc so it doesn't miss the messages; this feels like a potential can of worms with ordering/sync stuff though
• Ignore the problem and have something like resyncInterval that re-attempts the sync step at some interval; the existing resyncInterval API fixes it but then we're resyncing all the time for no good reason. We'd just need to reattempt syncing before the initial provider.synced = true flag got set

Let me know what you think @threepointone 👋🏻

Be clear about what this is good for. Be very clear about what this probably isn't good for.

seems broadly useful

some browsers dont supply it in localhost https://discord.com/channels/1051830863576453180/1051830865111547986/1069019338507558963

either we should allow looser room names (with special characters), or we should do proper validation and error properly on them.

specifically, long polling when websockets aren't available
we'll probably have to do this behind an abstraction like socket dot io

they don't work right now? fix 'em.

https://twitter.com/threepointone/status/1636699522649784320

As a service, it would cause less traffic if messages from multiple tabs would be sent over one websocket connection.

1. This and would reduce the amount of messages being sent to and from multiple tabs and the server
2. Would be faster to see changes across tabs
3. Would work even when you lost connection and your app supports offline-mode

I know this idea from https://logux.io/. It elects the leading tab which holds the websocket connection and instead of creating own connection, it uses postMessage to send the message to the elected tab, which then sends the message to the server and also receives messages from the elected tab instead of its own websocket connection.

If you are in the same room you want sync to happen between all tabs as quickly as possible, without network delay (even though workers are quick, this will be quicker, especially if you are working on a weak connection or you lost connection).

so you can set env vars during publish/dev time

yjs/y-websocket#81 (comment)

In dev, we should shut down a runtime for a room when all websockets disconnect from it. this is closer to the behaviour on the platform.

we'd like to be able to deploy multiple variants of a script. this is most useful for "previews" in PRs, where folks could check how a thing behaves before landing it. it could also be used for prototypes and experiments when combined with routing

phoenix live view is dope.

• can we deploy liveview directly on to partykit?
• can we build a "liveview, but in js"?
• what happens if react's server components were built for push instead of pull?

deploys/previews

right now an error can crash the whole server, we should be able to guard against that.

fixes #102

We should be able to tap into a running script's production logs. This ties into the broader debugging story, but this specific item would be very useful

export type PartyKitServer<Initial = unknown> = {
  onConnect: (ws: WebSocket, room: PartyKitRoom) => void | Promise<void>;
  onBeforeConnect?: (req: Request) => Initial | Promise<Initial>;
  // this part is new
  onRequest?: (
    req: Request,
    room: PartyKitRoom
  ) => Response | Promise<Response>;
  onBeforeRequest?: (req: Request) => void | Request | Promise<void | Request>;
  // ^ notice how this returns a Request, which is then passed to onRequest
  // giving you an opportunity to pass data in the request itself
};

twitter thread with some more context https://twitter.com/partykit_io/status/1618620327915032576

format, options, etc