The problem I am facing

I am noticing I sometimes get corrupted documents where certain properties are missing, this is a hard error and results in total loss of work. To combat this I would like to validate my updates being received by clients and optionally do nothing (and tell the sending client to reset their state).

While I would like to fix the underlying issue and not have documents corrupted ever I also would like some resiliency to this as it is somewhat inevitable that bugs like this find their way in.

Given I have a method for validating the JSON representation of my document this is how the flow would work in YJS:

• 1. Turn GC off on document
• 2. Take snapshot of document
• 3. Apply update
• 4. Validate JSON representation of document
  • 1. If invalid, revert document to snapshot
• 5. Turn GC back on (sidenote: is there some way to kick off a GC check now as this looks like it may never GC as its always off when the update is applied?)

The solution I would like

I can't see any way of fitting this flow into @hocuspocus/server at the moment as there is no method of listening in to both before and after an update is applied or method of preventing an update being sent to other clients.

Something that does the above would be great but I am open to suggestions on alternatives. The issue I can see with snapshots is I think currently the only methods available are to create a new document out of that snapshot. Which would require replacing the in memory document and telling connected clients to reset their state 🤔

Alternatives I have considered

The above assumes validation on every change, which I don't think should be a problem with a precompiled validation function using something like https://github.com/ajv-validator/ajv. However an alternative flow is to do validation when persisting to long term storage and resetting the server and client state to the last valid one from long term storage when an invalid document is received.

This would look something like:

• 1. Debounced validation in onChange
  • 1. If validation fails, reset the document and force open clients to do the same.

Currently I am preventing corrupted documents being saved into long term storage but am lacking a way of force restarting the server and currently connected clients.

We want to setup a heartbeat to detect disconnects that is shorter than the 15s period that the awareness protocol uses.

Description

onCreateDocument does not have the initial value from the client

Steps to reproduce the bug

1. update a document in a client
2. start the server
3. see an empty value in onCreateDocument. then that value shows up in onChange immediately afterwards.

@hocuspocus/server": "^1.0.0-alpha.53

import { Server } from "@hocuspocus/server";

const hocuspocus = Server.configure({
  port: 1234,

  async onCreateDocument(data) {
    console.log("create is empty:", data.document.isEmpty("monaco"));
    console.log("create value:", data.document.getText("monaco").toString());
    return data.document;
  },

  async onChange(data) {
    console.log("change is empty:", data.document.isEmpty("monaco"));
    console.log("change value:", data.document.getText("monaco").toString());
  },
});

hocuspocus.listen().then(() => console.log("started"));

Expected behavior

onCreateDocument should have the value

Screenshot, video, or GIF

Hi There,

I am experimenting with the onConnect hook to see what's possible. I am looking at ways to enforce document naming conventions to ensure they follow a pattern. In our use case it's a course name followed by a lecture date (IE. 'math101.2021apr21')

Anyway I was attempting to do so in the onConnect hook to abort a connection if say if just 'math101' was entered. I had the following thus far as a rudimentary test:

const server = Server.configure({
  port:1234,
  async onConnect(data) {
    const { documentName } = data

    const documentSpace = documentName.split('/')
    const entityId = documentSpace[1].split('.')
    if ( entityId.length !== 2) {
      throw new Error('Document Name does not match naming convention')
    }
  },

I also tried returning a reject() on the promise.

However it seems when I try the above it does indeed throw the error, but the server process is terminated as well, not just the connection. Any guidance on what I should do differently?

Description
When a Heading node is positioned in the middle/end of a document, pressing Enter at the beginning of it's line causes an error.

Steps to reproduce the bug
Steps to reproduce the behavior:

1. Go to https://codesandbox.io/s/simplest-tiptap-issue-knscq?file=/src/App.tsx
2. Put your cursor at the beginning of Heading
3. Hit enter twice
4. See error message

Expected behavior
The Heading content is moved down.

Additional context
We noticed this issue with StarterKit v70 as well.

Description

I have had to replace @hocuspocus/provider with y-websocket as when it shares a document with y-indexeddb and I replace an existing key in a Y.Map inside that document the key is deleted (instead of replaced) when using @hocuspocus/provider.

When using y-websocket the problem does not exist.

Note:

I do not instantiate the @hocuspocus/provider until after y-indexeddb has synced.

Steps to reproduce the bug

I am working on this part, we have a complicated application so working out the best way to reduce this into something that can be verified.

Expected behavior

Should work correctly with y-indexeddb and replace not delete the key.

Environment?

• operating system: macOS 11.2
• browser: Chrome 91
• hocuspocus version:
  • @hocuspocus/server: 1.0.0-alpha.60
  • @hocuspocus/provider: 1.0.0-alpha.8

Additional context

Seeing as it works correctly with y-websocket instead of @hocuspocs/provider and they should both be applying / sending updates in the same way, it suggests the issue lies somewhere there?

Note I mentioned corrupted documents in #135, this is not the only source of them (I am getting ones from undo/redo in YJS which are unrelated to hocuspocus) so I am still interested in that topic even if this bug is resolved.

As an additional question, what is the current benefit of using @hocuspocus/provider over y-websocket? Is it just something that will be expanded on in future or does it already offer different features?

That’s not very helpful:

hocuspocus_1  | [2021-05-25T07:09:01.550Z] Created document "App%5CModels%5CNewsletter%3A1" …
hocuspocus_1  | [2021-05-25T07:09:01.621Z] Document "App%5CModels%5CNewsletter%3A1" changed …
hocuspocus_1  | /var/www/node_modules/prosemirror-model/dist/index.js:1368
hocuspocus_1  |   if (!json) { throw new RangeError("Invalid input for Node.fromJSON") }
hocuspocus_1  |                      ^
hocuspocus_1  |
hocuspocus_1  | RangeError: Invalid input for Node.fromJSON
hocuspocus_1  |     at Function.fromJSON (/var/www/node_modules/prosemirror-model/dist/index.js:1368:22)
hocuspocus_1  |     at prosemirrorJSONToYDoc (/var/www/node_modules/y-prosemirror/dist/y-prosemirror.cjs:1060:27)
hocuspocus_1  |     at Prosemirror.toYdoc (file:///var/www/node_modules/@hocuspocus/transformer/dist/hocuspocus-transformer.esm.js:38:20)
hocuspocus_1  |     at Tiptap.toYdoc (file:///var/www/node_modules/@hocuspocus/transformer/dist/hocuspocus-transformer.esm.js:10193:39)
hocuspocus_1  |     at Webhook.onCreateDocument (file:///var/www/node_modules/@hocuspocus/extension-webhook/dist/hocuspocus-webhook.esm.js:110:68)
hocuspocus_1  |     at processTicksAndRejections (node:internal/process/task_queues:96:5)

If the JSON is invalid, the exception should output the JSON. :)

Depending on how the promise is rejected in the onConnect hook the server will crash.

If the async function returns Promise.reject(); it's fine, if it returns a rejection message or an Error (should usually be an error) it will crash the server.

Steps to reproduce the bug

const hocuspocus: Hocuspocus = Server.configure({
  async onConnect(data) {
    return Promise.reject(new Error('paul')); // <== crash
    return Promise.reject('paul'); // <== crash
    throw new Error('paul'); // <==== crash
    return Promise.reject(); // <== works as expected, close the connection and does not crash.
  },
});

The problem I am facing
I'm unable to set a client to read only in methods other than onConnect

The solution I would like
I'd like to have access to the client connection instance in all of the lifecycle hooks.

Alternatives I have considered
I've implemented a workaround that involves passing a getter to the entire Hocuspocus instance to my extension in order to iterate through the connections:

const document = this.getInstance().documents.get(documentName)
document.connections.forEach(({ connection }) => {
  const thisClientsVersion = connection.context.SCHEMA_VERSION
  if (!thisClientsVersion || thisClientsVersion < requiredVersion) {
    this.logger.warn({
      label: `${MODULE_NAME}.setOutdatedClientsToReadOnly`,
      message: `(${connection.socketId}) Client schema is out of date (v${thisClientsVersion}). Setting it to read only`,
    })
    connection.readOnly = true
  }
})

Additional context
I need to be able to set some connections to read only when certain events happen.

I think we’re not using status codes for the close event right now. Leaving this here as a note to check what we’re doing now, and if we can improve that.

https://developer.mozilla.org/en-US/docs/Web/API/CloseEvent

Couldn't create a PR as can't fork the repo but the types field is incorrect in @hocuspocus/provider.

This:

Should be:

  "types": "dist/packages/provider/src/index.d.ts",

Part of the documentation?
I’ve read the following page of the documentation: https://www.hocuspocus.dev/examples/monaco

Really helpful parts
The demo works perfectly

Hard to understand, missing or misleading
It would be good to have inlined example code for the editor integrations. I'm interested in the Monaco one at the moment. There is no link to the source code and the page source is obfuscated so I couldn't grab it that way.

Description

I'm trying to contribute to Hocuspocus however I only have access to an M1 machine as my development environment, currently RocksDB will not build on such a machine. It looks like a fix was merged in the last two weeks but we're waiting on a release here:

Level/rocksdb#171

Once a release is out we can updated RocksDB and all will be well, however it does look like that will mean a major jump from 4.X -> 5.X.

Update: Looks like Gridsome -> Sharp also has an M1 issue blocking install, this could be fixed using resolutions to force the sharp version

gridsome/gridsome#1519

Further update: node-sass @ v5 is also an issue, but it looks like a clean update to v6

When the onConnect hook is really slow and two or more users are connected and one starts to type before the connection is established, everything breaks:

1. The backend throws an exception:

[1] incoming message 0
[1] Caught error while handling a Yjs update Error: Integer out of range!
[1]     at Module.readVarUint 

2. Changes aren’t persisted through RocksDB anymore.

• 2 instances connected with Redis

Error:
[2021-04-22T22:09:08.111Z] New connection to "new-doc" …
[2021-04-22T22:09:08.111Z] Created document "new-doc" …
(node:53740) UnhandledPromiseRejectionWarning: Error: "new-doc" is already bound to this RedisPersistence instance
    at Object.error.create (~/repo/hocuspocus/node_modules/lib0/error.js:12:28)
    at RedisPersistence.bindState (~/repo/hocuspocus/node_modules/y-redis/src/y-redis.js:158:13)
    at Redis.onCreateDocument (file://~/repo/hocuspocus/node_modules/@hocuspocus/extension-redis/src/Redis.ts:45:28)
    at file://~/repo/hocuspocus/node_modules/@hocuspocus/server/src/Hocuspocus.ts:288:74
...

Setup:
extensions: [
    // Log level is not configurable at the moment.
    new Logger(),
    new Redis({
      host: process.env.REDIS_HOST,
      port: process.env.REDIS_PORT,
    }),
    new RocksDB({
      // [required] Path to the directory to store the actual data in
      path: "./database",

      // [optional] Configuration options for theRocksDB adapter, defaults to "{}“
      options: {
      // This option is only a example. See here for a full list:
      // https://www.npmjs.com/package/leveldown#options
        createIfMissing: true,
      },
   }),
],

We tried to test hocuspocus in a Node.js, but that doesn’t seem to be reliable enough, and it’s hard to write tests like this.

I tried to avoid it, but I think we should set up Cypress to do testing in a browser environment. That’s slower, but it’s probably easier to write tests and see what’s wrong when something fails.

I use a collaboration extension so that many users can work together on the same document, but users connecting to the same document see different content and they are out of sync with the rest.

In this video you can see how one user opened the same document in 4 browsers - chrome, opera, firefox and safari.

At first they all synced fine, but after restarting all pages the safari connected to another document and was not synced with the other browsers.

Then we switched to another document and saw the same picture, safari connected to another document.

Then we switched to another document, and we can see all the browsers were synchronized.

This also happens when different users connect to the same document.

Please let me know if you need my code.

Hi guys!

I am trying to connect your hocuspocus library to my koa server and it does not work as expected.
After some time, the connection to the socket is interrupted and not restored.

I made a connection, as described in the documentation.

I use:

FRONTEND:

• vue 3

BACKEND:

• koa
• koa-easy-ws
• @hocuspocus/extension-rocksdb: "^1.0.0-alpha.49",
• @hocuspocus/server: "^1.0.0-alpha.47",
• @hocuspocus/transformer: "^1.0.0-alpha.5",

And this is my setup:

const server = Server.configure({
  extensions: [
    new RocksDB({
      path: './tiptap-store'
    })
  ],

  timeout: 30000
});

router.get('/:projectId', async ctx => {
  try {
    if (ctx.ws) {
      const ws = await ctx.ws();

      server.handleConnection(ws, ctx.request, ctx.params.projectId);
    }
  } catch (err) {
    ctx.bad(400, err);
  }
});

Could you please add support for Koa as you have for Express?

Part of the documentation?

On hocuspocus.dev, there are links to /extensions/rocksdb which need updating to /api/extensions/rocksdb (i.e. https://www.hocuspocus.dev/api/extensions/rocksdb/ )

No worries, we have you covered! We built an extension that's meant to be used as primary storage:
the [RocksDB extension](/extensions/rocksdb). It's just a couple of lines to integrate.

There are ~4 occurrences: https://github.com/ueberdosis/hocuspocus/search?q=%2Fextensions%2Frocksdb

Additional context
New here. Very excited about hocuspocus and tiptap!

Description
Packaging bug :
Since package.json states type: module, the require import hint cannot be of a .js file or the server fails to import :

Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: .../node_modules/@hocuspocus/server/dist/hocuspocus-server.js
require() of ES modules is not supported.
require() of .../node_modules/@hocuspocus/server/dist/hocuspocus-server.js from .../hocuspocus-bug.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename hocuspocus-server.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from .../node_modules/@hocuspocus/server/package.json.

Steps to reproduce the bug
Steps to reproduce the behavior:

Create a single line script :

var h = require('@hocuspocus/server');

Run it

Expected behavior
Import works fine

Environment?

• node 16.1.0

Additional context
Changing @hocuspocus/server/package.json this way

-    "require": "./dist/hocuspocus-server.js"
+    "require": "./dist/hocuspocus-server.cjs"

And moving the file appropriately fixes the problem.

The problem I am facing
When a new code release of Hocuspocus goes out, I need to be able to close all open Hocuspocus connections to the old instance and force them to reconnect to the new one. This is necessary to ensure that clients can only connect to 1 Hocuspocus instance at a time for a given doc.

We're using AWS blue/green deployments to cutover traffic, but the websocket connections are not dropped, allowing existing clients to remain connected to the old instance and new clients to the new one.

The solution I would like
An instance method for Hocuspocus to close all open connections. Server.closeAllConnections

This is slightly different from Server.destroy, which would shutdown the http server and prevent a rollback from being possible.

Alternatives I have considered
We've worked around similar problems in the past using a reference to the instance and reaching into its internals:

this.getInstance().documents.forEach((document) => {
  document.connections.forEach((conn: Connection) => {
    conn.connection.close()
  })
})

Additional context
It's possible that this is solved later down the road once the Hocuspocus scaling work is complete (#87).

s/timout/timeout/

https://github.com/ueberdosis/hocuspocus/blob/main/docs/src/docPages/api/settings.md

provider.ws.send('ping') but that actually crashed our hocuspocus server due to not being encoded.

I have noticed that the provider triggers the synced and message events before actually applying the updates.

For context I am trying to determine when a document has loaded by looking at the contents of it (as I noticed onCreateDocument fires after the synced event so I need to listen to synced and message and check the contents of my document until it is not empty).

I was seeing a 15 second load time but it was because I was not aware the provider.on() callbacks trigger before any updates received are actually applied to the document.

For now I have wrapped my code in setTimeout(() => {}, 0) that checks if the document is loaded but am wondering why it runs in this order / what the use case is for having the event be emitted before the data is in the document?

And if this is intended perhaps this behaviour should be documented as it is somewhat counter intuitive to how the other YJS providers work.

Description

This is either a server bug, or a documentation issue. Documentation suggests that the requestParameters payload passed to onConnect is an object, see:
https://www.hocuspocus.dev/guide/authentication-and-authorization/

However it is actually an instance of URLSearchParams and attributes must be accessed with get method, like:

requestParameters.get('access_token') !== 'super-secret-token'

Expected behavior

I think on the server it's more typical for this to be a plain object, leaving the documentation as-is but changing the implementation internally would be the ideal fix.

The problem I am facing
I’m used to set HTTP headers for authentication, but that’s not working[1] for WebSocket connections. Currently, we use URL parameters to send tokens, but that doesn’t feel right. Those URLs can land in server logs, and those shouldn’t have tokens.

[1] only through hacks, not in all browsers

The solution I would like
Maybe we could use WebSocket messages to send tokens to the server.

Alternatives I have considered
Cookies, HTTP headers

Additional context
Not my idea, read it here: yjs/y-websocket#8 (comment)

Description

According to the docs the way to do authentication is to throw an error in the onConnect callback if authentication checks fail.

When throwing an error (or rejecting the returned Promise), the connection to the client will be terminated.

Steps to reproduce the bug
Steps to reproduce the behavior:

1. Create a minimal repo (might be nice if we had one of these ready for bug reports)
2. Throw an error in onConnect and nothing else
3. Open client in two browsers
4. Awareness and state is transferred between browsers

https://github.com/ueberdosis/hocuspocus/blob/main/packages/server/src/Hocuspocus.ts#L190

^ This line is never called.

Expected behavior

The connection should be terminated.

It should also not be possible for state to transfer between unauthenticated clients at all, messages received before authentication has completed should probably be held in a queue and emitted once onConnect has resolved. edit: seems like this was the intended behavior

It would be nice if some users could only receive updates and not make changes to the document, like Google Docs read only mode.

Description
Monitor fails to render the entire page when a lot of events are logged (a few thousands in my case, which is actually not that much)

Look at the scrollbar on my screenshot. Only the top of the page is rendered properly, the rest is all blank.

Steps to reproduce the bug
Steps to reproduce the behavior:

1. Reproduce the y-websocket ddos (easy way to get a lot of events)
2. Open Monitor
3. Wait ~1 second for all the data to load, the scrollbar becomes huge and nothing is rendered.

Screenshot, video, or GIF

Expected behavior
There should be a limit on the number of events to render, optionally with search and pagination. A limit would be fine for a realtime monitor, I'm gessing ~100 events is a sensible limit.

Description
I tried to install extension-rocksdb on my M1 machine but it fails. Rosetta emulation for terminal doesn't help. It installs fine on intel mac.

Environment?

• operating system: MacOS Big Sur 11.3.1 (20E241)
• chip: M1
• node: v16.1.0
• hocuspocus version: 1.0.0-alpha.53

Additional context
Here is my entire log

Hey, I've found a small bug 🐛

If I call my api in onConnect for checking if the user has permission to view/edit the document, only the first connection get the correct state.

Steps to reproduce

1. Run this server

const server = Server.configure({
  async onConnect(data) {
    await new Promise((r) => setTimeout(r, 1337));
  }
})

2. Connect 2 clients from DIFFERENT browsers (otherwise it works fine with Kevin's y-websocket working with Broadcast Channel)

3. Type in one of the documents

4. Reload one of the browsers. Now the first changes are gone.

It looks like it's not sending the first message properly with initial state if onConnect is async.

I followed tutorial and done minimal setup, just to make it work, but i am receiving an exception on the front-end.

So, when i return document from onCreateDocument(data) on server side, i am receiving this error on front end:

Exception: 
TypeError: Cannot read property 'matchesNode' of null at EditorView.updateStateInner (webpack-internal:///./node_modules/prosemirror-view/dist/index.es.js:4765:43) at EditorView.updateState (webpack-internal:///./node_modules/prosemirror-view/dist/index.es.js:4736:8) at Editor.dispatchTransaction (webpack-internal:///./node_modules/@tiptap/core/dist/tiptap-core.esm.js:2407:19) at EditorView.dispatch (webpack-internal:///./node_modules/prosemirror-view/dist/index.es.js:5004:50) at eval (webpack-internal:///./node_modules/y-prosemirror/src/plugins/sync-plugin.js:301:28) at ProsemirrorBinding.eval [as mux] (webpack-internal:///./node_modules/lib0/mutex.js:37:9) at ProsemirrorBinding._forceRerender (webpack-internal:///./node_modules/y-prosemirror/src/plugins/sync-plugin.js:297:10) at eval (webpack-internal:///./node_modules/y-prosemirror/src/plugins/sync-plugin.js:156:17)
message: "Cannot read property 'matchesNode' of null"

Hocuspocus setup method:

async onCreateDocument(data) {
	try {
		const prosemirrorDocument = JSON.parse(
			`{
				"type": "doc",
				"content": [{ "type": "paragraph", "content": [] }]
			}`
		);

		// When using the tiptap editor we need the schema to create
		// a prosemirror JSON. You can use the `getSchema` method and
		// pass it all the tiptap extensions you're using in the frontend
		const schema = getSchema([ Document, Paragraph, Text ]);

		// Convert the prosemirror JSON to a Y-Doc and simply return it
		const result = prosemirrorJSONToYDoc(schema, prosemirrorDocument);
		return result;
		// return data.document;
	} catch(e) {
		console.log(e);
	}
}

This is front-end setup:

mounted() {
    const ydoc = new Y.Doc();
    this.provider = new WebsocketProvider(process.env.VUE_APP_WS_ENDPOINT, 'tiptap-collaboration-example', ydoc);
    this.provider.on('status', event => {
      this.status = event.status;
    });

   this.editor = new Editor({
      extensions: [
	Document,
        Paragraph,
        Text,
	CollaborationCursor.configure({
          provider: this.provider,
          user: this.currentUser,
          onUpdate: users => {
            this.users = users;
          },
        }),
        Collaboration.configure({ document: ydoc }),
	],
	});

			this.updateCurrentUser({
				name: this.getRandomName(),
				color: this.getRandomColor()
			});

		localStorage.setItem('currentUser', JSON.stringify(this.currentUser));
	},

Those are the library versions that i am using on the front-end:

"@tiptap/vue-2": "^2.0.0-beta.1",
"@tiptap/starter-kit": "^2.0.0-beta.3",
"@tiptap/core": "^2.0.0-beta.2",
"@tiptap/extension-underline": "^2.0.0-beta.1",
"@tiptap/extension-image": "^2.0.0-beta.1",
"prosemirror-commands": "^1.1.7",
"@tiptap/extension-collaboration": "^2.0.0-beta.3",
"@tiptap/extension-collaboration-cursor": "^2.0.0-beta.3",
"yjs": "^13.5.2",
"y-websocket": "^1.3.11"

Any thoughts what am i doing wrong?

Hi,

I am not sure if this is related to hocuspucus server, or to my config, but when i followed tutorial, i am not able to run my server, because of the added import.

I am receiving the following error:
Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: .../node_modules/@hocuspocus/server/dist/hocuspocus-server.js
require() of ES modules is not supported.

I am using Babel for build..

Update:

I found that this issue is related to the node version:
manuelbieh/geolib#208

After switching to node version 12.12.0, now i got error from your module:

import WebSocket from 'ws';
^^^^^^

SyntaxError: Cannot use import statement outside a module
....
....

There were a few questions already about how to scale hocuspocus. The Redis extension is a start, but we still need to figure a lot of things out. In my opinion, this is how it should be done at the moment:

• Have a load balancer as entrypoint
• Have multiple nodes with hocuspocus running only the Redis extension behind the load balancer - Let's call them "workers"
• Have a single node with hocuspocus running the Redis extension, RocksDB extension and any application integrations like the onConnect hook or the webhook extension. This node is not connected to the load balancer - Let's call it "manager"

This way, all incoming traffic is split equally between the workers. Those sync the changes between themselves and the clients nearly instantaneously. The changes will also be synced over Redis (pub/sub) to the manager node which will handle the storage and any integrations into existing applications. As all of this doesn't need to be instantaneous it's fine if this node is getting slow or runs at max CPU usage. RocksDB and any application integrations could be on two different manager nodes as well.

Things to figure out:

• We need to investigate whether the updates stored in Redis will be deleted after a certain amount of time. As things can queue up when the manager node becomes slow or unresponsive.
• We need to bench the whole system on common VMs (DigitalOcean / AWS) to see if we will hit a limit and what the limit actually is. We need to do this with a third party integration like the webhook extension and without it.
• Can the primary storage (RocksDB) be scaled horizontally? Are there alternatives that are faster and scalable?

Things to do:

• Built and provide an official hocuspocus Docker image that's super easy to extend
• Provide example configs for Docker Swarm, Kubernetes and other common orchestrators in the docs

What do you folks think? (cc: @hanspagel)
Any ideas/wishes/opinions on scaling?

Note to myself, I need to check that:
https://github.com/meatflavourdev/legendary-octo-chainsaw/blob/main/render.yaml

I am running into an issue where the redis extension works correctly the first time the server is restarted, however after the initial connection to a document is closed, whenever a new connection is reopened it no longer subscribes to changes of that document.

This is what I think is happening:

1. When you first restart the server a YJS doc is created, this is subscribed to by the y-redis package, because the y-redis package is subscribing to the same document that is being edited in memory it is aware of the changes
2. After that document is closed (all clients disconnect) the y-redis package still remembers the document but @hocuspocus/server forgets the document
3. When a client now connects to that document @hocuspocus/server creates a new in memory document and the y-redis package does not subscribe to it because it remembers it.
   The issue with 3. is that the onCreateDocument hook claims that if you return a document from that hook it is loaded. Which the @hocuspocus/redis package does (

   hocuspocus/packages/redis/src/Redis.ts

   Line 49 in 5c5f484

   return binding.doc

   ), however if you look at the implementation of that hook it doesn't swap the in-memory doc for the one you use it merges them together (

   hocuspocus/packages/server/src/Hocuspocus.ts

   Line 254 in 5c5f484

   applyUpdate(document, encodeStateAsUpdate(loadedDocument))

   ), which means you end up with the same in-memory doc being used by @hocuspocus/server which is not the document that y-redis is subscribed to.

I think this can be solved simply by having the @hocuspocus/redis move the creation of RedisPersistence to onConnect and destroy it in onDisconnect when its the last client:

    async onConnect(data) {
        if (!this.persistence) {
            this.persistence = new yRedis.RedisPersistence(
            // @ts-ignore
            this.cluster
                ? { redisClusterOpts: this.configuration }
                : { redisOpts: this.configuration });
        }
    }
    async onDisconnect(data) {
        if (data.clientsCount === 0) {
            this.persistence.destroy();
            this.persistence = undefined;
        }
    }

I have tested this locally and it works for me

hocuspocus/packages/rocksdb/package.json

16: "types": "dist/packages/leveldb/src/index.d.ts"
👆is wrong, it should be 👇
16: "types": "dist/packages/rocksdb/src/index.d.ts"

Sorry couldn't make a PR 😂

Description
Somewhere in the y-websocket/hocuspocus client/server stack (sorry for not knowing exactly where it comes from) :

When a connection is established with hocuspocus and reset quickly after the instantiation from the client the document's data is not sent properly to the client.

Steps to reproduce the bug see #122 (comment) , more accurate after investigation
hocuspocus + rocksdb server
tiptap + y-websocket frontend

let provider,
  ydoc = new Y.Doc(),
  docId = 'somedoc';
const registerProvider = () => {
  provider = new WebsocketProvider('ws://somehost', docId, ydoc);
};

const resetConnection = () => {
  provider.destroy();
  registerProvider();
};
// <===== HERE, resetting the provider on the document 
// before the document is properly fed from the backend prevents it to ever get its full data
// try with different values between 1 and 100 and you'll reproduce
setTimeout(resetConnection, 10); 

new Editor({
  extensions: [
    StarterKit,
    SomeCustomExtension,
    Collaboration.configure({ document: ydoc }),
   ]
});

Expected behavior
Should initialize editor content properly even when connection is resetting a few times quickly

Additional context
I am using firebase authentication to authenticate the user before handing the connection over to hocuspocus, but firebase authentication tokens can change often. So, if the client disconnects and tries to reconnect using y-websocket internal reconnection logic the token may be expired and the server will reject the connection. To avoid that I reset the connection on every onIdTokenChanged , and one of those happen within the first second of a page being loaded.

So I can work around that by preventing the connection reset within the first few seconds, or even better by allowing y-websocket query params to be functions that are called every time a connection attempt is made.

Description
Within a custom extension, an onRequest method with a response and rejected promise crashes Node.js with an unhandled error event. The result is the same for a @hocuspocus/server onRequest method when run via Node.js 16.6.1, or a very verbose deprecation warning on 14.17.4.

Steps to reproduce the bug

1. Create a custom extension having only one non-empty method, like so:

async onRequest(data: onRequestPayload): Promise<any> {
  return new Promise<void>((resolve, reject) => {
    const { request, response } = data
    response.writeHead(200, { 'Content-Type': 'text/plain' })
    response.end('EXTENSION RESPONSE HERE')

    return reject()
  })
}

2. Make any cURL request to the local server, e.g. curl -v http://127.0.0.1:4000/request
3. Receive the expected "EXTENSION RESPONSE HERE" response, but find this error at the Node.js server:

[onRequest] Cannot read property 'message' of undefined
events.js:377
      throw er; // Unhandled 'error' event
      ^

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at writeAfterEnd (_http_outgoing.js:694:15)
    at ServerResponse.end (_http_outgoing.js:815:7)
    at file:///.../node_modules/@hocuspocus/server/dist/hocuspocus-server.esm.js:1529:26
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
Emitted 'error' event on ServerResponse instance at:
    at writeAfterEndNT (_http_outgoing.js:753:7)
    at processTicksAndRejections (internal/process/task_queues.js:83:21) {
  code: 'ERR_STREAM_WRITE_AFTER_END'
}
error Command failed with exit code 1.

Or, for the server method run the example from https://www.hocuspocus.dev/api/on-request and find the same error on the latest Node.js, or the below from version 14:

(node:38547) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'message' of undefined
    at file:///.../node_modules/@hocuspocus/server/dist/hocuspocus-server.esm.js:1721:27
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:38547) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 12)
(node:38547) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Expected behavior
The custom extension onRequest method sends its response and promise reject to prevent further handling and server output without resulting in a Node.js exit.

Not rejecting the promise prevented crashing, but resulted in a malformed extra "OK" response via

Environment?

• operating system: macOS 11.5.1
• browser: cURL
• hocuspocus version: 1.0.0-alpha.60

Hi guys! First off all, thanks for your work. It's amazing!

Description
In my project, I can give users access to edit projects or not.

For example, I gave the user access to edit the project, the user connects to the project and edits it. After a while, I decided to take away the ability to edit from the user. The user should see all updates, but shouldn't edit anything.

How can I immediately inform the client from the server about this, in order to block the editor? I don't need to close his connection, I just need to block his editor.

Also, if the auth token has expired, I would like to close the connection. I can do this with connection.close(), but in that case, I cannot send an error code and message.

On the server, I check user permission in the onChange hook, and if the user no longer has the ability to edit, I give him a connection.readOnly = true. And it works, what the user writes is not saved on the server. But I need also block his editor on the client.

And I also check user permission in the onConnect hook, and I need the same instruments here, but currently, I don't have connection as an instance of Connection here, I can only change readOnly mode

This is my code from server

async onConnect (data) {
    const permission = await checkPermission(someData);

     if (permission.code === 4000) { // read only
        data.connection.readOnly = true;
        // here I need to send to client message - like 'Read Only'
      } else if (permission.code >= 4400) { 
        // here I need to close connection, but with code and message.
        // data.connection.close(permission.code, permission.message) - don't work, because data.connection.close() is not a function
      }
}

async onChange (data) {
    for (const { connection } of data.document.connections.values()) {
      if (connection.socketId === data.socketId) {
        const permission = await checkPermission(someData);

        if (permission.code === 4300) { // can only read the document
          connection.readOnly = true;
          // connection.send('test');
        } else if (permission.code >= 4400) { // connection must be closed and never reconnect
          // here I need to close connection, but with code and message.
          // connection.close(permission.code, permission.message);
        }
      }
    }
}

Please, let me know if you have any ideas about it or if you have a better solution for solving this problem.

Thanks!

Description
When using the onConnect hook to authenticate a client the messages sent by this client before he is disconnected will still be applied to the underlying Y doc.

Steps to reproduce the bug

Client

tiptap collaboration sample should work

Server

const hocuspocus: Hocuspocus = Server.configure({
  async onConnect(data) {
    // simulate a very slow authentication process that takes 10 seconds (or more if you want to type more)
    await new Promise((resolve: Function) => {
      setTimeout(() => { resolve(); }, 10000);
    });
    return Promise.reject();
  },
  extensions: [
    new Logger(),
    new RocksDB({
      path: './rocksdb',
    }),
  ],
});

1. Open the client, write some stuff before the connection is dropped by the server
2. Remove the "authentication failure" return Promise.reject()
3. Document content written between the connection establishment and the Promise rejection is saved in the document

Expected behavior
Changes sent by a client that fail to authenticate are dropped into the infinite void of the non-existing-anymore stuff. Or optionally logged somewhere if you don't want to get metaphysical.

Environment?

• @hocuspocus/server alpha.58

Hi,

I occurred one more issue during collaboration setup :/ After hocuspocus restarts, existing content from tiptap v2 is sent again, which results doubling the data in the content.

Steps to reproduce:

1. Open tiptap v2 editor
2. Enter some text, for example 'Test'
3. Stop hocuspocus server
4. Start hocuspocus server again

Error: Front end sends 'Test' content again to hocuspocus, which results with two 'Test' content paragraphs in database / file.

NB: After you restart it again, two 'Test' paragraphs will be sent, which results with 4 'Test' items in the database..

Front end is configured in the following way:

const ydoc = new Y.Doc();
const token = user.token;
const uri =  'ws://localhost';
this.provider = new WebsocketProvider(uri, 'tiptap-example', ydoc, {
	params: {
		token
	}
});
this.provider.on('status', event => {
	this.status = event.status;
});

this.editor = new Editor({
	extensions: [
		...defaultExtensions().filter(extension => extension.config.name !== 'history'),
		Underline,
		Image,
		Document,
		Paragraph,
		Text,
		CollaborationCursor.configure({
			provider: this.provider,
			user: this.currentUser,
			onUpdate: users => {
				this.users = users;
			},
		}),
		Collaboration.configure({
			provider: this.provider,
			document: ydoc
		}),
	],
});

Thank you for help in advance!

For people using Socket.io it would be great to integrate hocuspocus, like it’s already possible with Express.

Description
onConnect hooks, no matter what they return or throw, cannot interrupt the hook chain which results in subsequent hooks being run as though everything is OK.

This is especially problematic for auth, as it means there is no way to prevent the createDocument/createConnection blocks from being reached:

Steps to reproduce the bug
Steps to reproduce the behavior:

Try to setup the onConnect example: https://www.hocuspocus.dev/api/on-connect#example

Just throw an error in an extension with onConnect

Observe that it does not prevent downstream onConnect hooks or the onCreateDocument hook from running.

Expected behavior
The chain should be interrupted and the connection should be closed (This was previous behavior).

Additional context
I believe this is a regression introduced in 450e5ad

calling .catch returns a new promise that will continue the chain, just like catching an error. This is described in with examples in the MDN docs: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/catch#using_and_chaining_the_catch_method

The problem I am facing
I’m always frustrated when … I need access to my Tiptap extension schema on the server (Hocuspocus or my own API) when the extensions all live in my client package.

The goal is to be able to create documents on the server side with some initial content, which leverages the prosemirrorJSONToYDoc utility, which requires the schema.

The solution I would like
A method like getSchemaJSON(extensions) and extensionsFromJSON(jsonSchema).

These methods would still need to be run in a browser environment, but having them would make it easy to checkin a copy of the JSON or serve it from an endpoint and then consume it somewhere else.

Alternatives I have considered
We've implemented this by hand and it works, but it feels a little brittle:

import { getSchema } from '@tiptap/core'

export const getSchemaJSON = (extensions) => {
  const { spec } = getSchema(extensions)
  const marks: any[] = []
  const nodes: any[] = []

  spec.marks?.forEach((key: string, value: any) => {
    marks.push({
      name: key,
      ...value,
    })
  })

  spec.nodes?.forEach((key: string, value: any) => {
    nodes.push({
      name: key,
      ...value,
    })
  })
  return {
    topNode: spec.topNode,
    marks,
    nodes,
  }
}

import { Node, Mark } from '@tiptap/core'

export const extensionsFromJSON = (jsonSchema) => {
  return [
      ...jsonSchema.marks.map((mark) => Mark.create(mark)),
      ...jsonSchema.nodes.map((node) => {
        const { attrs = {}, ...rest } = node
        return Node.create({ ...rest, addAttributes: () => attrs })
      }),
    ]
}

Would it be possible to provide a cjs version of the packages?

I am not sure if it is bug or a feature... but I think document name should be URL decoded just in case. If user use some unsafe characters in document name, they are auto encoded in URL and they are used as the document name on server.

Some of those characters I tried are: space { } < > | ^. Surprisingly % was ok. You could argue these are very weird chars to have in document name though but I had them and that's how I found out.

Hi,

Would it be possible to get the documentName from context and default to parsing the request URL if none is provided? This would help when integrating with express for example

const hookPayload = {
    documentName: context?.documentName || Hocuspocus.getDocumentName(request),
    requestHeaders: request.headers,
    requestParameters: Hocuspocus.getParameters(request),
    socketId,
}

Description

When the onConnect is not synchronous, awareness state leaks to other rooms.

Steps to reproduce the bug

See https://github.com/jperl/yjs-repro and run npm run ok and npm run bug.

Expected behavior

I would expect there to only be 1 user state on this document ever.

Hi guys! Thanks for your work!

I have set up a hocusfocus server in my application and it works well. But I have one problem. Socket connection never reconnects.

I opened your sample site and left it open for 2 days, after I came back I can still write something. That's cool. On the network tab, I can see how some socket connections were closed and one is always open.

In my application, I cannot do this. My socket connection always closes after some time of inactivity and never creates a new connection. How can I fix this?

My server code

const server = Server.configure({
  async onCreateDocument (data) {
    const fieldName = 'default';

    if (data.document.isEmpty(fieldName)) {
      return;
    }

    const project = await ProjectService.getById({
      id: data.context.projectId,
      userId: data.context.userId
    });

    let prosemirrorJSON = project.document;

    if (!prosemirrorJSON) {
      const { document } = (new JSDOM(project.body.trim())).window;

      const schema = getSchema(extensions);

      prosemirrorJSON = DOMParser
        .fromSchema(schema)
        .parse(document, { preserveWhitespace: true })
        .toJSON();
    }

    return TiptapTransformer.toYdoc(prosemirrorJSON, extensions, fieldName);
  },

  async onConnect (data) {
    const { requestParameters } = data;

    const token = requestParameters.get('token');

    const user = await UserService.getUserByToken(token);
    return {
      userId: user.id,
      token: token
    };
  },

  async onChange (data) {
    if (data.context.ws) {
      try {
        await CollaborationService.checkPermission({
          token: data.context.token,
          ws: data.context.ws
        });
      } catch (error) {
        data.context.ws.close(4000, error.Error);
        return;
      }
    }

    const save = async () => {
      const prosemirrorJSON = TiptapTransformer.fromYdoc(data.document, 'default');

      try {
        await ProjectService.update({
          id: data.context.projectId,
          userId: data.context.userId,
          document: prosemirrorJSON
        });
      } catch (error) {
        data.context.ws.close(4000, error.Error);
      }
    };

    debounced?.clear();
    debounced = debounce(() => save(), 1000);
    debounced();
  },

  timeout: 8640000 // 24hour
});


router.get('/:projectId', async ctx => {
  const projectId = parseInt(ctx.params.projectId);

  try {
    if (ctx.ws) {
      const ws = await ctx.ws();

      const context = {
        projectId: projectId,
        ws
      };

      server.handleConnection(ws, ctx.request, context);
    }
  } catch (err) {
    ctx.bad(400, err);
  }
});

Thanks for your help in advance!