Permissions are currently enforced a bit brutally on the rest endpoints for streaming objects; i.e. public stream with link sharing on can't be viewed in the 3d viewer, even if logged in.

I suspect that this might be the case as well in grasshopper/dynamo too 😬

Fix: check permissions on the stream same as the gql endpoints do.

Expected vs. Actual Behavior

We should be able to tell users who shared a given stream with them. This is useful for both auditing purposes, as well as simply displaying "Stream X was shared with you by Y" in the frontend.

Proposed Solution (if any)

Add a sharedBy field in the ACL list for streams, field to be populated with the userId of grantor. This will allow us to work some magic in the queries to be able to say who granted what permissions to whom!

This is a bug in the v-autocomplete component: vuetifyjs/vuetify#11365

Current workaround: I've added a filter function to remove users that already are contributors from the list but this doesn't work if there is no text in the search box, so as you click on it it still shows the previous search cached results.

Once the autocomplete bug is resolved, the filter should be removed and the no-filter prop should be added.

Small enhancement suggestion: the viewer should inform the user when there are no visible elements in a commit, or maybe the viewer could be disabled alltogether.

Currently, it is possible to accidentally delete the main branch. I was able to do this on the front end when manipulating multiple branches. For the front end specifically, I think this was due to the fact that it doesn't always update on stream changes without your refreshing the page.

Deleting the main branch should not be possible as it will make your stream inaccessible! A check for this should be introduced in the server to block this from happening.

Expected vs. Actual Behavior

All base object should have a property for their speckle type. This property should of course have a consistent name. At the moment, root base objects have a speckleType property while all child objects have a speckle_type property.

Reproduction Steps & System Config (win, osx, web, etc.)

Upload a base object with nested children within it.

Proposed Solution (if any)

Pick one of the names (prob speckle_type) and enforce it across all base objects.

Expected vs. Actual Behavior

Running a completely fresh server stack via docker-compose fails for the first run. The server in most cases boots up quicker than the database container finishes with first time init. This crashes the server.

The whole stack should just run on the first try.

Reproduction Steps & System Config (win, osx, web, etc.)

Currently, check out #76 and run docker-compose up

Proposed Solution (if any)

There is a wait solution for this.

This is a bigger issue, spanning across more than one repo ultimately.

Expected Behaviour Proposal

The goal is to make an end-user's account creation/login/registration flow as easily as possible. In 1.0, it's a bit of a mess as you can add accounts from all over the place, and it's rather confusing.

In 2.0, we'd want to have the experience streamlined. Here's a preliminary proposal:

1. User navigates to a server. He is then prompted to log in or register via the available strategies on that server.
2. On successful authentication, the user is redirected to the main web application (the frontend).

Upon reaching this stage we need to determine if it's the user's first session on that server.

3. If it's the user's first session, display a nice dismissible banner/ui element asking to

• (a) download and install the speckle manager
• (b) or, if they have done that already, "click here" to register their desktop connectors.

4. If it's not the first time, carry on as usual :)

The (3.b) behaviour entails opening up a custom application scheme, the way slack and others do it (ie, speckle://manager?args, and launching into the oauth2 flow from the manager itself.

The Gist

• each individual server's frontend becomes the one stop place to add an account to that server.
• you cannot add accounts from clients, etc.

This helps us keep a secure auth flow for clients (prevents impersonation by other apps), and hopefully simplifies confusion on the user's side, as much as we can in the distributed many speckle servers scenario.

TODOs

To make this work, several things need to happen:

• update the manager to register a custom app scheme on windows, so it can be launched from the browser
• update the manager to be able to dance the oauth flow & store an account on the user's computer
• dynamic client registration for connectors on the server the first time a user authenticates
• frontend needs to check wether it's the user's first time here (server models need to be updated, or use a cookie?)

Optional: Affected Projects

Server, Frontend, Core and the Speckle Manager.

Here I am getting following error when I start the server, though my postgres and redis is running. Can you help ?

C:\ONX\Speckle\speckle-server-master\packages\server>npm run dev

@speckle/[email protected] dev C:\ONX\Speckle\speckle-server-master\packages\server
cross-env NODE_ENV=development DEBUG=speckle:* nodemon ./bin/www --watch . --watch ./bin/www -e js,graphql,env

[nodemon] 2.0.6
[nodemon] to restart at any time, enter rs
[nodemon] watching path(s): . bin\www
[nodemon] watching extensions: js,graphql,env
[nodemon] starting node ./bin/www
speckle:db-startup Loaded knex conf for development +0ms
speckle:www KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
speckle:www at Client_PG.acquireConnection (C:\ONX\Speckle\speckle-server-master\packages\server\node_modules\knex\lib\client.js:348:26)
speckle:www at async listCompleted (C:\ONX\Speckle\speckle-server-master\packages\server\node_modules\knex\lib\migrate\migration-list-resolver.js:12:3)
speckle:www at async Promise.all (index 1)
speckle:www at async Migrator.latest (C:\ONX\Speckle\speckle-server-master\packages\server\node_modules\knex\lib\migrate\Migrator.js:64:29)
speckle:www at async exports.init (C:\ONX\Speckle\speckle-server-master\packages\server\app.js:38:3) +0ms

Commits should not be editble, we should then remove or disable the CommitUpdate mutation and subscription.

Behaviour

Frontend auth lock behaviour borkage on localhost bound redirects.

• if the app is a web app, window.location redirect is the expected behaviour
• if the app is desktop bound (ie, connectors), the redirecting the browser to window.location is confusing (shows a blank page/404)

Potential solution

Differentiate apps based on their nature. This might need modifying the current schema & needs propagation throughout the apps api.

Expected vs. Actual Behavior

Commits should track which application they came from.

Proposed Solution (if any)

Add a "sourceApplication" to the commit table.

Optional: Affected Projects

All clients should, when possible, populate this field.

Implement rate limiting on endpoints. TBD its implementation.

Test stream: https://staging.speckle.dev/streams/76ad4bd698

Sent curves:

There's a wee bit of a dependency mess that causes problems at the moment:

• Node: we're stuck at v12.X because of old knex version
• Old knex version is kept because:
• Remote db connection needs to be figured out so as to allow upgrading pg to latest (^7 > ^8). There's some surmountable incompatibilities with our current DO's staging Postgresql db.

To start a discussion on this, I've added my insigts to the question:

To make deployment and development significantly simpler, there should be a single command that with only providing environment variables starts up all the components and configures a fully functional SpeckleServer.

Additionally automatic HTTPS & TLS with auto renewing certs makes things production ready.

Problem definition

This document details a plan to add a fully automated OCI container based runtime to the 2.0 Speckle server. Automatically running all the components together or only parts (for development) is the task. The server consist of 4 distinct parts, frontend, backend, persistent storage and cache.

Frontend

The frontend is a static vue application. This means, that the artifacts of a production build are a set of html js and css files, that can directly be served to web clients without any server side processing.

Currently the backend nodejs server is responsible for serving these files (but it is not strictly required, the frontend may be served from anywhere as long as it is able to talk to the backend).

With the current setup of the backend server serving the static files of the fronted, it is not required to create a separate container for the frontend. Only the build artifacts are needed to be copied into the backend server container.

Backend

The backend is a nodejs express application. This application provides an auth application, a rest api for the connectors to talk to the server, a graphql api for frontend applications to querry data, and serves the static frontend application.

What else is needed here?

Persistent storage

The data storage layer for the server is a Postgresql database. Currently there is a need for one SpeckleDev database and a user that has table creation and alteration rights in the DB (it should not be the root user in the db).

Currently the tables and relationships are created by knex database migrations, which should stay that way.

The missing piece on the DB side is the automated creation of an application database and db user

with proper permissions.

For development purposes a database admin tool is also helpful to get a full dev env up and running.

Cache

Currently cache is used for auth session data storage.

Only session secret is used to authenticate? Should there be more steps to make Redis production ready and safe?

Reverse proxy

From the services above only 2 should be publicly accessible, the backend api and the frontend app. These public resources should also have automatic tls cert handling (its 2021).

This should be handled by a reverse proxy.

Proposed solution

Overview

Today the industry is heading into a cloud native future, where everything is ran either serverless, managed or in an OCI container.

For development pruposes and some (if not all) production deployment cases a fully container based solution should be preffered.

Running Postgres on windows is a pain, WSL2 is widely available. With a good guide, I think anyone can get a local environment with docker up and running.

Possible exceptions from this is a managed database instance and a statically server frontend from some an edge/cdn hosting provider.

There is a solution that ties all requirements together with a stack:

• Node/express app in docker container serving frontend and backend
• Postgres running in a docker container with an attached volume on the host for safe backup.
• Redis in container is a no brainer
• Traefik for reverse proxy, works well with docker compose and has 0 config tls certs.
• Docker compose for tying all services together
• Configuration of environment with .env file

Many of the pieces are present in the current state of server 2.0, these are the missing tasks:

Nodejs app

1. The container build definition (Dockerfile) is currently not developet for the 2.0 server.
2. The express app must not be ran by the container root user, to avoid root user exploits. This can be done by adding a user with limited permissions in the Dockerfile and switching to that user after build steps are defined (Pretty straight forward).
   For nodejs it seams to be the consensus when using the official base image is to use the node user provided in the base image to run the application.
3. Use a multi stage build to ensure small final container size.
4. Production build configuration for lerna managed apps?

Postgres

For most usecases connecting to a hosted database is the preffered deployment. So connecting to a Postgres instance by using environment variables to configure the db connection string is the baseline.
To make things a bit more flexible, the connection string should be built up from a few separate env variables, like BD_URL, DB_USER, DB_PASSWORD. With these variables separate connecting to a hosted instance or bootstrapping a complete local environment is only a question of configuration.

Creating a db container is fairly straight forward, there are some missing automation pieces. These have to be planned out a bit:

1. Root user configuration with env variables
2. Automatic database creation script
3. Automatic application user creation based on environment variables

Maybe something like this. Issue here, is that the sql files have to be present on the host OS to be mountable. This means cloning the repos to some degree. I'd preffer some sort of env variable based solution. This could even be a custom speckle container image of Postgres hosted on docker hub...

Postgres admin

To enable easy database administration the official Postgres admin should also be part of initial environment setup.
It also has an official container image.
This is a question of habit tough, many people would use some sort of db management application, but giving a good default solution out of the box is a good experience.

Redis

Quite good prod ready redis docs here. Note the separate docker network between the cache and the components that need to talk to the cache.

Docker compose

Why use docker compose?

• Works well with .env files
• For fairly simple apps like Speckle Server it is robust enough to provide a lot of functionality and not too complicated.
• Works well for production deployment and dev environment too.

Traefik

Basically adding traefik as reverse proxy is done in docker compose. Good docs.

The App Authorization page should display the currently logged user (if any) and provide a way to log out / switch accounts.
Right now it just prompts the user to approve or deny access:

Expected vs. Actual Behaviour

Default apps that come pre-registered on the server (the main frontend, the explorer, and the desktop manager) have no registered author they can be attributed to, so we can't really display their full credentials :/

Proposed Solution (if any)

Pre-register a default "Speckle" user on each server, with randomly generated id and password. The default apps can then be attributed to it thereafter.

Note: the "first registered user becomes admin" rule needs to change to "the second registered user becomes admin".

New users who sign up in the web app see only empty pages, it would be a great way to get them started if all new accounts had a couple of sample streams.
We have considered implementing this feature by:

• add a couple of sample streams to the server
• each time a user registers, clone the sample streams and share them

space-in-parens & etc.

Expected vs. Actual Behavior

Allow easy retrieval of commit total object count.

Proposed Solution (if any)

Implement in services:

• getCommitById
• getCommitsByStreamId
• getCommitsByBranchName

As well as:

• Implement in gql schema

Alternative solution: rather than implementing in all the services above, add a new field to the table and populate that on commit creation (as the referenced object should already exist then, since we just fixed the race condition we had).

Expected vs. Actual Behavior

Voila, in a nutshell, the actual behaviour (curtsey @teocomi ):

Apps that come pre-registered with the server (the web app, the manager and the explorer) have no author. That's because, when the migrations run the first time ever, there is no user to assign them to.

Proposed Solution (if any)

We want these apps to be registered to "Speckle", and not a random person. We're responsible for them ultimately! What we need to do:

• in the core migrations add a new first user ("Speckle" or "Aec Systems"?) with a random password and random email.
• in the apps migrations, set the authorId of the pre-registered apps to the above user's id.

This will add one extra user to the db before anyone else has already registered, hence the "first user becomes admin" check needs to become "the second user becomes admin". This is checked for in the user services in the core module.

Tests will need to be updated as well, at least on the users (i don't think we're checking for app authors in the app-related tests).

The viewer keeps loading "stale" data even if you've moved away from the page displaying it.

A stream owner should not be allowed to revoke their own stream permissions in case they are the only owners of the stream, otherwise we could end up with orphaned streams.

Expected vs. Actual Behavior

Docker build fails because of missing viewer packages.
The current command used to install packages to build the frontend, npm ci will stop if the package-lock.json is not up to date.

Reproduction Steps & System Config (win, osx, web, etc.)

Just run docker build .

Proposed Solution (if any)

Until viewer is published as an npm package, it has to be build as part of the build process.

Docker build should be part of the CI process to check for basic errors.

Minimal onboarding flow for new users. What's been discussed is that we'll have a little small wizard that shows up on first login, which will guide users through three main steps (so far):

1. Downloading the manager
2. Adding the account to the manager
3. Link them to the docs

Thanks in advance for any and all help!

Expected vs. Actual Behavior

In speckle-server/packages/frontend running:
npm i

throws error:

...
npm ERR! 404 Not Found - GET https://registry.npmjs.org/@speckle%2fviewer - Not found
npm ERR! 404 
npm ERR! 404  '@speckle/viewer@^2.0.0' is not in the npm registry.
...

Reproduction Steps & System Config (win, osx, web, etc.)

Pull repo, cd to speckle-server/packages/frontend and run npm i.

Proposed Solution (if any)

So I got this working by the following steps:

1. remove @speckle/viewer from package.json
2. cd speckle-server/frontend
3. npm i
4. sudo npm link ../viewer

Don't know what the timeline for publishing the viewer on npm will be but this seems like a workable short term fix that could be documented?

Currently this is shown:

Expected vs. Actual Behavior

Currently, the way to find your role on a given stream is to find yourself in the list of contributors and get the role from there. It would be simpler if your role was easily retrievable from the stream object itself.

Proposed Solution (if any)

Add a role property to the stream query so the user's role can be easily accessed from the stream object.

Optional: Affected Projects

This will need to be propagated to the Models and ClientOperations in Core.

Expected vs. Actual Behaviour

Hitting the enter key on dialogs (eg create stream or create branch) will close the dialog and refresh the page without submitting the inputs. This is pretty counterintuitive and the behaviour should be modified to submit the form on enter

Basically this, currently blocked by #43

Schema directives should have additional checks on the input names for branches and streams, if directives can't be used, validation will have to happen in the resolvers.
Below my suggested rules.

Branches
Currently implemented in the frontend:

• min length 3 (currently a branch with no name is allowed on the server!)
• max length 25

Optional rules to make branches URL and CLI friendly:

• only lowercase, numbers and dashes
• can't start with a dash -

Streams
Currently implemented in the frontend:

• min length 3
• max length 25

Streams visibility is not affected when setting the isPublic property on a StreamUpdate mutation.
This is because the mutation currently only works on name and description.
There is a updateStreamPrivacy service on the server but it's not exposed via the API.

The resolution is to either expose it and remove isPublic from StreamUpdateInput or have the StreamUpdate mutation also change visibility.

Tests currently rely on a mixture of chai-http and superagent/supertest. We should move all of them to use supertest for consistency; and it's also much nicer to use.

The stream updatedAt property is not being set correctly.
It should update if:

• name or description change
• branches change
• new commits are pushed
• sharing changes
• ...

Public streams cannot be accessed via the frontend as the auth screen is shown instead.
Ultimately, this could be a feature or better a setting:

• if enabled public streams would be visible only to logged in users
• if disabled they would be visible to anyone

Currently, if you are already logged into the server in Chrome, the consent page will not display any information of which user is active.

We should display at least the avatar + username/email to prevent users from consenting with the wrong account (has happened to me twice already 😅 )

There should also be some option to log-out or "switch user" to allow for account changing without having to kill the Manager's flow.

They don't look nice in dark mode.

Let's ensure quality! There's quite a few places we need to plug this in.

• update user profile
• update server info (if role permits)
• add subscriptions
• manage stream permissions
  • change link sharing
  • add/remove collaborators
• create /revoke tokens
• display more detailed commit info
• delete streams & branches
• pagination on streams & commits
• tooltips
• (optional) create commits
• (optional) proper home / feed page
• (REQUIRED) Easter egg of sorts

For the essential needs of (1) sharing streams with other users and (2) finding your own streams!

• user search
• stream search

Dependency for finishing off specklesystems/speckle-sharp#5

Expected vs. Actual Behavior

The branch updated sub should be the only subscription that fires upon a branch update so that a user can handle it on its own. However, the branch deleted sub also fires after the branch updated sub. This is confusing and means a user can't handle the branch updated case effectively. All 3 (create/update/delete) cases have to be handled by fetching an updated branch list from the server because handling the updated / deleted cases separately would lead to unintended deletion of updated branches.

Optional: Affected Projects

None, but DesktopUI can change the way it is handling branch subscriptions if this is fixed.

Currently the frontend doesn't do anything if the email or password are wrong when signing-in.

Expected vs. Actual Behaviour

It would be fab to be able to send emails to people inviting them to easily create an account to the server. This would allow speckle user A to share a stream with non-speckle user B by providing B's email. From B's perspective, they should be directed to a signup flow and automatically added as a collaborator to the stream A wanted them to see.

Optional: Affected Projects

This would allow the DesktopUi and other connectors to share streams with people who don't have a speckle account yet.

Implement relevant gql api subscriptions, specifically for:

• streams & all subsequent operations
• users & subequent operations (new stream added, etc.)
• more that i can't think of right now

Several minor UI tweaks and improvements:

• Limit number of collaborators in Stream card to 8 (or 7 and a +XX button.s

• Prevent stream name from overflowing when using long non spaced strings.

• Add loading state to 'Stream has no data' and description cards while stream is loading

Expected vs. Actual Behavior

When querying a stream, you can grab a branch by name and a commit by commit id. It would be neat if we could default to grabbing the main branch and the latest commit on main if no argument is given to the branch or commit fields.

Proposed Solution (if any)

Make the arguments for the stream fields branch and commit optional and add logic to default to the desired behaviour if no argument is given.

Implement a section box as seen in this demo:
Demo

Interaction should follow this pattern (repo for demo above):
Clipping With Caps

Another useful demo

Steps to implement (approximate):

• Implement SectionBox class that displays geometry, add to Viewer scene
• Implement face picking & hover state using SelectionHelper.js ideally
• Implement dragging interaction
• Align clip planes with box faces
• Apply clip planes to materials
• Limit planes so they can't cross each other while draggin
• Set from bounding box
• Translate, Rotate (not going to get into ui for this yet)
• Stencils

I'm actively working on this :) please let me know if you have any thoughts or advice as to how this should work / be implemented.

Some inputs types are not uniform across the various calls eg:

For streams we have (Stream) CreateInput & UpdateInput, but for branches and commits also a DeleteInput. (Stream) GrantPermissions and RevokePermission do not have input types either.

What should be the general approach?