Giter VIP home page Giter VIP logo

clowdr-app / clowdr Goto Github PK

View Code? Open in Web Editor NEW
60.0 6.0 20.0 76.88 MB

Midspace (formerly Clowdr) is a completely open-source virtual conference platform. Host complex events with hundreds of authors and presenters. Midspace includes automated livestreams, video chat, text chat, randomised networking and much more.

Home Page: https://www.midspace.app/

License: GNU Affero General Public License v3.0

JavaScript 0.07% TypeScript 98.15% Shell 0.04% HTML 0.01% CSS 0.01% PLpgSQL 1.71% Erlang 0.01% Batchfile 0.01% Procfile 0.01%
clowdr virtual conference video video-conference virtual-conference

clowdr's Introduction

Midspace virtual conference platform


Note: the official Midspace instance at midspace.app is no longer available. You can still host your own Midspace instance using the open-source code. Thanks for using Midspace!


This is version 3 of Midspace - more functional, more robust, more scalable, and 100% open source!

If you want to contribute to Midspace, please read our contribution guidelines.

Structure

Folder Contents ReadMe
aws Infrastructure as code for AWS AWS Readme
frontend The frontend React web app Frontend Readme
hasura The Hasura GraphQL configuration, actions and seed data. Hasura Readme
services Micro-services
services/actions A service that handles most Hasura actions. Actions service readme
services/auth Auth service readme
services/caches Caches service readme
services/realtime A service that handles realtime interactions like chat and presence. Realtime service readme
services/playout A service that controls video broadcast pipelines. Playout service readme

Quick Setup vs. Full Setup vs. Production

For contributors that only want to play with the user interface, the "Quick" version of the following instructions should get you a minimal working setup. Just skip over the steps marked Full Setup or Production and follow the steps marked Quick Setup.

Warning: The Quick Setup instructions are still being debugged and may or may not work yet!!

Caveat: The word "Quick" should be taken with a grain of salt. Even this streamlined path is likely to take you a day or two.

To test changes that affect other parts of the platform, skip the steps marked Quick Setup and follow steps marked Full Setup. To deploy Midspace publicly or run your own conference on Midspace, follow the steps marked Production as well.

Local Software Installation

  1. VSCode
    • We also recommend you install the "recommended extensions" listed in the .vscode/extensions folder. VSCode may offer to install them automatically.
    • If using Windows:
      1. Note that you may encounter issues setting up a development environment that are specific to Windows, in which case README pull requests are welcome!
      2. Install Git command line if you haven't already.
      3. Open or restart VSCode, and open a terminal with menu Terminal -> New Terminal
      4. Next to the big + sign in the right side of the terminal header, there's a dropdown with tooltip "Launch Profile...". Click it and select Git Bash.
  2. Node.js 17
  3. pnpm 16.30 or later
  4. Docker Desktop - Midspace uses Docker Compose, now included in the Docker CLI.
  5. Hasura CLI
  6. Full Setup: AWS CLI and awsvault
  7. Production: Heroku CLI if you will be deploying Midspace publicly.

Generate Hasura Admin Secret

The Hasura GraphQL engine will be configured with an arbitrary admin secret value, and this value is needed for configuring several other services that make up Midspace, so generate a secure secret value now and make note of it. A suggested method is a hex string representing a 128-bit value, which can be generated using one of the methods below depending on which tools are available on your system.

  • openssl rand -hex 16
  • node -e "console.log(require('crypto').randomBytes(16).toString('hex'))"

Several other setup steps require an arbitrarily selected secret value shared between services, and this method may be used for generating those values as well.

Install Packages

Install all node packages (pnpm descends into subdirectories and creates all necessary node_modules directories):

pnpm i

Cloud Services

Midspace relies on various cloud services, which will need to be configured for local development as well.

  1. Full Setup but not Production: Set up tunnels for enabling callbacks to services running locally
  2. Set up user authentication
  3. Full Setup: Set up AWS account and Deploy AWS CloudFormation Stacks
  4. Set up video chat service
  5. Full Setup: Create a SendGrid account and an API key for it.
    1. Take note of the API key when creating the account; you will not be able to view it later without updating it to a new value.
    2. Also set up Signed Event Webhook Requests and generate a verification key pair. You will need the public key, but you can come back to the Sendgrid mail settings page → Signed Event Webhook Requests modal to view it later.

Setting Up Local Working Copy

  1. Follow the Hasura setup: Midspace Hasura ReadMe
  2. Follow the Actions Service setup: Midspace Actions Service ReadMe
  3. Follow the Auth Service setup: Midspace Auth Service ReadMe
  4. Follow the Caches Service setup: Midspace Caches Service ReadMe
  5. Follow the Playout Service setup: Midspace Playout Service ReadMe
  6. Follow the Realtime Service setup: Midspace Realtime Service ReadMe
  7. Follow the Frontend setup: Midspace Frontend ReadMe
  8. Generate all GraphQL client code
    1. cd packages/shared/graphql
    2. cp .env.example .env
    3. Set environment variable values in .env as per other parts of Midspace, likely just HASURA_ADMIN_SECRET.
    4. Run the VSCode task All -- GraphQL Codegen. If there are errors, check that .env files have been populated in all relevant subdirectories as described in previous steps.
    5. Fixup currently required after running GraphQL Codegen: Some changes are made to the file frontend/src/generate/graphql.tsx that need to be undone:
      1. Restore this line at the beginning of the file: import { useQuery } from "@midspace/urql-hasura-cache-generic-resolver/useQuery";
      2. Revert all instances of Urql.useQuery< back to useQuery<.

Starting your local working copy

Once you have finished setup, it's easy to run the entire environment with a single VSCode task: "Run All -- Local Development". This task starts PacketRiot tunnels as well as all Midspace services.

Final Steps

  1. Once the system is up and running, open the app in your browser, log in, then navigate to https://<your-domain>/su and follow the instructions to set up a superuser.
    • In a production environment, we recommend using separate infrequently-accessed accounts for superusers, to reduce the risk and impact of security breaches.
    • In a production environment, we recommend using separate accounts for the various privileges available to superusers. For example, keep accounts with the ability to create conference codes separate from those able to modify the set of superusers.
  2. If running this software in a production environment, you will need to use the superuser configuration pages to initialise the System Configuration.
    • Fill out values for all available keys.
    • Refer to the description field of each key (in system.ConfigurationKey) for expected values.

If you alter environment config, Docker Compose config, etc., then all tasks must be restarted. Tasks can be killed in VSCode using Ctrl+C or by closing the terminal window they are running in. To kill Docker containers, you will need to manually terminate the container (e.g. by pressing the stop button in Docker Desktop)

Create a conference

When you log into Clowdr for the first time, there will be no conferences listed. You will need a demo code to create a conference, and this cannot yet be done through the Clowdr UI. To create a demo code:

  1. Go to the Data tab in the Hasura console.
  2. Open the conference > DemoCode table.
  3. Open the Insert Row tab. Ensure that id is set to Default and click Save. There is no need to enter any values manually.
  4. A code has now been created. Open the Browse Rows tab and find the new row in the table.
  5. Copy the id column of the new row. This is your demo code - you can use it to create a conference in the Midspace UI.

Modifying the default security settings

By default, only the creator of a conference has permission to view its elements. You can give permissions to other groups of users by opening the Content admin panel in Midspace. Click the yellow button with a lock icon to open the (conference-)global element security settings.

You probably want to at least add an entry for 'Organiser' permissions with the 'Organisers' group.

Formatting

This repository uses Prettier for auto-formatting and checks for both pushes and PRs. You may need to configure your editor to use Prettier (for example, by using the VSCode extension)

Notes

  • The various Procfiles are used by Heroku to determine what services to run.
  • In production deployments, you will want to regularly clear out old Hasura event logs or you will encounter performance issues. There is a stored procedure public.truncate_hasura_logs to clear out logs older than a week. You could, for example, deploy a Cron To Go task in Heroku that calls psql $DATABASE_URL -c "CALL public.truncate_hasura_logs()".

GitHub Actions Configuration (/Secrets)

If you want to configure the GitHub actions for CI, you will need to set the following secrets:

Secret Value
HASURA_ADMIN_SECRET The value of Hasura Cloud
HASURA_ENDPOINT The GraphQL API URL from Hasura Cloud but without the /v1/graphql path.
ACTION_BASE_URL As-per the Hasura Cloud environment variable.
REALTIME_BASE_URL As-per the Hasura Cloud environment variable.
HASURA_PERSONAL_ACCESS_TOKEN The value from Hasura Cloud
HASURA_PROJECT_ID The value from Hasura Cloud

Note:: The HASURA_ENDPOINT changes when if rename your project inside Hasura Cloud.

clowdr's People

Contributors

bcpierce00 avatar ednutting avatar hadley0828 avatar its-nate avatar jon-bell avatar nguerette avatar rossng avatar shawnsp avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

clowdr's Issues

Video archiving (to YouTube)

Implement the mechanism for archiving videos to a provided YouTube channel including:

  • Stitching together the lightning talk and Q&A
  • Auto-uploading the video
  • Setting the description to the abstract
    • including links to the papers
  • Auto-creating playlists based on:
    • the session period
    • the content's tag
  • Auto-emailing authors to notify them of where their videos can be found
  • Ensuring existing subtitles are transferred across (after we've cleaned up ones that went wrong)

[Feature Request] Chat width resize?

On maximized windows, the chat takes considerable space that could be used for the videocall instead. Maybe if we could close the chat just like the sidebar or resize the chat it would be nice.

Login to Clowdr space broken

After entering login credentials (using the link with the invite code embedded, or just by using the link that does not contain the code), only the top of the main page is shown with the buttons (incl "LogOut") near the top, but the rest of the page just shows the spinning wheel. Clearing history, quitting browser etc doesn't solve the issue - the only way to participate in the conference is to by pass Clowdr and go to the zoom room of a session (if you happen to know the zoom room...).This issue has been preventing me now to log back into Clowdr for 2h. I've given a few more details in a mail to Ed, Jon, Crista.

[Feature Request] Permanent Sidebar

On a big screen, it would be nice to have the sidebar there permanently rather than needing to click the icon in the top-left corner every time.

Shuffle Spaces: Allocation algorithm

The the moment shuffles queue are processed first-come first-served. It is fairly simple to substitute the algorithm used to do the allocation in the backend, provided said algorithm is a pure function over the existing queue state.

Various suggestions have been made, all including some element of randomisation.

The key problem to solve is what happens when a room ends and a group of 3 to 6 people all go straight back into the queue. They would rather not end up all in another room together (as that defeats the purpose of the system). Some kind of historic knowledge of room allocations may need to be included in the allocation algorithm to solve this.

Admin interface: rooms changes and saved data

The app is saving at every character typed! Make it save more sparsely.
I think saving should not be automatic, and, instead, it should be a explicit action by the user. This is because we can make mistakes, and there is no "undo." So it's better if saves are explicit rather than automatic. (like the content and people panels)

Sponsors access to only their booths

There is a suggestion from FPGA that it may be useful to have the ability to give certain attendees access to run their sponsor booth but not access to the rest of the conference.

Livestreamed subtitles loopback on old feed

When two feeds are played one after the other in the same livestream and the second does not have embedded captions/subtitles, AWS seems to loop back on the previous video's subtitles.

We also need to go back and fix this for the existing Q&A recordings from POPL, which have ended up with incorrect subtitles embedded.

Broad Design Concerns

I'm making this an issue so it's a bit easier for you to triage, but feel free to close it and move this somewhere else if needed.

I have some broad, design-based feedback that I hope you'll keep in mind as Clowdr evolves. I imagine you've already thought about some of these points, but hopefully some of them are novel.

Modelling Concepts for the User

At one point I heard an attendee say “there’s more on the screen now [versus at ICFP]”. This is patently false, but that was the perception. I propose that the issue here is concept modelling.

Having fewer concepts isn’t necessarily better.

Unifying DMs, rooms, breakouts, etc into a single "concept" seems nice on the surface (there's only one thing to learn), but I suspect users don't see it that way. Users have already learned what a DM is---it’s a little chat box somewhere that sends a message to a particular person. By unifying DMs with other concepts, we actually put more burden on the user.

Unified concepts in the back-end need not be unified in the front-end.

This is related to the previous point, but worth stating on its own. It’s totally fine that every chat room has an associated video room, but the “announcements” channel (for example) should not have that video room exposed. (I’ve seen confused people sitting around in the announcements room.)

More Focus on High-Level Views

Users have high-level questions that they want to answer. Where is everyone? What are all of the events happening today? etc. The current interface has some of these views (if you know where to look), but they’re not up-front and obvious.

This could actually be fixed fairly easily with a few UI tweaks. I think the schedule should automatically start more “zoomed out” so the first thing the user sees is a birds-eye view. The rooms tab should sort social rooms to the top and set them apart, making it clear where to find people.

Personally I’d even like to see these things together somewhere. Dashboards make me feel in control of a piece of software.

Minor Points

  • Viewer counts - I don't think they actually add anything (especially when they're negative)
  • Smaller border radii - Seems like a silly point, but I think shrinking the border radii a bit will make everything feel more professional
  • Fix elements relative to page size - Getting a new message modifies everything on the page; chat windows should probably be fixed-size, rather than growing with the content

[Bug] Duplicated video

When you toggle light mode/dark mode while you're in a room, your video and audio gets duplicated and people see multiples copies of your user

Chat: Re-implement backend to meet real-time requirements

Chat is currently built upon Hasura Subscriptions, which although they run over a websocket, Hasura is ultimately just polling the Postgres db at a defined rate (1s -- though for much of POPL we had to throttle this to 1.5s). This isn't real time and murders our Postgres load.

We should re-implement chat on a socket.io / Redis layer on top of Heroku with a write-back buffer that batches writes into Postgres.

We should aim initially to:

  • support chats with ~2,000 members, with 10% sending a message every 10s to be received by all others. This will be a sufficient target for our upcoming conferences.
  • have happens-before consistency, though eventual-consistency may be sufficient
  • have the ability to scale the backend servers horizontally using Heroku's automated scaling

Chat archiving

Implement archiving of chats:

  • Public chats (papers/keynotes/co-events items but not public breakout rooms) need digests sent via email as file attachments to the relevant people
  • Private chats (DMs, private groups) need digests sent to their respective participants

Bad error message when not logged in

At POPL 2021, when I try to navigate directly to the "Rooms" page when not logged in, the error message about trouble loading data is quite opaque. I wish it would say "log in, dude"!

Admin interface: Schedule: don't auto-save

Auto-save here us a nuisance! Consider this use case:

  • Create a new event. This creates an entry with times that default to the current time. The entry ends up somewhere on the table.
  • Change the day/time. Poof! the entry suddenly disappears out of focus and goes somewhere else on the table. We need to visually search for it.

Here's what I suggest:

  • Creating a new event places a new entry at the very top of the table
  • Let me edit it to my heart's content, and then let me press a Save button
  • (Ideally, but not as important as the last two things) If the entry changed places on the table, show me where it went

Trackers?

Why send all the information to doubleclick

Immediate go-to-Q&A options

  1. A button to immediately end pre-recorded playback and go into Q&A mode
  2. Pre-recorded videos that last longer than their allocated time slot should go straight to Q&A after finishing (if one is scheduled afterwards) rather than padding with filler video

Miscellaneous small improvements to shuffle spaces

  • When you get moved from the waiting queue into the Join Room page, you should get an audible signal (in case you have switched to looking at email while waiting).
  • Instead of "Rooms limited to..." it should say "Rooms will reshuffle after X minutes"
  • When it goes into "Sorry, it may not be possible to match you with anyone at the moment" mode, it should either tell me that it is still trying to match me or else give me a button to leave.
  • "Please wait" should be renamed to "Waiting for more people to join"

Basic analytics

We don't currently gather any analytics data. We should collect/provide at least the following:

  • Video view counts
  • Video watch time graphs
  • Paper-link click counts
  • Viewer count graphs for livestreams

Rooms can have negative people

Currently, the POPL room "Hallway 1" has -4 people in it. This has bounced between -4 and 0 for the last hour or so as people have come and gone.

I'm not sure what's causing it, but it's probably something to look into.

Own messages in chat appear only after browser refresh

If I send a message in a chat of a room, I only see the message after clicking refresh. I see messages of others immediately.

I'm on Mozilla Firefox 84.0 and can send more information on the browser / plugins / OS if you tell me what you need to know

Thanks a lot for the great work!

Room garbage collection

Lots of rooms hang around after their events. Make a flag? Have a GC button that lets admins destroy rooms that (a) have no future events scheduled, (b) aren't DMs or whatever, and (c) haven't been used in X minutes.

Sign up process fails in some cases

It's unclear what the exact sequence of events is that leads to this, but some people are still encountering problems signing up when they receive the initial invitation. The Clowdr app gets into a state that doesn't let them signup and/or accept the invitation. In some cases, they need to enter the registration code by hand, and it's not clear what that registration code is.

Hint: this may be related to the use of Safari first. Not sure.

Two action items here:

  • Include the invitation code in the email, additionally to the invitation link, so that at least it's easier for people to copy it

  • These issues need to be investigated, reproduced, and fixed.

Webcam on when it shouldn't be?

At POPL 2021, I jumped from a hallway room to a lightning talk, and my webcam is telling me it's still on, with its usual light. Is Clowdr keeping video/audio recording on for longer than needed, representing a privacy risk in case of bugs?

Metrics board in admin interface

Would be great to see number of users online, users in Q&A rooms, chats per minute, views per video (which need some custom tracking, I think!).

Add mechanism to request user to refresh page

For various reasons, we might need to get a user to click “refresh” in clowdr. We need a system to 1: set these bits for each session of each user, and 2: cease all requests from that session until the page is refreshed.

Ed wrote in Slack:

We probably just need a backend var that has a timestamp for 'last required refresh time' and a frontend timestamp in local storage that stores the last time we asked the user to refresh. And the rest is obvious.

I would suggest alternatively using semantic versioning for frontend deployments. Minor deployments do not require a refresh (but suggest to the user that they might want to click refresh to get the latest and greatest, or cancel and keep doing what they were doing), major deployments put up a modal warning, requiring a refresh and also cut off all requests until refresh occurs. This provides an easy way to do hot-fixes during an event and get people to hit refresh overnight, and to distinguish between “if this user doesn’t refresh our system will suffer for it” and “we have a new version of the shuffle space interface and we think you’ll find it better, but it’s not obligatory that you refresh”.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.