element-hq / element-call Goto Github PK

on the allow guest access checkbox:

We should have a view where a presenter is made the focus of the call. By default this will use an algorithm to detect who is the active speaker. Otherwise users can be pinned as a speaker in the room. Users can also locally focus a user and have them as the presenter.

Most of the conference call logic currently resides in this project. We've proven this concept now and the the conference call logic should make its way back into the matrix-js-sdk.

Safari is not sending its webcam feed reliably. At first I thought this was due to reusing the user's MediaStream for each outbound call. This issue is outlined here. I made sure that these streams are now cloned, however it's still sometimes not sending the stream. What's even more odd is that when you change your webcam during this state, your local feed goes blank and the other participants can see you.

I noticed that all of my remote video feeds froze when I went to screenshare. I think maybe it's stopping them or not updating properly when renegotiating, audio kept working fine. I remember seeing in matrix-js-sdk something about stopping video feeds when screensharing, so maybe this isn't supported yet? If not I can add support for it.

You should be able to pick which audio output device to use if your browser supports it (Chrome)

You should be able to share your screen in a conference room.

I recently added support for group call participants, however they are not being rendered for participants in rooms when you first view the home page, making them relatively useless. There must be a bug with emitting the participants changed event or something along those lines of announcing participant changes.

It would be nice to be able to send a link to a user and have them join the conference all as a guest, without needing them to register.

We should add indicators to the participant's nametag that shows if a user has muted themselves. We can use the CallFeed state/events for this.

Right now we need to link in the matrix-js-sdk and matrix-react-sdk projects because they are not exporting their dependencies properly. It would be nice to be able to use a git branch on the staging server as well. I think in order to do that we should use a prepare script in the package's package.json like so. We'll also probably need typescript and webpack aliases set up for these projects to reference typescript source files in development.

You should be able to set/change your user avatar in the conference UI. The avatars should be shown when your webcam is disabled or if you are using audio only.

MSC3401 is set up so that you should be able to dial the conference from multiple devices as the same user, but this doesn't seem to be wired up right. Therefore it's very easy to get very confused (c.f. #37) if you try to join the same room under the same user from multiple devices.

Currently we display the room initials in a default room avatar, but don't show room avatars when they exist. Now that we support a public room list and signing in with other homeservers we need to support room avatars.

This is highly dependent on #9

If we don't need to worry about signalling messages not being received due to the number of debug messages, we can safely turn debug messages back on.

At least for ConferenceCallManager and similar as I assume it will become a part of the js-sdk which is close to 100% TS

Currently we do not allow for accounts other than the server the client is configured for.

Video tiles should be able to locally be made bigger/smaller. You should be able to drag/drop tiles into these larger spots to have them take up the expanded space.

If you hit "Enter call" before the "accept webcam permissions" prompt disappears, you end up in a call with blank local video. The "enter call" button presumably shouldn't be accessible until permissions are granted and the camera has spun up. This on iOS15

It should be possible to join a room by only allowing microphone permissions. We could request microphone and video permissions separately to do this.

Prior to switching to to-device signalling, logging in with an account that you use with Element would prompt you to receive 1-1 group call rings. We need to test to ensure that this is fixed with the to-device messages.

Rather than prompting guest users to enter a display name, we can randomly generate a guest username for them so that joining requires one less step. They can then change it once in the room in the settings modal or in the microphone/webcam setup screen.

When users join & part the conference we should clearly play vworping sound effects

You should be able to change your display name from the microphone/webcam screen and in the call in the settings modal

Rather than flickering up the name of the user who's currently speaking, i think it might be better to show a 'vector green' #0DBD8B glow of a few pixels around the border of the tile, CSS fading in & out over ~100ms to avoid it being too shouty.

Similarly, if the remote user is muted, we should show that they are muted constantly rather than just on hoverover.

From MSC3401:

When initiating a 1:1 call, the m.call.invite is sent to * devices of the intended target user.
Once the user answers the call from the device, the sender should rescind the other pending to-device messages, ensuring that other devices don't get spammed about long-obsolete 1:1 calls. XXX: We will need a way to rescind pending to-device msgs.
Subsequent candidates and other events are sent only to the device who answered.

We should determine if a user is currently in the call via timeouts, likely by sending ping/pong to-device messages to the currently connected device.

Describe the bug
When leaving the call via navigation, the webcam indicator doesn't turn off.

To Reproduce
Steps to reproduce the behavior:

1. In your mobile or desktop browser, join a call and then click the Element logo in the top right corner.
2. Observe that the webcam indicator is still on

Expected behavior
The webcam indicator should turn off.

Desktop (please complete the following information):

• Seen in Safari iOS and Desktop Firefox on macOS

Additional context
To properly disable the webcam status indicator, all video media stream tracks need to be stopped by calling .stop() on them. I thought I was doing this, but that's most likely not the case.

Currently participants are laid out in a naive fashion and there are updated designs with more intuitive per-device layouts. We need to update the layout algorithm to reflect these.

MacBook Pro (13-inch, M1, 2020)
MacOS v11.4
Chrome Version 92.0.4515.131

To-device signalling is currently sent via plaintext. We should switch to sending them with Olm

We should talking indicators to the user's nametag.

I think we can use matrix-org/matrix-js-sdk#1865 for talking indicators once it is finished.

The way we determine active speaker / presenter should be updated with the following behaviors:

• You shouldn't focus yourself as the active speaker
• Double clicking in presenter mode should still let you focus other users. (Need additional design help on this)
• Screenshares should take priority over webcam tiles in the presenter layout

Tried to join General on robertlong.dev from both macOS Chrome and Mobile Safari (iOS 15) but looks like signalling didn't work; the other call didn't turn up at all. Reproducible.

It would be nice to be able to set room aliases in the UI so that urls are more memorable.

We should use hash routing # for the single page app which will better hide the room id from the server. This also lets us use proper room aliases in room urls and simplifies the reverse proxy setup.

Facepiles show who is currently in a room that you've already joined. This will not work for public rooms until we have proper room peeking.

We should move to using device events instead of room events for signalling/keep alive messages. This data doesn't is ephemeral and doesn't need to be stored. It also can cause problems if too many events are sent in the room and signalling messages are not fetched when joining. By moving to device events, we will stop storing these signalling events and will not run up against bugs due to the number of events fetched on join.

You should be able to change your webcam/microphone/audio device when joining a call and during the call. Changing what audio output device is used will be limited to Chrome only.

In the case where you are joining a call with an active screenshare and you are responsible for calling the other user, your call will not succeed.

Thank you @SimonBrandner for the bug report:

When joining a call where someone is screen-sharing, if we are the party sending the invite, the call will fail. This is caused by the fact that our offer doesn't include anything for the screen-sharing track/stream, the remote party then immediately tries negotiate but we ignore that because we're the impolite party and we don't have a stable connection at that moment.

One solution I found is to hangup the call and let the other party which is screen-sharing send the offer and let us answer. While this covers most cases, there could be a case where one party would be only sending video and the other only audio, so they would never agree.

Maybe there is a sane way to start re-negotiation right after the call connects? Or maybe there is a generally saner way to do this? 🤔

The debugger panel currently takes up a fixed height. It'd be great to make the top of the panel draggable so it can be resized.

When you or someone else is speaking there's a green highlight around your tile. This highlight goes away too quickly during pauses and is quite distracting. It should only fade away after a second of inactivity and the opacity should smoothly fade out over a longer period of time.

We should add Sentry for error reporting in production

MSC 3401 specifies a sources array that allows for a single user to join a group call from multiple devices.

When we transitioned to to-device events users who leave a call without properly removing the call id from their member state event get stuck where they will not be called by other users in a call if their username is also lexicographically lesser than the other members currently in the room.

Here's an example:

I have a call I'd like to join with 3 users: User 1, User 2, and User 3

All three users join the call.

One user drops but does not reset their member event to remove the call. The members events now look like this:

{
  "@user1": { "m.calls": ["123"] },
  "@user2": { "m.calls": ["123"] },
  "@user3": { "m.calls": ["123"] },
}

User 3's video tile still appears on User 1 and User 2's screen until the WebRTC call drops.

User 3 enters the call again 2 minutes later and the member state is still set as it was before.

User 3 expects users 1 and 2 to start the call with them because they have lexicographically lesser user ids. They never receive a call invite because to users 1 and 2, that user's WebRTC call was disconnected and their state event never changes.

I see a couple different solutions to this:

1. Once a WebRTC call drops but the user's state hasn't been reset, keep sending invite messages to all of their devices.
2. Store a session id in the member event. This is how I was handing things before. When you join, you set a new session id. If the session id changes, the other participants can initiate a new call with that user.
3. If you are about to enter a group call and there's already that call's id in your member state, remove that call send it, and then add it and send it again.