dolbyio-samples / rts-app-react-publisher-viewer Goto Github PK

On Publisher app, the version is set to 0.0.1.
When PR is merged, the minor version number of app should be incremented. This would help the QA team to log the bug against the version number and it would be easy to track the bugs based on the version number.

Netlify will keep deploying our website by ignoring the build and test results. We should use CLI and deploy it manually.

Publisher is not able to stream the contents and error 'Invalid codec' is displayed in the browser console when the Publisher is disabling the simulcast after enabling it.

STEPS:

1. Open the publisher app on Chrome https://astounding-lily-fa33fc.netlify.app/
2. Click on the settings icon
3. Enable the simulcast and select H264 as codec
4. Click on Go Live Button
5. Publisher should be able to stream the contents
6. Click on Stop Live Button
7. Publisher should be navigated to the Setup Page
8. Click on the settings icon
9. Disable the simulcast and select "Select Codec" option as a codec
10. Click on Go Live Button
11. Publisher is not able to stream the contents and below error is shown in the browser console.

Error: Invalid codec. Possible values are: vp8,vp9,h264,av1
at @millicast.fc59a0e8.js:26:5969
at Generator.next ()
at Co (@millicast.fc59a0e8.js:1:616)
at c (@millicast.fc59a0e8.js:1:819)
at @millicast.fc59a0e8.js:1:880
at new Promise ()
at @millicast.fc59a0e8.js:1:760
at tp.publish (@millicast.fc59a0e8.js:26:6863)
at @millicast.fc59a0e8.js:26:24435
at Generator.next ()

Screen Reference: https://dolby.box.com/s/6vr54ysmmv32ck5ojus8ftah7u39e0pt

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

Create a Viewer app inside the mono repo with a working link for Netlify deployment.

Acceptance Criteria

• The Mono repo must have a new workspace for the Viewer
• Import chakra, react and the Millicast SDK
• Netlify deployment for viewer app.

Out of scope

• Do not update the viewer link in the publisher to point to the new deployment just yet
• Do not add any additional components etc.

When simulcast is on, user can select different video quality from a dropdown list

• add stream quality list to useViewer hook
• Add an option to toggle the quality of the incoming stream, options include - 'Low', 'Medium' and 'High'.

https://docs.dolby.io/streaming-apis/docs/source-and-layer-selection

to be further refined

We need a GitHub action to deploy our app into netlify

STEPS:

1. Open the publisher app https://astounding-lily-fa33fc.netlify.app/
2. Click on the settings icon
3. Enable the simulcast and select VP9 as codec
4. Click on Go Live button
5. On the viewer app, participant is not shown with the Video Quality option under setting icons
6. VP9 is used as video codec for streaming. Viewer-> Click on Setting icon -> Media Stats -> Codecs

Also with the VP8 simulcast works.

Publisher Codec. Viewer. Status
Chrome. VP8. Chrome. Worked
Chrome. VP8. Firefox. Worked

Chrome. H264. Chrome. Worked
Chrome. H264. Firefox. Worked

Chrome. VP9. Chrome. No option to select Video Quality
Chrome. VP9. Firefox. No option to select Video Quality

Firefox. * Chrome. No option to select Video Quality
Firefox. * Firefox. No option to select Video Quality

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

Hook

• A function to generate the link for Viewer in useStreamInfo

UI Component

• A copy button, check this hook
• A text link to open it from browser
• Add test-id for components

When the user selects a different camera or microphone, the popup always jumps up. While the expected behavior is it should show at the original position.

https://astounding-lily-fa33fc.netlify.app/

The publisher needs to be updated to support Simulcast.

• Update BroadcastOptions with a boolean for simulcast
• Pass simulcast to publisher.connect in the usePublisher hook
• Ensure the publisher settings pop up has the option to enable or disable simulcast.
• Resolution, codec, bitrate and simulcast cannot be toggled after stream has started.
• DO NOT SURFACE AV-1 CODEC OPTION. Dolby.io open source does not have permission to use AV-1 codec.
• Only show the option to enable Simulcast if browser is chrome.

Note :

• Simulcast is only available on Chrome when using the h.264 or VP8 codec. No other browser or codec is supported.
• Refer to the millicast demo app for what's possible

Developer needs a useViewer hook so that the incoming stream can be played.

Acceptance Criteria

• Write a useViewer hook that accept stream name and account id
• Source id and MediaStream should be returned from useViewer hook so that the incoming stream can be played

Copy needs to be refined, but we need placeholder text that needs to be shown in place of the VideoView

Acceptance Criteria

• Add text placeholder
• Add test id
• Text should only be available when the view state is inactive

See this for guidance

• listen to [deviceChange](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/devicechange_event) event then update the device list
• keep current device if it's not removed
• replace removed device with default device or first device in the list

When the external devices such as camera or microphone is disconnected during the stream, viewer participant is not able to hear the publisher audio and video. Ideally when the external devices are disconnected during the streaming, it should fallback to the build-in devices.

STEPS:

1. Connect the external camera and external microphone
2. Open the publisher app https://astounding-lily-fa33fc.netlify.app/
3. Click on the settings icon
4. Select the external camera and microphone for streaming
5. Click on Go Live button
6. Disconnect the external microphone during the streaming
7. On the viewer app, participant is not able to hear the publisher audio
8. Disconnect the external camera during the streaming
9. On the viewer app, participant is not able to view the publisher video

Ideally when the external devices are disconnected during the streaming, it should fallback to the build-in devices.

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

User needs to refresh the page every time when the external devices are connected/disconnected, ideally App should be able to detect the devices dynamically when connected/disconnected.

STEPS:

1. Open the publisher app https://astounding-lily-fa33fc.netlify.app/
2. Click on the settings icon
3. Only the build-in camera and microphone should be displayed
4. Now connect the external microphone/camera devices
5. Click on the settings icon
6. External connected devices are not shown in the microphone/camera drop-downs

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

When the publisher copies the live event viewer link to share with the attendies, the copied viewer link has streamId as undefined
Copied Viewer Link: https://viewer.millicast.com/?streamId=undefined/l9c58zkw

STEPS:

1. Open the Publisher app
2. Click on the Copy Link button
3. Open new tab/page/browser and paste the URL in the site bar
4. Copied viewer link has streamId as undefined
   Copied Viewer Link: https://viewer.millicast.com/?streamId=undefined/l9c58zkw

Publisher App URL: https://deploy-preview-34--astounding-lily-fa33fc.netlify.app/

Tested On:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

Add a button to both the viewer and publisher that closes the tab when pressed. The UX is not currently final and is still awaited.

Acceptance Criteria

• Add a test-id
• Button needs to be present on both viewer and publisher

When the Publisher starts the streaming and views the stream information, following stream information is missing from the stats and certain attributes needs unit conversion dynamically based on its value.

Missing Stats Information:

1. FPS: Calculated FPS info is missing
2. Server: Info is missing [Not Required - Confirmed by Shivank]
3. Cluster: Info is missing [Not Required - Confirmed by Shivank]

Dynamic Unit Display:

1. Available Outgoing Bitrate: Bitrate unit is dynamic, based on its value it shows kbps or mbps
2. Video Bitrate : Unit is dynamic, based on its value it shows kbps or mbps
3. VideoTotal Sent: Unit should be in MB
4. Timestamp: Timestamp is shown in epco time instead of formatted time

Main Branch Reference App: https://astounding-lily-fa33fc.netlify.app/
Millicast Reference App: https://streaming.dolby.io/#/broadcast-new?id=<TOKEN_ID>&s=<STREAM_NAME>

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

Screenshot:
Main Branch Reference App:

Millicast Reference App:

• Dropdown select for codec
• Dropdown select for resolution

Note: return Codec, resolution, from useMediaDevices hook
If one of the capabilities is not supported, we can hide the options in settings

For resolution, we should filter out the unsupported ones from the list below:

2160p/4K
1440p/2k
1080p/ Full HD
720p/ HD
480p/SD

Out of scope -

• Dropdown select for bitrates
• FPS selection,
• Portrait resolutions in mobile

For getting a list of codecs - please use this method

Refer to this screenshot from the sample app for the sort of settings we need to expose

Designs will be confirmed soon. Will update tickets with updated designs.

Hook

• Have a function to provide the latest statistic information in useViewer hook
• Update UI as new event comes through with stats

UI Component

• an overlay or popover which is always at the front of VideoView
• Add test-id for components
• Show timestamp in the correct timezone and not as epoch time.

The current Millicast project shows the following settings - please try and replicate all/as many as possible

Out of scope

• Cluster and server information

We need a VideoView to preview the current video stream

• should be mirrored - mirrored by default, with no interface for users to change
• could be fullscreen in browser, fullscreen button should disappear after 2 seconds and appear when cursor hovers on or tap on mobile
• (Optional) have network connection indicator
• (Optional) have an audio activity indicator (wave animation in the video-call app)
• Add test-id for components

We need to have the ability to select a video file as the source for Millicast playback.

Questions to be resolved -

• What if the video file also has an audio track? Does the audio matter?

Stream info button is not visible to Publisher in following scenario:

1. When the video is turned off
2. When the video is in full screen mode

** Stream info button is shown when the user hovers over the area where button is present

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

Viewer needs to select Video/Audio Source from the Setting panel to view the streaming content.

STEPS:

1. Open the Publisher App https://astounding-lily-fa33fc.netlify.app/
2. Start the streaming
3. Open the View App https://viewer.millicast.com/?streamId=KtpPTK/l9c58zkw
4. Viewer is not able to view the Publisher Video
5. Viewer needs to explicitly select Video/Audio Source from the Setting panel to view the streaming content

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

We need to have an icon button for video view to mute/unmute the audio track

• Implement updateMediaStream in usePublisher hook
• Merge CameraSelect and MicrophoneSelect to one component then reuse it

We need a timer component to show the duration of the streaming session
Only surface this while stream is on-going.
Add test-id for components

Codec dropdown should be disabled when simulcast is disabled

• When the simulcast is disabled, codec dropdown component should also be disabled.

• When the simulcast is enabled, default codec should be set in the dropdown
  when the user enables the simulcast by turning on the simulcast toogle, and without selecting codec, publisher can go live with the vp8 as codec, better to set the default codec in the dropdown when the simulcast is enabled.

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

** Update** not needed for milestone 1

Hook

• start/stop recording in usePublisher hook

UI Component

• A switch to control recording start/stop.
• Add test-id for components

Support Chromium & Safari

Hook

• it should return an array of cameras
• it should return an array of microphones
• it should return a MediaStream with select camera as video source and microphone as audio source
• it should have a function to set the video source to a specified camera
• it should have a function to set the audio source to a specified microphone

UI Components

• Simple video preview using <video> HTML tag
• CameraSelect
• MicSelect
• Add test-id for components

Reference:
https://github.com/millicast/millicast-sdk/blob/18f118ca28f0075cac3e776132fc94af5923700f/packages/millicast-publisher-demo/src/js/MillicastMedia.js
https://stackoverflow.com/questions/68047599/how-to-switch-cameras-in-pwa-app-built-with-reactjs
Browsers: Chrome & Safari

We need to show the settings UI in a popup

Note: We nee another Publisher to present the shared screen as a video track

• A share button to start/stop the sharing session with a screen/app picker which the browser should provide
• A UI component to display the shared screen/app

Ref: https://developer.mozilla.org/en-US/docs/Web/API/Screen_Capture_API/Using_Screen_Capture
Chromium & Safari

Rough guide

Note, need to enable the multi-source option from the dashboard

Hooks

• mute/unmute audio in useMedia hook
• turn on/off camera in useMedia hook

UI components

• mute/unmute microphone button
• turn on/off camera button
• Add test-id for components

Support Chormium and Safari

In lieu of designs/icons, add a text based indicator to the viewer app to indicate that the publisher is publishing.

This can be done by subscribing to broadcast events on millicastView

• The active and inactive views are what you want to subscribe to

Acceptance Criteria

• add live/not live to useViewer hook
• Position : Top right above the participant count
• Just use Text ("Live"/"Not Live") to indicate status in lieu of icons
• Add test-id

Refer to this for more details - https://www.figma.com/file/mpGO2yc1EBeDKgcvhmm5CA/Dolby.io-%2F-Real-time-Streaming-%E2%80%93-MVP?node-id=5826%3A19616

Currently for simulcast, we use a property VideoCodec of type string. VideoCodec is a bit generic. Constrain it like the SDK does.

Hook

• Have a function to provide the latest statistic information in usePublisher
• Update UI as new event comes through with stats

UI Component

• an overlay or popover which is always at the front of VideoView
• Add test-id for components

The current Millicast project shows the following settings - please try and replicate all/as many as possible

Use the current skeleton app as a baseline

• Add Cucumber and Playwright to project
• Add one feature file
• Create and run the test case

Since millicast sdk still doesn't support typescript, so we need to add the type definitions

• Director
• Publisher
• RTCPeerConnection
• Logger

When the publisher selects the camera/microphone devices from the Manage Your Devices popup, following issues are observed:

• Components within the Setup screen are resized when the publisher selects the build-in camera from the external camera
• Components are reloaded again when the publisher selects camera/microphone
• Manage Your Devices popup jumps up when the camera/microphone is selected for the first time
• video view on the setup page is small and back when the app is open or refreshed

STEPS:

1. Open the publisher app https://astounding-lily-fa33fc.netlify.app/
2. Click on the settings icon
3. Select microphone/camera from the Manage Your Devices popup

Screen Recording: https://dolby.box.com/s/ld8mdw522b9p5qbjfoyyu3k053rqi09m

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

Subsribe to broadcastEvents from the viewer. This will allow you to subscribe to viewercount events.

Acceptance Criteria

• add participant count to useViewer hook
• Add test-id
• Use same component as that used in Publisher app
• position - top/right corner of the screen until the complete UX is ready
• This should only be visible when the stream has started. Otherwise invisible

out of scope

• UI/UX improvements including responsive design

Test the publisher functionality

Testing the below functionality on Publisher side:

• Camera Select
• Microphone Select
• Stream using Selected Camera and Microphone
• Participants Count
• Toggle Audio and Video
• Viewer Link button and Link functionality
• Setting Popup [Camera Select and Microphone Select]
• Static Information
• Change Audio and Video Src

• On Setup
• After going live

• Permission

• Don't give Camera and Microphone
• Don't give Camera
• Don't give Microphone
• Remove permission during live stream

• External Audio and Video Src

• Dynamic detection
• On Setup
• After going live
• Connect/Disconnect on Setup
• Connect/Disconnect during Live

Netlify has a default security setting that prints out a log in the console

[Report Only] Refused to connect to ‘https://cdn.segment.com/v1/projects/PzoD1qlC1wpvDGhNckresPQM3zcX8I1s/settings’ because it violates the following Content Security Policy directive: “connect-src ‘self’ data: *.amazonaws.com *.bugsnag.com *.firebaseio.com *.giphy.com *.launchdarkly.com *.netlify.com .segment.io netlify-cocoon.netlify.app netlify-slapp.netlify.app netlilink.netlify.app ws://localhost:3000/ wss://.services.netlify.com”.

This does not block the application from running in any way, it simply controls the number of domains that the app is allowed to connect.

We need to fix the error

Hook

• a function to provide the latest participant count in usePublisher

UI Component

• A participants icon and text
• Add test-id for components

A hook to provide Millicast SDK's publish function

• create a publisher with a token and stream in ready state
• start/stop streaming
• should return states: ready | connecting | streaming

UI component

• A Button to Toggle stream start/stop
• Add test-id for components

Github secret

• Add token and stream as github env secrets

Note:

• please use plain JS to get audio&video stream until issue #1 is done
• please declare the types when necessary: https://www.typescriptlang.org/docs/handbook/declaration-files/by-example.html

Reference: https://docs.dolby.io/streaming-apis/docs/web
Browsers: Chrome & Safari

Show broadcast in videoview

Design a component to mute/unmute the speaker on the viewer side. This will then allow us to set muted on the VideoView component. By default the incoming stream's audio is not muted.

** Acceptance Criteria **

• By default the incoming audio is not muted
• Add a test-id
• The icon should update as the voice is muted/unmuted

Millicast does not offer session continuance; if you close the browser window or hit refresh during a stream, the stream is disconnected.

As an enhancement, later, we would like to have the ability to tell the user what will happen if the browser tab is closed/reloaded so that they can make an informed decision.

When working on #82 please make sure this warning is shown to the viewer