Giter VIP home page Giter VIP logo

rts-app-react-publisher-viewer's Introduction

Real-time Streaming - Publisher & Viewer Sample Apps

Overview

Quickly stand-up a lightning fast, broadcast quality, real-time streaming app with ultra low-latency (sub 500ms).

Getting Started

Prerequisites

  • Node v16.16.0
  • Yarn v1.22.19
  • A Dolby.io account

How to get a Dolby.io account

To setup your Dolby.io account, go to the Dolby.io dashboard and complete the form. After confirming your email address, you will be logged in.

Project structure

This project is setup as a monorepo, and contains the following:

  • Publisher app (under apps/publisher)
  • Viewer/subscriber app (under apps/viewer)
  • Tests (under apps/e2e-test/* )
  • Shared components/hooks (under libs/*)
  • Storybook stories (under .apps/stories/*)

Note: We chose to use yarn and nx in to our sample app because it helps with our workflow. You are welcome to choose tooling that suits your needs.

The RTS solution is broken down into 2 parts - the publisher (or the broadcaster) and the subscriber (or the viewer). This repository contains sample code for both parts.

Installation and setup

To install the libraries necessary, clone the repo. Then from the root directory of the project, run the following command in a terminal window:

yarn

RTS Tokens

In order for this application to work, you need three tokens - stream name, stream account id and stream publishing token. These tokens can be found in your Dolby.io dashboard, under the Streaming tab for a given token.

token create

To allow publishing from multiple sources to work with your token, remember to go to advanced settings and enable "Multisource".

multi-source

Once created, you can grab the stream name, publisher token and stream ID from the Token details page, under the API tab.

tokens screenshot

For your application to pick these values up, you can either set them as environment variables or store them in a .env file. The .env file must be under apps/publisher/ and apps/viewer/.

VITE_RTS_STREAM_NAME=<your stream name>
VITE_RTS_STREAM_PUBLISHING_TOKEN=<your stream token>
VITE_RTS_ACCOUNT_ID=<your account id>

note : This file is typically not added to git. When you clone the repo, you will not find this file and will have to create one yourself. The .env.example app can be used as a template with basic keys. Rename the file to .env and insert your tokens to get the file up and running.

env file folder structure

Note: Please set these env variables before you launch the apps. You can either set them via the command line in your shell environment or by entering them in your .env file.

Configuring the viewer link

The viewer link is configured in the .env file in your apps/publisher directory, or in your shell/systen environment variables. To set the URL, insert/update the following variable:

VITE_RTS_VIEWER_BASE_URL=<Your URL goes here>

If you are using a particular port number, please add this to the URL. The best way to do this is to first run the viewer app, grab the URL from your browser and add it to your .env file as such.

VITE_RTS_VIEWER_BASE_URL=http://localhost:5174/

Running the publisher app

yarn nx serve publisher

Note: Remember to grant the publisher app all the necessary camera and microphone permissions.

permissions

Running the viewer app

To run your app in dev mode, run the following command in your terminal and open the browser:

yarn nx serve viewer

Features, roadmap and limitations

RTS Publisher

  • Publish streams
  • Publish with simulcast (only on Chrome)
  • Microphone and camera device selection
  • Invite viewers to watch your stream
  • Screen sharing
  • Mute audio and video
  • Resolution, bandwidth, codec, and bitrate selection
  • Ability to stream from up to 4 sources. See here for bandwidth limitations.
  • Ability to stream a local video file
  • Observing stream statistics for each simulcast layer
  • Recording your streams

RTS Viewer

  • View streams
  • Mute audio and video
  • Select incoming stream quality with simulcast
  • Ability to see stats for each individual source
  • Chromecast support
  • Picture in picture support

Issues and Bugs

The apps and components provided here should be used as reference material. Although we have taken great care in creating these, please note that they are not intended for real production use and there may be some bugs.

Please report any issues under Issues on our GitHub and appropriately label them. Please review the existing open issues before raising a new one.

Browser compatibility

While utmost care has been taken to ensure this works across all browsers, please note that there are some limitations and therefore, we recommend Chromium based browsers or Safari as the best means to test out these applications. Features such as simulcast only work on Chromium based browsers, for example.

Generally speaking, our app has been tested to work on the following browser versions:

  • Google Chrome v100+
  • Apple Safari v16.x
  • Edge v107.x

There are known limitations with webRTC and Mozilla Firefox, and therefore the browser is not recommended.

More resources

Want to learn more? Check out the Real Time Streaming app gallery page.

Looking for more sample apps and projects? Head to the Project Gallery.

rts-app-react-publisher-viewer's People

Contributors

adamlorek-mq avatar dependabot[bot] avatar dubeyshivank avatar dzfill avatar gsimplicio-emed avatar jackshen avatar vincentsong avatar vishalkharge avatar xinran-ma avatar xxmadolby avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

rts-app-react-publisher-viewer's Issues

ViewerLinkButton

Hook

  • A function to generate the link for Viewer in useStreamInfo

UI Component

  • A copy button, check this hook
  • A text link to open it from browser
  • Add test-id for components

[Publisher] Add video file as source

We need to have the ability to select a video file as the source for Millicast playback.

Questions to be resolved -

  • What if the video file also has an audio track? Does the audio matter?

[Publisher] Browser Console: Error: Invalid codec. Possible values are: vp8,vp9,h264,av1

Publisher is not able to stream the contents and error 'Invalid codec' is displayed in the browser console when the Publisher is disabling the simulcast after enabling it.

STEPS:

  1. Open the publisher app on Chrome https://astounding-lily-fa33fc.netlify.app/
  2. Click on the settings icon
  3. Enable the simulcast and select H264 as codec
  4. Click on Go Live Button
  5. Publisher should be able to stream the contents
  6. Click on Stop Live Button
  7. Publisher should be navigated to the Setup Page
  8. Click on the settings icon
  9. Disable the simulcast and select "Select Codec" option as a codec
  10. Click on Go Live Button
  11. Publisher is not able to stream the contents and below error is shown in the browser console.

Error: Invalid codec. Possible values are: vp8,vp9,h264,av1
at @millicast.fc59a0e8.js:26:5969
at Generator.next ()
at Co (@millicast.fc59a0e8.js:1:616)
at c (@millicast.fc59a0e8.js:1:819)
at @millicast.fc59a0e8.js:1:880
at new Promise ()
at @millicast.fc59a0e8.js:1:760
at tp.publish (@millicast.fc59a0e8.js:26:6863)
at @millicast.fc59a0e8.js:26:24435
at Generator.next ()

Screen Reference: https://dolby.box.com/s/6vr54ysmmv32ck5ojus8ftah7u39e0pt

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

[Publisher] App should be able to detect the devices dynamically when connected/disconnected

User needs to refresh the page every time when the external devices are connected/disconnected, ideally App should be able to detect the devices dynamically when connected/disconnected.

STEPS:

  1. Open the publisher app https://astounding-lily-fa33fc.netlify.app/
  2. Click on the settings icon
  3. Only the build-in camera and microphone should be displayed
  4. Now connect the external microphone/camera devices
  5. Click on the settings icon
  6. External connected devices are not shown in the microphone/camera drop-downs

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

Setting component for simulcast

  • Dropdown select for codec
  • Dropdown select for resolution

Note: return Codec, resolution, from useMediaDevices hook
If one of the capabilities is not supported, we can hide the options in settings

For resolution, we should filter out the unsupported ones from the list below:

2160p/4K
1440p/2k
1080p/ Full HD
720p/ HD
480p/SD

Out of scope -

  • Dropdown select for bitrates
  • FPS selection,
  • Portrait resolutions in mobile

For getting a list of codecs - please use this method

Refer to this screenshot from the sample app for the sort of settings we need to expose

Image

Fix Netlify Content Security Policy

Netlify has a default security setting that prints out a log in the console

[Report Only] Refused to connect to ‘https://cdn.segment.com/v1/projects/PzoD1qlC1wpvDGhNckresPQM3zcX8I1s/settings’ because it violates the following Content Security Policy directive: “connect-src ‘self’ data: *.amazonaws.com *.bugsnag.com *.firebaseio.com *.giphy.com *.launchdarkly.com *.netlify.com .segment.io netlify-cocoon.netlify.app netlify-slapp.netlify.app netlilink.netlify.app ws://localhost:3000/ wss://.services.netlify.com”.

This does not block the application from running in any way, it simply controls the number of domains that the app is allowed to connect.

We need to fix the error

By default viewer is not able to view Publisher video/audio, selection of video/audio src is required

Viewer needs to select Video/Audio Source from the Setting panel to view the streaming content.

STEPS:

  1. Open the Publisher App https://astounding-lily-fa33fc.netlify.app/
  2. Start the streaming
  3. Open the View App https://viewer.millicast.com/?streamId=KtpPTK/l9c58zkw
  4. Viewer is not able to view the Publisher Video
  5. Viewer needs to explicitly select Video/Audio Source from the Setting panel to view the streaming content

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

useMedia hook, CameraSelect and MicSelect components

Hook

  • it should return an array of cameras
  • it should return an array of microphones
  • it should return a MediaStream with select camera as video source and microphone as audio source
  • it should have a function to set the video source to a specified camera
  • it should have a function to set the audio source to a specified microphone

UI Components

  • Simple video preview using <video> HTML tag
  • CameraSelect
  • MicSelect
  • Add test-id for components

Reference:
https://github.com/millicast/millicast-sdk/blob/18f118ca28f0075cac3e776132fc94af5923700f/packages/millicast-publisher-demo/src/js/MillicastMedia.js
https://stackoverflow.com/questions/68047599/how-to-switch-cameras-in-pwa-app-built-with-reactjs
Browsers: Chrome & Safari

[Viewer] Create Monorepo workspace for Viewer app

Create a Viewer app inside the mono repo with a working link for Netlify deployment.

Acceptance Criteria

  • The Mono repo must have a new workspace for the Viewer
  • Import chakra, react and the Millicast SDK
  • Netlify deployment for viewer app.

Out of scope

  • Do not update the viewer link in the publisher to point to the new deployment just yet
  • Do not add any additional components etc.

[Publisher] Setup: Components are resized and loaded again when device is selected

When the publisher selects the camera/microphone devices from the Manage Your Devices popup, following issues are observed:

  • Components within the Setup screen are resized when the publisher selects the build-in camera from the external camera
  • Components are reloaded again when the publisher selects camera/microphone
  • Manage Your Devices popup jumps up when the camera/microphone is selected for the first time
  • video view on the setup page is small and back when the app is open or refreshed

STEPS:

  1. Open the publisher app https://astounding-lily-fa33fc.netlify.app/
  2. Click on the settings icon
  3. Select microphone/camera from the Manage Your Devices popup

Screen Recording: https://dolby.box.com/s/ld8mdw522b9p5qbjfoyyu3k053rqi09m

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

Add simulcast support

The publisher needs to be updated to support Simulcast.

  • Update BroadcastOptions with a boolean for simulcast
  • Pass simulcast to publisher.connect in the usePublisher hook
  • Ensure the publisher settings pop up has the option to enable or disable simulcast.
  • Resolution, codec, bitrate and simulcast cannot be toggled after stream has started.
  • DO NOT SURFACE AV-1 CODEC OPTION. Dolby.io open source does not have permission to use AV-1 codec.
  • Only show the option to enable Simulcast if browser is chrome.

Note :

  • Simulcast is only available on Chrome when using the h.264 or VP8 codec. No other browser or codec is supported.
  • Refer to the millicast demo app for what's possible

Image

[Viewer] [Publisher] Exit button

Add a button to both the viewer and publisher that closes the tab when pressed. The UX is not currently final and is still awaited.

Acceptance Criteria

  • Add a test-id
  • Button needs to be present on both viewer and publisher

usePublisher hook and StreamToggle button

A hook to provide Millicast SDK's publish function

  • create a publisher with a token and stream in ready state
  • start/stop streaming
  • should return states: ready | connecting | streaming

UI component

  • A Button to Toggle stream start/stop
  • Add test-id for components

Github secret

  • Add token and stream as github env secrets

Note:

Reference: https://docs.dolby.io/streaming-apis/docs/web
Browsers: Chrome & Safari

Warn people from hitting refresh or closing browser tab when stream is in progress

Millicast does not offer session continuance; if you close the browser window or hit refresh during a stream, the stream is disconnected.

As an enhancement, later, we would like to have the ability to tell the user what will happen if the browser tab is closed/reloaded so that they can make an informed decision.

When working on #82 please make sure this warning is shown to the viewer

[Viewer] Add Live/Not LIve Indicator

In lieu of designs/icons, add a text based indicator to the viewer app to indicate that the publisher is publishing.

This can be done by subscribing to broadcast events on millicastView

  • The active and inactive views are what you want to subscribe to

Acceptance Criteria

  • add live/not live to useViewer hook
  • Position : Top right above the participant count
  • Just use Text ("Live"/"Not Live") to indicate status in lieu of icons
  • Add test-id

Refer to this for more details - https://www.figma.com/file/mpGO2yc1EBeDKgcvhmm5CA/Dolby.io-%2F-Real-time-Streaming-%E2%80%93-MVP?node-id=5826%3A19616

[Viewer] Audio on/off button

Design a component to mute/unmute the speaker on the viewer side. This will then allow us to set muted on the VideoView component. By default the incoming stream's audio is not muted.

** Acceptance Criteria **

  • By default the incoming audio is not muted
  • Add a test-id
  • The icon should update as the voice is muted/unmuted

[Publisher] Copied viewer link has streamId as undefined

When the publisher copies the live event viewer link to share with the attendies, the copied viewer link has streamId as undefined
Copied Viewer Link: https://viewer.millicast.com/?streamId=undefined/l9c58zkw

STEPS:

  1. Open the Publisher app
  2. Click on the Copy Link button
  3. Open new tab/page/browser and paste the URL in the site bar
  4. Copied viewer link has streamId as undefined
    Copied Viewer Link: https://viewer.millicast.com/?streamId=undefined/l9c58zkw

Publisher App URL: https://deploy-preview-34--astounding-lily-fa33fc.netlify.app/

Tested On:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

Record stream function in hook and Switch UI

** Update** not needed for milestone 1

Hook

  • start/stop recording in usePublisher hook

UI Component

  • A switch to control recording start/stop.
  • Add test-id for components

Support Chromium & Safari

[Publisher] Missing stats information and dynamic unit display

When the Publisher starts the streaming and views the stream information, following stream information is missing from the stats and certain attributes needs unit conversion dynamically based on its value.

Missing Stats Information:

  1. FPS: Calculated FPS info is missing
  2. Server: Info is missing [Not Required - Confirmed by Shivank]
  3. Cluster: Info is missing [Not Required - Confirmed by Shivank]

Dynamic Unit Display:

  1. Available Outgoing Bitrate: Bitrate unit is dynamic, based on its value it shows kbps or mbps
  2. Video Bitrate : Unit is dynamic, based on its value it shows kbps or mbps
  3. VideoTotal Sent: Unit should be in MB
  4. Timestamp: Timestamp is shown in epco time instead of formatted time

Main Branch Reference App: https://astounding-lily-fa33fc.netlify.app/
Millicast Reference App: https://streaming.dolby.io/#/broadcast-new?id=<TOKEN_ID>&s=<STREAM_NAME>

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

Screenshot:
Main Branch Reference App:
Millicast_Actual

Millicast Reference App:
Millicast_Expected

Timer

We need a timer component to show the duration of the streaming session
Only surface this while stream is on-going.
Add test-id for components

Statistic Information

Hook

  • Have a function to provide the latest statistic information in usePublisher
  • Update UI as new event comes through with stats

UI Component

  • an overlay or popover which is always at the front of VideoView
  • Add test-id for components

The current Millicast project shows the following settings - please try and replicate all/as many as possible

Image

[Viewer] Stream statistics

Hook

  • Have a function to provide the latest statistic information in useViewer hook
  • Update UI as new event comes through with stats

UI Component

  • an overlay or popover which is always at the front of VideoView
  • Add test-id for components
  • Show timestamp in the correct timezone and not as epoch time.

The current Millicast project shows the following settings - please try and replicate all/as many as possible

Out of scope

  • Cluster and server information

Image

Simulcast with VP9 and Chrome does not work

STEPS:

  1. Open the publisher app https://astounding-lily-fa33fc.netlify.app/
  2. Click on the settings icon
  3. Enable the simulcast and select VP9 as codec
  4. Click on Go Live button
  5. On the viewer app, participant is not shown with the Video Quality option under setting icons
  6. VP9 is used as video codec for streaming. Viewer-> Click on Setting icon -> Media Stats -> Codecs

Also with the VP8 simulcast works.

Publisher Codec. Viewer. Status
Chrome. VP8. Chrome. Worked
Chrome. VP8. Firefox. Worked

Chrome. H264. Chrome. Worked
Chrome. H264. Firefox. Worked

Chrome. VP9. Chrome. No option to select Video Quality
Chrome. VP9. Firefox. No option to select Video Quality

Firefox. * Chrome. No option to select Video Quality
Firefox. * Firefox. No option to select Video Quality

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

[Publisher] Fallback to build-in devices when the external device is disconnected during stream

When the external devices such as camera or microphone is disconnected during the stream, viewer participant is not able to hear the publisher audio and video. Ideally when the external devices are disconnected during the streaming, it should fallback to the build-in devices.

STEPS:

  1. Connect the external camera and external microphone
  2. Open the publisher app https://astounding-lily-fa33fc.netlify.app/
  3. Click on the settings icon
  4. Select the external camera and microphone for streaming
  5. Click on Go Live button
  6. Disconnect the external microphone during the streaming
  7. On the viewer app, participant is not able to hear the publisher audio
  8. Disconnect the external camera during the streaming
  9. On the viewer app, participant is not able to view the publisher video

Ideally when the external devices are disconnected during the streaming, it should fallback to the build-in devices.

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

Toggle Audio/Video

Hooks

  • mute/unmute audio in useMedia hook
  • turn on/off camera in useMedia hook

UI components

  • mute/unmute microphone button
  • turn on/off camera button
  • Add test-id for components

Support Chormium and Safari

[Viewer] Add Participant Count Indicator

Subsribe to broadcastEvents from the viewer. This will allow you to subscribe to viewercount events.

Acceptance Criteria

  • add participant count to useViewer hook
  • Add test-id
  • Use same component as that used in Publisher app
  • position - top/right corner of the screen until the complete UX is ready
  • This should only be visible when the stream has started. Otherwise invisible

out of scope

  • UI/UX improvements including responsive design

QA: Test the publisher functionality

Test the publisher functionality

Testing the below functionality on Publisher side:

  • Camera Select
  • Microphone Select
  • Stream using Selected Camera and Microphone
  • Participants Count
  • Toggle Audio and Video
  • Viewer Link button and Link functionality
  • Setting Popup [Camera Select and Microphone Select]
  • Static Information
  • Change Audio and Video Src
  • On Setup
  • After going live
  • Permission
  • Don't give Camera and Microphone
  • Don't give Camera
  • Don't give Microphone
  • Remove permission during live stream
  • External Audio and Video Src
  • Dynamic detection
  • On Setup
  • After going live
  • Connect/Disconnect on Setup
  • Connect/Disconnect during Live

Participant Count

Hook

  • a function to provide the latest participant count in usePublisher

UI Component

  • A participants icon and text
  • Add test-id for components

[Viewer] Add useViewer hook

Developer needs a useViewer hook so that the incoming stream can be played.

Acceptance Criteria

  • Write a useViewer hook that accept stream name and account id
  • Source id and MediaStream should be returned from useViewer hook so that the incoming stream can be played

[Viewer] Placeholder screen before stream starts

Copy needs to be refined, but we need placeholder text that needs to be shown in place of the VideoView

Acceptance Criteria

  • Add text placeholder
  • Add test id
  • Text should only be available when the view state is inactive

See this for guidance

Image

VideoView Component

We need a VideoView to preview the current video stream

  • should be mirrored - mirrored by default, with no interface for users to change
  • could be fullscreen in browser, fullscreen button should disappear after 2 seconds and appear when cursor hovers on or tap on mobile
  • (Optional) have network connection indicator
  • (Optional) have an audio activity indicator (wave animation in the video-call app)
  • Add test-id for components

[Publisher] Codec dropdown should be disabled when simulcast is disabled

Codec dropdown should be disabled when simulcast is disabled

  • When the simulcast is disabled, codec dropdown component should also be disabled.

  • When the simulcast is enabled, default codec should be set in the dropdown
    when the user enables the simulcast by turning on the simulcast toogle, and without selecting codec, publisher can go live with the vp8 as codec, better to set the default codec in the dropdown when the simulcast is enabled.

Environment:
OS: MAC [macOS Monterey Version 12.3.1]
Browser: Chrome [Version 106.0.5249.119 (Official Build) (x86_64)]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.