Giter VIP home page Giter VIP logo

client-sdk-unity-web's Introduction

The LiveKit icon, the name of the repository and some sample code in the background.

LiveKit: Real-time video, audio and data for developers

LiveKit is an open source project that provides scalable, multi-user conferencing based on WebRTC. It's designed to provide everything you need to build real-time video audio data capabilities in your applications.

LiveKit's server is written in Go, using the awesome Pion WebRTC implementation.

GitHub stars Slack community Twitter Follow GitHub release (latest SemVer) GitHub Workflow Status License

Features

Documentation & Guides

https://docs.livekit.io

Live Demos

Ecosystem

  • Agents: build real-time multimodal AI applications with programmable backend participants
  • Egress: record or multi-stream rooms and export individual tracks
  • Ingress: ingest streams from external sources like RTMP, WHIP, HLS, or OBS Studio

SDKs & Tools

Client SDKs

Client SDKs enable your frontend to include interactive, multi-user experiences.

Language Repo Declarative UI Links
JavaScript (TypeScript) client-sdk-js React docs | JS example | React example
Swift (iOS / MacOS) client-sdk-swift Swift UI docs | example
Kotlin (Android) client-sdk-android Compose docs | example | Compose example
Flutter (all platforms) client-sdk-flutter native docs | example
Unity WebGL client-sdk-unity-web docs
React Native (beta) client-sdk-react-native native
Rust client-sdk-rust

Server SDKs

Server SDKs enable your backend to generate access tokens, call server APIs, and receive webhooks. In addition, the Go SDK includes client capabilities, enabling you to build automations that behave like end-users.

Language Repo Docs
Go server-sdk-go docs
JavaScript (TypeScript) server-sdk-js docs
Ruby server-sdk-ruby
Java (Kotlin) server-sdk-kotlin
Python (community) python-sdks
PHP (community) agence104/livekit-server-sdk-php

Tools

Install

Tip

We recommend installing LiveKit CLI along with the server. It lets you access server APIs, create tokens, and generate test traffic.

The following will install LiveKit's media server:

MacOS

brew install livekit

Linux

curl -sSL https://get.livekit.io | bash

Windows

Download the latest release here

Getting Started

Starting LiveKit

Start LiveKit in development mode by running livekit-server --dev. It'll use a placeholder API key/secret pair.

API Key: devkey
API Secret: secret

To customize your setup for production, refer to our deployment docs

Creating access token

A user connecting to a LiveKit room requires an access token. Access tokens (JWT) encode the user's identity and the room permissions they've been granted. You can generate a token with our CLI:

livekit-cli create-token \
    --api-key devkey --api-secret secret \
    --join --room my-first-room --identity user1 \
    --valid-for 24h

Test with example app

Head over to our example app and enter a generated token to connect to your LiveKit server. This app is built with our React SDK.

Once connected, your video and audio are now being published to your new LiveKit instance!

Simulating a test publisher

livekit-cli join-room \
    --url ws://localhost:7880 \
    --api-key devkey --api-secret secret \
    --room my-first-room --identity bot-user1 \
    --publish-demo

This command publishes a looped demo video to a room. Due to how the video clip was encoded (keyframes every 3s), there's a slight delay before the browser has sufficient data to begin rendering frames. This is an artifact of the simulation.

Deployment

Use LiveKit Cloud

LiveKit Cloud is the fastest and most reliable way to run LiveKit. Every project gets free monthly bandwidth and transcoding credits.

Sign up for LiveKit Cloud.

Self-host

Read our deployment docs for more information.

Building from source

Pre-requisites:

  • Go 1.20+ is installed
  • GOPATH/bin is in your PATH

Then run

git clone https://github.com/livekit/livekit
cd livekit
./bootstrap.sh
mage

Contributing

We welcome your contributions toward improving LiveKit! Please join us on Slack to discuss your ideas and/or PRs.

License

LiveKit server is licensed under Apache License v2.0.


LiveKit Ecosystem
Real-time SDKsReact Components · JavaScript · iOS/macOS · Android · Flutter · React Native · Rust · Python · Unity (web) · Unity (beta)
Server APIsNode.js · Golang · Ruby · Java/Kotlin · Python · Rust · PHP (community)
Agents FrameworksPython · Playground
ServicesLivekit server · Egress · Ingress · SIP
ResourcesDocs · Example apps · Cloud · Self-hosting · CLI

client-sdk-unity-web's People

Contributors

d4rwinss avatar davidzhao avatar dependabot[bot] avatar dsa avatar marianmatees avatar mohelm97 avatar ocupe avatar theomonnom avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

client-sdk-unity-web's Issues

Spatial Audio

Can I implement 3D / Spatial Audio with this plugin?

After an user is diconnected, the unpublish track has no information about the participant or the track

Hi there!

Imagine that there is 2 users in the same room. They are user A and user B

If the user B quit the platform, the user A received that the tracks form user B has been unpublished, however those tracks comes with the Track.Sid as null or the parcicipant.Identity as null as well.

So I can't clean all the things related to the tracks cince I don't know which track to dispose

The log No track id will be shown in the following sniped

  private void HandleUnpub(RemoteTrackPublication publication, RemoteParticipant participant)
        {
            // Note: there is a bug in livekit where there the tracks doens't come with the ID, so we just try it and if not, just let it go
            try
            {
                if (publication.Kind == TrackKind.Video)
                    VideoTrackUnpublished(publication.Track.Sid);
                else if (publication.Kind == TrackKind.Audio)
                {
                    Debug.Log("Audio unpublished");
                    Debug.Log("Audio unpublished track " + publication.Track);
                    try
                    {
                        Debug.Log("Audio unpublished track " + publication.Track.Sid));
                    }
                    catch (Exception e)
                    {
                        Debug.Log("No track id");
                    }
                    Debug.Log("Audio unpublished owner id " + Convert.ToInt32(participant.Identity));

                    // Note: There is a bug that the track Sid comes empty, so we use the participant id to solve the issue
                    VoiceChatTrackUnpublished(Convert.ToInt32(participant.Identity));
                }
            }
            catch { }
        }


Code to create the room and connect

    m_Room = new LiveKit.Room();
            m_Room.ParticipantConnected += (p) =>
            {
                Debug.Log($"Participant connected: {p.Sid}");
            };

            m_Room.TrackMuted += VideoTrackMuteChanged;
            m_Room.TrackUnmuted += VideoTrackMuteChanged;
            m_Room.LocalTrackPublished += (publication, participant) => HandleRawTrack(publication.Track, publication);
            m_Room.TrackSubscribed += (track, publication, participant) => HandleOthersRawTrack(track, publication, participant); 
            m_Room.TrackUnsubscribed += (track, publication, participant) => HandleUnsubTrack(track, publication, participant);
            m_Room.TrackUnpublished += (track, publication) => HandleUnpub(track, publication);
            m_Room.LocalTrackUnpublished += (track, publication) => LocalTrackUnpublished(track, publication);
            
             var c = m_Room.Connect(_serverUrl, _roomToken, new RoomConnectOptions()
            {
                AutoSubscribe = true
            });
            yield return c;

            if (c.IsError)
            {
                Debug.Log("Failed to connect to the room !");
                yield break;
            }

            yield return m_Room.LocalParticipant.EnableCameraAndMicrophone();

Cannot initialize non-default texture with negative or zero width on Unity 2023.

Hello,
I'm having an issue while trying to use this SDK on Unity 2023.
I'm getting this error while trying to start a video using the webcam or screen sharing:
livekit_error_unity2023

Everything works as intended on 2022 LTS.
It seems like it is not possible anymore to create or resize a 2D texture to size 0. I think this is related to the problem I'm having in the above screenshot.
This problem is little bit related to this one i found on the official unity forum found HERE.

Looking at the code HERE it seems that the VideoWidth and VideoHeight properties are set to 0 when the first SetupTexture() is called.
I'm gonna create a pull request with a simple solution that helped me overcome this issue.

Kind Regards

Screenshare Video texture no longer updating after entering WebXR mode

We are using SDK version 1.5.0 in a Unity project version 2021.3.21f1 and are encountering an issue, screenshare video texture no longer updating after entering WebXR mode
I have several textures, these play videos perfectly fine in non WebXR mode.

Below is the code I use for the LocalTrackPublished and TrackSubscribed events:

private void HandleAddedTrack(Track track, TrackPublication publication, Participant? participant = null)
{
    if (track.Kind == TrackKind.Video && publication.TrackName == TrackName.ScreenSharel)
    {
        
        if (ScreenShareContainer.transform.childCount >= 2)
            return; // No space to show more than 1 tracks

        var video = track.Attach() as HTMLVideoElement;
        var newView = Instantiate(ScreenShareViewPrefab, ScreenShareContainer.transform);
        video.VideoReceived += tex =>
        {
            newView.texture = tex;
            Debug.Log("Reciving ScreenShare");
        };
        Canvas screenShareCanvans = GameObject.Find("Screenshare").GetComponent<Canvas>();
        screenShareCanvans.enabled = true;
    }
}

Any help is appreciated.

hardware camera not releasing when disconnect and reconnect in another room

If you connect a room
room.Connect(url, token)
then you disconnect from room
room.Disconect(true)

and so connect again in a different room with other token and roomName, the video track stack, and camera hardware never turn off.

publishing track {localTrack: LocalVideoTrack}
publishing track {localTrack: LocalVideoTrack}
publishing track {localTrack: LocalVideoTrack}

and no way to unpublish and release de camera.
even doing a disconnection again, the track keep stacking

First participant audio publish issue

I was able to setup the project and take a WebGL build in Unity without any issues but came across a bug in publishing the audio (I'm only trying to publish the audio tracks, video is disabled). The issue seems to be only affecting the first participants in a room where in the first participant in the room can hear the audio of all other participants, but other participants are not able to hear the audio of the first participant. I've tested the following scenarios:

  • 2 users - 1st user can hear audio from 2nd user, but 2nd user cannot hear audio from 1st user
  • 3 users - 1st user can hear audio from 2nd & 3rd user, 2nd & 3rd user can't hear audio from 1st user but can hear each other's audio.
  • 3 users and 1st user leaves - When the 3 users were in the room, the observation was same as above. Let's say the users were abc, efg & xyz in their order of joining, after abc leaves the room, xyz can't hear the audio from efg but efg can hear the audio from xyz.

My initial assumption was that maybe the first participant doesn't have publishing permissions or maybe the SDP lines were set to recvonly but all of these seems to be set correctly. All participants are having the same permissions and SDP lines were also set to sendrecv.

I've tried the similar implementation using JS SDK as well and everything was working well without any issues. This issue seems to be only happening for Unity WebGL builds.

Tested browsers

  • Chrome
  • Firefox
  • Brave
  • Edge
  • Safari

Linear colorspace intead of Gamma colorspace

Hi there! We are a metaverse company that is using livekit to implement the voice chat, and share screen/webcam in our unity project. We are having a problem that your system is setup in the Gamma colorspace while our project is setup in Linear color space. This is making the textures to be wash out.

If you want to have a realsitic look in unity you need to go with the linear colorspace. You only want to go for the Gamma approach on old mobile devices but most of the devices today supports linear.

Is there any chance that you can support linear colorspace for web?

image

https://docs.unity3d.com/Manual/LinearRendering-LinearOrGammaWorkflow.html#:~:text=Working%20in%20linear%20color%20space,linear%20or%20gamma%20color%20space.

Can't Achieve High-Fidelity Audio for Music Streaming

Currently it is not possible to achieve high-fidelity audio quality for music streaming using the LiveKit Unity Web SDK. I discussed this in Slack with @theomonnom, @noahlt, and @davidzhao and am filing an issue report after confirming the bug.

This issue is quite problematic for our team because when someone tries to stream music via screen share (for example a DJ streaming a live performance into a virtual world) the music quality is very "garbled-sounding" and low quality and there is currently no way to fix it.

I am aware of the Hi-fi audio documentation, however this documentation does not cover how to implement hi-fi audio in the Unity Web SDK. I attempted to translate the documentation into Unity C# as best I could, however when doing so this causes screen share to break (no errors or warnings, screen share simply does not initialize at all).

Here is my working code to start a screen share with audio (not hi-fi audio):

private IEnumerator BeginScreenShareRoutine()
{
    TrackPublishOptions trackPublishOptions = new TrackPublishOptions
    {
        Name = "ScreenShare_" + GameManager.Instance.GetUserID()  // Append the instigator's user ID to the name of the screen share.
    };

    ScreenShareCaptureOptions screenShareCaptureOptions = new ScreenShareCaptureOptions
    {
        Audio = true    // *Need* to set 'Audio = true' in order for screen share to include audio.
    };

    yield return room.LocalParticipant.SetScreenShareEnabled(true, screenShareCaptureOptions, trackPublishOptions);
}

The above code works fine to initiate a screen share - the window pops up and you can choose what to share and whether or not to include system audio. Below is my attempt to adjust the settings to use high-fidelity audio:

private IEnumerator BeginScreenShareRoutineNew()
{
    TrackPublishOptions trackPublishOptions = new TrackPublishOptions
    {
        Name = "ScreenShare_" + GameManager.Instance.GetUserID(),  // Append the instigator's user ID to the name of the screen share.
        Source = TrackSource.ScreenShare,
                
        videoEncoding = new VideoEncoding{ MaxBitrate = 1_700_000, MaxFramerate = 30 },
        ScreenShareEncoding = new VideoEncoding { MaxBitrate = 2_500_000, MaxFramerate = 15 },
        VideoCodec = LiveKit.VideoCodec.VP8,
        AudioBitrate = 32_000,
        DTX = false,
        Simulcast = true,
        StopMicTrackOnMute = false
        //VideoSimulcastLayers = ,  // ???
        //ScreenShareSimulcastLayers =  // ???
    };

    ScreenShareCaptureOptions screenShareCaptureOptions = new ScreenShareCaptureOptions
    {
        Audio = true,    // *Need* to set 'Audio = true' in order for screen share to include audio.
        AudioOptions = new AudioCaptureOptions    // *** Setting an AudioCaptureOptions within ScreenShareCaptureOptions BREAKS screen share. ***
        {
            AutoGainControl = true,
            ChannelCount = 2,
            //DeviceId = , // ???
            EchoCancellation = false,
            //Latency = ,   // ???
            NoiseSuppression = false
            //SampleRate = ,    // ???
            //SampleSize =      // ???
        }
        //Resolution = new VideoResolution
        //{
            //Width = , // Does all of this get automatically set?
            //Height = ,
            //FrameRate = ,
            //AspectRatio = 
        //}
    };

    yield return room.LocalParticipant.SetScreenShareEnabled(true, screenShareCaptureOptions, trackPublishOptions);
}

The above code breaks screen share. There is no error or warning - screen share simply does not initialize at all. No window pops up when I start the above coroutine.

Hopefully this is a fairly simple issue to fix? I assume some wires just need to be connected up from C# to JS in order to bring things in line with the Hi-fi audio documentation? And hopefully this would be a good opportunity to update the documentation to include Unity. :)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.