Giter VIP home page Giter VIP logo

Comments (4)

ritchielozada avatar ritchielozada commented on July 30, 2024

The video/audio/data are effectively separate streams. In browser based implementations (and original UWP/XAML samples) you typically do not even get the video frames since the camera/media object is handled directly by WebRTC and playback is passed to a peer media object without the raw frames.
The UWP WebRTC package and version I am using with Visual Studio introduced raw and encoded frame handling so we can process incoming video frames and render in a Unity Texture as a viewing peer, it does not send video frames back,

There are 3 approaches (top of my head response) you can consider if you need to do subtitles.

  • You can hard-sub (text embedded on video) in realtime updating your video frames and send that to the peer as video. Your complexity increases since you would need to create a virtual camera/video source that overlays your subtitles to the video and send that out as the media object to WebRTC but your video viewing peer can be a browser or client app.
  • If you need to keep your video and data separate, you always have to option of only using the data channel to send your video frame and data, you would end up losing the video optimizations and video performance management of WebRTC so it's not an ideal approach you and would have to handle the video frame render.
  • Using the UWP library you can buffer the incoming frames so you can hold the video update to sync with your subtitle data if you want video frame level precision. You would have to manage the timecode/markers on your video frames and match it with your subtitle data in the frame receive events.

from unitywithwebrtc.

NinoStella avatar NinoStella commented on July 30, 2024

I modified your sample and enabled video sending (the underline uwp webrtc library supports this feature) so i can have two peers with bidirectional audio/video.
But there is no way to update outgoing frames so i can't store data in them.

About your suggestions:
The first and third are not applicable because of what i wrote before (can't update outgoing frames)
The second is absolutely not to consider

from unitywithwebrtc.

ritchielozada avatar ritchielozada commented on July 30, 2024

If you can retrieve the frames from the webcam or video file, you can update/edit them as long as you can pass the updated frame through a media source object. If you only have the device source passed directly to WebRTC then you'll end up having to implement the media reader/viewer first.
The non-trivial work routes are (1) create a virtual camera/video device using the updated frames and use that as the video source or (2) update WebRTC lib integration to use video frames. Check out the https://github.com/CatalystCode/3dtoolkit project for sample implementations.

The second option is not so bad as a real-time data transport solution :) you lose the "real-time" managed video but binary data still get passed to the peers.

from unitywithwebrtc.

ritchielozada avatar ritchielozada commented on July 30, 2024

Closing discussion on feature/functionality.

from unitywithwebrtc.

Related Issues (16)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.