Sync data with frames about unitywithwebrtc HOT 4 CLOSED

The video/audio/data are effectively separate streams. In browser based implementations (and original UWP/XAML samples) you typically do not even get the video frames since the camera/media object is handled directly by WebRTC and playback is passed to a peer media object without the raw frames.
The UWP WebRTC package and version I am using with Visual Studio introduced raw and encoded frame handling so we can process incoming video frames and render in a Unity Texture as a viewing peer, it does not send video frames back,

There are 3 approaches (top of my head response) you can consider if you need to do subtitles.

• You can hard-sub (text embedded on video) in realtime updating your video frames and send that to the peer as video. Your complexity increases since you would need to create a virtual camera/video source that overlays your subtitles to the video and send that out as the media object to WebRTC but your video viewing peer can be a browser or client app.
• If you need to keep your video and data separate, you always have to option of only using the data channel to send your video frame and data, you would end up losing the video optimizations and video performance management of WebRTC so it's not an ideal approach you and would have to handle the video frame render.
• Using the UWP library you can buffer the incoming frames so you can hold the video update to sync with your subtitle data if you want video frame level precision. You would have to manage the timecode/markers on your video frames and match it with your subtitle data in the frame receive events.

from unitywithwebrtc.

I modified your sample and enabled video sending (the underline uwp webrtc library supports this feature) so i can have two peers with bidirectional audio/video.
But there is no way to update outgoing frames so i can't store data in them.

About your suggestions:
The first and third are not applicable because of what i wrote before (can't update outgoing frames)
The second is absolutely not to consider

from unitywithwebrtc.

If you can retrieve the frames from the webcam or video file, you can update/edit them as long as you can pass the updated frame through a media source object. If you only have the device source passed directly to WebRTC then you'll end up having to implement the media reader/viewer first.
The non-trivial work routes are (1) create a virtual camera/video device using the updated frames and use that as the video source or (2) update WebRTC lib integration to use video frames. Check out the https://github.com/CatalystCode/3dtoolkit project for sample implementations.

The second option is not so bad as a real-time data transport solution :) you lose the "real-time" managed video but binary data still get passed to the peers.

from unitywithwebrtc.

Closing discussion on feature/functionality.

from unitywithwebrtc.