deepch / vdk Goto Github PK

Related to deepch/RTSPtoWebRTC#121 (comment)

Packets inspected here are having negative duration.
Video is stuttering, about:webrtc reports framerate around, 3.

after abs'ing Duration` before sending

if pck.Duration < 0 {
	pck.Duration = -pck.Duration
}
err = muxerWebRTC.WritePacket(pck)

frame rate reported in about:webrtc is around 23. Video is "smoother" - all frames seem to be displayed, they are jumping but in different way (back and forth), I guess that is because Durations is not correct.

Hi,

We are using RTSPtoWeb to stream cams RTSp flux, but we need to support the audio format G711. Is it possible to implement it or to provide a documentation to how we can add a new audio/video format.

Regards.

There exist multiple build issues:

go version go1.21.1 linux/amd64
master at 9d21d05

$ go build ./...
# github.com/deepch/vdk/format/mp4/mp4io/gen
format/mp4/mp4io/gen/pattern.go:4:2: undefined: atom
format/mp4/mp4io/gen/pattern.go:4:7: undefined: Header
format/mp4/mp4io/gen/pattern.go:4:15: undefined: MovieHeader
format/mp4/mp4io/gen/pattern.go:5:2: undefined: atom
format/mp4/mp4io/gen/pattern.go:5:7: undefined: MovieExtend
format/mp4/mp4io/gen/pattern.go:6:2: undefined: atoms
format/mp4/mp4io/gen/pattern.go:6:8: undefined: Tracks
format/mp4/mp4io/gen/pattern.go:6:16: undefined: Track
format/mp4/mp4io/gen/pattern.go:7:2: undefined: _unknowns
format/mp4/mp4io/gen/pattern.go:11:8: undefined: Version
format/mp4/mp4io/gen/pattern.go:11:8: too many errors
# github.com/deepch/vdk/format/mp4m/mp4io/gen
format/mp4m/mp4io/gen/pattern.go:4:2: undefined: atom
format/mp4m/mp4io/gen/pattern.go:4:7: undefined: Header
format/mp4m/mp4io/gen/pattern.go:4:15: undefined: MovieHeader
format/mp4m/mp4io/gen/pattern.go:5:2: undefined: atom
format/mp4m/mp4io/gen/pattern.go:5:7: undefined: MovieExtend
format/mp4m/mp4io/gen/pattern.go:6:2: undefined: atoms
format/mp4m/mp4io/gen/pattern.go:6:8: undefined: Tracks
format/mp4m/mp4io/gen/pattern.go:6:16: undefined: Track
format/mp4m/mp4io/gen/pattern.go:7:2: undefined: _unknowns
format/mp4m/mp4io/gen/pattern.go:11:8: undefined: Version
format/mp4m/mp4io/gen/pattern.go:11:8: too many errors
# github.com/deepch/vdk/format/mp4f/mp4fio/gen
format/mp4f/mp4fio/gen/pattern.go:4:2: undefined: atom
format/mp4f/mp4fio/gen/pattern.go:4:7: undefined: Header
format/mp4f/mp4fio/gen/pattern.go:4:15: undefined: MovieHeader
format/mp4f/mp4fio/gen/pattern.go:5:2: undefined: atom
format/mp4f/mp4fio/gen/pattern.go:5:7: undefined: MovieExtend
format/mp4f/mp4fio/gen/pattern.go:6:2: undefined: atoms
format/mp4f/mp4fio/gen/pattern.go:6:8: undefined: Tracks
format/mp4f/mp4fio/gen/pattern.go:6:16: undefined: Track
format/mp4f/mp4fio/gen/pattern.go:7:2: undefined: _unknowns
format/mp4f/mp4fio/gen/pattern.go:11:8: undefined: Version
format/mp4f/mp4fio/gen/pattern.go:11:8: too many errors
# github.com/pion/srtp
../../go/pkg/mod/github.com/pion/[email protected]/session_srtp.go:138:12: assignment mismatch: 1 variable but h.Unmarshal returns 2 values
../../go/pkg/mod/github.com/pion/[email protected]/srtp.go:37:12: assignment mismatch: 1 variable but header.Unmarshal returns 2 values
../../go/pkg/mod/github.com/pion/[email protected]/srtp.go:52:12: assignment mismatch: 1 variable but header.Unmarshal returns 2 values
../../go/pkg/mod/github.com/pion/[email protected]/srtp.go:56:52: header.PayloadOffset undefined (type *rtp.Header has no field or method PayloadOffset)
../../go/pkg/mod/github.com/pion/[email protected]/srtp_cipher_aead_aes_gcm.go:81:59: header.PayloadOffset undefined (type *rtp.Header has no field or method PayloadOffset)
../../go/pkg/mod/github.com/pion/[email protected]/srtp_cipher_aead_aes_gcm.go:86:35: header.PayloadOffset undefined (type *rtp.Header has no field or method PayloadOffset)
../../go/pkg/mod/github.com/pion/[email protected]/srtp_cipher_aead_aes_gcm.go:87:31: header.PayloadOffset undefined (type *rtp.Header has no field or method PayloadOffset)
../../go/pkg/mod/github.com/pion/[email protected]/srtp_cipher_aes_cm_hmac_sha1.go:117:31: header.PayloadOffset undefined (type *rtp.Header has no field or method PayloadOffset)
../../go/pkg/mod/github.com/pion/[email protected]/srtp_cipher_aes_cm_hmac_sha1.go:122:33: header.PayloadOffset undefined (type *rtp.Header has no field or method PayloadOffset)
../../go/pkg/mod/github.com/pion/[email protected]/stream_srtp.go:81:8: assignment mismatch: 1 variable but header.Unmarshal returns 2 values
../../go/pkg/mod/github.com/pion/[email protected]/stream_srtp.go:81:8: too many errors
# github.com/pion/ice
../../go/pkg/mod/github.com/pion/[email protected]/gather.go:398:21: cannot use a.net (variable of type *"github.com/pion/transport/vnet".Net) as transport.Net value in struct literal: *"github.com/pion/transport/vnet".Net does not implement transport.Net (wrong type for method CreateDialer)
                have CreateDialer(*net.Dialer) "github.com/pion/transport/vnet".Dialer
                want CreateDialer(*net.Dialer) transport.Dialer

I am trying to convert a RTSP stream from some Hikvision camera to webrtc using https://github.com/deepch/RTSPtoWeb . This is successful in converting the rtsp source to a webrtc, and when i open it in my browser it is streaming. Now when I am opening it in iOS it is failing to stream and shows a black screen. Previously I was using https://github.com/deepch/RTSPtoWebRTC to do the exact thing, but some of the camera were unable to stream video, so I tried https://github.com/deepch/RTSPtoWeb to stream and it was successful. It looks like https://github.com/deepch/vdk used for rtsp streaming is having different version used in RTSPtoWeb is different than RTSPtoWebRTC. I tried changing them, but the code was not running.

Thank you for vdk.

Some functions of vdk used to packet []byte. for example) func ParseSliceHeaderFromNALU(packet []byte) (sliceType SliceType, err error)

These functions expect the input arguments to be in the form of an rbsp from the h264 specification.
rbsp is pre-processed packet (emulation prevention byte).

I think I have to remove emulation prevention byte from NALU packet because of NALU is not rbsp.
Therefore, In order to notice It is need to remove emulation prevention byte, I suggest to rename of some functions of parameter that packet -> rbsp

format/webrtc3/adapter.go and format/webrtc/adapter.go assume that the SDP message is base-64 encoded.

func (element *Muxer) WriteHeader(streams []av.CodecData, sdp64 string) (string, error) {
	...
	sdpB, err := base64.StdEncoding.DecodeString(sdp64)

However, I do not think this is required in the specifications.

I see that RTSPtoWebRTC/web/static/js/app.js uses this encoding as well. Perhaps the encoding could be isolated to only the RTSPtoWebRTC project?

If it is not required by the relevant specifications, I think this encoding should be optional / configurable, or removed entirely.

希望rtspv2,添加pcm alaw 和 pcm mulaw的支持

Thanks for the library @deepch it is helping me a lot on a project :)

Would you be open to a PR that adds the ability to send RTCP? I am working on a WebRTC<->RTSP bridge, and would like to forward PLI/NACK to RTSP from WebRTC.

thanks!

format/rtspv2/client.go: 561 in master branch

stack is below:

panic: runtime error: slice bounds out of range [60:56]

goroutine 13411 [running]:
github.com/deepch/vdk/format/rtspv2.(*RTSPClient).RTPDemuxer(0xc00047d2c0, 0xc000310120, 0xc0006f4004, 0x34, 0x34, 0x34)
        /root/go/pkg/mod/github.com/deepch/[email protected]/format/rtspv2/client.go:559 +0x2aaa
github.com/deepch/vdk/format/rtspv2.(*RTSPClient).startStream(0xc00047d2c0)
        /root/go/pkg/mod/github.com/deepch/[email protected]/format/rtspv2/client.go:292 +0x678
created by github.com/deepch/vdk/format/rtspv2.Dial
        /root/go/pkg/mod/github.com/deepch/[email protected]/format/rtspv2/client.go:222 +0x7a8

Is it possible using /example/transcoder/main.go
to keep the h264 video codec and transcode AAC to OPUS ?
I tried this :
cmd := exec.CommandContext(ctx, "ffmpeg", "-flags", "low_delay", "-analyzeduration", "1", "-fflags", "-nobuffer", "-probesize", "1024k", "-f", "mpegts", "-i", "-", "-vcodec", "libx264", "-preset", "ultrafast", "-bf", "0", "-f", "mpegts", "-max_muxing_queue_size", "400", "-pes_payload_size", "0", "-acodec", "libopus", "-b:a", "128k", "pipe:1")

but demuxer error : new codec data [] invalid PMT

Hi,

I'm wondering if it is possible to support OPUS encoded audio with different opus frame sizes.
At the moment RTSPtoWebRTC has issues with anything other than 20ms frame sizes, and i'm wondering if this is hard coded in VDK as per this line in codec.go?

func (self OpusCodecData) PacketDuration(data []byte) (time.Duration, error) {
return time.Duration(20) * time.Millisecond, nil
}

Below are two audio only streams encoded with 20ms and 5ms opus frame sizes. Both streams play fine in ffplay/gst-launch.

rtsp://35.213.221.96:554/test // (20ms)
rtsp://35.213.221.96:554/test_5 // (5ms)

Thanks

vdk/codec/h264parser/parser.go line 230 nalus = append(nalus, _b[:_val4]) Index out of bounds
original:
if val4 <= uint32(len(b)) {
_val4 := val4
_b := b[4:]
nalus := [][]byte{}
for {
nalus = append(nalus, _b[:_val4])
_b = _b[_val4:]
if len(_b) < 4 {
break
}
_val4 = pio.U32BE(_b)
_b = _b[4:]
if _val4 > uint32(len(_b)) {
break
}
}
if len(_b) == 0 {
return nalus, NALU_AVCC
}
}

modify:
if val4 <= uint32(len(b)) {
_val4 := val4
_b := b[4:]
nalus := [][]byte{}
for {
if _val4 > uint32(len(_b)) {
break
}
nalus = append(nalus, _b[:_val4])
_b = _b[_val4:]
if len(_b) < 4 {
break
}
_val4 = pio.U32BE(_b)
_b = _b[4:]
}
if len(_b) == 0 {
return nalus, NALU_AVCC
}
}

push stream to rtmp server. will be error as:

2021/03/10 09:20:50 [alert] 8#8: *10 too big RTMP chunk size:134217728, client: 172.17.0.1, server: 0.0.0.0:1935 2021/03/10 09:21:14 [alert] 8#8: *11 too big RTMP chunk size:134217728, client: 172.17.0.1, server: 0.0.0.0:1935

and connect to play this stream be reject. return EOF.

In PCM_MULAW of commit 981746c

On line 211:

// format/webrtcv3/adapter.go:
    case av.PCM_ALAW:
    case av.OPUS:
    case av.PCM_ALAW: // line 211
    case av.AAC:
// ...

Should be:

// format/webrtcv3/adapter.go:
    case av.PCM_ALAW:
    case av.OPUS:
    case av.PCM_MULAW: // line 211
    case av.AAC:
// ...

It's probably a typo

Hi,
thanks for sharing this code, it is awesome!
I'm try to hack it a bit, but unfortunately i'm not a golang pro coder, so i'm not able to understand how to save the stream (the source as flv or the wp8 generated) and i don't understand if this libs is transcoding the stream (when it is possible and needed) or mp4 rtsp media streams is repackaged to an webrtc container when codec match the ones supported without transcoding (or both).

can you help me to understand which method/library i need to use to store the (source) stream in chunks?

The version I use is [email protected] and the version of go is 1.18.
On the LAN and not available when there is no internet, then I modified /format/webrtcv3/adapter.go

This method

func (element *Muxer) WriteHeader(streams []av.CodecData, sdp64 string) (string, error)

About line 197

	// gatherCompletePromise := webrtc.GatheringCompletePromise(peerConnection)
	answer, err := peerConnection.CreateAnswer(nil)
	if err != nil {
		return "", err
	}
	if err = peerConnection.SetLocalDescription(answer); err != nil {
		return "", err
	}

	element.pc = peerConnection
	// waitT := time.NewTimer(time.Second * 10)
	// select {
	// case <-waitT.C:
	// 	// return "", errors.New("gatherCompletePromise wait")
	// case <-gatherCompletePromise:
	// 	//Connected
	// }
	time.Sleep(time.Millisecond * 1)
	resp := peerConnection.LocalDescription()
	WriteHeaderSuccess = true

	return base64.StdEncoding.EncodeToString([]byte(resp.SDP)), nil

so modify hereafter in lan without the use of the internet, but I don't know what is the cause of this problem, so... can you check what is the reason, and then repair under, thank you.

hope add hls(m3u8) format support

package main

import (
	"fmt"
	"os"
	"time"

	"github.com/deepch/vdk/format/mp4"
	"github.com/deepch/vdk/format/rtspv2"
)

func main() {
	c, err := rtspv2.Dial(rtspv2.RTSPClientOptions{URL: "rtsp://admin:[email protected]:554/LiveMedia/ch1/Media2", DisableAudio: true, DialTimeout: 3 * time.Second, ReadWriteTimeout: 3 * time.Second, Debug: false})
	if err != nil {
		fmt.Println("失败", err)
		return
	}
	defer c.Close()
	fmt.Println("连接成功")
	fmt.Println(c.CodecData[0].Type())

	f, err2 := os.Create("./test.mp4")
	if err2 != nil {
		fmt.Println("创建文件失败", err2)
		return
	}
	defer f.Close()

	m := mp4.NewMuxer(f)
	err = m.WriteHeader(c.CodecData)
	if err != nil {
		fmt.Println("WriteHeader失败", err)
		return
	} else {
		fmt.Println("WriteHeader成功")
	}
	i := 0
	for {
		select {
		case signals := <-c.Signals:
			fmt.Println("信号：", signals)
			switch signals {
			case rtspv2.SignalCodecUpdate:
				fmt.Println("开始")

			case rtspv2.SignalStreamRTPStop:
				return
			}
		case packetAV := <-c.OutgoingPacketQueue:
			fmt.Println("数据：", packetAV.IsKeyFrame, i)
			i++
			err = m.WritePacket(*packetAV)
			if err != nil {
				fmt.Println("WritePacket失败", err)
				return
			} else {
				fmt.Println(packetAV.CompositionTime)
				fmt.Println(packetAV.Duration)
				fmt.Println(packetAV.Idx)
				fmt.Println(packetAV.IsKeyFrame)
				fmt.Println(packetAV.Time)
				fmt.Println("WritePacket成功", len(packetAV.Data))
			}
		}
		if i >= 251 {
			break
		}
	}
	m.WriteTrailer()
	fmt.Println("WriteTrailer成功")
	fmt.Println("结束")

	fmt.Println("")
}

用vlc播放视频文件，看不到画面，但能看到时间。看文件大小，明显不对。是不是代码写得有问题？谢谢了。

When debug messages are enabled in rtspv2.RTSPClientOptions, I see many messages of the form:

2021/04/17 07:10:12 [Unsupported NAL Type 6]

These appear to be Supplemental Enhancement Information (SEI) NAL types for H.264.

Would it be beneficial to simply note in the code that these are not currently supported, instead of logging these messages for NAL Type 6 specifically? Other unsupported NAL types would continue to be logged.

‘avcodec_decode_video2’ is deprecated
‘avresample_open’ is deprecated
‘avresample_get_out_samples’ is deprecated and so on.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update module github.com/pion/interceptor to v0.1.29
• Update module github.com/pion/webrtc/v3 to v3.2.43
• Update module github.com/google/uuid to v1.6.0
• Update actions/checkout action to v4
• Update github/codeql-action action to v3
• Update module github.com/pion/webrtc/v2 to v3
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

when i use hikvision RTSP url in my demo, the following code will cause exception. In my case, when pkt.Idx=1, but len(element.streams) is still 1.

my fix is

if int(pkt.Idx) >= len(element.streams) {
  return false, []byte{}, fmt.Errorf("pkt's Idx is array bound ")
}
stream := element.streams[pkt.Idx]	

and it's work.

type Packet struct {
IsKeyFrame bool // video packet is key frame
Idx int8 // stream index in container format
CompositionTime time.Duration // packet presentation time minus decode time for H264 B-Frame
Time time.Duration // packet decode time
Data []byte // packet data
}

I want to know the Data is a nv12 or yuv data?

The player can render H.265 codec on all browser (Chrome, Edge) except Safari in Mac.
It can just render the very first frame on Safari, and raised InvalidStateError when rendering subsequence frame.

AVPacket can't write to WebRTC directly because of this.

test code

func TestRTMPConnect(t *testing.T) {
	assert := assert.New(t)
	conn, err := rtmpUtil.Dial("some url")
	assert.NoError(err)

	_, err = conn.Streams()
	assert.NoError(err)

	for i := 0; i < 10; i++ {
		pkt, err := conn.ReadPacket()
		assert.NoError(err)
                 // break here
		assert.NotEqual(time.Duration(0), pkt.Duration)
	}
}

Nice project to stream rtsp/rtmp to a browser!

where is DASH code?

ts := uint64(tm * PCR_HZ / time.Second)
ts := uint64(tm / time.Second * PCR_HZ)

tsio.go. func TimeToPCR

These two values are not the same, resulting in pcr error and vlc cannot play

In file included from /usr/local/gopath/pkg/mod/github.com/deepch/[email protected]/cgo/ffmpeg/audio.go:4:
./ffmpeg.h:2:10: fatal error: libavformat/avformat.h: No such file or directory
2 | #include <libavformat/avformat.h>

можно сделать простую проверку на длину
if len(nal) == 0 { return nil, false } naluType := nal[0] & 0x1f

Hello,
I am getting error in demuxer.

demuxer := mkv.NewDemuxer(reader)

`
panic: runtime error: slice bounds out of range [6:4]

goroutine 66 [running]:
format/mkv.(*Demuxer).probe(0xc0000a6000)
format/mkv/demuxer.go:63 +0x45a
format/mkv.(*Demuxer).Streams(0xc0000a6000)
format/mkv/demuxer.go:36 +0x33`

ffmpeg ;
"-c:v", "prores", "-pix_fmt", "yuva444p10le", "-f", "matroska",

using example/transcode/main.go find some h265 not find keyframe , some camera is ok

In format/webrtcv3/adapter.go a data-channel is required to keep the WebRTC connection open.

	peerConnection.OnDataChannel(func(d *webrtc.DataChannel) {
		d.OnMessage(func(msg webrtc.DataChannelMessage) {
			element.ClientACK.Reset(5 * time.Second)
		})
	})

I see RTSPtoWebRTC/web/static/js/app.js this note:

      //send ping becouse PION not handle RTCSessionDescription.close()
      sendChannel = pc.createDataChannel('foo');
      sendChannel.onclose = () => console.log('sendChannel has closed');
      sendChannel.onopen = () => {
        console.log('sendChannel has opened');
        sendChannel.send('ping');
        setInterval(() => {
          sendChannel.send('ping');
        }, 1000)
      }
      sendChannel.onmessage = e => log(`Message from DataChannel '${sendChannel.label}' payload '${e.data}'`);

If this is a bug/limitation in pion/webrtc, it would need to be fixed there first. Is there an issue filed with that project? I could not find one.

Hi,

Here is the logs:

# github.com/deepch/vdk/cgo/ffmpeg
In file included from ../../../../pkg/mod/github.com/deepch/[email protected]/cgo/ffmpeg/audio.go:4:
./ffmpeg.h:2:10: fatal error: libavformat/avformat.h: No such file or directory
    2 | #include <libavformat/avformat.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

While streaming with rtsp url, I got an error on io.ReadFull function. That function was returned io.UnexpectedEOF and packet that caught on wireshark shown "tcp previous segment not captured" perspectively. You can see it at line 265 and 284 of https://github.com/deepch/vdk/blob/master/format/rtspv2/client.go. Hope you reply, thanks!!!

是否可以提供一个h264解码后,抽取一帧转图片的功能

Hi,
I used the RTSPToHls to play H.264 camera.
is it still pull rtsp video stream if i won't to preview it?
the OutgoingPacketQueue will take ~2000M(if one packet is 1M) memory?

client := &RTSPClient{
	headers:             make(map[string]string),
	Signals:             make(chan int, 100),
	OutgoingProxyQueue:  make(chan *[]byte, 3000),
	OutgoingPacketQueue: make(chan *av.Packet, 3000),
	BufferRtpPacket:     bytes.NewBuffer([]byte{}),
	videoID:             -1,
	audioID:             -2,
	videoIDX:            -1,
	audioIDX:            -2,
	options:             options,
	AudioTimeScale:      8000,
}

		pkt, got := client.RTPDemuxer(&content)
		if !got {
			continue
		}
		for _, i2 := range pkt {
			if len(client.OutgoingPacketQueue) > 2000 {
				client.Println("RTSP Client OutgoingPacket Chanel Full")
				return
			}
			client.OutgoingPacketQueue <- i2
		}

rtspv2 can't parse rtsp url with username and password?

for example, "rtsp://admin:pass123##[email protected]:554/cam/realmonitor?channel=1&subtype=1"

thx.

CodecData SPS() index error

src file path

• adapter.go: line:218 codec.SPS() panic. console like this

How can I fix this problem

thank you

Thanks for your great library.
But I didn't find an example how to use it?
I want transcode g711 to aac?

Any examples? Thank you

Hi @deepch，

I'm a new to rtsp protocol user, in my scenario I wanna re-stream audio only from video, audio stream.

I have tried to add option: disableVideo and skipped video track request, but it seems doesn't works. I got nothing from tcp, and a error was found: read tcp i/o timeout.

would you have any suggestion?

Hello,

You've created a cool library.

Do you have plans to implement other protocols (udp, http) than tcp for rtsp?

Thanks for this project!

There's a minor error in format/mp4/muxer.go line 124:

		codec := self.CodecData.(h264parser.CodecData)

Should be:

		codec := self.CodecData.(h265parser.CodecData)