sparanoid / live-dl Goto Github PK

View Code? Open in Web Editor NEW

265.0 20.0 37.0 111 KB

Monitor and download live streams from YouTube

License: GNU Affero General Public License v3.0

Shell 97.66% Dockerfile 0.87% Makefile 0.61% HCL 0.86%

youtube youtube-dl live livestream streamlink ffmpeg mp4 aria2 bash jq

live-dl's Introduction

live-dl

Monitor and download live streams from YouTube.

Features

Monitor your favorite YouTube channels and download streams when it starts
Email/Slack notifications when the stream starts or finish downloading
Writing streamer metadata (author/channel name, description, year) via FFmpeg
Keywords filter: only download streams that match specific keywords (or regular expressions) in the title
Download subtitles if available
Convert TS to MP4 automatically after downloading
URL guessing: this script will try its best to guess what you pass to it, the following URLs/URIs should all work:
- https://www.youtube.com/channel/UC1opHUrw8rvnsadT-iGp7Cg/live
- https://www.youtube.com/channel/UC1opHUrw8rvnsadT-iGp7Cg
- https://www.youtube.com/watch?v=S3CAGeeMRvo
- https://www.youtube.com/playlist?list=UU1opHUrw8rvnsadT-iGp7Cg
- S3CAGeeMRvo
- UC1opHUrw8rvnsadT-iGp7Cg

System Requirements

Tested on macOS up to 11.2.3, should be working on Ubuntu and RHEL. Running live-dl inside a container is recommended.

Available Tags

latest: the latest tagged release
nightly: the nightly build with latest youtube-dl and streamlink bundles

Run `live-dl` Inside a Container with Docker

The simplest way to use live-dl is executing it inside a container. The following command will print the help message of live-dl:

docker run --rm -it sparanoid/live-dl:latest

Run live-dl in interactive mode:

docker run --rm -it --init \
  -v $(pwd)/downloads:/app/downloads \
  sparanoid/live-dl:latest \
  'UC1opHUrw8rvnsadT-iGp7Cg'

Run live-dl as a Docker daemon:

docker run --rm -itd --init \
  -v $(pwd)/downloads:/app/downloads \
  sparanoid/live-dl:latest \
  'UC1opHUrw8rvnsadT-iGp7Cg'

Run live-dl with host volume mounted:

# Mount host volume for custom config.yml:
docker run --rm -itd --init \
  -v $(pwd)/downloads:/app/downloads \
  -v $(pwd)/config.yml:/app/config.yml \
  sparanoid/live-dl:latest \
  'UC1opHUrw8rvnsadT-iGp7Cg'

# Mount host volume for custom cookies.txt:
docker run --rm -itd --init \
  -v $(pwd)/downloads:/app/downloads \
  -v $(pwd)/youtube.com_cookies.txt:/app/cookies.txt \
  sparanoid/live-dl:latest \
  'UC1opHUrw8rvnsadT-iGp7Cg'

You can extract your current cookies simply via Get cookies.txt Google Chrome extension.

Run `live-dl` with Docker Compose

version: '3'

x-defaults: &defaults
  image: sparanoid/live-dl:latest
  restart: always
  volumes:
    - ./config.yml:/app/config.yml
    - ./youtube.com_cookies.txt:/app/cookies.txt
    - ./downloads:/app/downloads

services:
  minatoaqua:
    <<: *defaults
    command: https://www.youtube.com/channel/UC1opHUrw8rvnsadT-iGp7Cg --debug

  uruharushia:
    <<: *defaults
    command: https://www.youtube.com/channel/UCl_gCybOJRIgOXw6Qb4qJzQ --debug

Run `live-dl` Locally

Run ./live-dl without any parameter to print help message.

You can run this script in background with nohup:

# Start process
nohup bash live-dl https://www.youtube.com/channel/UC1opHUrw8rvnsadT-iGp7Cg &>/tmp/live-dl-minatoaqua.log &

# View processes
ps aux | grep live-dl
501 94552   964   0  9:38PM ttys009    0:00.06 bash live-dl https://www.youtube.com/channel/UC1opHUrw8rvnsadT-iGp7Cg
501 94765   964   0  9:39PM ttys009    0:00.00 grep live-dl

# Stop process
kill 94552

Configurations

All configurations are defined in config.yml file. Some options can be overwritten by command line arguments.

Run ./live-dl without any parameter to print the help message.

Contributing

It's recommended to use Docker for development and testing. You should simply mount all your files to the container.

# Method 1: Use compose file
docker compose up [--build]

# Method 2: Use Dockerfile
docker build -t sparanoid/live-dl:local .
docker run -it -v $(pwd):/app sparanoid/live-dl:local

If you'd like to run it locally. You need to install all the dependencies defined in live-dl.

License

AGPL-3.0

live-dl's People

Contributors

Stargazers

Watchers

live-dl's Issues

Non-supported youtube url

yea, with youtube's recent updates with handles and all, it kinda broke the functionality of this auto downloader. just needs some URL path changes to the script

youtube-dl fails second live check

I know the stream is live and the first check successfully finds the stream and everything, however youtube-dl throws an error on the second check. Also, this does not happen on every stream. For example, the nasa livestream works. But other streams also fail this same way, for example the lofi beats 24/7 channel.

I am running this inside a docker container with docker compose

I only modified the config to include a discord webhook

Use youtube-dlc instead?

I think you should take a look at youtube-dlc.
It's a newer version of youtube-dl and works with playlists.

https://github.com/blackjack4494/yt-dlc

Multiple channels?

Is monitoring multiple channels something that is supported?
If it is, how is it meant to be done?
At the moment, I just launch 4 instances of live-dl with a large sleep interval and staggered by some time to not spam YouTube.
I would like to monitor more channels, but I'd rather not have my IP banned, and what I'm currently doing seems unsafe as it is, haha.
Is there no feasible way to monitor a bunch of channels with a single instance?

yt-dlp --live-from-start option error

When I modify config.yml file to use yt-dlp and add option argument for --live-from-start, it gives me an error message as follows:

yt-dlp: error: no such option: --live-from-start

Any idea why this option is not available?

Purpose of the map section in the config file

Thanks a lot for this simple yet amazing tool, it looks like it's exactly what I was looking for and you did it POSIX only.

I just have a quick question regarding the config file and its map section.
What is the purpose of declaring channel specific config for a many channels in each container when only one is used per instance?
Wouldn't we be able to state the extra bits in each command in the yml?
Or am I misunderstanding and there's a way to monitor more than one channel in a single instance?

Cheers

Error when attempting to download 404

i have run this successfully with yt-dlp which is a fork of youtube-dl but now im running into an issue that i cannot debug myself:
Here is my log.

https://pastebin.com/7TT8T8n1

im unable to find out why it worked on some live streams but not others.
Due note this error also comes up when i use youtube-dl

this looks to be a streamlink problem as if i switch to youtube-dl to hand the stream it works.

i am running this locally and its only a bit modified but running a fresh one from git clone i get the same error i am unable to bypass this so far.

i was hoping to use this tool to download archived streams but it seems like i cant.

[Fri 18 Jun 2021 08:24:08 PM EDT] Stop downloading
[Fri 18 Jun 2021 08:24:08 PM EDT] File summary not available due to missing files.

i get this when i run ./live-dl $CHANNEL

not sure what files are missing.

parse error after "Re-checking via yt-dlp..."

It seems there is an error while the JSON is parsed:

[Wed Apr 13 18:17:37 UTC 2022] �[1;31;33mUpcoming event detected: 1649873700�[0m
[Wed Apr 13 18:18:09 UTC 2022] �[1;31;33mUpcoming event detected: 1649873700�[0m
[Wed Apr 13 18:18:40 UTC 2022] cURL check seems goes live now, continue checking...
[Wed Apr 13 18:18:40 UTC 2022] Re-checking via yt-dlp...
parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 2, column 10

parse error: Invalid numeric literal at line 1, column 25453

./downloads/xxx/20220413 xxx.jpg: No Exif data found in the file
[Wed Apr 13 18:18:45 UTC 2022] �[1;31;32mPrepare downloading...�[0m
ID: xxx
Title: xxx 2022-04-13 18:18
URL: https://www.youtube.com/watch?v=xxx
Author: xxx
Channel: https://www.youtube.com/channel/xxx
Date: 20220413
Thumbnail: https://i.ytimg.com/vi/xxx/maxresdefault.jpg
Filename: 20220413 xxx (xxx)
Location: ./downloads/xxx/
Hostname: 1e214fed04ae

The title of the video has Greek characters. Could that be breaking something? I'm using sparanoid/live-dl:latest.

Fails to download live music video

I'm using the following and it's stopping for an unknown reason: docker run --rm -it -v $(pwd)/downloads:/opt/live-dl/downloads sparanoid/live-dl:latest "MSMjU5Z9_Ug"

docker run --rm -it -v $(pwd)/downloads:/opt/live-dl/downloads sparanoid/live-dl:latest "MSMjU5Z9_Ug"
[Thu Sep 16 13:30:16 UTC 2021] URL (URI) to process           : MSMjU5Z9_Ug
[Thu Sep 16 13:30:16 UTC 2021] Mode                           : download
[Thu Sep 16 13:30:16 UTC 2021] Output base dir                : ./downloads
[Thu Sep 16 13:30:16 UTC 2021] Run interval (when loop)       : 10
[Thu Sep 16 13:30:16 UTC 2021] Use youtube-dl for streams     : false
[Thu Sep 16 13:30:16 UTC 2021] Skip convert                   : false
[Thu Sep 16 13:30:16 UTC 2021] Skip embedding metadata        : false
[Thu Sep 16 13:30:16 UTC 2021] Skip email notification        : false
[Thu Sep 16 13:30:16 UTC 2021] Skip Slack notification        : false
[Thu Sep 16 13:30:16 UTC 2021] Skip Telegram notification     : false
[Thu Sep 16 13:30:16 UTC 2021] Skip Discord notification      : 
[Thu Sep 16 13:30:16 UTC 2021] Email provider                 : ses
[Thu Sep 16 13:30:16 UTC 2021] One-time execution             : false
[Thu Sep 16 13:30:16 UTC 2021] URI detected, guessing... (Use full URL to avoid guessing URL)
[Thu Sep 16 13:30:16 UTC 2021] YouTube video ID detected
[Thu Sep 16 13:30:16 UTC 2021] Checking https://www.youtube.com/watch?v=MSMjU5Z9_Ug...
parse error: Invalid numeric literal at line 2, column 17
[Thu Sep 16 13:30:17 UTC 2021] cURL check seems goes live now, continue checking...
[Thu Sep 16 13:30:17 UTC 2021] Re-checking via youtube-dl...
./downloads/PMB Music/20210826 Background Music - Relaxing Covers 🍹 2021-09-16 13_30 (MSMjU5Z9_Ug).jpg: No Exif data found in the file
[Thu Sep 16 13:30:20 UTC 2021] Prepare downloading...

ID:        MSMjU5Z9_Ug
Title:     Background Music - Relaxing Covers 🍹 2021-09-16 13:30
URL:       https://www.youtube.com/watch?v=MSMjU5Z9_Ug
Author:    PMB Music
Channel:   https://www.youtube.com/channel/UCFm4zM6-08kb9abXDk4fxXQ
Date:      20210826
Thumbnail: https://i.ytimg.com/vi/MSMjU5Z9_Ug/maxresdefault.jpg?v=613a630b
Filename:  20210826 Background Music - Relaxing Covers 🍹 2021-09-16 13_30 (MSMjU5Z9_Ug)
Location:  ./downloads/PMB Music/
Hostname:  03abd7c24db3

[Thu Sep 16 13:30:20 UTC 2021] Start downloading 20210826 Background Music - Relaxing Covers 🍹 2021-09-16 13_30 (MSMjU5Z9_Ug)
[Thu Sep 16 13:30:20 UTC 2021] Logging: tail -f "./downloads/PMB Music/20210826 Background Music - Relaxing Covers 🍹 2021-09-16 13_30 (MSMjU5Z9_Ug).log"
[Thu Sep 16 13:30:21 UTC 2021] Stop downloading
[Thu Sep 16 13:30:21 UTC 2021] File summary not available due to missing files.

Record at Constant Frame Rate

Is there any way to record at a constant frame rate, instead of variable? Files are in variable frame rates, making them impossible to work with in editing software. Transcoding videos to constant takes far too long, wondering if there's a way to just capture them at constant in the first place, even if it sacrifices quality or whatever. Thanks.

Windows Build

Please consider a Windows build.

yt-dlp option -o has to be without extension

live-dl/live-dl

Line 1289 in a16f899

$YTDL_ARGS -o "$OUTPUT_PATH.ts" \

$YTDL_ARGS -o "$OUTPUT_PATH"

live-dl/live-dl

Line 1329 in a16f899

$YTDL_ARGS -o "$OUTPUT_PATH.ts" \

$YTDL_ARGS -o "$OUTPUT_PATH"

live-dl/live-dl

Line 1343 in a16f899

$YTDL_ARGS -o "$OUTPUT_PATH.mp4" \

$YTDL_ARGS -o "$OUTPUT_PATH"

Unescaped left brance in regex

Whenever i run it with any video, this shows up repeatedly:

Unescaped left brace in regex is deprecated here (and will be fatal in Perl 5.32), passed through in regex; marked by <-- HERE in m/^.*?ytInitialPlayerResponse = ({ <-- HERE ".*]});var .*/ at -e line 1.
parse error: Invalid numeric literal at line 1, column 17
[Thu Jan 21 05:57:52 CET 2021] Not a valid page, something wrong with your internet connection?

yq Error on startup

I'm using ubuntu (64x) on rpi. I've initialized the script and installed all the dependences.

[Mon Dec 20 00:36:34 CET 2021] Checking dependencies... [Mon Dec 20 00:36:34 CET 2021] Found: ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers [Mon Dec 20 00:36:37 CET 2021] Found: yt-dlp 2021.12.01 [Mon Dec 20 00:36:37 CET 2021] Found: jq-1.6 [Mon Dec 20 00:36:38 CET 2021] Found: yq 2.13.0 [Mon Dec 20 00:36:38 CET 2021] Found: exiv2 0.27.2 [Mon Dec 20 00:36:38 CET 2021] Found: aria2 version 1.35.0 [Mon Dec 20 00:36:40 CET 2021] Found: streamlink 2.4.0

After executing this script with an url this error message is shown:

yq: Error running jq: ParserError: while parsing a block mapping in "<stdin>", line 7, column 3 did not find expected key in "<stdin>", line 97, column 3.
Further more, this shows up:

Try 'sleep --help' for more information. sleep: missing operand Try 'sleep --help' for more information. sleep: missing operand Try 'sleep --help' for more information. sleep: missing operand Try 'sleep --help' for more information. sleep: missing operand Try 'sleep --help' for more information. sleep: missing operand Try 'sleep --help' for more information. sleep: missing operand Try 'sleep --help' for more information.

Support Youtube Cookies

Hi,
I want to monitor/download membership stream or over 18 years olds stream (some game stream need it) on Youtube. I see you are using streamlink to download "status live", but streamlink is not support cookies like youtube-dl (I'm using youtube-dl --cookies /cookies.txt to download live.ts).
Do you really need to use streamlink to download live stream?

Thank you for creating this live-dl.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

chore(deps): update python docker tag to v3.12.5
chore(deps): update actions/checkout action to v4
chore(deps): update docker/bake-action action to v5
chore(deps): update docker/login-action action to v3
chore(deps): update docker/metadata-action action to v5
chore(deps): update docker/setup-buildx-action action to v3
chore(deps): update docker/setup-qemu-action action to v3
Click on this checkbox to rebase all open PRs at once

Detected dependencies

docker-compose

docker-compose.yml

dockerfile

Dockerfile

python 3.8.5-alpine

github-actions

.github/workflows/build.yml

actions/checkout v2

docker/setup-qemu-action v1

docker/setup-buildx-action v1

docker/login-action v1

docker/login-action v1

docker/metadata-action v3

docker/bake-action v1

Check this box to trigger a request for Renovate to run again on this repository

How to set the Video Quality

I don't see the option to set the video quality in the config.template.

Failed to start container on latest version

docker-compose.yml:

version: '3'

x-defaults: &defaults
  image: sparanoid/live-dl:latest
  restart: always
  volumes:
    - /volume2/NAS/live-dl/config.yml:/app/config.yml
    - /volume2/NAS/live-dl/youtube.com_cookies.txt:/app/cookies.txt
    - /volume2/NAS/Downloads/live-dl:/app/downloads

services:
  minatoaqua:
    <<: *defaults
    command: https://www.youtube.com/channel/UC1opHUrw8rvnsadT-iGp7Cg --debug

docker-compose logs

root@Alex-NAS:/volume2/NAS/live-dl# docker-compose logs
Attaching to live-dl_minatoaqua_1
minatoaqua_1  | standard_init_linux.go:230: exec user process caused: no such file or directory
minatoaqua_1  | standard_init_linux.go:230: exec user process caused: no such file or directory
minatoaqua_1  | standard_init_linux.go:230: exec user process caused: no such file or directory

It just kept restarting and didn't print any other error logs.

btw, how can I add a twicasting recording task by editing the docker-compose file?

Unbound variable SKIP_ALIAS

Variable SKIP_ALIAS is set on L1358, however it's used on L317 via func_check_deps() on L1277.

Thus variable SKIP_ALIAS is unset when --init is on.

As a result, the alias is always added to the shell profile, even if --skip-alias is specified.

Callback function

I want the video being uploaded to my GDrive after downloaded.
Is it possible to call a callback after the download has finished?
Example: live-dl $URL --callback "./upload.sh {CHANNEL_ID} {FILENAME}"

Thanks!

Live-dl probably fried my sd-card ... twice

Hi there,
i love the project so far but recently, my favourite streamer who streams mostly when I'm not home, my 2 sd cards died after the stream ended, also my CPU is going through the roof during downloading/converting.

I'm using Docker on a Raspberry PI 8gb with Ubuntu 21.10.

I don't know if the two factors are correlating but maybe put a Disclosure in the ReadMe so it wont happen to somebody else.

Download did not start

I experienced trouble that the download did not begin when the live stream has started. Instead, there was always the same error message. I was able to download with another tool then.

[Wed Apr 21 18:02:46 UTC 2021] URL (URI) to process           : https://www.youtube.com/watch?v=bvxA9yzUpSo
[Wed Apr 21 18:02:46 UTC 2021] Mode                           : download
[Wed Apr 21 18:02:46 UTC 2021] Output base dir                : ./downloads
[Wed Apr 21 18:02:46 UTC 2021] Run interval (when loop)       : 10
[Wed Apr 21 18:02:46 UTC 2021] Use youtube-dl for streams     : false
[Wed Apr 21 18:02:46 UTC 2021] Skip convert                   : false
[Wed Apr 21 18:02:46 UTC 2021] Skip embedding metadata        : false
[Wed Apr 21 18:02:46 UTC 2021] Skip email notification        : false
[Wed Apr 21 18:02:46 UTC 2021] Skip Slack notification        : false
[Wed Apr 21 18:02:46 UTC 2021] Skip Telegram notification     : false
[Wed Apr 21 18:02:46 UTC 2021] Email provider                 : ses
[Wed Apr 21 18:02:46 UTC 2021] One-time execution             : false
[Wed Apr 21 18:02:46 UTC 2021] YouTube URL detected
[Wed Apr 21 18:02:46 UTC 2021] YouTube video detected
[Wed Apr 21 18:02:46 UTC 2021] Checking https://www.youtube.com/watch?v=bvxA9yzUpSo...
[Wed Apr 21 18:02:46 UTC 2021] DEBUG: start func_check_state
[Wed Apr 21 18:02:46 UTC 2021] Not a valid page, something wrong with your internet connection?
[Wed Apr 21 18:02:46 UTC 2021] DEBUG: Not valid at the moment, continue running
[Wed Apr 21 18:02:46 UTC 2021] DEBUG: Stream unavailable, retry in 10s...
[Wed Apr 21 18:02:56 UTC 2021] DEBUG: start func_check_state
[Wed Apr 21 18:02:56 UTC 2021] Not a valid page, something wrong with your internet connection?
[Wed Apr 21 18:02:56 UTC 2021] DEBUG: Not valid at the moment, continue running
[Wed Apr 21 18:02:56 UTC 2021] DEBUG: Stream unavailable, retry in 10s...
[Wed Apr 21 18:03:06 UTC 2021] DEBUG: start func_check_state
[Wed Apr 21 18:03:06 UTC 2021] Not a valid page, something wrong with your internet connection?
[Wed Apr 21 18:03:06 UTC 2021] DEBUG: Not valid at the moment, continue running
[Wed Apr 21 18:03:06 UTC 2021] DEBUG: Stream unavailable, retry in 10s...
[Wed Apr 21 18:03:16 UTC 2021] DEBUG: start func_check_state
[Wed Apr 21 18:03:17 UTC 2021] Not a valid page, something wrong with your internet connection?
[Wed Apr 21 18:03:17 UTC 2021] DEBUG: Not valid at the moment, continue running
[Wed Apr 21 18:03:17 UTC 2021] DEBUG: Stream unavailable, retry in 10s...

[Solution] restream to multiple destination instead of download

Hi, how to pass custom ffmpeg to docker container to start multi streaming to various platforms such as facebook, youtube, twitch etc.

I created following script and it works well when I do it manually. ~~I am looking for add channel monitoring ability, so that whole thing can be automated without needing to start manually start live.~~

Update:
I was able to get .m3u8 file URL from live-dl output which is used as input for ffmpeg command to start live stream on channels configured into script with stream key.

AUTO_YT_VIDEO_ID=$(/bin/bash /home/USER/live-dl/live-dl --skip-metadata -1 https://www.youtube.com/channel/UC9OM-qeiYIPtAkBe9veG5uw | grep '.m3u8' | head -1 | awk '{print $NF}') #Replace channel URL with your preference to capture Live URL as soon as channel goes live. This command runs only once to avoid banning by YouTube, feel free to change.

#YT_SOURCE_STREAM=https://www.youtube.com/watch?v=vfNy33eoS_U # OR use existing video link and replace ffmpeg -i variable
#VIDEO_SOURCE="$(youtube-dl -f best -g $YT_SOURCE_STREAM)" #only if you want to stream video from exiting channel
VBR="4.5M"
FPS="30"
QUAL="ultrafast"
BUFF="8M"
AUDIO_ENCODER="aac"

# Stream Keys and URLs
RESTREAM_URL="rtmp://live.restream.io/live"
FACEBOOK_URL="rtmps://live-api-s.facebook.com:443/rtmp/"
YOUTUBE_URL="rtmp://a.rtmp.youtube.com/live2"
CHANNE1_RESTREAM_KEY="key-key-key1"
CHANNE2_RESTREAM_KEY="key2"
CHANNE3_RESTREAM_KEY="key3"
PAGE1_FACEBOOK_KEY="fb-key1"
PAGE2_FACEBOOK_KEY="fb-key2"

## FFMPEG
ffmpeg   -re   -v ${$2:-info}   -i "${AUTO_YT_VIDEO_ID}"   -map 0 -flags +global_header -c:v libx264 -preset $QUAL -r $FPS -g $(($FPS*2)) -b:v $VBR -maxrate:6 4.8M   -c:a $AUDIO_ENCODER -threads 6 -ar 44100 -b:a 128k -flags +global_header -bsf:a aac_adtstoasc -bufsize $BUFF -pix_fmt yuv420p   -flags +global_header   -f tee    "[f=flv:onfail=ignore]$RESTREAM_URL/$CHANNE1_RESTREAM_KEY|[f=flv:onfail=ignore]$RESTREAM_URL/$CHANNE2_RESTREAM_KEY|[f=flv:onfail=ignore]$RESTREAM_URL/$CHANNE3_RESTREAM_KEY|[f=flv:onfail=ignore]$FACEBOOK_URL/$PAGE1_FACEBOOK_KEY|[f=flv:onfail=ignore]$FACEBOOK_URL/$PAGE2_FACEBOOK_KEY|

Thank you for your time and sharing..

skip_convert flag is unsed

Flag skip_convert is read from config file and saved to SKIP_CONVERT variable but isn't used for func_finalize_download function

Passing cookie to monitor Youtube page

Would it be possible to pass a (youtube) cookie to curl in order to monitor potential private live streams (restricted to paying members for example)?

From the git logs I noticed there was a "cookie" argument at some point but got removed?

I know it's possible to pass cookie to youtube-dl from #3 but is it supposed to work for monitoring?

Sorry if I sound a bit confused, just tried fiddling with the script but got nowhere so far, my $METADATA_CURL_RAW is empty due to some parsing error. Still investigating.

Edit: actually my question probably entails more work than I anticipated, since the private streams never appear on the channel/live endpoint, only in the channel/community one.

It occurs to me that youtube-dl should be able to do that already, especially by using the download-archive feature which keeps track of new videos from a playlist.

Time format of upcoming event is wrong

live-dl/live-dl

Line 561 in a16f899

__info "$(__yellow "Upcoming event detected: $UPCOMING_TIME")"

$_date_calc is missing

__info "$(__yellow "Upcoming event detected: $_date_calc ($UPCOMING_TIME)")"

YouTube Stream Stops Downloading After 6 Hours

I just encountered by first issue using this script when a YouTube stream that was being recorded exceeded 6 hours.

The stream was 6 hours 17 minutes long, the recorded file produced by this script was only 17 minutes long. After some investigation I found that they YouTube URL is only valid for 6 hours (streamlink/streamlink#1380). According to the log file produced by live-dl the 6 hour vod successfully recorded, but was then overwritten by the latter 17 minute recording, resulting in the first 6 hours being lost.

Would there be any way of incorporating into the script to handle these cases? For now, I have just replaced the -y param for the ffmpeg call to -n to prevent overwrites but this isn't the best solution.

[Feature Request] Add support for inheriting user and group permissions

When running this in docker(-compose), it natively runs as root. This can cause some pretty unfortunate permissions issues when trying to later interact with those files as it will even override ACLs and setgid bits.

Could this be adapted to inherit a user and group id to avoid this? Spent some time hunting down ways of doing it, but its a bit over my head to push it over the finish line.

The method used by linuxserver.io is likely way to complex to implement and relies on their base images and another package when manages some of the permissions. I wasn't quite able to figure out how. https://github.com/just-containers/s6-overlay

Another Youtube-DL package which doesnt have any live or scheduling functions is a bit simpler, but is implemented in Bash before executing python. https://github.com/mikenye/docker-youtube-dl/blob/master/init#L44-L73

I think that something like this might work in your existing dockerfile, but have not been able to test it. Blatantly stolen from another dockerfile somewhere.

RUN \
 groupmod -g $PGID users && \
 useradd -u $PUID -U -d /config -s /bin/false abc && \
 usermod -G users abc
USER abc

PGID and PUID would need to be set as environment variables to the container in something like this:

x-defaults: &defaults
  image: sparanoid/live-dl:latest
  restart: always
  environment:
      - PUID=${PUID}
      - PGID=${PGID}
      - TZ=${TZ}
  volumes:
    - ${DOCKER_APPDATA}/live-dl/config.yml:/opt/live-dl/config.yml
    - ${DOCKER_APPDATA}/live-dl/youtube.com_cookies.txt:/opt/live-dl/cookies.txt
    - /Torrents/live-dl:/opt/live-dl/downloads

If you've got a better way, please do.

Download does not attempt to convert and embed metadata

See logs below. Download completes, all the files are in place, but it does not remux to mp4 or embed the metadata.

Stream is not a scheduled one, it just goes live and then it is taken down immediately after for several days.
Not trying to send any notifications to anything, just to download it live.

[Fri Nov  5 01:48:14 UTC 2021] URL (URI) to process           : https://www.youtube.com/channel/UCpXBGqwsBkpvcYjsJBQ7LEQ
[Fri Nov  5 01:48:14 UTC 2021] Mode                           : download
[Fri Nov  5 01:48:14 UTC 2021] Output base dir                : /opt/live-dl/downloads
[Fri Nov  5 01:48:14 UTC 2021] Run interval (when loop)       : 300
[Fri Nov  5 01:48:14 UTC 2021] Use youtube-dl for streams     : false
[Fri Nov  5 01:48:14 UTC 2021] Skip convert                   : false
[Fri Nov  5 01:48:14 UTC 2021] Skip embedding metadata        : false
[Fri Nov  5 01:48:14 UTC 2021] Skip email notification        : true
[Fri Nov  5 01:48:14 UTC 2021] Skip Slack notification        : true
[Fri Nov  5 01:48:14 UTC 2021] Skip Telegram notification     : true
[Fri Nov  5 01:48:14 UTC 2021] Skip Discord notification      :
[Fri Nov  5 01:48:14 UTC 2021] Email provider                 :
[Fri Nov  5 01:48:14 UTC 2021] One-time execution             : false
[Fri Nov  5 01:48:14 UTC 2021] YouTube URL detected
[Fri Nov  5 01:48:14 UTC 2021] YouTube video detected (alt)
[Fri Nov  5 01:48:14 UTC 2021] Checking https://www.youtube.com/channel/UCpXBGqwsBkpvcYjsJBQ7LEQ/live...
[Fri Nov  5 01:48:14 UTC 2021] DEBUG: start func_check_state
[Fri Nov  5 01:48:14 UTC 2021] DEBUG: Got valid cURL metadata
[Fri Nov  5 01:48:15 UTC 2021] DEBUG: Is live: true
[Fri Nov  5 01:48:15 UTC 2021] DEBUG: Playability: OK
[Fri Nov  5 01:48:15 UTC 2021] DEBUG: Playability Reason:
[Fri Nov  5 01:48:15 UTC 2021] DEBUG: Streamability ID: sgK82qYoFW4
[Fri Nov  5 01:48:15 UTC 2021] DEBUG: Channel: Critical Role
[Fri Nov  5 01:48:15 UTC 2021] DEBUG: Title: Critical Role Campaign 3, Episode 3 | Livestream
[Fri Nov  5 01:48:15 UTC 2021] cURL check seems goes live now, continue checking...
[Fri Nov  5 01:48:15 UTC 2021] DEBUG: User filter found: Critical Role | Campaign 3. Checking if video title matches...
[Fri Nov  5 01:48:15 UTC 2021] Video title matched, mark as downloadable...
[Fri Nov  5 01:48:15 UTC 2021] DEBUG: Current state for first check: live
[Fri Nov  5 01:48:15 UTC 2021] Re-checking via youtube-dl...
[Fri Nov  5 01:48:16 UTC 2021] DEBUG: Got valid youtube-dl metadata
jq: error (at <stdin>:23): Cannot iterate over null (null)
[Fri Nov  5 01:48:17 UTC 2021] DEBUG: Channel name mapping not found, consider adding it in config.yml
[Fri Nov  5 01:48:17 UTC 2021] DEBUG: I will use Critical Role for folder name, embedding metadata, etc.
[Fri Nov  5 01:48:17 UTC 2021] DEBUG: Telegram channel not found, consider adding it in config.yml
/opt/live-dl/downloads/Critical Role/20211028 Critical Role Campaign 3, Episode 3 _ Livestream 2021-11-05 01_48 (sgK82qYoFW4).jpg: No Exif data found in the file
[Fri Nov  5 01:48:17 UTC 2021] DEBUG: Got state: live
[Fri Nov  5 01:48:17 UTC 2021] DEBUG: Current state for second check: live
[Fri Nov  5 01:48:17 UTC 2021] Prepare downloading...
[Fri Nov  5 01:48:17 UTC 2021] DEBUG: Download platform: YouTube
[Fri Nov  5 01:48:17 UTC 2021] DEBUG: Download type: channel
[Fri Nov  5 01:48:17 UTC 2021] DEBUG: Download mode: download

ID:        sgK82qYoFW4
Title:     Critical Role Campaign 3, Episode 3 | Livestream 2021-11-05 01:48
URL:       https://www.youtube.com/watch?v=sgK82qYoFW4
Author:    Critical Role
Channel:   https://www.youtube.com/channel/UCpXBGqwsBkpvcYjsJBQ7LEQ
Date:      20211028
Thumbnail: https://i.ytimg.com/vi/sgK82qYoFW4/maxresdefault.jpg?v=617b9134
Filename:  20211028 Critical Role Campaign 3, Episode 3 _ Livestream 2021-11-05 01_48 (sgK82qYoFW4)
Location:  /opt/live-dl/downloads/Critical Role/
Hostname:  044461897c81

[Fri Nov  5 01:48:17 UTC 2021] No Discord credentials found, skip sending Discord notification
[Fri Nov  5 01:48:17 UTC 2021] Start downloading 20211028 Critical Role Campaign 3, Episode 3 _ Livestream 2021-11-05 01_48 (sgK82qYoFW4)
[Fri Nov  5 01:48:17 UTC 2021] Logging: tail -f "/opt/live-dl/downloads/Critical Role/20211028 Critical Role Campaign 3, Episode 3 _ Livestream 2021-11-05 01_48 (sgK82qYoFW4).log"
[Fri Nov  5 06:14:39 UTC 2021] Stop downloading

Duration:  04:25:20
Size:      2.6GiB
Path:      044461897c81:/opt/live-dl/downloads/Critical Role/20211028 Critical Role Campaign 3, Episode 3 _ Livestream 2021-11-05 01_48 (sgK82qYoFW4).ts

[Fri Nov  5 06:14:49 UTC 2021] No Discord credentials found, skip sending Discord notification
[Fri Nov  5 06:14:49 UTC 2021] Checking https://www.youtube.com/channel/UCpXBGqwsBkpvcYjsJBQ7LEQ/live...
[Fri Nov  5 06:14:49 UTC 2021] DEBUG: start func_check_state
[Fri Nov  5 06:14:50 UTC 2021] Not a valid page, something wrong with your internet connection?
[Fri Nov  5 06:14:50 UTC 2021] DEBUG: Not valid at the moment, continue running
[Fri Nov  5 06:14:50 UTC 2021] DEBUG: Stream unavailable, retry in 300s...

-

Fail to start with "Not a valid page"

It seems that YouTube changed the page for some channels. Trying with this channel fails with the following:

[Thu Oct 21 13:58:38 UTC 2021] URL (URI) to process           : UCJrwvP4Hgkq2_YJB8veP-4w
[Thu Oct 21 13:58:38 UTC 2021] Mode                           : download
[Thu Oct 21 13:58:38 UTC 2021] Output base dir                : ./downloads
[Thu Oct 21 13:58:38 UTC 2021] Run interval (when loop)       : 10
[Thu Oct 21 13:58:38 UTC 2021] Use youtube-dl for streams     : false
[Thu Oct 21 13:58:38 UTC 2021] Skip convert                   : false
[Thu Oct 21 13:58:38 UTC 2021] Skip embedding metadata        : false
[Thu Oct 21 13:58:38 UTC 2021] Skip email notification        : false
[Thu Oct 21 13:58:38 UTC 2021] Skip Slack notification        : false
[Thu Oct 21 13:58:38 UTC 2021] Skip Telegram notification     : false
[Thu Oct 21 13:58:38 UTC 2021] Skip Discord notification      :
[Thu Oct 21 13:58:38 UTC 2021] Email provider                 : ses
[Thu Oct 21 13:58:38 UTC 2021] One-time execution             : false
[Thu Oct 21 13:58:38 UTC 2021] URI detected, guessing... (Use full URL to avoid guessing URL)
[Thu Oct 21 13:58:38 UTC 2021] YouTube channel ID detected
[Thu Oct 21 13:58:38 UTC 2021] Checking https://www.youtube.com/channel/UCJrwvP4Hgkq2_YJB8veP-4w/live...
[Thu Oct 21 13:58:39 UTC 2021] Not a valid page, something wrong with your internet connection?
[Thu Oct 21 13:58:49 UTC 2021] Not a valid page, something wrong with your internet connection?

I've also tried merging this PR but didn't have any luck. Using a custom built image with python:3.8.12-alpine.

Testing with:

docker run --rm -it live-dl:latest 'UCJrwvP4Hgkq2_YJB8veP-4w'

Unknown mode, check your config

Keeps saying unknown mode. Log below.

Config is an exact copy of the example one

LOG:
~/live/live-dl$ sudo bash live-dl https://www.youtube.com/channel/UC-hM6YJuNYVAmUWxeIr9FeA/live
live-dl: line 97: cd: live-dl: Not a directory
live-dl: line 1540: yq: command not found
[Sat Nov 6 15:00:05 +08 2021] URL (URI) to process : https://www.youtube.com/channel/UC-hM6YJuNYVAmUWxeIr9FeA/live
[Sat Nov 6 15:00:05 +08 2021] Mode :
[Sat Nov 6 15:00:05 +08 2021] Output base dir :
[Sat Nov 6 15:00:05 +08 2021] Run interval (when loop) :
[Sat Nov 6 15:00:05 +08 2021] Use youtube-dl for streams :
[Sat Nov 6 15:00:05 +08 2021] Skip convert :
[Sat Nov 6 15:00:05 +08 2021] Skip embedding metadata :
[Sat Nov 6 15:00:05 +08 2021] Skip email notification :
[Sat Nov 6 15:00:05 +08 2021] Skip Slack notification :
[Sat Nov 6 15:00:05 +08 2021] Skip Telegram notification :
[Sat Nov 6 15:00:05 +08 2021] Skip Discord notification :
[Sat Nov 6 15:00:05 +08 2021] Email provider :
[Sat Nov 6 15:00:05 +08 2021] One-time execution :
[Sat Nov 6 15:00:05 +08 2021] YouTube URL detected
[Sat Nov 6 15:00:05 +08 2021] YouTube channel detected
[Sat Nov 6 15:00:05 +08 2021] Checking https://www.youtube.com/channel/UC-hM6YJuNYVAmUWxeIr9FeA/live...
[Sat Nov 6 15:00:06 +08 2021] cURL check seems goes live now, continue checking...
[Sat Nov 6 15:00:06 +08 2021] Re-checking via youtube-dl...
/Miko Ch. さくらみこ/20211104 【 #ホロライブ大**会2021 】ホロライブ集結‼秋のマイクラ**会🏳 - 主催者視点【Minecraft_さくらみこ】 2021-11-06 15_00 (wks67WHSGDw).jpg: No Exif data found in the file
[Sat Nov 6 15:00:08 +08 2021] Prepare downloading...

ID: wks67WHSGDw
Title: 【 #ホロライブ大**会2021 】ホロライブ集結‼秋のマイクラ**会🏳 - 主催者視点【Minecraft/さくらみこ】 2021-11-06 15:00
URL: https://www.youtube.com/watch?v=wks67WHSGDw
Author: Miko Ch. さくらみこ
Channel: https://www.youtube.com/channel/UC-hM6YJuNYVAmUWxeIr9FeA
Date: 20211104
Thumbnail: https://i.ytimg.com/vi/wks67WHSGDw/maxresdefault.jpg?v=61848ae8
Filename: 20211104 【 #ホロライブ大**会2021 】ホロライブ集結‼秋のマイクラ運動会🏳 - 主催者視点【Minecraft_さくらみこ】 2021-11-06 15_00 (wks67WHSGDw)
Location: /Miko Ch. さくらみこ/
Hostname: xxxx(removed on purpose)

Unknown mode, check your config

func_check_state() METADATA_CURL parsing failed

Hello, I want to report the error, this program has not worked since yesterday. Either the channel url or the video url will not work.

This is the log that appears when the live broadcast starts.

I'm almost sure that the regex replacement here failed. It may be that Youtube changed the page.
I cut a simple script to verify this.

I am not familiar with regex and bash, I found the problem but not able to fix it.
Hope you can handle it, thank you.

parse error: Invalid numeric literal at line 2, column 17 -p destination: Broken pipe

[Thu Aug 12 03:15:43 IST 2021] URL (URI) to process           : https://www.youtube.com/channel/UC9OM-qeiYIPtAkBe9veG5uw
[Thu Aug 12 03:15:43 IST 2021] Mode                           : notifier
[Thu Aug 12 03:15:43 IST 2021] Output base dir                : ./downloads
[Thu Aug 12 03:15:43 IST 2021] Run interval (when loop)       : 30
[Thu Aug 12 03:15:43 IST 2021] Use youtube-dl for streams     : true
[Thu Aug 12 03:15:43 IST 2021] Skip convert                   : true
[Thu Aug 12 03:15:43 IST 2021] Skip embedding metadata        : true
[Thu Aug 12 03:15:43 IST 2021] Skip email notification        : true
[Thu Aug 12 03:15:43 IST 2021] Skip Slack notification        : true
[Thu Aug 12 03:15:43 IST 2021] Skip Telegram notification     : false
[Thu Aug 12 03:15:43 IST 2021] Skip Discord notification      :
[Thu Aug 12 03:15:43 IST 2021] Email provider                 :
[Thu Aug 12 03:15:43 IST 2021] One-time execution             : true
[Thu Aug 12 03:15:43 IST 2021] YouTube URL detected
[Thu Aug 12 03:15:43 IST 2021] YouTube video detected (alt)
[Thu Aug 12 03:15:43 IST 2021] Checking https://www.youtube.com/channel/UC9OM-qeiYIPtAkBe9veG5uw/live...
parse error: Invalid numeric literal at line 2, column 17
-p destination: Broken pipe
[Thu Aug 12 03:15:44 IST 2021] cURL check seems goes live now, continue checking...
[Thu Aug 12 03:15:44 IST 2021] Re-checking via youtube-dl...
[Thu Aug 12 03:15:47 IST 2021] Unknown metadata
[Thu Aug 12 03:15:47 IST 2021] https://manifest.googlevideo.com/api/manifest/hls_playlist/expire/1628739945/ei/CUUUYa-kIdmGz7sPssGsuAo/ip/xxx.xx.xxx.xxx/id/0h2e0tKR7rA.1/itag/96/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D137/hls_chunk_host/rr5---sn-cvh7knes.googlevideo.com/playlist_duration/30/manifest_duration/30/vprv/1/playlist_type/DVR/mh/HS/mm/44/mn/sn-cvh7knes/ms/lva/mv/u/mvi/5/pl/22/dover/11/keepalive/yes/fexp/24001373,24007246/mt/1628717582/sparams/expire,ei,ip,id,itag,source,requiressl,ratebypass,live,sgoap,sgovp,playlist_duration,manifest_duration,vprv,playlist_type/sig/AOq0QJ8wRAIgYu6UN4T20_oPOZ3Tf1HfLsf01yfp0RBvgBWrXVWeMDgCIBqse3EueLhZrylp6Vd85e0iBXQgVTNudsMooMX-5zU0/lsparams/hls_chunk_host,mh,mm,mn,ms,mv,mvi,pl/lsig/AG3C_xAwRAIgVJ1-2FhqNKd2IAwjbCLS-qrVQewP89_ulk_3W9gqU9gCIDYMeRoeItLST-mvqTL9ph0WN02dlw8ItQvu0GinAWT6/playlist/index.m3u8
...

sparanoid / live-dl Goto Github PK

live-dl's Introduction

live-dl

Features

System Requirements

Available Tags

Run live-dl Inside a Container with Docker

Run live-dl with Docker Compose

Run live-dl Locally

Configurations

Contributing

License

live-dl's People

Contributors

Stargazers

Watchers

Forkers

live-dl's Issues

Open

Detected dependencies

Recommend Projects

Recommend Topics

Recommend Org

Run `live-dl` Inside a Container with Docker

Run `live-dl` with Docker Compose

Run `live-dl` Locally