dzenanjupic / rustube Goto Github PK

View Code? Open in Web Editor NEW

216.0 9.0 43.0 295 KB

A YouTube video downloader, originally inspired by pytube, written in Rust.

License: Apache License 2.0

Rust 100.00%

youtube rust youtube-downloader

rustube's People

Stargazers

Watchers

rustube's Issues

Fetch video info for offline livestreams

I wanted to fetch the VideoDetails/VideoInfo struct for an offline stream and realized, that fetch is pub but fails on Self::check_availability and get_video_info is private so I can't call it. By looking into the json arguments from an offline live stream I don't think they're missing anything except for the streams themselves.

How do you specify the download path for the video?

Description

How can I specify where to download the videos to?

Version

0.6.0

empty video_info_raw

Lastly I got QueryDeserialization(Custom("missing field 'player_response'")) quite often, seemingly at random. The direct cause of this error is that video_info_raw in crate::fetcher::VideoFetcher::get_video_info is an empty string. I haven't found any cause for this and trying to fetch it again works most of the time.

add dashManifestUrl && hlsManifestUrl in to stream

Description

add dashManifestUrl && hlsManifestUrl in to stream like

use rustube::{Id, Video};

#[tokio::main]
async fn main() {
    let id = Id::from_raw("https://www.youtube.com/watch?v=7tNtU5XFwrU").unwrap();
    let descrambler = Video::from_id(id.into_owned())
            .await
            .unwrap();
    let  hls_stream_url = descrambler.hlsManifestUrl;
    let m3u8_stream_url = descrambler.dashManifestUrl ;
}

Handle geoblocking

I tried to fetch information for https://www.youtube.com/watch?v=hQ4n4zHdrwA and it returned a watch html did not contain a PlayabilityStatus when fetched from Germany. As far as I can tell from the corosponding youtube site it's geoblocked

unknown variant `1440p60`

I got a QueryDeserialization(Custom("unknown variant '1440p60', expected one of '144p', '240p', '360p', '480p', '720p', '720p50', '720p60', '1080p', '1080p60', '1440p', '2160p' at line 1 column 5612")) for the youtube video.
Occured while trying to do Video::from_id(id.as_owned()).

docs.rs build is failing

Our doc.rs build is currently failing:

docs.rs Build output

# rustc version
rustc 1.53.0-nightly (07e0e2ec2 2021-03-24)
# docs.rs version
docsrs 0.6.0 (9708001 2021-04-08)

# build log
[INFO] running `Command { std: "docker" "create" "-v" "/home/cratesfyi/workspace/builds/rustube-0.2.8/target:/opt/rustwide/target:rw,Z" "-v" "/home/cratesfyi/workspace/builds/rustube-0.2.8/source:/opt/rustwide/workdir:ro,Z" "-v" "/home/cratesfyi/workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/home/cratesfyi/workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "DOCS_RS=1" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "3221225472" "--cpus" "2" "--user" "1001:1001" "--network" "none" "rustops/crates-build-env@sha256:abe1b87bf6b7c66130a0dd9bb19216a8db0cc381a1b41d68bf7d83b4bc040efb" "/opt/rustwide/cargo-home/bin/cargo" "+nightly" "rustdoc" "--lib" "-Zrustdoc-map" "-Zunstable-options" "--config=doc.extern-map.registries.crates-io=\"https://docs.rs\"" "-j2" "--" "-Z" "unstable-options" "--resource-suffix" "-20210324-1.53.0-nightly-07e0e2ec2" "--static-root-path" "/" "--cap-lints" "warn" "--disable-per-crate-search", kill_on_drop: false }`
[INFO] [stdout] 02c17a05f57e5b87a48eb73dbc42b8647a7faddaa87157fcd0be8504ef8a3908
[INFO] [stderr] WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
[INFO] running `Command { std: "docker" "start" "-a" "02c17a05f57e5b87a48eb73dbc42b8647a7faddaa87157fcd0be8504ef8a3908", kill_on_drop: false }`
[INFO] [stderr] /opt/crates-build-env/entrypoint.sh: line 7: /etc/hosts: Permission denied
[INFO] [stderr]    Compiling proc-macro2 v1.0.26
[INFO] [stderr]    Compiling unicode-xid v0.2.1
[INFO] [stderr]    Compiling syn v1.0.69
[INFO] [stderr]    Compiling autocfg v1.0.1
[INFO] [stderr]    Compiling libc v0.2.93
[INFO] [stderr]     Checking cfg-if v1.0.0
[INFO] [stderr]    Compiling log v0.4.14
[INFO] [stderr]    Compiling version_check v0.9.3
[INFO] [stderr]    Compiling memchr v2.3.4
[INFO] [stderr]     Checking pin-project-lite v0.2.6
[INFO] [stderr]    Compiling serde_derive v1.0.125
[INFO] [stderr]     Checking smallvec v1.6.1
[INFO] [stderr]     Checking once_cell v1.7.2
[INFO] [stderr]     Checking scopeguard v1.1.0
[INFO] [stderr]     Checking bytes v1.0.1
[INFO] [stderr]    Compiling serde v1.0.125
[INFO] [stderr]     Checking itoa v0.4.7
[INFO] [stderr]    Compiling pkg-config v0.3.19
[INFO] [stderr]    Compiling cc v1.0.67
[INFO] [stderr]     Checking futures-core v0.3.14
[INFO] [stderr]     Checking matches v0.1.8
[INFO] [stderr]     Checking percent-encoding v2.1.0
[INFO] [stderr]    Compiling proc-macro-hack v0.5.19
[INFO] [stderr]    Compiling ryu v1.0.5
[INFO] [stderr]    Compiling bitflags v1.2.1
[INFO] [stderr]     Checking tinyvec_macros v0.1.0
[INFO] [stderr]     Checking lazy_static v1.4.0
[INFO] [stderr]     Checking fnv v1.0.7
[INFO] [stderr]    Compiling openssl v0.10.33
[INFO] [stderr]    Compiling const_fn v0.4.6
[INFO] [stderr]     Checking foreign-types-shared v0.1.1
[INFO] [stderr]    Compiling serde_json v1.0.64
[INFO] [stderr]     Checking futures-task v0.3.14
[INFO] [stderr]    Compiling httparse v1.3.6
[INFO] [stderr]     Checking futures-sink v0.3.14
[INFO] [stderr]    Compiling ident_case v1.0.1
[INFO] [stderr]     Checking pin-utils v0.1.0
[INFO] [stderr]    Compiling crc32fast v1.2.1
[INFO] [stderr]    Compiling strsim v0.10.0
[INFO] [stderr]    Compiling native-tls v0.2.7
[INFO] [stderr]     Checking hashbrown v0.9.1
[INFO] [stderr]     Checking adler v1.0.2
[INFO] [stderr]     Checking openssl-probe v0.1.2
[INFO] [stderr]     Checking slab v0.4.2
[INFO] [stderr]     Checking try-lock v0.2.3
[INFO] [stderr]     Checking tower-service v0.3.1
[INFO] [stderr]    Compiling rustversion v1.0.4
[INFO] [stderr]    Compiling encoding_rs v0.8.28
[INFO] [stderr]     Checking httpdate v0.3.2
[INFO] [stderr]     Checking regex-syntax v0.6.23
[INFO] [stderr]     Checking mime v0.3.16
[INFO] [stderr]    Compiling convert_case v0.4.0
[INFO] [stderr]     Checking ipnet v2.3.0
[INFO] [stderr]     Checking base64 v0.13.0
[INFO] [stderr]    Compiling tokio v1.4.0
[INFO] [stderr]    Compiling indexmap v1.6.2
[INFO] [stderr]    Compiling miniz_oxide v0.4.4
[INFO] [stderr]    Compiling num-traits v0.2.14
[INFO] [stderr]    Compiling num-integer v0.1.44
[INFO] [stderr]     Checking instant v0.1.9
[INFO] [stderr]    Compiling standback v0.2.17
[INFO] [stderr]    Compiling time v0.2.26
[INFO] [stderr]    Compiling cookie v0.14.4
[INFO] [stderr]     Checking lock_api v0.4.3
[INFO] [stderr]     Checking futures-channel v0.3.14
[INFO] [stderr]     Checking unicode-bidi v0.3.5
[INFO] [stderr]     Checking form_urlencoded v1.0.1
[INFO] [stderr]    Compiling openssl-sys v0.9.61
[INFO] [stderr]     Checking tinyvec v1.2.0
[INFO] [stderr]     Checking tracing-core v0.1.17
[INFO] [stderr]     Checking http v0.2.4
[INFO] [stderr]     Checking foreign-types v0.3.2
[INFO] [stderr]     Checking futures-util v0.3.14
[INFO] [stderr]     Checking tracing v0.1.25
[INFO] [stderr]     Checking unicode-normalization v0.1.17
[INFO] [stderr]     Checking http-body v0.4.1
[INFO] [stderr]    Compiling quote v1.0.9
[INFO] [stderr]     Checking parking_lot_core v0.8.3
[INFO] [stderr]     Checking signal-hook-registry v1.3.0
[INFO] [stderr]     Checking num_cpus v1.13.0
[INFO] [stderr]     Checking socket2 v0.4.0
[INFO] [stderr]     Checking time v0.1.44
[INFO] [stderr]     Checking mio v0.7.11
[INFO] [stderr]     Checking want v0.3.0
[INFO] [stderr]     Checking aho-corasick v0.7.15
[INFO] [stderr]     Checking idna v0.2.2
[INFO] [stderr]     Checking parking_lot v0.11.1
[INFO] [stderr]     Checking flate2 v1.0.20
[INFO] [stderr]     Checking regex v1.4.5
[INFO] [stderr]     Checking chrono v0.4.19
[INFO] [stderr]    Compiling darling_core v0.12.3
[INFO] [stderr]    Compiling tokio-macros v1.1.0
[INFO] [stderr]    Compiling time-macros-impl v0.1.1
[INFO] [stderr]    Compiling pin-project-internal v1.0.6
[INFO] [stderr]    Compiling darling_macro v0.12.3
[INFO] [stderr]    Compiling thiserror-impl v1.0.24
[INFO] [stderr]    Compiling derive_more v0.99.13
[INFO] [stderr]    Compiling derivative v2.2.0
[INFO] [stderr]     Checking time-macros v0.1.1
[INFO] [stderr]     Checking pin-project v1.0.6
[INFO] [stderr]    Compiling darling v0.12.3
[INFO] [stderr]     Checking thiserror v1.0.24
[INFO] [stderr]    Compiling serde_with_macros v1.4.1
[INFO] [stderr]     Checking tokio-util v0.6.5
[INFO] [stderr]     Checking tokio-native-tls v0.3.0
[INFO] [stderr]     Checking async-compression v0.3.7
[INFO] [stderr]     Checking tokio-stream v0.1.5
[INFO] [stderr]     Checking url v2.2.1
[INFO] [stderr]     Checking serde_urlencoded v0.7.0
[INFO] [stderr]     Checking serde_qs v0.8.3
[INFO] [stderr]     Checking h2 v0.3.2
[INFO] [stderr]     Checking publicsuffix v1.5.6
[INFO] [stderr]     Checking serde_with v1.8.0
[INFO] [stderr]     Checking cookie_store v0.12.0
[INFO] [stderr]     Checking hyper v0.14.5
[INFO] [stderr]     Checking hyper-tls v0.5.0
[INFO] [stderr]     Checking reqwest v0.11.2
[INFO] [stderr]  Documenting rustube v0.2.8 (/opt/rustwide/workdir)
[INFO] [stderr] error[E0432]: unresolved import `futures`
[INFO] [stderr]  --> src/stream/callback.rs:6:5
[INFO] [stderr]   |
[INFO] [stderr] 6 | use futures::FutureExt;
[INFO] [stderr]   |     ^^^^^^^ use of undeclared crate or module `futures`
[INFO] [stderr] 
[INFO] [stderr] error: Compilation failed, aborting rustdoc
[INFO] [stderr] 
[INFO] [stderr] error: aborting due to 2 previous errors
[INFO] [stderr] 
[INFO] [stderr] For more information about this error, try `rustc --explain E0432`.
[INFO] [stderr] error: could not document `rustube`
[INFO] [stderr] 
[INFO] [stderr] Caused by:
[INFO] [stderr]   process didn't exit successfully: `rustdoc --edition=2018 --crate-type lib --crate-name rustube src/lib.rs -o /opt/rustwide/target/doc --cfg 'feature="bytes"' --cfg 'feature="chrono"' --cfg 'feature="default"' --cfg 'feature="descramble"' --cfg 'feature="download"' --cfg 'feature="fetch"' --cfg 'feature="mime"' --cfg 'feature="regex"' --cfg 'feature="reqwest"' --cfg 'feature="serde_json"' --cfg 'feature="serde_qs"' --cfg 'feature="serde_with"' --cfg 'feature="std"' --cfg 'feature="stream"' --cfg 'feature="thiserror"' --cfg 'feature="tokio"' --cfg 'feature="tokio-stream"' --error-format=json --json=diagnostic-rendered-ansi -Z unstable-options --resource-suffix -20210324-1.53.0-nightly-07e0e2ec2 --static-root-path / --cap-lints warn --disable-per-crate-search -L dependency=/opt/rustwide/target/debug/deps --extern bytes=/opt/rustwide/target/debug/deps/libbytes-c231ffc79de07686.rmeta --extern chrono=/opt/rustwide/target/debug/deps/libchrono-a0656ae707e7a9b9.rmeta --extern derivative=/opt/rustwide/target/debug/deps/libderivative-ca5f19c6a9bbd80c.so --extern derive_more=/opt/rustwide/target/debug/deps/libderive_more-e06769ccb5ae922e.so --extern log=/opt/rustwide/target/debug/deps/liblog-1666592c90e6faaa.rmeta --extern mime=/opt/rustwide/target/debug/deps/libmime-a872b864d5c8734e.rmeta --extern regex=/opt/rustwide/target/debug/deps/libregex-6db8dbd31a03a9a4.rmeta --extern reqwest=/opt/rustwide/target/debug/deps/libreqwest-86c7e4f557ee5c81.rmeta --extern serde=/opt/rustwide/target/debug/deps/libserde-794eec93d12d5d4d.rmeta --extern serde_json=/opt/rustwide/target/debug/deps/libserde_json-3a423f61b942e42c.rmeta --extern serde_qs=/opt/rustwide/target/debug/deps/libserde_qs-5bb7d59fb2fdccd4.rmeta --extern serde_with=/opt/rustwide/target/debug/deps/libserde_with-9c63c14ba4157bcb.rmeta --extern thiserror=/opt/rustwide/target/debug/deps/libthiserror-b3682012c025bb68.rmeta --extern tokio=/opt/rustwide/target/debug/deps/libtokio-6ddeadb1f0b3ca27.rmeta --extern tokio_stream=/opt/rustwide/target/debug/deps/libtokio_stream-59542b729bff818b.rmeta --extern url=/opt/rustwide/target/debug/deps/liburl-962c77f799fcfc0b.rmeta --extern-html-root-url 'bytes=https://docs.rs/bytes/1.0.1/' --extern-html-root-url 'chrono=https://docs.rs/chrono/0.4.19/' --extern-html-root-url 'derivative=https://docs.rs/derivative/2.2.0/' --extern-html-root-url 'derive_more=https://docs.rs/derive_more/0.99.13/' --extern-html-root-url 'log=https://docs.rs/log/0.4.14/' --extern-html-root-url 'mime=https://docs.rs/mime/0.3.16/' --extern-html-root-url 'regex=https://docs.rs/regex/1.4.5/' --extern-html-root-url 'reqwest=https://docs.rs/reqwest/0.11.2/' --extern-html-root-url 'serde=https://docs.rs/serde/1.0.125/' --extern-html-root-url 'serde_json=https://docs.rs/serde_json/1.0.64/' --extern-html-root-url 'serde_qs=https://docs.rs/serde_qs/0.8.3/' --extern-html-root-url 'serde_with=https://docs.rs/serde_with/1.8.0/' --extern-html-root-url 'thiserror=https://docs.rs/thiserror/1.0.24/' --extern-html-root-url 'tokio=https://docs.rs/tokio/1.4.0/' --extern-html-root-url 'tokio_stream=https://docs.rs/tokio-stream/0.1.5/' --extern-html-root-url 'url=https://docs.rs/url/2.2.1/' -Zunstable-options --crate-version 0.2.8` (exit code: 1)
[INFO] running `Command { std: "docker" "inspect" "02c17a05f57e5b87a48eb73dbc42b8647a7faddaa87157fcd0be8504ef8a3908", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "02c17a05f57e5b87a48eb73dbc42b8647a7faddaa87157fcd0be8504ef8a3908", kill_on_drop: false }`
[INFO] [stdout] 02c17a05f57e5b87a48eb73dbc42b8647a7faddaa87157fcd0be8504ef8a3908

That's bad, since it makes the documentation inaccessible, and might scare off new comers.

Stream URLs for direct viewing

Is there an API in rustube that allowed me to get the stream URL directly, so I can view it directly in a <video> tag?
Youtube-dl supports this with the -g flag.

Make high quality the default (or add documentation on how to choose that)

I don't know if this is related to the split_once, but trying with a couple videos, it consistently does the following:

Rustube chooses P360
640x360 h.264 at average bitrate around 700kbps
96kbps VBR aac 44100Hz (not normalized)

Youtube chooses P1080
1920x1080 VP9/H264
Opus 48000Hz (normalized)

With rustube check <url> I notice two things:

The correct stream types are showing up, but P360 is the first in the list and out of order from the rest.
The loudness_db is not being correctly applied

Videos are listed before audio and shows:

        loudness_db: None

so if there's an audio stream with some ridiculous loudness_db:

        loudness_db: Some(
            8.46,
        ),

then the final audio is way too loud and clips. I'm going to guess it's related to my issue with it choosing 360p

watch html did not contain a PlayabilityStatus

Seems like YouTube just changed their API, since I started getting this error:

Error: Could not fetch or descramble the video information

Caused by:
    YouTube returned an unexpected response: `watch html did not contain a PlayabilityStatus`

when trying to download various videos.

I won't have time to investigate it instantly, so information about the problem and PRs are welcome.

add option for stdout

Description

this is a feature request.

change the write_stream_to_file to accept AsyncWrite and make the cli able to write to stdout. I want to use this library on https://github.com/aquelemiguel/parrot but it needed stdout capabilities.

Version

No response

Problem downloading best quality?

Description

I am writing an app that needs to download google play store app promo videos. The videos are hosted on youtube and I am trying to download Clash of Clans promo video(https://www.youtube.com/embed/QHnxnzrwQmk?ps=play&vq=large&rel=0&autohide=1&showinfo=0), but Rustube keeps downloading it at 720p.

On Youtube I see an option for 1080p and when downloading via 4k Video Downloader, I also get 1080p.

My Code:
let video = Video::from_url(&url).await.unwrap();
video
.best_quality()
.unwrap()
.download_to(format!("{}_vid.mp4", id))
.await
.unwrap();

Version

0.3.8

403: Forbidden when trying to download any video

Summary

I am unable to download any video with rustube, since youtube always returns 403

Video IDs

DB7rUoN7ES8
Ub_NI1lCrdA
mBA31Jr1THc
c13gpBrnGEw

Version

rustube 0.6.0

Log output or error

log

  reqwest::Error { kind: Status(403), url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("rr3---sn-2gb7sn7y.googlevideo.com")), port: None, path: "/videoplayback", query: Some("expire=1719070626&ei=Qpt2ZobyKoO56dsPx7KAiAU&ip=185.167.209.67&id=o-ALf2FLXpwBC-hU8SCzlRF2zhA2QNimpEr4sgNqqup8v3&itag=251&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=Et&mm=31%2C26&mn=sn-2gb7sn7y%2Csn-4g5lzner&ms=au%2Conr&mv=m&mvi=3&pl=22&gcr=cz&initcwndbps=1346250&bui=AbKP-1P5pMnlEmTMja0b4_H_n-YhqDptRj9Mjnts0jUAvV_gHnulTWP2hbHDaLQWnhJpIuwmC0NCyoaf&spc=UWF9f_BrJ_4Y8TFca0WBuHlMzS7F4sxUiVb_3wV1HLz8deGtLDkRQIOXc0jH&vprv=1&svpuc=1&mime=audio%2Fwebm&ns=gApVCO3OfPx1vzJ7mX0fRoYQ&rqh=1&gir=yes&clen=4401953&dur=266.281&lmt=1714541503865233&mt=1719048541&fvip=1&keepalive=yes&c=WEB&sefc=1&txp=5432434&n=yIVU8ecpD_Ldxwibol&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cgcr%2Cbui%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHlkHjAwRgIhAOYbBvRD9pj03CnIID3vjQA_7dEVVrJUOw1rGxI1pDcBAiEA8ZteVvV6S4hzNDtZ2uLxpBsVrHwMTThvXaX8whBiwg4%3D&sig=AJfQdSswRQIhAOJMwsUni1zk6oQe0AccL-hu0ear5uW8T-cmeXkw4K-PAiB7M_3URord44sLdPKT8xxrbPyLudCVNjf60_FjCx4Z_g%3D%3D"), fragment: None } }

Additional notes or ideas

No response

VideoDetails.average_rating does no longer exist

Summary

YouTube recently removed the like button. This broke our VideoDetails struct.

Video IDs

all videos

Version

0.3.6

Log output or error

No response

Additional notes or ideas

No response

Can't download a specific video (id=whoI4EuC0yg)

Summary

Error downloading whoI4EuC0yg: YouTube returned an unexpected response: Could not acquire the player response from the watch html! It looks like YouTube changed it's API again :-/ If this not yet reported, it would be great if you could file an issue: (https://github.com/DzenanJupic/rustube/issues/new?assignees=&labels=youtube-api-changed&template=youtube_api_changed.yml).

Video IDs

whoI4EuC0yg
i3lilyB2RGg

Version

- 0.6.0 (https://github.com/Discursif/rustube fork to allow downloading at high-speed)

Log output or error

log.txt

Additional notes or ideas

Maybe it's due to the high speed downloading fix I applied but this is the only video unable to get downloaded

Is download speed throttled by YouTube?

pytube (#1033) and youtube-dl (#29326) have open issues regarding YouTube throttling the download speed.

It would be interesting to know if that also affects us!

This comment from youtube-dl has a potential solution.

Cargo install doesn't work

I'm just going to git clone this and then build it, but I thought I should let you know that if you just run the cargo install commands, it does not install:

cargo +nightly install rustube-cli
    Updating crates.io index
  Installing rustube-cli v0.2.4
   Compiling proc-macro2 v1.0.26
   Compiling unicode-xid v0.2.1
   Compiling syn v1.0.68
   Compiling winapi-x86_64-pc-windows-gnu v0.4.0
   Compiling winapi v0.3.9
   Compiling autocfg v1.0.1
   Compiling version_check v0.9.3
   Compiling cfg-if v1.0.0
   Compiling libc v0.2.92
   Compiling log v0.4.14
   Compiling memchr v2.3.4
   Compiling pin-project-lite v0.2.6
   Compiling ntapi v0.3.6
   Compiling bytes v1.0.1
   Compiling serde_derive v1.0.125
   Compiling smallvec v1.6.1
   Compiling scopeguard v1.1.0
   Compiling lazy_static v1.4.0
   Compiling serde v1.0.125
   Compiling itoa v0.4.7
   Compiling futures-core v0.3.13
   Compiling proc-macro-hack v0.5.19
   Compiling matches v0.1.8
   Compiling percent-encoding v2.1.0
   Compiling tinyvec_macros v0.1.0
   Compiling ryu v1.0.5
   Compiling const_fn v0.4.6
   Compiling fnv v1.0.7
   Compiling hashbrown v0.9.1
   Compiling serde_json v1.0.64
   Compiling strsim v0.10.0
   Compiling pin-utils v0.1.0
   Compiling crc32fast v1.2.1
   Compiling ident_case v1.0.1
   Compiling futures-task v0.3.13
   Compiling native-tls v0.2.7
   Compiling futures-sink v0.3.13
   Compiling httparse v1.3.5
   Compiling slab v0.4.2
   Compiling try-lock v0.2.3
   Compiling adler v1.0.2
   Compiling rustversion v1.0.4
   Compiling tower-service v0.3.1
   Compiling encoding_rs v0.8.28
   Compiling httpdate v0.3.2
   Compiling bitflags v1.2.1
   Compiling unicode-segmentation v1.7.1
   Compiling convert_case v0.4.0
   Compiling ipnet v2.3.0
   Compiling base64 v0.13.0
   Compiling mime v0.3.16
   Compiling unicode-width v0.1.8
   Compiling anyhow v1.0.40
   Compiling regex-syntax v0.6.23
   Compiling vec_map v0.8.2
   Compiling os_str_bytes v2.4.0
   Compiling instant v0.1.9
   Compiling standback v0.2.17
   Compiling time v0.2.26
   Compiling proc-macro-error-attr v1.0.4
   Compiling cookie v0.14.4
   Compiling proc-macro-error v1.0.4
   Compiling lock_api v0.4.3
   Compiling tracing-core v0.1.17
   Compiling unicode-bidi v0.3.4
   Compiling futures-channel v0.3.13
   Compiling tinyvec v1.2.0
   Compiling tokio v1.4.0
   Compiling indexmap v1.6.2
   Compiling miniz_oxide v0.4.4
   Compiling num-traits v0.2.14
   Compiling num-integer v0.1.44
   Compiling form_urlencoded v1.0.1
   Compiling http v0.2.3
   Compiling futures-util v0.3.13
   Compiling textwrap v0.12.1
   Compiling heck v0.3.2
   Compiling tracing v0.1.25
   Compiling unicode-normalization v0.1.17
   Compiling want v0.3.0
   Compiling aho-corasick v0.7.15
   Compiling num_cpus v1.13.0
   Compiling http-body v0.4.1
   Compiling quote v1.0.9
   Compiling idna v0.2.2
   Compiling regex v1.4.5
   Compiling flate2 v1.0.20
   Compiling miow v0.3.7
   Compiling parking_lot_core v0.8.3
   Compiling schannel v0.1.19
   Compiling socket2 v0.4.0
   Compiling winreg v0.7.0
   Compiling winapi-util v0.1.5
   Compiling time v0.1.44
   Compiling atty v0.2.14
   Compiling parking_lot v0.11.1
   Compiling termcolor v1.1.2
   Compiling darling_core v0.12.2
   Compiling chrono v0.4.19
   Compiling mio v0.7.11
   Compiling tokio-macros v1.1.0
   Compiling time-macros-impl v0.1.1
   Compiling pin-project-internal v1.0.6
   Compiling thiserror-impl v1.0.24
   Compiling clap_derive v3.0.0-beta.2
   Compiling derivative v2.2.0
   Compiling derive_more v0.99.13
   Compiling darling_macro v0.12.2
   Compiling time-macros v0.1.1
   Compiling darling v0.12.2
   Compiling serde_with_macros v1.4.1
   Compiling thiserror v1.0.24
   Compiling pin-project v1.0.6
   Compiling clap v3.0.0-beta.2
   Compiling tokio-util v0.6.5
   Compiling tokio-native-tls v0.3.0
   Compiling async-compression v0.3.7
   Compiling tokio-stream v0.1.5
   Compiling h2 v0.3.2
   Compiling url v2.2.1
   Compiling serde_urlencoded v0.7.0
   Compiling serde_qs v0.8.3
   Compiling publicsuffix v1.5.6
   Compiling cookie_store v0.12.0
   Compiling serde_with v1.8.0
   Compiling hyper v0.14.5
   Compiling hyper-tls v0.5.0
   Compiling reqwest v0.11.2
   Compiling rustube v0.2.5
error[E0658]: use of unstable library feature 'str_split_once': newly added
   --> C:\Users\Lyam\.cargo\registry\src\github.com-1ecc6299db9ec823\rustube-0.2.5\src\descrambler\cipher.rs:187:14
    |
187 |             .split_once(':')
    |              ^^^^^^^^^^
    |
    = note: see issue #74773 <https://github.com/rust-lang/rust/issues/74773> for more information
    = help: add `#![feature(str_split_once)]` to the crate attributes to enable

error: aborting due to previous error

For more information about this error, try `rustc --explain E0658`.
error: could not compile `rustube`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: failed to compile `rustube-cli v0.2.4`, intermediate artifacts can be found at `C:\Users\Lyam\AppData\Local\Temp\cargo-installMVWamh`

Caused by:
  build failed

Could not acquire the player response (age-restricted videos)

Edit

The issue was partially fixed. For now only age-restricted or otherwise unavailable videos should fail to download.
If you encounter this problem with a 'normal' video, please let us know about it.

Old issue:

Apparently, YouTube keeps removing the /get_video_info endpoint. pytube (#1060) has the same issue.

We have logs and backtraces of two failed actions:

I guess it gets more and more clear, that we have to find a way to make the youtube-api-v1 work since /get_video_info will probably not be around for much longer.

Could anyone recommend a tool for JS reverse engineering? This would help me to find out how youtube.com handles the video download.

Consider moving away from nightly-only features

First of all, thanks for creating and maintaining this library, which allows me to download youtube videos without introducing dependencies on python or other heavy stuff.

Currently this library uses a number of nightly-only features of the compiler. While they are nice, requiring nightly compiler threatens with code breakage and generally increases maintenance burden on the library users.

Looking at the features used, most (all?) of them can be removed by writing code that is a bit more verbose, adding a dependency (notably - there is once_cell crate) or using some attr tricks (for doc(cfg)).

How would you look at me landing a PR that would try to reduce number of nightly-only features used?

Downloading gives an error: thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: QueryDeserialization(Custom("missing field `player_response`"))'

Hello,
I am trying to use your crate for a simple project as it seems to be the most advanced one. I really like how it should work but I am unable to make it download a video. I've tried multible ways but they all go back to the same error :
thread 'main' panicked at 'called Result::unwrap() on an Err value: QueryDeserialization(Custom("missing field player_response"))'

the simple code is your example code:

#[rustube::tokio::main]
async fn main() {
    let url = "https://www.youtube.com/watch?v=Edx9D2yaOGs&ab_channel=CollegeHumor";
    println!("downloaded video to {:?}", rustube::download_best_quality(&url).await.unwrap());
}

I am pretty new to rust so this might be a simple fix, but I am unable to find what might be causing this.
Thanks a lot!

How to run the tests

I wanted to run the tests but didn't find any way to run them. Neither in the Readme, nor in the test source code. It would be nice to find such an explanation somewhere.

404 /get_video_info

And again there's a step away from /get_video_info. So it seems like we have to move to API v1 as fast as possible. But for now, pytube#1021 should do the job.

Default example from docs not working correctly.

Description

OS: Ubuntu 22.04.2 LTS
Executing following code:

#[tokio::main]
async fn main() {
    let url = "https://www.youtube.com/watch?v=Edx9D2yaOGs&ab_channel=CollegeHumor";
    println!("downloaded video to {:?}", [path](rustube::download_best_quality(&url).await.unwrap())
}

downloads and saves the video as expected, but freezes and println line not triggered

Version

rustube 0.6.0

Possible API change

Summary

These are random videos from my playlist with seemingly no correlation or anything

Video IDs

1WSTfgufuNc
C0ez15WnFKI
kP8AKjONNhs
BKl4gZDWP34
52aO5bvA31g
hK7hDvHbN_E
pItd34SK6h8
gccwWVqJ8BY
7Do70nztRNE
KnJ5gLxZZAU
8qSwIax_tgU
BvZO9Zx5Qdc
WtPH5OZwwHc
probably others

Version

0.6.0

Log output or error

log

  fetch() => UnexpectedResponse("Could not acquire the player response from the watch html!\nIt looks like YouTube changed it's API again :-/\nIf this not yet reported, it would be great if you could file an issue:\n            (https://github.com/DzenanJupic/rustube/issues/new?assignees=&labels=youtube-api-changed&template=youtube_api_changed.yml).")

Additional notes or ideas

Happy to provide additional information and continue testing, if any is needed

Return download url without actually executing .download()

I've read the documentation and couldn't find such an option. I guess building url by filtering output of Video would be an option but is there a better way to do this? Thanks!

callback cannot use in `tokio::spawn`

Description

I want to download video in an new task scope, but the Callback not implement Send.

#[tokio::main]
async fn main() -> Result<()> {
    let path = ".";
    let url = "xxxxxx"; // some url
    let id = Id::from_raw(url)?.into_owned();

    let cookie_jar = rustube::fetcher::recommended_cookies();
    let headers = rustube::fetcher::recommended_headers();

    let client = Client::builder()
        .default_headers(headers)
        .cookie_provider(std::sync::Arc::new(cookie_jar))
        .build()
        .unwrap();

    let join_handle = tokio::spawn(async move {
        let video = VideoFetcher::from_id_with_client(id, client.clone())
            .fetch()
            .await
            .unwrap()
            .descramble()
            .unwrap();
        let title = video.title().to_string();

        let callback = Callback::new().connect_on_progress_closure_slow(move |arg| {
            if let Some(content_length) = arg.content_length {
                println!("[{title}] download {}%", (arg.current_chunk * 100) as f64 / content_length as f64);
            }
        });
        let path = video
            .best_quality()
            .ok_or(Error::NoStreams)
            .unwrap()
            .download_to_dir_with_callback(path, callback)
            .await
            .unwrap();
        println!("downloaded video to {:?}", path);
    });

    join_handle.await.unwrap();

    Ok(())
}

will build error error: future cannot be sent between threads safely

future is not `Send` as this value is used across an await
let callback = Callback::new().connect_on_progress_closure_slow(move |arg| {
    |               -------- has type `Callback` which is not `Send`
...
54  |               .download_to_dir_with_callback(path, callback)
    |  ___________________________________________________________^
55  | |             .await
    | |__________________^ await occurs here, with `callback` maybe used later

Version

0.5.0

Cannot build with `blocking` feature enabled

Compiling with blocking feature enabled is showing this error (with --verbose):

    Checking rustube v0.2.4
     Running `rustc --crate-name rustube --edition=2018 /home/gokul/.cargo/registry/src/github.com-1ecc6299db9ec823/rustube-0.2.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="blocking"' --cfg 'feature="regex"' --cfg 'feature="std"' --cfg 'feature="thiserror"' --cfg 'feature="tokio"' -C metadata=98fe3e18e2a8bc75 -C extra-filename=-98fe3e18e2a8bc75 --out-dir /home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps -L dependency=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps --extern derivative=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/libderivative-71e4ead28806b1b4.so --extern derive_more=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/libderive_more-fec911f7eef0ec2c.so --extern log=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/liblog-1666592c90e6faaa.rmeta --extern regex=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/libregex-6db8dbd31a03a9a4.rmeta --extern serde=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/libserde-8c1f857ff956d42a.rmeta --extern thiserror=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/libthiserror-73dcfd75846dc732.rmeta --extern tokio=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/libtokio-1b17b73a3a47b471.rmeta --extern url=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/liburl-d2aee667742686db.rmeta --cap-lints allow`
error[E0599]: no function or associated item named `new` found for struct `Runtime` in the current scope
  --> /home/gokul/.cargo/registry/src/github.com-1ecc6299db9ec823/rustube-0.2.4/src/blocking/mod.rs:61:14
   |
61 |     Runtime::new().expect("Unable to start the tokio Runtime")
   |              ^^^ function or associated item not found in `Runtime`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0599`.
error: could not compile `rustube`

Caused by:
  process didn't exit successfully: `rustc --crate-name rustube --edition=2018 /home/gokul/.cargo/registry/src/github.com-1ecc6299db9ec823/rustube-0.2.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="blocking"' --cfg 'feature="regex"' --cfg 'feature="std"' --cfg 'feature="thiserror"' --cfg 'feature="tokio"' -C metadata=98fe3e18e2a8bc75 -C extra-filename=-98fe3e18e2a8bc75 --out-dir /home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps -L dependency=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps --extern derivative=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/libderivative-71e4ead28806b1b4.so --extern derive_more=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/libderive_more-fec911f7eef0ec2c.so --extern log=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/liblog-1666592c90e6faaa.rmeta --extern regex=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/libregex-6db8dbd31a03a9a4.rmeta --extern serde=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/libserde-8c1f857ff956d42a.rmeta --extern thiserror=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/libthiserror-73dcfd75846dc732.rmeta --extern tokio=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/libtokio-1b17b73a3a47b471.rmeta --extern url=/home/gokul/Projects/Rotten-Scripts/Rust/Youtube_Downloader/target/debug/deps/liburl-d2aee667742686db.rmeta --cap-lints allow` (exit code: 1)

blocking_download_to_with_callback is not found

Description

I've installed the rustube throw the cargo and also tried reference to the GitHub repository and still can not access the blocking_download_to_with_callback method to download the video will Callback.

Cargo.toml
rustube = { git = "https://github.com/DzenanJupic/rustube", branch = "master" }
rustube = "0.6.0"

Version

No response

watch html did not contain a PlayabilityStatus

I updated to 0.2.6 and most Videos are working again, exept for https://www.youtube.com/watch?v=OWDmwBPKR-4, which still throws this error. This will probably persist until the 12.4 when the premiere/live stream goes live

Tracking issue: YouTube API v1

This is the tracking issue for adapting from /get_video_info to YouTube's API v1.

Related issues:

Add a way to do something on progress

It would be nice to be able to do stuff on progress, like pytube provides the ability to add an on_download_progress callback. This would be helpful for displaying a progress bar.

Can you make url property in stream visible?

Description

I've tried fixes from #51, but it's not working for me. Can you make url accessible so i can implement download on my side?

Version

0.6.0

Files downloaded with best_audio or worst_audio are using the wrong extension

Media files downloaded with video.best_audio() or video.worst_audio() use the mp4 extension despite being webm files.

I just spend a couple of days trying to convert the produced mp4 to mp3 using various multimedia encoding/decoding libraries, only to discover that the file is actually webm file using the mp4 extension falsely.

Security issue around CVE-2020-26235 (RUSTSEC-2020-0071)

Description

chrono dependency for the rustube library brings time 0.1.45 as dependency which triggers security advisory related to a segfault (see issue at chronotope/chrono#602).

Additional cargo audit outcome for the repo:

cargo audit                                                                                                                                                                     master
    Fetching advisory database from `https://github.com/RustSec/advisory-db.git`
      Loaded 498 security advisories (from /home/X/.cargo/advisory-db)
    Updating crates.io index
    Scanning Cargo.lock for vulnerabilities (227 crate dependencies)
Crate:     time
Version:   0.1.45
Title:     Potential segfault in the time crate
Date:      2020-11-18
ID:        RUSTSEC-2020-0071
URL:       https://rustsec.org/advisories/RUSTSEC-2020-0071
Solution:  Upgrade to >=0.2.23
Dependency tree:
time 0.1.45
├── pbr 1.0.4
│   └── rustube-cli 0.6.0
└── chrono 0.4.23
    ├── rustube-cli 0.6.0
    └── rustube 0.6.0
        └── rustube-cli 0.6.0

error: 1 vulnerability found!

This issue can be avoided by setting default-features to false and possible adding std as the only feature on the crate.

rustube-cli requires additional changes related to pbr dependency.

Version

0.6.0

UnexpectedResponse("watch html did not contain a PlayabilityStatus")

Summary

No response

Video IDs

OxqNH2s74ZA
ZveUN5KRhuo
and well.. any video :(

Version

0.6.0

Log output or error

log

  path/to/project>cargo run -- ytd OxqNH2s74ZA
  Finished dev [unoptimized + debuginfo] target(s) in 0.31s
   Running `target\debug\sle.exe ytd OxqNH2s74ZA`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: UnexpectedResponse("watch html did not contain a PlayabilityStatus")', src\you_tube.rs:4:85
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: process didn't exit successfully: `target\debug\sle.exe ytd OxqNH2s74ZA` (exit code: 101)

Additional notes or ideas

No response

Spelling error in README and possibly in the CLI app

The word 'whether' has been spelled wrong as "weather" in the README. I haven't looked into the source code yet but it may be present there too. Small thing to fix

dzenanjupic / rustube Goto Github PK

rustube's People

Stargazers

Watchers

Forkers

rustube's Issues

Description

Version

Description

Description

Version

Description

Version

Summary

Video IDs

Version

Log output or error

Additional notes or ideas

Summary

Video IDs

Version

Log output or error

Additional notes or ideas

Summary

Video IDs

Version

Log output or error

Additional notes or ideas

Edit

Old issue:

Description

Version

Summary

Video IDs

Version

Log output or error

Additional notes or ideas

Description

Version

Description

Version

Description

Version

Description

Version

Summary

Video IDs

Version

Log output or error

Additional notes or ideas

Recommend Projects

Recommend Topics

Recommend Org