greatest-ape / aquatic Goto Github PK

I am getting a lot of errors when I try to cargo build --release -p aquatic_udp

Some errors have detailed explanations: E0283, E0412, E0425, E0432, E0433, E0599.
For more information about an error, try rustc --explain E0283.
error: could not compile privdrop due to 24 previous errors
warning: build failed, waiting for other jobs to finish...

Hi again,

When consuming the Prometheus metrics exported from aquaitc-ws, I have noted that over a long period of time, there are "ghost" entries in both the aquatic_active_connections and aquatic_peer_clients metrics, causing the numbers to not reflect aquatic_peers or the number of open tcp sockets on the server.

For example:

After a large spike in concurrent peers, all using a specific client, in this case WebTorrent 0.6, the number of peers reported in both the aquatic_peer_clients and aquatic_active_connections metrics do not appear to not fully reflect all of the peers disconnecting until aquatic-ws is restarted.

I believe this is a bug, due to the metrics being counts and not incremental values, but I have not been able to identify the specific condition needed to reproduce this issue, but have been noticing this consistently on aquaitc-ws since implementing Prometheus metrics.

The first 8 bytes of a client's peer_id will typically include information about what client/version that peer is running.

Is it possible to include statistics about how many peers are connected with a specific version of a client?

bittorrent-tracker currently exposes this data in its stats page in this format

Edit: For aquatic-ws and aquatic-http, it might be best to also collect the same data, but with user-agent.

Hi there,

I saw in issue #81 that you mentioned that you would want a way to monitor the service. I feel like having support for Prometheus would be a great idea.

Is this something you're still considering?

Hello. It would be wonderful if aquatic could also serve as a tracker to webtorrent folks.

At the moment there is still no BEP to guide this implementation. In the future webtorrent/webtorrent#168 may link to a specification.

Installed as per the docs here, tested on both x86_64 and arm64

After starting with ./target/release/aquatic_http -c "aquatic-http-config.toml" and giving it some traffic it will always crash after a few minutes with a socket worker panic with the message peer addr should already have been extracted by now.

I've tried playing around with the memlock and also running it as root, but no change, it always ends up crashing after some time.

Here's error with RUST_BACKTRACE=full :

02:21:41 [DEBUG] (socket-01) aquatic_http_protocol::request: request GET path: /announce?info_hash=%0b%10%0d%3f%da%a5%3c%3d%c6%98%10%02KTr%96%0a%2a8%c8&peer_id=-UT360W-%c4%b7%04%c9%e0%12%e6CX%ac%91%1c&port=44290&uploaded=0&downloaded=9006958060&left=0&corrupt=0&key=1CAD4AAF&numwant=200&compact=1&no_peer_id=1
02:21:41 [DEBUG] (socket-01) aquatic_http_protocol::request: ignored unrecognized key: corrupt
02:21:41 [DEBUG] (socket-01) aquatic_http_protocol::request: ignored unrecognized key: no_peer_id
02:21:42 [DEBUG] (socket-01) aquatic_http_protocol::request: request GET path: /scrape?info_hash=%0b%10%0d%3f%da%a5%3c%3d%c6%98%10%02KTr%96%0a%2a8%c8
02:21:43 [DEBUG] (socket-01) aquatic_http_protocol::request: request GET path: /announce?info_hash=%cf%bd%f1%a4%e08%19%b6Q%9b%af%27%84%bd%9b%c7A%f5%ff%8a&peer_id=-Tp0000-.pmdNGaF2t-r&port=39277&uploaded=0&downloaded=44040192&left=2104235913&corrupt=0&key=9C6413F0&numwant=200&compact=1&no_peer_id=1&supportcrypto=1&redundant=0
02:21:43 [DEBUG] (socket-01) aquatic_http_protocol::request: ignored unrecognized key: corrupt
02:21:43 [DEBUG] (socket-01) aquatic_http_protocol::request: ignored unrecognized key: no_peer_id
02:21:43 [DEBUG] (socket-01) aquatic_http_protocol::request: ignored unrecognized key: supportcrypto
02:21:43 [DEBUG] (socket-01) aquatic_http_protocol::request: ignored unrecognized key: redundant
02:21:43 [DEBUG] (socket-01) aquatic_http_protocol::request: request GET path: /announce
02:21:43 [DEBUG] (socket-01) aquatic_http::workers::socket::connection: Failed parsing request: no query string
02:21:43 [DEBUG] (socket-01) aquatic_http_protocol::request: request GET path: /announce?info_hash=%13%b2%91%0aVy%a4%0a%5c%ac%df%ab6%dd%ae%91%28UPP&peer_id=-UT360W-%c4%b7%04%c9%e0%12%e6CX%ac%91%1c&port=44290&uploaded=10797056&downloaded=1393622563&left=0&corrupt=0&key=1CAD4AAF&numwant=200&compact=1&no_peer_id=1
02:21:43 [DEBUG] (socket-01) aquatic_http_protocol::request: ignored unrecognized key: corrupt
02:21:43 [DEBUG] (socket-01) aquatic_http_protocol::request: ignored unrecognized key: no_peer_id
02:21:44 [DEBUG] (socket-01) aquatic_http_protocol::request: request GET path: /scrape?info_hash=%13%b2%91%0aVy%a4%0a%5c%ac%df%ab6%dd%ae%91%28UPP
02:21:45 [DEBUG] (socket-01) aquatic_http_protocol::request: request GET path: /announce
02:21:45 [DEBUG] (socket-01) aquatic_http::workers::socket::connection: Failed parsing request: no query string
02:21:45 [DEBUG] (socket-01) aquatic_http_protocol::request: request GET path: /announce
02:21:45 [DEBUG] (socket-01) aquatic_http::workers::socket::connection: Failed parsing request: no query string
thread 'socket-01' panicked at crates/http/src/workers/socket/connection.rs:446:18:
peer addr should already have been extracted by now
stack backtrace:
   0:     0xb4a18c199960 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hece1148e1500c9a2
   1:     0xb4a18c065e90 - core::fmt::write::hb4e3acbf271edb44
   2:     0xb4a18c196804 - std::io::Write::write_fmt::h8ca942adea865c1e
   3:     0xb4a18c19978c - std::sys_common::backtrace::print::h2b12b67a16e9135f
   4:     0xb4a18c19ac58 - std::panicking::default_hook::{{closure}}::hf72040aae8f7970a
   5:     0xb4a18c19a98c - std::panicking::default_hook::hff2a626330bfd796
   6:     0xb4a18c19b0c4 - std::panicking::rust_panic_with_hook::hdbe957ea7e1caf24
   7:     0xb4a18c19af94 - std::panicking::begin_panic_handler::{{closure}}::h5f3c13bf100f9a71
   8:     0xb4a18c199e34 - std::sys_common::backtrace::__rust_end_short_backtrace::hd8a0426804ef2ca9
   9:     0xb4a18c19ad34 - rust_begin_unwind
  10:     0xb4a18bf6cfc0 - core::panicking::panic_fmt::heae5498d3da202ba
  11:     0xb4a18bf6cf90 - core::option::expect_failed::h39c56eaa3bce6881
  12:     0xb4a18c00847c - aquatic_http::workers::socket::connection::run_connection::{{closure}}::h8c46d45577788a16
  13:     0xb4a18c008c34 - aquatic_http::workers::socket::run_socket_worker::{{closure}}::{{closure}}::{{closure}}::he8c6f8133edad29d
  14:     0xb4a18c001624 - <futures_lite::future::Race<F1,F2> as core::future::future::Future>::poll::hd7072edc84899fd5
  15:     0xb4a18bfdcf10 - <core::pin::Pin<P> as core::future::future::Future>::poll::h6bbae0ecb7f29323
  16:     0xb4a18bfaa0d8 - glommio::task::raw::RawTask<F,R,S>::run::hc26468ae5081c969
  17:     0xb4a18c0ad5ac - glommio::executor::LocalExecutor::run_task_queues::h85fe1fdb72baac06
  18:     0xb4a18bfc02ac - scoped_tls::ScopedKey<T>::set::ha1615bc524603fc0
  19:     0xb4a18bfe81b0 - std::sys_common::backtrace::__rust_begin_short_backtrace::h942b8cc9ce563b57
  20:     0xb4a18c00cc34 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h60de96789818e5fe
  21:     0xb4a18c19d494 - std::sys::pal::unix::thread::Thread::new::thread_start::h1a4b48df802ee0a8
  22:     0xe06961b4d5c8 - <unknown>
  23:     0xe06961bb5edc - <unknown>
  24:                0x0 - <unknown>
Error: Socket worker 1 panicked

After examining entrypoint.sh,
I understand it is used for testing aquatic via docker image you create and run.
In your entrypoint.sh you're installing some debian based packages and particularly rtorrent and mktorrent.
You create public cert, private key, tls.toml, udp.toml, ws.toml and launch their respective http, udp and ws aquatic trackers.

Where I start losing comprehension is the seed, leech and torrents directories and associated setup there.
You created these directories then created some test content files in seed directory: tls-test-ipv4, udp-test-ipv4, wss-test-ipv4. These threw me off because they don't have any extensions. It would have been preferred to have .txt extensions to denote these as test content files intended to make a torrent file from.

Then you proceeded to make the torrent files for these. I'll go through the different arguments here. The '-p' means private, not DTH nor PeerExchange. The '-o' means the destination output torrent file. The '-a' means the announced tracker url. The last argument is the source content to share and make a torrent out of. The source content may be a directory or a file. In this case you chose 3 different files for the three different http/udp/ws trackers.

mktorrent -p -o "torrents/tls-ipv4.torrent" -a "https://example.com:3001/announce" "seed/tls-test-ipv4"
mktorrent -p -o "torrents/udp-ipv4.torrent" -a "udp://127.0.0.1:3000" "seed/udp-test-ipv4"
mktorrent -p -o "torrents/wss-ipv4.torrent" -a "wss://example.com:3002" "seed/wss-test-ipv4"

My first question: The fog I have happens in the following steps. You copy the torrents directory into different seed/leech directories. Why?

cp -r torrents torrents-seed
cp -r torrents torrents-leech

I'm going to guess: seed is for the trackers, and leech is for the torrent clients?

My second question: You redirected content twice to the same file: "~/.rtorrent.rc".
Why didn't you redirect that content to two different files?
For example:

~/.seeding_rtorrent_clients.rc
~/.leeching_rtorrent_clients.rc

I believe some of the above could have been documented within the usage in the README.txt to help lift the fog for aquatic beginners because I only saw how to compile it and not how to use it within the README.txt. By placing it there, you will help more people get up to speed with this more quickly. Thank you in advance.

Hi there,

I noticed that the Tracker is designed to receive ip parameter, but it is optional.

And the most widely used Bittorrent library, libtorrent, does not pass the ip parameter by default and states "normal trackers don't accept the IP parameter": https://www.libtorrent.org/reference-Settings.html#announce_ip

So I'm wondering if it will be able to HTTP headers like X-Real-IP and X-Forwarded-For if reverse proxying is done for aquatic? And whether WebTorrent can be reverse proxied, I don't really understand this. Of course, the UDP protocol is not considered here.

Best regards,
Cheriny

Hey all,

I've ran into an issue today while working with aquatic-ws@master in combination with libtorrent. Currently, libtorrent's websocket tracker client implementation is configured to perform a websocket ping every 10 seconds, which was not part of the current webtorrent/bittorrent-tracker client implementation.

In the current implementation of aquatic-ws, I can see that when a websocket ping message is received, and the log level is set to trace, we get the following messages:

2023-01-07T22:35:01.597Z TRACE [tungstenite::protocol] Received message                               
2023-01-07T22:35:01.597Z DEBUG [aquatic_ws::workers::socket] Couldn't parse in_message: Message is nether text nor binary                                                                                 
2023-01-07T22:35:01.597Z TRACE [tungstenite::protocol] Sending pong reply

Via a packet capture, I can confirm a websocket pong is returned.

In addition to the pong, due to the failure to parse the above empty message, in what appears to be this check within aquatic_ws_protocol, it also disconnects the client with the following error message:

{"failure reason":"Invalid request"}

As this is not an issue with Aquatic, but rather a non-standard enhancement made within within libtorrent, would websocket pings be something that Aquatic would be interested in supporting?

Hi, thanks for this great project!

While benchmarking the UDP implementation against chihaya I noticed some interesting results and decided to dig deeper. Mainly, the connect response handling looks very different. (please ignore raw results as this is a Qubes VM in a notebook. The interesting part is more about the magnitude and behavior over time)

The reason for that difference in connect handling seems to be related to stateless handling of connection ids through HMAC.
Something done here: https://github.com/chihaya/chihaya/blob/828edb8fd8bea3e86ba53fd39aa9a8b89c95a781/frontend/udp/connection_id.go

Initially, aquatic starts with a very high throughput for announces but low for connect.

Requests out: 154357.21/second          
Responses in: 154132.02/second                                                                
  - Connect responses:  221.60    
  - Announce responses: 128203.99                          
  - Scrape responses:   25706.44                                     
  - Error responses:    0.00                                   
Peers per announce response: 5.01   

Then, over some time, the number of peers finally gets closer to the 50 default limit while throughput drops as expected:

Requests out: 24783.82/second
Responses in: 24723.03/second
  - Connect responses:  60.79
  - Announce responses: 20552.83
  - Scrape responses:   4109.41
  - Error responses:    0.00
Peers per announce response: 45.61

chihaya starts with a higher connect handling, which makes it bring way more peers.

Requests out: 42182.74/second
Responses in: 33390.18/second
  - Connect responses:  6619.61
  - Announce responses: 22347.70
  - Scrape responses:   4422.87
  - Error responses:    0.00
Peers per announce response: 61.34

Requests out: 34582.55/second
Responses in: 27301.13/second
  - Connect responses:  5521.26
  - Announce responses: 18218.75
  - Scrape responses:   3561.11
  - Error responses:    0.00
Peers per announce response: 84.60

I believe making it stateless should help aquatic getting some nice boost in connect response handling. Hopefully reducing memory usage for millions of peers too, given we can remove ConnectionMap.

Hi there
do you have any solution to run aquatic as a systemd service?

It runs from CLI without problem, but if i put the code into a systemd file (ubuntu) it sucks

CLI:
/etc/aquatic# ./target/release/aquatic_http -c aquatic-http-config.toml

Systemd:

[Unit]
Description=opentracker
After=network.target

[Service]
Type=simple
ExecStart=/etc/aquatic/target/release/aquatic_http -c /etc/aquatic/aquatic-http-config.toml

Environment=RUST_BACKTRACE=full

Restart=on-failure

[Install]
WantedBy=multi-user.target

Greets

Andi

The error:

error[E0635]: unknown feature `stdsimd`
  --> /home/josecelano/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ahash-0.7.7/src/lib.rs:33:42
   |
33 | #![cfg_attr(feature = "stdsimd", feature(stdsimd))]
   |                                          ^^^^^^^

   Compiling rand_chacha v0.3.1
For more information about this error, try `rustc --explain E0635`.
error: could not compile `ahash` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...

active toolchain
----------------

nightly-x86_64-unknown-linux-gnu (default)
rustc 1.78.0-nightly (3b1717c05 2024-03-10)

I had the same problem and it was solved by upgrading the ahash crate.

Nice project! I've written a bittorrent tracker myself and am curious how it compares to other implementations.

I was wondering if you had automated or created documentation for your tracker benchmarks like the one found at aquatic/documents/aquatic-udp-load-test-2023-01-11.pdf so I can reproduce it and add my tracker.

Thanks.

hi there

nginx example config (running)

limit_req_zone $binary_remote_addr zone=req_limit_per_torrent:10m rate=2r/s; #limits rate per client
limit_conn_zone $binary_remote_addr zone=conn_torrent:10m; #limits connections per client

server {

        listen 6970 ssl reuseport; #important ssl parameter
        listen [::]:6970 ssl reuseport; #important ssl parameter

        include /etc/nginx/ssl.conf; #path to ssl configuration file

        access_log off;
        error_log off;

        server_name [your server name as in ssl configuration];

        location / {
                deny all;
                return 404;
        }

        location /announce {

         limit_req zone=req_limit_per_torrent;
         limit_conn conn_torrent 10;

                proxy_read_timeout     2;
                proxy_connect_timeout  2s;
                proxy_pass https://127.0.0.1:6967/announce; #important **https:**//....
        }

        location /scrape {
                limit_req zone=req_limit_per_torrent;
                limit_conn conn_torrent 10;
                proxy_read_timeout 2;
                proxy_connect_timeout 2s;
                proxy_pass https://127.0.0.1:6967/scrape; #important **https:**//....

        }
}

enjoy and have fun

Hey im trying to test the ws tracker but when compiling i get the following

process didn't exit successfully: rustc --crate-name aquatic --edition=2018 aquatic/src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C lto -C debuginfo=2 -C metadata=5e217a32f81b1595 -C extra-filename=-5e217a32f81b1595 --out-dir /root/aquatic/target/release/deps -L dependency=/root/aquatic/target/release/deps --extern aquatic_cli_helpers=/root/aquatic/target/release/deps/libaquatic_cli_helpers-0fff919043297a62.rlib --extern aquatic_http=/root/aquatic/target/release/deps/libaquatic_http-54a5b31abd59126b.rlib --extern aquatic_udp=/root/aquatic/target/release/deps/libaquatic_udp-c2a319b99805b7b0.rlib --extern aquatic_ws=/root/aquatic/target/release/deps/libaquatic_ws-2905b8d887a28bb8.rlib --extern mimalloc=/root/aquatic/target/release/deps/libmimalloc-c2c27c25cdb1fbeb.rlib -C target-cpu=native -C target-feature=-avx512bf16 -C target-feature=-avx512bitalg -C target-feature=-avx512bw -C target-feature=-avx512cd -C target-feature=-avx512dq -C target-feature=-avx512er -C target-feature=-avx512f -C target-feature=-avx512ifma -C target-feature=-avx512pf -C target-feature=-avx512vbmi -C target-feature=-avx512vbmi2 -C target-feature=-avx512vl -C target-feature=-avx512vnni -C target-feature=-avx512vp2intersect -C target-feature=-avx512vpopcntdq -L native=/root/aquatic/target/release/build/libmimalloc-sys-25042ae55c62c2e8/out/./build (signal: 9, SIGKILL: kill)

Hi,

I am trying to run the ws version with tls, using let's encrypt-provided certificate and private key in PEM format (I have both PKCS1 and PKCS8 versions).

I could convert them to DER format, but each time I run aquatic qith any combination of keys, I get this error:

Error: create rustls config: No private keys in file

Any ideas/guidance?

Hi there
other Question

as i understand and since 10 minutes running (https://ibksturm.synology.me/trackerstats/index.php) aquatic_udp got a html statistic output

does the aquatic_http got also one and i can't find it in config? oder does it simple are in pipeline?

I want to test aquatic and the torrent clients returns the "Tracker reply has no peers field" status.
Client: uTorrent 2.2.1

Thanks for the great work here on acquatic.

Would you consider creating a docker file so that it's easier to test this software (and become a contributor) without having to go through the Rust setup?

Rust Version: rustc 1.67.1

Crash :

aquatic_udp[2674874]: thread 'swarm-01' panicked at 'Out of bounds access', /.cargo/registry/src/github.com-1ecc6299db9ec823/indexmap-amortized-1.6.1/src/map/core.rs:615:14
aquatic_udp[2674874]: note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
aquatic_udp[2674874]: thread 'socket-02' panicked at 'Request channel 0 is disconnected', aquatic_udp/src/common.rs:80:17
aquatic_udp[2674874]: Error: worker thread panicked
aquatic_udp[2674874]: thread 'socket-01' panicked at 'Request channel 0 is disconnected', aquatic_udp/src/common.rs:80:17

Hey @greatest-ape,

Thank you for this awesome project!

I'm currently running a public aquatic-ws server over at tracker.webtorrent.dev and this is currently behind a Nginx at the moment so I can both have have certificates automatically managed by certbot, and so the tracker and a website can be served at the same subdomain and route, e.g.

• https://tracker.webtorrent.dev shows the metrics page.
• wss://tracker.webtorrent.dev is proxied to Aquatic.

(The forwarding logic is based on the upgrade header being present or not to determine if this is the start of a WebSocket connection.)

When thinking about the future of the tracker, I see myself running into issues with ephemeral port exhaustion due to the mentioned proxying when combined with the long living nature of WebSocket connections, so I'm looking to move remove the tracker from behind Nginx if feasible (so I don't have to use hacks like listening on [::] with Aquatic and forwarding to multiple loopback addresses to get more than ~31k ports/connections).

The one big and little problem that I'm contemplating when looking to move away from Nginx are:

1. No way to reload SSL certificate without restarting the server and losing all swarm state / disconnecting all peers.
2. No way to redirect HTTPs connections away from the tracker.

A few thoughts from my side:

• In regards to 1. I may be over thinking this, as unlike HTTPs trackers, which would really struggle with a restart due to there being no swarm information for new peers until clients re-announce, WebSocket peers have a very low default announce time and know when the tracker has restarted, so can attempt to re-connect and announce again immediately.
• In regards to 2. this may be a none problem, as I'm unsure how likely a HTTPs client will attempt to navigate to a WSS URL and this may simply be form over function and should be discarded.

Finally the question:

• Would 1. be a possibly considered feature for both aquatic-ws and aquatic-http?

Maybe a strange question, but do websockets trackers work real-time?

So if I connect to the ws tracker requesting data about info-hash, will seeding clients receive my leecher status immediately, bypassing the NAT ~instantly, or is there still some announce interval to wait?

Hi, I'm working on a simple UDP client script:

torrust/torrust-tracker#627

I want to have a simple command to make a default request to a UDP tracker. I'm using aquatic. I would like to print the response in JSON format. But UDP responses don't implement the serialize trait. I guess it could be derived automatically.

I've already created another client for HTTP trackers:

https://github.com/torrust/torrust-tracker/blob/develop/src/bin/http_tracker_client.rs

After leaving an instance of aquatic-ws @ e2a3211 running on a Ubuntu 22.04.3 LTS machine, with this configuration, it appears there is a memory leak in the aquatic_ws process, causing memory usage to increase over the span of multiple days, until the process crashes due to exhausting all system memory (in my case ~2.4GB free - total system memory of 4GB).

An example of peer counts, message throughput and usage can be seen in the following image:

It appears the memory leak is related to peer count in some way, as the memory usage rate increases more under high load, and slows during lower load, but none the less increases over time.

I plan to disable metrics to verify if this could be the cause and will report back if this helps alleviate the issue, but I thought it would be best to open the issue first and provide updates as I try and troubleshoot this.

Is private tracker support planned? Would be nice to implement in a project I maintain.

https://github.com/HDInnovations/UNIT3D-Community-Edition