distribyted / distribyted Goto Github PK

Torrent client with HTTP, fuse, and WebDAV interfaces. Start exploring your torrent files right away, even zip, rar, or 7zip archive contents!

Home Page: https://distribyted.com

License: GNU General Public License v3.0

Makefile 0.80% JavaScript 20.72% Go 63.59% HTML 14.03% Dockerfile 0.85%

torrent distributed filesystem p2p fuse fuse-filesystem go webdav-server webdav http

distribyted's Introduction

distribyted

Torrent client with on-demand file downloading as a filesystem.

Report a Bug · Request Feature

About The Project

Distribyted is an alternative torrent client. It can expose torrent files as a standard FUSE, webDAV or HTTP endpoint and download them on demand, allowing random reads using a fixed amount of disk space.

Distribyted tries to make easier integrations with other applications using torrent files, presenting them as a standard filesystem.

Note that distribyted is in beta version, it is a proof of concept with a lot of bugs.

Use Cases

Play multimedia files on your favorite video or audio player. These files will be downloaded on demand and only the needed parts.
Explore TBs of data from public datasets only downloading the parts you need. Use Jupyter Notebooks directly to process or analyze this data.
Give access to your latest dataset creation just by sharing a magnet link. People will start using your data in seconds.
Play your ROM backups directly from the torrent file. You can have virtually GBs in games and only downloaded the needed ones.

Documentation

Check here for further documentation.

Contributing

Contributions are what make the open-source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

Some areas need more care than others:

Windows and macOS tests and compatibility. I don't have any easy way to test distribyted on these operating systems.
Web interface. Web development is not my forte.
Tutorials. Share with the community your use case!

Special thanks

Anacrolix BitTorrent client package and utilities. An amazing torrent library with file seek support.
Nwaples rardecode library, experimental branch. The only go library that is able to seek over rar files and avoid to use io.Discard.
Bodgit 7zip library. Amazing library to decode 7zip files.

License

Distributed under the GPL3 license. See LICENSE for more information.

distribyted's People

Contributors

Stargazers

Watchers

distribyted's Issues

Create docker container and add documentation of how to use it.

Change how download and upload speed is calculated.

Every file and folder an automatic torrent

Found this project looking for torrent fuse mount projects, this one seems the most supported and active.

I am wondering about the feasibility of this, a fuse mount, where each file and folder gets their own torrent, such that I can have a free structured mount file system, and be able to get a torrent for any file or folder within it, and share it to others knowing the fuse mount is auto seeding/leeching it.

I can also open a torrent or magnet file in a helper, that will ask me where in the fuse mount I wish to save it to, which upon completion, generates torrents for each file and folder within that save.

This would enable a good cross-platform and cross-language ecosystem to be built atop of it, as applications would just need to add files/folders to the fuse mount then query the meta data for the torrent details.

downloading a torrent would still need an api it would seem

Torrents are not being added

I just updated distribyted to version 0.7.0.1 and none of my old config files are working anymore. The webserver is started but no torrents are mounted.

Here is an example


$  distribyted --config ~/.tmp/distribyted/core/config/fun.yml 
5:52PM INF All 0 tables opened in 0s component=item-store
5:52PM INF Discard stats nextEmptySlot: 0 component=item-store
5:52PM INF Set nextTxnTs to 0 component=item-store
5:52PM INF Deleting empty file: distribyted-data/metadata/items/000021.vlog component=item-store
5:52PM INF All 0 tables opened in 0s component=torrent-store
5:52PM INF Discard stats nextEmptySlot: 0 component=torrent-store
5:52PM INF Set nextTxnTs to 0 component=torrent-store
5:52PM INF Deleting empty file: distribyted-data/metadata/magnetdb/000021.vlog component=torrent-store
5:52PM INF setting cache size to 1024 MB
5:52PM INF adding torrents from configuration component=torrent-service
5:52PM INF adding torrents from database component=torrent-service
5:52PM INF starting FUSE mount path=./distribyted-data/mount
5:52PM WRN webDAV configuration not found!
5:52PM INF starting webserver host=0.0.0.0:4444

Allow pinning torrents

For seeding purposes

Improve configuration file structure to support several interfaces.

Right now, configuration structure is really tied to fuse implementation. We want to support several interfaces (fuse, WebDAV, HTTP, and so on), so we need to change the configuration file a bit.

Initial idea:

global_cache_size: 1024
metadata_folder: ./distribyted-data/metadata

routes:
  - name: multimedia
    torrents:
      - magnet_uri: "MAGNET_URI"
      - magnet_uri: "MAGNET_URI"
    fuse:
      allow_other: true
      path: ./path/to/mountpoint
    webdav:
      port: 1233

WebDAV https support

Add HTTPS support for WebDAV interface.

High CPU usage when disconnected from the internet

When the process is running, and the internet connection is lost, the distribyted process starts using almost the entire CPU (after a few minutes).

There is no different log in the terminal, only what I have already reported here

Operating System: Arch Linux
KDE Plasma Version: 5.21.4
Kernel Version: 5.11.13-arch1-1

WebDav ignores settings and always starts on 36911 port

Always uses 36911 port
Can't set interface to listen - always 0.0.0.0
Seems broken, at least webdav-js show error on dir HEAD and not usable

If you are mounting a directory anyway, isn’t it easier to take some ready-made webdav server (for example, GoWebDAV) and use it?

Using GoWebDAV also provides the advantage of a convenient in-browser experience and can replace HTTPFS.

To show what it looks like, I set up a test public server for a couple of days: https://a.jipok.ru/

Infinity spam in log: `WRN tracker client for "wss://tracker.openwebtorrent.com": ...`

1:01PM INF All 0 tables opened in 0s component=item-store
1:01PM INF Discard stats nextEmptySlot: 0 component=item-store
1:01PM INF Set nextTxnTs to 0 component=item-store
1:01PM INF Deleting empty file: /mainpool/trnt/dysttibyted-metadata/items/000005.vlog component=item-store
1:01PM INF starting new server folder component=server name=server
1:01PM INF server folder started component=server name=server
1:01PM INF starting serving new torrent component=server name=server
1:01PM INF not creating magnet from empty folder component=server name=server
1:01PM INF starting watcher component=server name=server
1:01PM INF All 1 tables opened in 0s component=torrent-store
1:01PM INF Discard stats nextEmptySlot: 0 component=torrent-store
1:01PM INF Set nextTxnTs to 1 component=torrent-store
1:01PM INF Deleting empty file: /mainpool/trnt/dysttibyted-metadata/magnetdb/000005.vlog component=torrent-store
1:01PM INF setting cache size to 1000000 MB
1:01PM INF adding torrents from configuration component=torrent-service
1:01PM INF getting torrent info component=torrent-service hash=c9e15763f722f23e98a29decdfae341b98d53056
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"ÉáWc÷\\\"ò>\u0098¢\u009dìß®4\\u001b\u0098Õ0V\",\"complete\":2,\"incomplete\":3}" component=torrent-client
1:01PM INF obtained torrent info component=torrent-service hash=c9e15763f722f23e98a29decdfae341b98d53056
1:01PM INF torrent added component=torrent-service name="Cosmos Laundromat" route=multimedia
1:01PM INF getting torrent info component=torrent-service hash=dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"Ý\u0082UìÜ|¥_°»ø\\u0013#ØpbÛ\\u001fm\\u001c\",\"complete\":2,\"incomplete\":11}" component=torrent-client
1:01PM INF obtained torrent info component=torrent-service hash=dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c
1:01PM INF torrent added component=torrent-service name="Big Buck Bunny" route=multimedia
1:01PM INF getting torrent info component=torrent-service hash=08ada5a7a6183aae1e09d831df6748d566095a10
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"\\b\u00ad¥§¦\\u0018:®\\u001e\\tØ1ßgHÕf\\tZ\\u0010\",\"complete\":29,\"incomplete\":648}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM INF obtained torrent info component=torrent-service hash=08ada5a7a6183aae1e09d831df6748d566095a10
1:01PM INF torrent added component=torrent-service name=Sintel route=multimedia
1:01PM INF getting torrent info component=torrent-service hash=209c8226b299b308beaf2b9cd3fb49212dbd13ec
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"Ý\u0082UìÜ|¥_°»ø\\u0013#ØpbÛ\\u001fm\\u001c\",\"complete\":2,\"incomplete\":11}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\" \u009c\u0082&²\u0099³\\b¾¯+\u009cÓûI!-½\\u0013ì\",\"complete\":1,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM INF obtained torrent info component=torrent-service hash=209c8226b299b308beaf2b9cd3fb49212dbd13ec
1:01PM INF torrent added component=torrent-service name="Tears of Steel" route=multimedia
1:01PM INF getting torrent info component=torrent-service hash=a88fda5954e89178c372716a6a78b8180ed4dad3
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\" \u009c\u0082&²\u0099³\\b¾¯+\u009cÓûI!-½\\u0013ì\",\"complete\":1,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM INF obtained torrent info component=torrent-service hash=a88fda5954e89178c372716a6a78b8180ed4dad3
1:01PM INF torrent added component=torrent-service name="The WIRED CD - Rip. Sample. Mash. Share" route=multimedia
1:01PM INF adding torrents from database component=torrent-service
1:01PM INF getting torrent info component=torrent-service hash=dc3e3fc801ae56d8b9f1830df6b7be689132238a
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\" \u009c\u0082&²\u0099³\\b¾¯+\u009cÓûI!-½\\u0013ì\",\"complete\":1,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\" \u009c\u0082&²\u0099³\\b¾¯+\u009cÓûI!-½\\u0013ì\",\"complete\":1,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connected component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\" \u009c\u0082&²\u0099³\\b¾¯+\u009cÓûI!-½\\u0013ì\",\"complete\":1,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\" \u009c\u0082&²\u0099³\\b¾¯+\u009cÓûI!-½\\u0013ì\",\"complete\":1,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\" \u009c\u0082&²\u0099³\\b¾¯+\u009cÓûI!-½\\u0013ì\",\"complete\":1,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\" \u009c\u0082&²\u0099³\\b¾¯+\u009cÓûI!-½\\u0013ì\",\"complete\":1,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\" \u009c\u0082&²\u0099³\\b¾¯+\u009cÓûI!-½\\u0013ì\",\"complete\":1,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM INF obtained torrent info component=torrent-service hash=dc3e3fc801ae56d8b9f1830df6b7be689132238a
1:01PM INF torrent added component=torrent-service name=Pantheon.S01.WEB-DL.1080p.NewStation route=multimedia
1:01PM INF starting HTTPFS host=127.0.0.1:8092/fs
1:01PM INF starting FUSE mount path=./mount
1:01PM INF starting webDAV server host=0.0.0.0:36911
1:01PM INF starting webserver host=127.0.0.1:8092
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to connecting component=torrent-client
1:01PM WRN tracker client for "wss://tracker.openwebtorrent.com": unhandled announce response "{\"action\":\"announce\",\"interval\":120,\"info_hash\":\"¨\u008fÚYTè\u0091xÃrqjjx¸\\u0018\\u000eÔÚÓ\",\"complete\":0,\"incomplete\":2}" component=torrent-client
...
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client
1:04PM WRN tracker client for "wss://tracker.openwebtorrent.com": webrtc PeerConnection state changed to failed component=torrent-client

I tested distribyted a lot, but these errors only started appearing in the logs recently. While the service is running, such errors will appear constantly and very, very often.

For Windows: is it possible to implement caching as in Linux?

For Windows: is it possible to implement caching as in Linux, but not by whole files, but by chunks?
In Linux - cache in chunks, full size 46 gb, save 120 mb (15 chunks * 8 mb). good

In Windows - cache in files, full size 3 gb, save 3 gb. Not good.

Update anacrolix/torrent to latest version

This is not something trivial.

After trying to update to the latest version of this library, the torrent file interface stops working.

Have a deeper look and make a PR to the library to fix the problem, or fix the problem from our side.

WebDAV auth implementation

Cache configuration per folder.

Now we have a global cache configuration. We want to be able to configure a specific cache per route to cover more use cases (per instance, seed the content of a specific route with cache-size:-1).

Bug: error reading chunk for peer Request

It's hard to catch, but in general it looks like this::

Add some torrents with big files, like film/anime
Open file in mounted dir with mpv
Play random parts of the file or just watch for a couple of minutes at a time
The entire folder of this torrent will become unavailable. You can't even view its contents. When trying to close the program hangs on the closing torrent client..

....
11:20AM WRN *torrent.PeerConn 0xc01190d508: error reading chunk for peer Request {1318 {16384 16384}}: EOF component=torrent-client
11:20AM WRN *torrent.PeerConn 0xc01190d508: error reading chunk for peer Request {1318 {180224 16384}}: EOF component=torrent-client
11:21AM WRN *torrent.PeerConn 0xc01b81c008: error reading chunk for peer Request {1324 {16384 16384}}: EOF component=torrent-client
11:21AM WRN *torrent.PeerConn 0xc01b81c008: error reading chunk for peer Request {1324 {81920 16384}}: EOF component=torrent-client
11:21AM WRN *torrent.PeerConn 0xc01b81c008: error reading chunk for peer Request {1324 {131072 16384}}: EOF component=torrent-client
11:21AM WRN *torrent.PeerConn 0xc01b81c008: error reading chunk for peer Request {1324 {0 16384}}: EOF component=torrent-client
11:21AM WRN *torrent.PeerConn 0xc01b8e0008: error reading chunk for peer Request {1325 {81920 16384}}: EOF component=torrent-client
11:21AM WRN *torrent.PeerConn 0xc01b8e0008: error reading chunk for peer Request {1325 {966656 16384}}: EOF component=torrent-client
11:21AM WRN *torrent.PeerConn 0xc01b8e0008: error reading chunk for peer Request {1325 {32768 16384}}: EOF component=torrent-client
11:21AM WRN *torrent.PeerConn 0xc01bf32008: error reading chunk for peer Request {1326 {16384 16384}}: EOF component=torrent-client
11:21AM ERR error reading data error="context canceled" path=/multimedia/Pantheon.S01.WEB-DL.1080p.NewStation/Pantheon.S01E01.WEB-DL.1080p.NewStation.mkv
^C11:22AM INF closing servers...
11:22AM INF closing items database...
11:22AM INF Lifetime L0 stalled for: 0s component=item-store
11:22AM INF Level 0 [ ]: NumTables: 00. Size: 0 B of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiBLevel 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiBLevel 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiBLevel 3 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiBLevel 4 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiBLevel 5 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiBLevel 6 [B]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiBLevel Done component=item-store
11:22AM INF closing magnet database...
11:22AM INF Lifetime L0 stalled for: 0s component=torrent-store
11:22AM INF Level 0 [ ]: NumTables: 01. Size: 375 B of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiBLevel 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiBLevel 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiBLevel 3 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiBLevel 4 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiBLevel 5 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiBLevel 6 [B]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiBLevel Done component=torrent-store
11:22AM INF closing torrent client...
^C^C^C^C^C^CKilled

Make torrent timeout errors on program boot not to be fatal

I my config.yaml file I have a torrent with hash a85de845618a87deacd8e90e3153696c1b6e8fb1.
If the torrent can't be resolved (ie. found) on boot, after the timeout expires, the program initialization stops.
I would expect the program continue the boot and proceed with the next file, as a minimum. Optionally, it could keep retrying at some defined interval in case the torrent can be resolved later.

6:36PM ERR timeout getting torrent info component=torrent-service hash=a85de845618a87deacd8e90e3153696c1b6e8fb1
6:36PM FTL problem starting application error="error when loading torrents: timeout getting torrent info"

Improve logging

WebDAV problem - file with spaces in the name is not available

The "Bullet Train.mkv" torrent has been added to the program. As a Webdav resource is visible, when you try to download an error, in the logs you can see that there was an attempt to download the file "Bullet"

Mounted torrents are not accessible from host and other containers in Docker.

I was trying to set up an on-demand media server with Distribyted, Sonarr, Jellyfin. But I got stuck at accessing mounts from outside container.

Other volume bindings are working normally but when I try to cd mounts folder, getting /bin/sh: 40: cd: can't cd to multimedia error. In host (Windows), multimedia folder is just empty.

Here's my docker-compose file:

version: "2"
services:
  distribyted:
    container_name: distribyted
    image: distribyted/distribyted:latest
    restart: unless-stopped
    ports:
      - "4444:4444/tcp"
    volumes:
      - ${MOUNT_POINT}/torrent-mounts:/distribyted-data/mountpoints:shared
      - ${MOUNT_POINT}/config/distribyted/metadata:/distribyted-data/metadata
      - ${MOUNT_POINT}/config/distribyted/config:/distribyted-data/config
    security_opt:
      - apparmor:unconfined
    devices:
      - /dev/fuse
    cap_add:
      - SYS_ADMIN

  sonarr:
    image: ghcr.io/linuxserver/sonarr
    container_name: sonarr
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Istanbul
    volumes:
      - ${MOUNT_POINT}/config/sonarr/config:/config
      - ${MOUNT_POINT}/tvseries:/tv
      - ${MOUNT_POINT}/torrent-mounts:/downloads
    ports:
      - 8989:8989
    restart: unless-stopped

unable to build locally

by the way, this looks like a fun project, thanks for writing it.

here are my commands:

# go generate ...
# go build -o distribyted-amd64 -tags "release" cmd/distribyted/main.go

^^ this results in this error:

go: github.com/grpc-ecosystem/grpc-gateway/examples/clients/abe imports
        github.com/go-resty/resty: github.com/go-resty/[email protected]: parsing go.mod:
        module declares its path as: gopkg.in/resty.v1
                but was required as: github.com/go-resty/resty

i have also tried:

# make build

^^ this results in this error:

  >  Generating code files...
go: GOPATH entry cannot start with shell metacharacter '~': "~/go"
make: *** [Makefile:36: go-generate] Error 2

i am using the latest version of golang

module declares its path as: github.com/ajnavarro/distribyted

go 1.16
MacOS 10.15.7

$ go get github.com/distribyted/distribyted
go get: github.com/distribyted/distribyted@none updating to
	github.com/distribyted/[email protected]: parsing go.mod:
	module declares its path as: github.com/ajnavarro/distribyted
	        but was required as: github.com/distribyted/distribyted

Load torrent files from specific folder

Unable to play media

Cloned the repo and did a build make (running linux amd)

Looking at the routes, it does establish a connection fine and I do see a healthy number of seeders and peers for the torrents.

I tried adding a new magnet uri and that works fine. The folder gets created but trying to play the media (via vlc) doesn't work, i just get a vlc error saying unable to open the file.

I tried using one of the default media in the configs and that doesn't work either. The only one that had some progress was the .webm file (tears of steel) that opens a webpage and loads. I see data being downloaded but it isn't streaming to the player.

I don't see any obvious errors/logs relating to my issue

Option to disable IPV6

My network does not have IPV6 support, and when I start distribyted in this environment, the console displays a huge amount of errors.

Example:

go-libutp: 2021/04/01 12:07:49 callbacks.go:73: error sending packet: write udp6 [::]:42069->[2804:14d:78c4:42be:5c6f:1f28:595d:6a11]:47107: sendto: cannot assign requested address
go-libutp: 2021/04/01 12:07:52 callbacks.go:73: error sending packet: write udp6 [::]:42069->[2804:14d:78c4:42be:5c6f:1f28:595d:6a11]:47107: sendto: cannot assign requested address
go-libutp: 2021/04/01 12:07:56 callbacks.go:73: error sending packet: write udp6 [::]:42069->[2804:14d:78c4:42be:5c6f:1f28:595d:6a11]:47107: sendto: cannot assign requested address

I would like to take this opportunity and thank you for this amazing tool. I have been looking for something like this for a long time and finally found it. THANKS!

Improve web user interface

Problems with cross-compiling

Windows is not generated correctly when cross-compiled. Fix it or compile it from a Windows CI node.

mount fs with fuse error under Windows

Hi there. I'm trying to mount with winfsp installed but i always get errors under Windows. Every path i set in the config.yaml i always get this error:

6:04PM INFCannot create WinFsp-FUSE file system: mount point in use.

and if i try to specify a drive letter like

path: "X:"

i get this error:

6:07PM INF error mounting filesystems error="mkdir X:.: Unable to find the specified path"

I'm using last distribyted 0.9.1 version and i've installed the last WinFSP too.

Improve error handling

mount fs from user

Is it possible to mount files not from the user's root?
Plex does not see files of other users.

Multiple file access

Thanks for the project.
If you watch a video and then open a folder or do any action with another file, the video will freeze and the speed graph, after loading the folder, will show 0B/s.
I understand correctly that at the moment it is possible to work with only one file?
What about buffering the video so that chunks are downloaded in advance? I think it's worth putting it into the config.

How to view zip files

Could you please ask when to write the tutorial

runtime error: invalid memory address or nil pointer dereference

Yesterday I started the service, and today it stopped working.
I opened the logs, but there was an endless repetition of this:

2024/02/25 14:00:46 [Recovery] 2024/02/25 - 14:00:46 panic recovered:
runtime error: invalid memory address or nil pointer dereference
/usr/lib/go/src/runtime/panic.go:261 (0x455c97)
        panicmem: panic(memoryError)
/usr/lib/go/src/runtime/signal_unix.go:861 (0x455c65)
        sigpanic: panicmem()
/home/kiv/go/pkg/mod/github.com/anacrolix/[email protected]/webrtc.go:56 (0xbc615d)
/home/kiv/go/pkg/mod/github.com/anacrolix/[email protected]/torrent.go:1950 (0xbbafb4)
/home/kiv/go/pkg/mod/github.com/anacrolix/[email protected]/torrent.go:1934 (0xbbadd3)
/home/kiv/go/pkg/mod/github.com/anacrolix/[email protected]/torrent.go:1927 (0xbbabf9)
/home/kiv/dev/distribyted/torrent/stats.go:204 (0xd92a4d)
/home/kiv/dev/distribyted/torrent/stats.go:154 (0xd924b5)
/home/kiv/dev/distribyted/http/api.go:40 (0xec7544)
/home/kiv/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 (0xec95b8)
/home/kiv/dev/distribyted/http/http.go:77 (0xec9587)
/home/kiv/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 (0xeb04ca)
/home/kiv/go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:172 (0xec91a4)
/home/kiv/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 (0xebc4f9)
/home/kiv/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:102 (0xebc4e7)
/home/kiv/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 (0xebb3da)
/home/kiv/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620 (0xebb06d)
/home/kiv/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576 (0xebab9c)
/usr/lib/go/src/net/http/server.go:2938 (0x72a2ad)
        serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
/usr/lib/go/src/net/http/server.go:2009 (0x726193)
        (*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
/usr/lib/go/src/runtime/asm_amd64.s:1650 (0x4751a0)
        goexit: BYTE    $0x90   // NOP