is it possible to add flag for this. I want to avoid spam on terminal for folders with many files.

Thanks for tool. it's really fast.

Commands to reproduce: yay -S torrenttools
Errors:

/usr/include/fmt/core.h:1757:7: note: ‘formattable’ evaluates to false
make[3]: *** [external/dottorrent/CMakeFiles/dottorrent.dir/build.make:132: external/dottorrent/CMakeFiles/dottorrent.dir/src/chunk_reader.cpp.o] Error 1
make[3]: *** [external/dottorrent/CMakeFiles/dottorrent.dir/build.make:104: external/dottorrent/CMakeFiles/dottorrent.dir/src/chunk_hasher_single_buffer.cpp.o] Error 1
make[3]: *** [external/dottorrent/CMakeFiles/dottorrent.dir/build.make:118: external/dottorrent/CMakeFiles/dottorrent.dir/src/chunk_processor_base.cpp.o] Error 1
make[2]: *** [CMakeFiles/Makefile2:1258: external/dottorrent/CMakeFiles/dottorrent.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:1109: CMakeFiles/torrenttools.dir/rule] Error 2
make: *** [Makefile:577: torrenttools] Error 2
==> ERROR: A failure occurred in build().
    Aborting...
 -> error making: torrenttools

Aur package version: torrenttools-0.6.0-1

For example like this

root@ubuntu:/home/ubuntu# torrenttools create "/var/filebrowser/srv/VMware-workstation-full-16.2.1-18811642.exe /var/filebrowser/srv/VMware-workstation-full-16.2.1-18811642.exe.md5" -a "https://tracker.tamersunion.org/announce" "udp://opentor.org:2710/announce" -d "router.utorrent.com:6881" "router.bittorrent.com:6881" "dht.transmissionbt.com:6881" "router.bitcomet.com:6881" "dht.aelitis.com:6881" -o "VMware-workstation-full-16.2.1-18811642.exe.torrent"
Error: Path does not exist: /var/filebrowser/srv/VMware-workstation-full-16.2.1-18811642.exe /var/filebrowser/srv/VMware-workstation-full-16.2.1-18811642.exe.md5

I want to package and distribute the file with its md5 file, but it looks like it cannot be created at this stage

There's no option like that in profile config.

Trying to create a torrents with ~200,000 files, 2 TiB in total, and the memory usage surged up to 12 GB when hashing. The progress isn't moving and stuck at 0 B/s. Any ideas for optimizing this?

Edit: The progress began moving after a few minutes, at a reasonable rate. But the memory usage is still high.

Edit2: No. The progress is actually broken. It first went extremely fast to 100%, and freezes. But the network activity shows the progress is still running.

Thanks in advance.

# torrenttools create --config ~/config.yml --trackers-config ~/trackers.json -t 1 -o ~/torrenttools-ftp.torrent -a udp://tracker.opentrackr.org:1337/announce 03.Audio\ Courses,\ Textbooks/
Scanning target directory: 0 files processed
Sorting file list... Done.
Adding files to metafile... Done.
Metafile:          /root/torrenttools-ftp.torrent
Protocol version:  v1
Piece size:        16 MiB (16777216 bytes)
Piece count:       567
Created by:        torrenttools/0.5.0
Created on:        2023-01-15 02:52:50 UTC
Private:           false
Name:              03.Audio Courses, Textbooks
Source:
Comment:
Announce-urls:     tier 1  - udp://tracker.opentrackr.org:1337/announce
DHT nodes:
Web seeds:
HTTP seeds:
Similar torrents:
Collections:
Error: wstring_convert::from_bytes

Happens both on Alpine container and with AppImage download.

'torrenttools show size' displays wrong filesize if the torrent contains a single file above 4GB. This torrent has a piece length of 4KB. Need a solution. Thank you in advance.

Using the hash of the announce URL as entropy for cross-seeding can lead to a potential privacy violation. Consider an anonymous uploader on a private tracker. If they use torrenttools's cross-seed feature, it becomes possible to tell if two anonymous uploads came from the same person, unless the uploader regularly rotates their passkey (which is usually a painful process). Please consider just using a random value for this like pyrocore does.

I'm having a problem verifying some large torrents that a (doubled, tripled) verified by transmission but have either their unique large file or the last of their files listed as 99% by torrenttools.

I was wondering if I had a way to get detailed output and/or instructions to debug this ?

[I then picked torf (from your performance chart) and verified one torrent. It agrees with transmission so I'm inclined to think I found a bug. How can I help?]

Hey! The project looked extremely promising so I had to give it a go. Unfortunately I have run into an error on my first torrent creator.

torrenttools create "E:\Docker\Deluge\Data\linux.iso" --piece-size 4M --announce "XXX" -p on -o "E:\Docker\Deluge\Data\linux.torrent"

16.00 EiB in 0 directories, 1 files

Error: std::bad_alloc

Hi,

Regardless of whether I use the default config.yml, or some configured one, I am getting this error.

Error: [json.exception.parse_error.101] parse error at line 10, column 15: syntax error while parsing object - unexpected string literal; expected '}'

If I edit line 10 of the config.yml to enclose the {pid}, like so
pid: {example_pid}

Then I receive this error.
Error: yaml-cpp: error at line 10, column 10: bad conversion

All binary releases should be signed to allow users to validate the contents of packages.
Currently only deb packages are signed since it is required for building on launchpad.

[snk-tv.pro].t150.torrent
torrenttools-0.6.0-1.fc33.x86_64.rpm

torrenttools info [snk-tv.pro].t150.torrent
Metafile: /var/data/Upload/[snk-tv.pro].t150.torrent
Protocol version: v1
Infohash: 14570758501b25e736a1b29ad8b08dfbeb16fe5c
Piece size: 1 MiB (1048576 bytes)
Piece count: 802
Created by: uTorrent/3.4.5
Created on: 2021-08-05 15:16:48 UTC
Private: false
Name: King Gary s2 1080p SNK-TV
...

transmission-show [snk-tv.pro].t150.torrent
Name: King Gary s2 1080p SNK-TV
File: [snk-tv.pro].t150.torrent

GENERAL

Name: King Gary s2 1080p SNK-TV
Hash: 79cdf0937d3a6e35200aac218a36a6abb8e4fa33
Created by: uTorrent/3.4.5
Created on: Thu Aug 5 22:16:48 2021
Comment: http://snk-tv.pro/viewtopic.php?t=150
Piece Count: 802
Piece Size: 1.00 MiB
Total Size: 840.0 MB
Privacy: Public torrent
...

This is related to this comment in a Reddit post.

https://github.com/fbdtemme/torrenttools/blob/main/src/create.cpp#L392
is that a typo?

Using:

--comment "this is a comment"  \
--source "testsource" 

Ouput:

Source:           
Comment:          this is a comment

Using only:
--source "testsource"

Ouput:

Source:           
Comment:          this is a comment

Add an options to disable the progress bar.
Simple progress reporting is already implemented when the output is piped to a file but there is currently no option to enable it manually.

Hi,
I 'm using torrenttools-0.3.2 and it works fine on my centos 7 server.
But when i upgrade it to torrenttools-0.5.1 it is not working any more.
It stuck on this

Scanning target directory: 0 files processed
Sorting file list... Done.
Adding files to metafile... Done.

ldd /usr/bin/torrenttools

linux-vdso.so.1 =>  (0x00007ffeb21f0000)
libtbb.so.2 => /lib64/libtbb.so.2 (0x00007f596e813000)
libcrypto.so.1.1 => /usr/local/openssl/lib/libcrypto.so.1.1 (0x00007f596e34d000)
libstdc++.so.6 => /usr/local/gcc8/lib64/libstdc++.so.6 (0x00007f596dfc9000)
libm.so.6 => /lib64/libm.so.6 (0x00007f596dcc7000)
libgcc_s.so.1 => /usr/local/gcc8/lib64/libgcc_s.so.1 (0x00007f596daaf000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f596d893000)
libc.so.6 => /lib64/libc.so.6 (0x00007f596d4c5000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f596d2c1000)
librt.so.1 => /lib64/librt.so.1 (0x00007f596d0b9000)
libz.so.1 => /lib64/libz.so.1 (0x00007f596cea3000)
/lib64/ld-linux-x86-64.so.2 (0x00007f596ea55000)

I attached strace.log please take a look.
Please any advise, thank you.
strace.log

Thanks for what looks to be a fantastic tool.

There are some cases where having a Docker image to easily run torrenttools might be more convenient / desirable than the existing COPR / launchpad repos.

Would you be willing to consider publishing an official Docker image that is updated as part of releases?

I set up a docker container with docker-compose but the docker shuts down immediately since he is not receiving a command. I would like him to idle.

when I solely run docker run -it --rm fbdtemme/torrenttools this appears and the container exits:

A subcommand is required
torrenttools v0.5.0
Tools for inspecting, creating and modifying bittorrent metafiles.

Usage: torrenttools [OPTIONS] SUBCOMMAND

Options:
  -h,--help                        Print this help message and exit
  --checksum-algorithms            List the supported hash functions for per file checksums.
  -V,--version                     Show program version and exit.
  --config <path>                  Path to custom location for the config.yml file.
  --trackers-config <path>         Path to custom location for the trackers.json file.

Subcommands:
  info                             General information about BitTorrent metafiles.
  create                           Create BitTorrent metafiles.
  verify                           Verify local data against BitTorrent metafiles.
  show                             Show specific fields of BitTorrent metafiles.
  edit                             Edit BitTorrent metafiles.
  magnet                           Generate a magnet URI for BitTorrent metafiles.
  pad                              Generate padding files for a BitTorrent metafile.

But I have not even entered the container.

Here my docker-compose.yml:

version: "3.8"
services:
  torrenttools:
    image: fbdtemme/torrenttools
    container_name: torrenttools
    stdin_open: true
    tty: true 
    volumes:
      - "REDACTED:REDACTED"
    restart: unless-stopped

Same if I want to start the container with docker-compose file.

Seesm to me like the container expects an immediate command, but nothing is written in the documentary. The docker setup is not even mentioned there, nor anything additional on DockerHub.

Please make the docker container persistent, to I can launch him. And then exec into him with:

docker exec -it torrenttools /bin/bash

Or something like that, and then I can do my thing there.

The current build fail at many steps.

1. I assume that Bencode recently changed it's master branch to main. But the torrentools makefile still search for a master tag in external/bencode.cmake: GIT_TAG master

   • Solution: by changing it to "main".

2. Then is fails because upstream errors with bencode compilation. I opened an issue about this upstream : Bencode: Build errors with deps and instructions

   • Current solution: by compiling and installing the deps manually.

3. Then it fails because it does not find nlohmann/json.

   • Current solution: by compiling and installing manually.

[ 81%] Building CXX object CMakeFiles/torrenttools.dir/src/common.cpp.o
In file included from /home/tomasz/Builds/torrenttools/include/common.hpp:8,
                 from /home/tomasz/Builds/torrenttools/src/common.cpp:4:
/home/tomasz/Builds/torrenttools/include/tracker_database.hpp:11:10: fatal error: nlohmann/json.hpp: Aucun fichier ou dossier de ce type
   11 | #include <nlohmann/json.hpp>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.

4. Some compilation errors with a pattern in lines 50 & 65 of include/file_matcher.hpp :
   • Unresolved: I stopped here because I'm not good enough with C++

[ 81%] Building CXX object CMakeFiles/torrenttools.dir/src/create.cpp.o                                                                                                                      
In file included from /home/tomasz/Builds/torrenttools/src/create.cpp:22:                                                                                                                    
/home/tomasz/Builds/torrenttools/include/file_matcher.hpp: In member function ‘void torrenttools::file_matcher::allow_extension(std::string_view)’:                                          
/home/tomasz/Builds/torrenttools/include/file_matcher.hpp:50:31: error: unable to find string literal operator ‘operator""_format’ with ‘const char [7]’, ‘long unsigned int’ arguments      
   50 |         std::string pattern = ".*.{}$"_format(extension);                                                                                                                            
      |                               ^~~~~~~~~~~~~~~                                                                                                                                        
/home/tomasz/Builds/torrenttools/include/file_matcher.hpp: In member function ‘void torrenttools::file_matcher::block_extension(std::string_view)’:                                          
/home/tomasz/Builds/torrenttools/include/file_matcher.hpp:65:31: error: unable to find string literal operator ‘operator""_format’ with ‘const char [8]’, ‘long unsigned int’ arguments      
   65 |         std::string pattern = ".*\\.{}$"_format(extension);                                                                                                                          
      |                               ^~~~~~~~~~~~~~~~~

5. It's not an error but you should add this dependancy in the Readme : termcontrol

When I try to create a .torrent on Fedora 37 with last version, I get following error:
/usr/include/c++/11/string_view:233: constexpr const value_type& std::basic_string_view<_CharT, _Traits>::operator[](std::basic_string_view<_CharT, _Traits>::size_type) const [with _CharT = char; _Traits = std::char_traits<char>; std::basic_string_view<_CharT, _Traits>::const_reference = const char&; std::basic_string_view<_CharT, _Traits>::size_type = long unsigned int]: Assertion '__pos < this->_M_len' failed. Aborted (core dumped)

I’ve tried different files, but no luck. Other functions seem to be working. Any help?

Whether I use utf-8 or gb2312, it produces garbled characters.

Hello,

@fbdtemme mentioned DEB packages in #11.

Since the PPA cannot be used in Ubuntu 21.10, where can I find those DEB files ?

I looked at launchpad.net/torrenttools but :

torrenttools does not have any download files registered with Launchpad.

Thanks

It would be great if more private trackers are included in the default trackers configuration file.
If you want to have a tracker added you can post the information here or send a pull request.

When creating a torrent file the time to collect all files that match the filters can take a long time if the input directory contains many files. torrenttools should run this step in a separate thread instead of on the main thread and provide simple reporting of how many files are process. This will avoid giving the impression of a freeze if this steps takes longer then a few seconds.

I would love a silent option for torrent creation that displays nothing.
Like the other OP asking for silent mentioned. It is crazy stupid fast dude. I love it. Saves me a TON of work.

It seems there isn't a release of torrenttools for the Ubuntu jammy in the launchpad repo, and as such causes an error upon attempting to update repos

This option allows users to easily cross-seed torrents without having to manually define a source flag.
A unique id made from the hash of all announce-urls is embedded inside the info dict.
Behaviour is almost the same as the --source options except that the user does not care what the value of the unique value is exactly.

Apparently there are trackers that strip the source field so a different flag is needed.

This option is implemented by pyrocore with the x_cross_seed field.

After executing torrenttools, the arrow keys no longer move the cursor left and right, but produce the escape sequences ("^[[D^" and the like). Backspace is also affected, now deleting entire words rather than single characters.

When you want to specify a custom announce URL but some part of the URL can be matched to an existing entry in trackers.json, then trackers.json will take precedence.

Using:
torrenttools create -a 'http://please.passthepopcorn.me:2710/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/announce' -p -l 21 -v '1' -o ~/somewhere/ ~/somewhere/something.iso

Result:
Announces:
tier 1 - http://please.passthepopcorn.me:2710/01qwqwdlc922_sample_pid_d93fqd6fji9/announce

The reason for this is that the "passthepopcorn" portion of the URL matches with the name of an entry in trackers.json. This shouldn't happen.

Changing the URL to "please.passthepopcornxxx.me" ... Result:
Announces:
tier 1 - http://please.passthepopcornxxx.me:2710/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/announce

Of course, I can remove the entry from trackers.json (since I won't be using it), but it would nice if they can co-exist

I would think that, checking if the announce value starts with http, and if yes, then do not check in trackers.json

Hi, while installing torrenttools-0.6.2-macos-x86_64.pkg on Mac OS 13.1, the following error message occurs:

There is no other information provided.

Hi,

I'm not sure what I'm doing wrong, but I just can't get it to work :

$ docker run -v /media:/media --name torrenttools --rm fbdtemme/torrenttools create -v hybrid -a udp://... udp://... -o /media/ '/media/shows/...'
Configuration error: no configuration was found
Warning: could not find config.yml file: not all features will be available.
$ 

It just exits without generating a torrent.
I've tried mapping an empty file using -v ./empty:/etc/torrenttools/config.yml but it makes no difference, exact same error.

https://ptpimg.me/k07nar.png

On the Windows version, I am using the command torrenttools edit -a "http://newtracker.url/announce.php" "A.Release.by.a-GROUP.torrent" , which crashes, producing no output. Afterwards, using torrenttools info "A.Release.by.a-GROUP.torrent" shows that the announce URL has not been edited. I have also tried using the -o option, as well as using a named tracker (which doesn't update any of the relevant parts of the torrent).

On Linux, this command works as intended.

Event Viewer log:

Faulting application name: torrenttools.exe, version: 0.0.0.0, time stamp: 0x610e32a1
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x00007ff7ab3c0000
Faulting process ID: 0x3680
Faulting application start time: 0x01d797908bc46f20
Faulting application path: C:\Program Files\torrenttools\torrenttools.exe
Faulting module path: unknown
Report ID: 541fa805-e161-4d41-8cea-92aeb9bce5e1
Faulting package full name: 
Faulting package-relative application ID: 

macOS is left out of the following

# This is an example configuration file.
# It should be copied to the suitable directory for your OS and modified there.
# Linux : $HOME/.config/torrenttools/config.yml
# Windows : %AppData%/torrenttools/config.yml

I use the include filter to select 1700 files from a folder with many more. A 700 file filter is no problem. I get the error Error: re2 compiler out of memory. I have 5GB available so I think I have enough for the regexp, but perhaps there's a configured cap somewhere?

What about giving an option of increasing the max memory usage for re2?

cat /proc/meminfo
MemTotal:        8040328 kB
MemFree:          134448 kB
MemAvailable:    5704240 kB

I followed instructions how to build on Ubuntu (I use Debian 11) this command:
cmake --build . --target torrenttools

Gave me this error:
-- Using cryptographic library: OpenSSL -- Could NOT find TBB (missing: TBB_DIR) CMake Error at cmake/FindTBB.cmake:142 (message): Required library TBB not found. Call Stack (most recent call first): cmake/FindTBB.cmake:378 (findpkg_finish) CMakeLists.txt:54 (find_package)

I used torrenttools to create a hybrid torrent on about 1TiB of data. The resulting file is about 20MB and can't be opened by qBittorrent ("the v1 and v2 metadata does not match"), but actually even torrenttools info out.torrent fails with an exception:

Catchpoint 1 (exception thrown), 0x00007ffff76a91b1 in __cxxabiv1::__cxa_throw (obj=0x555555989dd0, tinfo=0x55555593f598 <typeinfo for bencode::parsing_error>, dest=0x55555568b080 <bencode::parsing_error::~parsing_error()>) at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:81
81      ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc: No such file or directory.
(gdb) bt
#0  0x00007ffff76a91b1 in __cxxabiv1::__cxa_throw (obj=0x555555989dd0, tinfo=0x55555593f598 <typeinfo for bencode::parsing_error>, dest=0x55555568b080 <bencode::parsing_error::~parsing_error()>) at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:81
#1  0x000055555574f93e in bencode::decode_view (view="d8:announce34:udp://tracker.coppersurfer.tk:696913:announce-listll34:udp://tracker.coppersurfer.tk:6969el40:udp://tracker.leechers-paradise.org:6969el28:udp://exodus.desync.com:6969el42:udp://tracker."...)
    at /code/build/torrenttools/build/_deps/bencode-src/include/bencode/detail/decode_view.hpp:24
#2  0x000055555574d1ca in dottorrent::read_metafile (view="d8:announce34:udp://tracker.coppersurfer.tk:696913:announce-listll34:udp://tracker.coppersurfer.tk:6969el40:udp://tracker.leechers-paradise.org:6969el28:udp://exodus.desync.com:6969el42:udp://tracker."...)
    at /code/build/torrenttools/build/_deps/dottorrent-src/src/metafile.cpp:262
#3  0x000055555574d7d6 in dottorrent::load_metafile (path=filesystem::path "/some/redacted/path/out.torrent" = {...}) at /code/build/torrenttools/build/_deps/dottorrent-src/src/metafile.cpp:324
#4  0x0000555555674291 in run_info_app (main_options=..., options=...) at /code/build/torrenttools/src/info.cpp:52
#5  0x000055555568dd9c in main (argc=3, argv=0x7fffffffd408) at /code/build/torrenttools/src/main.cpp:71

The exception message is Error: parse error: invalid character read at position 9302219: value limit exceeded.
Here is the (lightly redacted) start of the log, during the torrent creation:

Scanning target directory: 56509 files processed
Sorting file list... Done.
Adding files to metafile... Done.
Metafile:          /some/redacted/path/out.torrent
Protocol version:  v1 + v2 (hybrid)
Piece size:        4 MiB (4194304 bytes)
Piece count:       263162
Created by:        torrenttools/0.6.2
Created on:        2022-07-18 19:46:21 UTC
Private:           false
Name:              test
Source:            
Comment:           
Announce-urls:     tier 1  - udp://tracker.coppersurfer.tk:6969
                   tier 2  - udp://tracker.leechers-paradise.org:6969
                   tier 3  - udp://exodus.desync.com:6969
                   tier 4  - udp://tracker.opentrackr.org:1337/announce
                   tier 5  - udp://tracker.uw0.xyz:6969/announce
DHT nodes:         
Web seeds:         
HTTP seeds:        
Similar torrents:  
Collections:       Test collection
Other info fields: 

Files:

Metafile contains more than 1000 files: skipping file tree ...

  1 TiB in 6516 directories, 57385 files


Hashing files...
[progress snipped]

I can provide the 20MB output torrent privately if necessary.

this needs to be main, currently torrenttools doest build because of this

honestly you should either copy 3rd party libs into the project or add it as a git submodule fixed at a specific tag to avoid stuff like this in the future

Using:
torrenttools create -a "https://blah1.com" -p -l 21 -v '1' -o ~/somewhere/ ~/somewhere/something

Results in no announce URL being added to the metafile

Announces:

Using:
torrenttools create -a "https://blah1.com" "https://blah2.com" -p -l 21 -v '1' -o ~/somewhere/ ~/somewhere/something

Results in both URLs being added to the metafile

Announces:
tier 1 - https://blah1.com
tier 2 - https://blah2.com

Hi

I have installed "sudo apt install torrenttools" and then installed ISA-L Crypto from their github. does that mean i have torrenttools with ISA-L Crypto support? or am i still using openssl

Hi, I tried building following the instructions but it fails:
torrenttools/src/tracker_database.cpp:7:10: fatal error: fmt/args.h: No such file or directory

I am on Arch Linux, and fmt is installed: fmt 7.1.3-1

macOS fails with the following error

xxx@xxx-Mac-mini ~/Downloads> /usr/local/bin/torrenttools
dyld: lazy symbol binding failed: Symbol not found: ___stack_logging_early_finished
  Referenced from: /Library/torrenttools/bin/../lib/libSystem.B.dylib
  Expected in: /Library/torrenttools/bin/../lib/libSystem.B.dylib

dyld: Symbol not found: ___stack_logging_early_finished
  Referenced from: /Library/torrenttools/bin/../lib/libSystem.B.dylib
  Expected in: /Library/torrenttools/bin/../lib/libSystem.B.dylib

fish: Job 1, '/usr/local/bin/torrenttools' terminated by signal SIGABRT (Abort)

This is on an M1 mac with Rosetta installed

Some trackers (such as HDB) use the full name there, while others (such as PTP) use the abbreviation. It would be useful to be able to match it exactly without specifying -s each time, this way there's no need to redownload the torrent to start seeding.

Currently there is support for BEP-19: WebSeed - HTTP/FTP Seeding (GetRight style), but not for the other HTTP Seeding mechanism specified in BEP-17 which adds the "url-list" key.

BEP-17 HTTP seeding is superseeded by BEP-19 and should not be used in new torrents, but it is still usefull to support this for editing and inspecting older torrent files.