Expected behaviour: Just as Torrent.read(), but from bytes object.

example:

var t = bytes('from.some.torrent')
rtb = Torrent.readBytes(t)

Returns: New Torrent instance

I'm creating single file torrents for large files, and the files are stored on a remote server (an S3 bucket). Rather than downloading the large file and requiring the disk space for it, I'd like to be able to stream the content directly into the generator. Is that possible? It almost looked possible to me reading through the internals, but at the end of the day it seems like the exposed interface really expects a file on disk.

Ideally it would be great to do this for a multi-file torrent, but that would understandably be more difficult.

I'm struggling to make a torrent that does not create a directory for the download

for example I have video.mp4 in files : path='files/video.mp4' #Relative path from the script

When people download the torrent, I want them to only have video.mp4 in their download folder, not files/video.mp4

My files folder always contains only 1 video, but the video name can change so I prefer not to hardcode the filename in the program

Is it possible to do this ?

__enter and __exit method are missed in class Torrent, and 'with' cannot be used.
code:
with Torrent(....) as t:
----....
----t.write()
I got an error:
AttributeError: __enter__

On linux, the default limit is 1024 open files ulimit -n
This error can be temporarily remedied by running ulimit -n 5000 or whatever number

I imagine it's not a simple fix within torf, so this is mainly for anyone else that runs into this problem.

Traceback (most recent call last):
  File "/home/w/.local/lib/python3.10/site-packages/torf/_stream.py", line 382, in _get_open_file
    self._open_files[filepath] = open(filepath, 'rb')
OSError: [Errno 24] Too many open files: "FILENAME"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/h/convert/out/torrent.py", line 35, in <module>
    t.generate(callback=cb, interval=0.5)
  File "/home/w/.local/lib/python3.10/site-packages/torf/_torrent.py", line 1114, in generate
    piece_hashes = collector.collect()
  File "/home/w/.local/lib/python3.10/site-packages/torf/_generate.py", line 337, in collect
    self._collect(*task)
  File "/home/w/.local/lib/python3.10/site-packages/torf/_generate.py", line 367, in _collect
    maybe_cancel = self._callback(
  File "/home/w/.local/lib/python3.10/site-packages/torf/_generate.py", line 427, in __call__
    return self._intervaled_callback(piece_index, pieces_done, pieces_total,
  File "/home/w/.local/lib/python3.10/site-packages/torf/_generate.py", line 412, in __call__
    return self._callback(*args)
  File "/home/w/.local/lib/python3.10/site-packages/torf/_generate.py", line 455, in _call_callback
    raise exceptions[0]
  File "/home/w/.local/lib/python3.10/site-packages/torf/_stream.py", line 445, in iter_pieces
    fh = self._get_open_file(filepath)
  File "/home/w/.local/lib/python3.10/site-packages/torf/_stream.py", line 384, in _get_open_file
    raise error.ReadError(e.errno, filepath)
torf._errors.ReadError: FILENAME: Too many open files

Thank you! This tool is too powerful。

Hi,
Might you consider switching back to MIT or switching to LGPL so we can use this as a package?
Thank you!

All modern clients support this so please support creating 32MB piece size torrents.

Sorry for the question as i am new to python, i work with the code:

t = Torrent(path=folder_path,
trackers=['https://example:2910/announce'],                                     
t.private = True
t.generate()

and it works, but if i want to remove the file who as an extension .nfo per example i write the t.exclude_globs('*.nfo'), and it gives me an error, what i am doing wrong please.

t.exclude_globs('*.nfo') gives an error:
maybe its not torf fault:

i get the folder path from pysimplegui, like this:
file_path = values[1]

the debug:

torrent file's infohash got change after load, found at least 2 torrent file.

test code:

import torf

t = print(torf.Torrent.read('/config/torrents/sample1.torrent'))
print(t.infohash)

result:

sample.zip

I recently encountered this torrent file in the wild and it has one invalid tracker url out of many.

Trying to read it raises a metainfo error

Traceback (most recent call last):
  File "C:\Users\raven\Documents\GitHub\TEST\test.py", line 10, in <module>
    torrent = Torrent.read_stream(BytesIO(open(path, "rb").read()))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\raven\Documents\GitHub\TEST\.venv\Lib\site-packages\torf\_torrent.py", line 1632, in read_stream
    torrent.validate()
  File "c:\Users\raven\Documents\GitHub\TEST\.venv\Lib\site-packages\torf\_torrent.py", line 1388, in validate
    utils.assert_type(md, ('announce-list', i, j), (str,), check=utils.is_url)
  File "c:\Users\raven\Documents\GitHub\TEST\.venv\Lib\site-packages\torf\_utils.py", line 727, in assert_type
    raise error.MetainfoError(f"{keychain_str}[{key!r}] is invalid: {obj[key]!r}")
torf._errors.MetainfoError: Invalid metainfo: ['announce-list'][4][0] is invalid: '*udp://9.rarbg.to:2710/announce'

I understand this is an invalid URL but is it possible to drop invalid metainfo instead of erroring? especially non critical ones like this where there are several more working trackers. This is a perfectly downloadable torrent and clients like Qbit simply report it as unsupported while continuing to download it

Possibly a strict: bool = True flag in read() and read_stream() where:

• True behaves like the current behavior, erroring at any metainfo error
• False drops non critical info instead of erroring

Thank you

I'm on Arch Linux, and python-torf installs the file /usr/lib/python<version>/site-packages/tests/__init__.py. This same file is also installed by package python-hypothesis. Since these two packages installs the same file, this results in a package conflict (where both packages cannot be installed at the same time).

The list of files installed by package python-hypothesis on Arch Linux can be seen here. I've noticed that this file is being installed by python-hypothesis since its version 5.37.0.

Can this file conflict with python-hypothesis be fixed here?

I'm trying to create a torrent with a single announce url but I'm having issues.

try: torrent = torf.Torrent(path=path, trackers='https://example.com/announce/dhjhj5h4hfh4fkejfs9dhfgj0', private=True, comment='This is a comment', piece_size=piece_size, created_by='torf/3.0.0', source='EXP')

Gives this error...

Traceback (most recent call last): File "/home/<REDACTED>/bin/autoup3.py", line 383, in <module> torrent_path = str(create_torrent(path, args.overwrite)) File "/home/<REDACTED>/bin/autoup3.py", line 348, in create_torrent source='EXP') File "/home/<REDACTED>/env/lib/python3.6/site-packages/torf/_torrent.py", line 96, in __init__ self.trackers = trackers File "/home/<REDACTED>/env/lib/python3.6/site-packages/torf/_torrent.py", line 390, in trackers tier = [utils.validated_url(str(item))] File "/home/<REDACTED>/env/lib/python3.6/site-packages/torf/_utils.py", line 47, in validated_url raise error.URLError(url) torf._errors.URLError: h: Invalid URL

The only way I can make it create a torrent successfully is to make trackers a list with the single tracker url and that adds announce-list to the torrent file.

I only want announce in the torrent file with a single url.

Am I doing this wrong?

For example, I have a file named "out.html" and want it to be named "index.html" inside the torrent file.
I am loading the paths by setting filepaths, so I figured it could be possible to either have it when setting or change it after under metainfo["info"]["files"].

MetainfoError is raised if piece size is not a power of two. This behaviour is incorrect.

From https://blog.libtorrent.org/2020/09/bittorrent-v2/
In v1 torrents, the size of a piece is not restricted by the specification. It doesn’t make much sense to have a piece smaller than the block size of 16 kiB, but it’s not disallowed. The vast majority of torrents that are created use a power-of-two piece size, but there are a few outliers that are not, but still divisible by 16 kiB

I propose the is_power_of_2 check be replaced by a is_divisible_by_16_kib check.

Does it only works on python 3.6??
I installed torf using
pip install torf

I am just trying out the example provided When I try this on Python 3.6 virtualenv

`import torf

t = torf.Torrent(path='1.jpg',
trackers=['https://tracker1.example.org:1234/announce',
'https://tracker1.example.org:1234/announce'],
comment='This is a comment')
t.private = True`

I get this

Traceback (most recent call last): File "/home/vaibhav/projects/test/py36test/torf.py", line 1, in <module> import torf File "/home/vaibhav/projects/test/py36test/torf.py", line 3, in <module> t = torf.Torrent(path='1.jpg', AttributeError: module 'torf' has no attribute 'Torrent'

I am getting this on every python envs.

I tried this on python3.3
from torf import Torrent

`Python 3.3.5 |Continuum Analytics, Inc.| (default, Jun 4 2015, 15:22:11)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.

from torf import Torrent
Traceback (most recent call last):
File "", line 1, in
File "/home/vaibhav/anaconda3/envs/iag/lib/python3.3/site-packages/torf/init.py", line 26, in
from ._errors import *
File "/home/vaibhav/anaconda3/envs/iag/lib/python3.3/site-packages/torf/_errors.py", line 37
super().init(f'{url}: Invalid URL')
^
SyntaxError: invalid syntax
`

Am I missing out something?

Hi @rndusr,

Thanks for your hard working.
I want to download several files in a torrent which contains a lot of files.
Can a torrent file be modified by torf? I want to remove some files in the torrent.

a test

from torf import Torrent                                                  
                                                                          
torrent = Torrent.read('./temp/old.torrent')                              
torrent.files = [torrent.files[0], torrent.files[2], torrent.files[-1]]   
torrent.write('./temp/new.torrent')

but the output is:

Traceback (most recent call last):
File "test.py", line 5, in
torrent.write('./temp/new.torrent')
File "D:\work\02work\work\009torrent\rw-torrent\venv\lib\site-packages\torf_torrent.py", line 1477, in write
self.write_stream(content, validate=validate)
File "D:\work\02work\work\009torrent\rw-torrent\venv\lib\site-packages\torf_torrent.py", line 1448, in write_stream
content = self.dump(validate=validate)
File "D:\work\02work\work\009torrent\rw-torrent\venv\lib\site-packages\torf_torrent.py", line 1435, in dump
self.validate()
File "D:\work\02work\work\009torrent\rw-torrent\venv\lib\site-packages\torf_torrent.py", line 1331, in validate
utils.assert_type(md, ('info', 'pieces'), (abc.ByteString,), must_exist=True)
File "D:\work\02work\work\009torrent\rw-torrent\venv\lib\site-packages\torf_utils.py", line 718, in assert_type
raise error.MetainfoError(f'Missing {key!r} in {keychain_str}')
torf._errors.MetainfoError: Invalid metainfo: Missing 'pieces' in ['info']

Looking forward to your reply.

I realize that 32MB piece size support was added recently but it seems the new piece size support standard for most clients currently in use is 64MB.

• qBittorrent supports 64MB pieces and larger (and will create up to to 256MB piece sizes in qbittorrent/qBittorrent#19535)

• Transmission 4.0 supports up to 64MB pieces (as shown in transmission/transmission#3768)

• Deluge supports 64 MB piece size torrents

• BiglyBT supports 64 MB piece size torrents.

• https://github.com/kz26/dottorrent already supported creation of 64MB but that ability was lost in torf.

• py3createtorrent recently went to 64 MB in rsnitsch/py3createtorrent#34

• uTorrent 3.x supports 64MB piece size torrents

• Tixati 3.x supports 64MB piece size torrents as of https://forum.tixati.com/support/7137

That's all the major clients in use today.

Please allow us to keep up with these clients by giving us the ability to create 64MB piece size torrents. That should be sufficient for years to come as torrents keep getting bigger and bigger.

Torrent class allow read torrent from file. But sometimes we need read from memory like StringIO or BytesIO.

I'm not sure how to best explain this. If I supply path to a directory containing a single file, the end result is a torrent with the content matching the relative paths to the file. So the content ends up looking like, and saving like, the directory structure.

If i pass a path to a single file, no matter what name i give it, its always in the "root" of the content directory, if that makes sense. I want the single file to match the relative paths like in the multi-file.

I tried doing the following but it still includes all files

t = Torrent(
    path='/media/instructionals/tryer/',
    include_globs=('2015/2015-01-06_Update_23.mp4',),
    # exclude_globs=('*.mp4',),
    trackers=['https://localhost:123/announce'],
    comment='test torrent'
)

This ends up with the content/directory structure I want but it includes all files not just the one I've listed in include_globs. If I uncomment the exclude_globs then it complains of no files because all files are excluded.

I'm having issues creating large torrents due to max piece size limits.

torf._errors.PieceSizeError: Piece size must be between 16384 and 16777216: 67108864

My piece size calc code can use piece sizes from 16KB to 256MB

The above error was trying to create a torrent with a 64MB piece size

I saw in the docs that the values can be set here...
https://torf.readthedocs.io/en/latest/#torf.Torrent.piece_size_min

But I guess I'm not implementing it correctly.

My code...

torrent = torf.Torrent(path=path,
trackers=torrent_announce,
private=True,
comment='This is a comment',
piece_size=piece_size,
piece_size_min='16384',
piece_size_max='268435456',
created_by='torf/3.0.0',
source=torrent_source)

But get TypeError: __init__() got an unexpected keyword argument 'piece_size_min'

How am I supposed to set piece_size_min, piece_size_max?

Thanks for bearing with my ignorance and for any help you can provide.

Hello,
Using latest version in python.
Getting "Invalid metainfo: Missing 'pieces' in ['info']" at torrent file loading.

Example torrent file:
https://libtorrent.org/bittorrent-v2-test.torrent

Hi. I believe this module doesn't support magnet link parsing? Is it possible for you to implement it in the future?

Thanks

It appears that for large torrents, 64 MiB is the maximum piece size that can be chosen.

While I realize this can be overridden by hand, I'd suggest to lower this bound by default, as uTorrent, one of the most popular clients, seems to be unable to handle such large piece sizes. Naturally, that's their problem and not this library's, but in the end it's the users (the ones who download the torrent) who are hurt the most and I discovered it the hard way ^^

def callback(torrent, filepath, pieces_done, pieces_total):
     print(f'{pieces_done / pieces_total * 100:3.0f} % done')


 torrent = Torrent(path=path, trackers='https://tracker.example.to', private=True)

 success = torrent.generate(callback, interval=1)

outputs a TypeError

Traceback (most recent call last):
  File "tests.py", line 16, in <module>
    success = torrent.generate(callback, interval=1)
  File "/usr/local/lib/python3.8/site-packages/torf/_torrent.py", line 972, in generate
    queue_size=threads*3)
TypeError: unsupported operand type(s) for *: 'function' and 'int'

I tested this with version 2.1.0 and it works fine and gives me the expected output

filename.file
  1 % done
100 % done
100 % done
True

Hi. I have been testing multiple torrent and suddenly 1 of the torrents returning a wrong hash.

Manjaro%20Kde%2015.12%20x86_64.zip (or download from here)

If I added in qbittorrent, the hash registered is db4d75e4f9e846e641d043a36f73187bb364c111.
But somehow in torf, hash returned is 1925c683d9fe4156b8e4be0a193a157acd6582ed.

This is the code I am using

#!/usr/bin/env python3.8

from torf import Torrent
import json
import sys

if len(sys.argv) == 2:
    try:
        torrent = Torrent.read(sys.argv[1])
        torrentlist = {'hash': torrent.infohash, 'files': list(torrent.files)}
        print(json.dumps(torrentlist))
    except Exception as e:
        print(e)
else:
    print('No .torrent detected')

I am running the latest code.

As the title suggests, from some reason, when I try to run the install script setup.py, I get -rw------ permissions on the egg-info folder of torf.

It doesn't happen so while installing with pip --root <>. Any idea why this is happening?

The announce URL is serialized as a list containing a single element.
I think it should be serialized as a string since there's announce-list.

Sources:

• http://www.bittorrent.org/beps/bep_0003.html
• http://www.bittorrent.org/beps/bep_0012.html

After create a torrent file ,How to seed these files ?

use what kind of software ?

Can python handle it ?
with some code

the torrent file can download from https://nyaa.si/view/1284768

torf package version is 3.1.1

the infohash using torf is 95f3619484546e6735b0837e58d003cd0c93775f
but the correct value is fb8899e9faff4c0682fec40b0446cfbccbbbed18

Hello

I usually create a torrent with Torf that includes a folder with all the files inside. How do I create a torrent with subfolders and their respective files as well?

I tried with 'include_globs,' but it keeps creating a torrent with all the files excluding those from the subfolders.

for example:

torf.Torrent(path="/home/ridente/FolderA/")

FolderA
     hello_1.mkv
     hello_2.mkv
     FolderB
           hello_3.mkv
     FolderC
           hello_4.mkv

It will create a torrent with only 'hello_1.mkv' and 'hello_2.mkv' without including FolderB and FolderC (subfolders and files)

Thank you

This library makes use of collections.abc.ByteString in several locations, but this type has been deprecated (and produces a warning) in Python 3.12 and is slated for removal in Python 3.14. See the link for advice on how to replace it.

Hey

I decided to port dottorrent-gui to torf

https://github.com/SavageCore/torf-gui

Do you want to add this to the readme?

Hi,

in the wild world out there torrent files exist which have additional .utf-8 variants for name and path keys. They were introduced by BT Inc. unofficially and not specified by BEP3. (see this)[https://github.com/BiglySoftware/BiglyBT/issues/1274]

I guess the background were character set related things on some systems (e.g. Asian systems - they had different problems switching to UTF-8 as far as I know). The name and path keys in these situation can contain every character set out there (I guess normally the system character set or ASCII).

torf can correctly deserialize these torrents and also write new ones without that keys (see this)[https://github.com/casey/intermodal/issues/534]. But torf fails e.g. in a verify run with paths it can't find. I guess the reason is that because of the unknown and not correctly handled character set a wrong path is searched (output with strange characters indicate this). So torf uses the non .utf-8 variants.

torf should prefer .utf-8 key variants when they exist like clients do. I think it's not really BEP3 violation to do this.

Best Regards

I came across some trackers that use source flag, i think for cross seeding and to download directly from a .torrent file created, instead of downloaded from the tracker, is it an option in torf?
Sorry if this question is dumb.