rndusr / torf Goto Github PK
View Code? Open in Web Editor NEWPython module to create, parse and edit torrent files and magnet links
License: GNU General Public License v3.0
Python module to create, parse and edit torrent files and magnet links
License: GNU General Public License v3.0
I'm having issues creating large torrents due to max piece size limits.
torf._errors.PieceSizeError: Piece size must be between 16384 and 16777216: 67108864
My piece size calc code can use piece sizes from 16KB to 256MB
The above error was trying to create a torrent with a 64MB piece size
I saw in the docs that the values can be set here...
https://torf.readthedocs.io/en/latest/#torf.Torrent.piece_size_min
But I guess I'm not implementing it correctly.
My code...
torrent = torf.Torrent(path=path,
trackers=torrent_announce,
private=True,
comment='This is a comment',
piece_size=piece_size,
piece_size_min='16384',
piece_size_max='268435456',
created_by='torf/3.0.0',
source=torrent_source)
But get TypeError: __init__() got an unexpected keyword argument 'piece_size_min'
How am I supposed to set piece_size_min, piece_size_max?
Thanks for bearing with my ignorance and for any help you can provide.
I'm not sure how to best explain this. If I supply path to a directory containing a single file, the end result is a torrent with the content matching the relative paths to the file. So the content ends up looking like, and saving like, the directory structure.
If i pass a path to a single file, no matter what name i give it, its always in the "root" of the content directory, if that makes sense. I want the single file to match the relative paths like in the multi-file.
I tried doing the following but it still includes all files
t = Torrent(
path='/media/instructionals/tryer/',
include_globs=('2015/2015-01-06_Update_23.mp4',),
# exclude_globs=('*.mp4',),
trackers=['https://localhost:123/announce'],
comment='test torrent'
)
This ends up with the content/directory structure I want but it includes all files not just the one I've listed in include_globs. If I uncomment the exclude_globs then it complains of no files because all files are excluded.
It appears that for large torrents, 64 MiB is the maximum piece size that can be chosen.
While I realize this can be overridden by hand, I'd suggest to lower this bound by default, as uTorrent, one of the most popular clients, seems to be unable to handle such large piece sizes. Naturally, that's their problem and not this library's, but in the end it's the users (the ones who download the torrent) who are hurt the most and I discovered it the hard way ^^
I'm trying to create a torrent with a single announce url but I'm having issues.
try: torrent = torf.Torrent(path=path, trackers='https://example.com/announce/dhjhj5h4hfh4fkejfs9dhfgj0', private=True, comment='This is a comment', piece_size=piece_size, created_by='torf/3.0.0', source='EXP')
Gives this error...
Traceback (most recent call last): File "/home/<REDACTED>/bin/autoup3.py", line 383, in <module> torrent_path = str(create_torrent(path, args.overwrite)) File "/home/<REDACTED>/bin/autoup3.py", line 348, in create_torrent source='EXP') File "/home/<REDACTED>/env/lib/python3.6/site-packages/torf/_torrent.py", line 96, in __init__ self.trackers = trackers File "/home/<REDACTED>/env/lib/python3.6/site-packages/torf/_torrent.py", line 390, in trackers tier = [utils.validated_url(str(item))] File "/home/<REDACTED>/env/lib/python3.6/site-packages/torf/_utils.py", line 47, in validated_url raise error.URLError(url) torf._errors.URLError: h: Invalid URL
The only way I can make it create a torrent successfully is to make trackers a list with the single tracker url and that adds announce-list to the torrent file.
I only want announce in the torrent file with a single url.
Am I doing this wrong?
MetainfoError is raised if piece size is not a power of two. This behaviour is incorrect.
From https://blog.libtorrent.org/2020/09/bittorrent-v2/
In v1 torrents, the size of a piece is not restricted by the specification. It doesn’t make much sense to have a piece smaller than the block size of 16 kiB, but it’s not disallowed. The vast majority of torrents that are created use a power-of-two piece size, but there are a few outliers that are not, but still divisible by 16 kiB
I propose the is_power_of_2 check be replaced by a is_divisible_by_16_kib check.
Hi,
in the wild world out there torrent files exist which have additional .utf-8 variants for name and path keys. They were introduced by BT Inc. unofficially and not specified by BEP3. (see this)[https://github.com/BiglySoftware/BiglyBT/issues/1274]
I guess the background were character set related things on some systems (e.g. Asian systems - they had different problems switching to UTF-8 as far as I know). The name and path keys in these situation can contain every character set out there (I guess normally the system character set or ASCII).
torf can correctly deserialize these torrents and also write new ones without that keys (see this)[https://github.com/casey/intermodal/issues/534]. But torf fails e.g. in a verify run with paths it can't find. I guess the reason is that because of the unknown and not correctly handled character set a wrong path is searched (output with strange characters indicate this). So torf uses the non .utf-8 variants.
torf should prefer .utf-8 key variants when they exist like clients do. I think it's not really BEP3 violation to do this.
Best Regards
Sorry for the question as i am new to python, i work with the code:
t = Torrent(path=folder_path,
trackers=['https://example:2910/announce'],
t.private = True
t.generate()
and it works, but if i want to remove the file who as an extension .nfo per example i write the t.exclude_globs('*.nfo')
, and it gives me an error, what i am doing wrong please.
t.exclude_globs('*.nfo')
gives an error:
maybe its not torf fault:
i get the folder path from pysimplegui, like this:
file_path = values[1]
Hi,
Might you consider switching back to MIT or switching to LGPL so we can use this as a package?
Thank you!
Thank you! This tool is too powerful。
Torrent
class allow read torrent from file. But sometimes we need read from memory like StringIO
or BytesIO
.
I'm creating single file torrents for large files, and the files are stored on a remote server (an S3 bucket). Rather than downloading the large file and requiring the disk space for it, I'd like to be able to stream the content directly into the generator. Is that possible? It almost looked possible to me reading through the internals, but at the end of the day it seems like the exposed interface really expects a file on disk.
Ideally it would be great to do this for a multi-file torrent, but that would understandably be more difficult.
I'm struggling to make a torrent that does not create a directory for the download
for example I have video.mp4 in files : path='files/video.mp4' #Relative path from the script
When people download the torrent, I want them to only have video.mp4 in their download folder, not files/video.mp4
My files folder always contains only 1 video, but the video name can change so I prefer not to hardcode the filename in the program
Is it possible to do this ?
Hello
I usually create a torrent with Torf that includes a folder with all the files inside. How do I create a torrent with subfolders and their respective files as well?
I tried with 'include_globs,' but it keeps creating a torrent with all the files excluding those from the subfolders.
for example:
torf.Torrent(path="/home/ridente/FolderA/")
FolderA
hello_1.mkv
hello_2.mkv
FolderB
hello_3.mkv
FolderC
hello_4.mkv
It will create a torrent with only 'hello_1.mkv' and 'hello_2.mkv' without including FolderB and FolderC (subfolders and files)
Thank you
As the title suggests, from some reason, when I try to run the install script setup.py
, I get -rw------
permissions on the egg-info folder of torf
.
It doesn't happen so while installing with pip --root <>
. Any idea why this is happening?
This library makes use of collections.abc.ByteString
in several locations, but this type has been deprecated (and produces a warning) in Python 3.12 and is slated for removal in Python 3.14. See the link for advice on how to replace it.
I came across some trackers that use source flag, i think for cross seeding and to download directly from a .torrent file created, instead of downloaded from the tracker, is it an option in torf?
Sorry if this question is dumb.
Hi. I believe this module doesn't support magnet link parsing? Is it possible for you to implement it in the future?
Thanks
I'm on Arch Linux, and python-torf installs the file /usr/lib/python<version>/site-packages/tests/__init__.py
. This same file is also installed by package python-hypothesis. Since these two packages installs the same file, this results in a package conflict (where both packages cannot be installed at the same time).
The list of files installed by package python-hypothesis on Arch Linux can be seen here. I've noticed that this file is being installed by python-hypothesis since its version 5.37.0.
Can this file conflict with python-hypothesis be fixed here?
def callback(torrent, filepath, pieces_done, pieces_total):
print(f'{pieces_done / pieces_total * 100:3.0f} % done')
torrent = Torrent(path=path, trackers='https://tracker.example.to', private=True)
success = torrent.generate(callback, interval=1)
outputs a TypeError
Traceback (most recent call last):
File "tests.py", line 16, in <module>
success = torrent.generate(callback, interval=1)
File "/usr/local/lib/python3.8/site-packages/torf/_torrent.py", line 972, in generate
queue_size=threads*3)
TypeError: unsupported operand type(s) for *: 'function' and 'int'
I tested this with version 2.1.0
and it works fine and gives me the expected output
filename.file
1 % done
100 % done
100 % done
True
All modern clients support this so please support creating 32MB piece size torrents.
I realize that 32MB piece size support was added recently but it seems the new piece size support standard for most clients currently in use is 64MB.
qBittorrent supports 64MB pieces and larger (and will create up to to 256MB piece sizes in qbittorrent/qBittorrent#19535)
Transmission 4.0 supports up to 64MB pieces (as shown in transmission/transmission#3768)
Deluge supports 64 MB piece size torrents
BiglyBT supports 64 MB piece size torrents.
https://github.com/kz26/dottorrent already supported creation of 64MB but that ability was lost in torf.
py3createtorrent recently went to 64 MB in rsnitsch/py3createtorrent#34
uTorrent 3.x supports 64MB piece size torrents
Tixati 3.x supports 64MB piece size torrents as of https://forum.tixati.com/support/7137
That's all the major clients in use today.
Please allow us to keep up with these clients by giving us the ability to create 64MB piece size torrents. That should be sufficient for years to come as torrents keep getting bigger and bigger.
Hey
I decided to port dottorrent-gui to torf
https://github.com/SavageCore/torf-gui
Do you want to add this to the readme?
Hi. I have been testing multiple torrent and suddenly 1 of the torrents returning a wrong hash.
Manjaro%20Kde%2015.12%20x86_64.zip (or download from here)
If I added in qbittorrent, the hash registered is db4d75e4f9e846e641d043a36f73187bb364c111
.
But somehow in torf, hash returned is 1925c683d9fe4156b8e4be0a193a157acd6582ed
.
This is the code I am using
#!/usr/bin/env python3.8
from torf import Torrent
import json
import sys
if len(sys.argv) == 2:
try:
torrent = Torrent.read(sys.argv[1])
torrentlist = {'hash': torrent.infohash, 'files': list(torrent.files)}
print(json.dumps(torrentlist))
except Exception as e:
print(e)
else:
print('No .torrent detected')
I am running the latest code.
On linux, the default limit is 1024 open files ulimit -n
This error can be temporarily remedied by running ulimit -n 5000
or whatever number
I imagine it's not a simple fix within torf, so this is mainly for anyone else that runs into this problem.
Traceback (most recent call last):
File "/home/w/.local/lib/python3.10/site-packages/torf/_stream.py", line 382, in _get_open_file
self._open_files[filepath] = open(filepath, 'rb')
OSError: [Errno 24] Too many open files: "FILENAME"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/h/convert/out/torrent.py", line 35, in <module>
t.generate(callback=cb, interval=0.5)
File "/home/w/.local/lib/python3.10/site-packages/torf/_torrent.py", line 1114, in generate
piece_hashes = collector.collect()
File "/home/w/.local/lib/python3.10/site-packages/torf/_generate.py", line 337, in collect
self._collect(*task)
File "/home/w/.local/lib/python3.10/site-packages/torf/_generate.py", line 367, in _collect
maybe_cancel = self._callback(
File "/home/w/.local/lib/python3.10/site-packages/torf/_generate.py", line 427, in __call__
return self._intervaled_callback(piece_index, pieces_done, pieces_total,
File "/home/w/.local/lib/python3.10/site-packages/torf/_generate.py", line 412, in __call__
return self._callback(*args)
File "/home/w/.local/lib/python3.10/site-packages/torf/_generate.py", line 455, in _call_callback
raise exceptions[0]
File "/home/w/.local/lib/python3.10/site-packages/torf/_stream.py", line 445, in iter_pieces
fh = self._get_open_file(filepath)
File "/home/w/.local/lib/python3.10/site-packages/torf/_stream.py", line 384, in _get_open_file
raise error.ReadError(e.errno, filepath)
torf._errors.ReadError: FILENAME: Too many open files
Hi @rndusr,
Thanks for your hard working.
I want to download several files in a torrent which contains a lot of files.
Can a torrent file be modified by torf? I want to remove some files in the torrent.
from torf import Torrent
torrent = Torrent.read('./temp/old.torrent')
torrent.files = [torrent.files[0], torrent.files[2], torrent.files[-1]]
torrent.write('./temp/new.torrent')
but the output is:
Traceback (most recent call last):
File "test.py", line 5, in
torrent.write('./temp/new.torrent')
File "D:\work\02work\work\009torrent\rw-torrent\venv\lib\site-packages\torf_torrent.py", line 1477, in write
self.write_stream(content, validate=validate)
File "D:\work\02work\work\009torrent\rw-torrent\venv\lib\site-packages\torf_torrent.py", line 1448, in write_stream
content = self.dump(validate=validate)
File "D:\work\02work\work\009torrent\rw-torrent\venv\lib\site-packages\torf_torrent.py", line 1435, in dump
self.validate()
File "D:\work\02work\work\009torrent\rw-torrent\venv\lib\site-packages\torf_torrent.py", line 1331, in validate
utils.assert_type(md, ('info', 'pieces'), (abc.ByteString,), must_exist=True)
File "D:\work\02work\work\009torrent\rw-torrent\venv\lib\site-packages\torf_utils.py", line 718, in assert_type
raise error.MetainfoError(f'Missing {key!r} in {keychain_str}')
torf._errors.MetainfoError: Invalid metainfo: Missing 'pieces' in ['info']
Looking forward to your reply.
I'd like to read the .torrent
in memory without first writing it to disk and write
seems to accept a a file path string only.
The announce URL is serialized as a list containing a single element.
I think it should be serialized as a string since there's announce-list
.
Sources:
Hello,
Using latest version in python.
Getting "Invalid metainfo: Missing 'pieces' in ['info']" at torrent file loading.
Example torrent file:
https://libtorrent.org/bittorrent-v2-test.torrent
For example, I have a file named "out.html" and want it to be named "index.html" inside the torrent file.
I am loading the paths by setting filepaths, so I figured it could be possible to either have it when setting or change it after under metainfo["info"]["files"].
torrent file's infohash got change after load, found at least 2 torrent file.
test code:
import torf
t = print(torf.Torrent.read('/config/torrents/sample1.torrent'))
print(t.infohash)
I have read over and tested the documentation when it comes to include/exclude globs/regexs. While I can achieve my goal utilizing these, is it possible to just directly modify the included filepaths/files to only include things that I want when it comes to a directory
.
Structure:
folder_a
file.mkv
file.srt
file.log
subfolder_a
file.h264
Let's say I want to use torf to open folder_a
and I know that I only want to include folder_a/
file.mkv
and file.srt
. Could I directly modify filepaths
and/or files
to only include that in the final output? Utilizing the globs/regexs feel like it could be error prone in certain situations when i know specifically before hand what files I'd like to have (say selecting from a filetree or something such as that).
After create a torrent file ,How to seed these files ?
use what kind of software ?
Can python handle it ?
with some code
I recently encountered this torrent file in the wild and it has one invalid tracker url out of many.
Trying to read it raises a metainfo error
Traceback (most recent call last):
File "C:\Users\raven\Documents\GitHub\TEST\test.py", line 10, in <module>
torrent = Torrent.read_stream(BytesIO(open(path, "rb").read()))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\raven\Documents\GitHub\TEST\.venv\Lib\site-packages\torf\_torrent.py", line 1632, in read_stream
torrent.validate()
File "c:\Users\raven\Documents\GitHub\TEST\.venv\Lib\site-packages\torf\_torrent.py", line 1388, in validate
utils.assert_type(md, ('announce-list', i, j), (str,), check=utils.is_url)
File "c:\Users\raven\Documents\GitHub\TEST\.venv\Lib\site-packages\torf\_utils.py", line 727, in assert_type
raise error.MetainfoError(f"{keychain_str}[{key!r}] is invalid: {obj[key]!r}")
torf._errors.MetainfoError: Invalid metainfo: ['announce-list'][4][0] is invalid: '*udp://9.rarbg.to:2710/announce'
I understand this is an invalid URL but is it possible to drop invalid metainfo instead of erroring? especially non critical ones like this where there are several more working trackers. This is a perfectly downloadable torrent and clients like Qbit simply report it as unsupported while continuing to download it
Possibly a strict: bool = True
flag in read()
and read_stream()
where:
Thank you
the torrent file can download from https://nyaa.si/view/1284768
torf package version is 3.1.1
the infohash using torf is 95f3619484546e6735b0837e58d003cd0c93775f
but the correct value is fb8899e9faff4c0682fec40b0446cfbccbbbed18
Currently, by installing torf, it creates /usr/lib/python3.12/site-packages/tests
directory that can cause conflicts with other Python libs.
Can you please exclude tests*
by pyproject.toml
file?
It can be fixed by: #43
If you agree, after merge, can you please release an updated minor version on PyPi?
Does it only works on python 3.6??
I installed torf using
pip install torf
I am just trying out the example provided When I try this on Python 3.6 virtualenv
`import torf
t = torf.Torrent(path='1.jpg',
trackers=['https://tracker1.example.org:1234/announce',
'https://tracker1.example.org:1234/announce'],
comment='This is a comment')
t.private = True`
I get this
Traceback (most recent call last): File "/home/vaibhav/projects/test/py36test/torf.py", line 1, in <module> import torf File "/home/vaibhav/projects/test/py36test/torf.py", line 3, in <module> t = torf.Torrent(path='1.jpg', AttributeError: module 'torf' has no attribute 'Torrent'
I am getting this on every python envs.
I tried this on python3.3
from torf import Torrent
`Python 3.3.5 |Continuum Analytics, Inc.| (default, Jun 4 2015, 15:22:11)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
from torf import Torrent
Traceback (most recent call last):
File "", line 1, in
File "/home/vaibhav/anaconda3/envs/iag/lib/python3.3/site-packages/torf/init.py", line 26, in
from ._errors import *
File "/home/vaibhav/anaconda3/envs/iag/lib/python3.3/site-packages/torf/_errors.py", line 37
super().init(f'{url}: Invalid URL')
^
SyntaxError: invalid syntax
`
Am I missing out something?
__enter and __exit method are missed in class Torrent, and 'with' cannot be used.
code:
with Torrent(....) as t:
----....
----t.write()
I got an error:
AttributeError: __enter__
Expected behaviour: Just as Torrent.read(), but from bytes object.
example:
var t = bytes('from.some.torrent')
rtb = Torrent.readBytes(t)
Returns: New Torrent instance
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.