Comments (4)
I have started uploading random torrents to the live demo:
2024-02-07T10:37:29.671005513+00:00 [seeder][INFO] Trying to login with username: admin ...
2024-02-07T10:37:30.280078305+00:00 [seeder][INFO] Logged as admin with account: admin
2024-02-07T10:37:30.280152064+00:00 [seeder][INFO] Uploading 2419200 random torrents to the Torrust Index with a 1 seconds interval...
2024-02-07T10:37:30.280172397+00:00 [seeder][INFO] Uploading torrent #1 ...
2024-02-07T10:37:30.280199768+00:00 [seeder][INFO] Uploading torrent with uuid: 538f2a1b-d048-4976-8bd8-b1a1bf73d27e ...
2024-02-07T10:37:31.896728858+00:00 [seeder][INFO] Uploaded torrent: {"torrent_id":24,"info_hash":"882b8399ba5c5b45a0e6503efd311acef13f957d"}
2024-02-07T10:37:32.897125478+00:00 [seeder][INFO] Uploading torrent #2 ...
2024-02-07T10:37:32.897204089+00:00 [seeder][INFO] Uploading torrent with uuid: 45e94cef-1ce3-4e91-b06f-36a43f50821f ...
2024-02-07T10:37:35.015353750+00:00 [seeder][INFO] Uploaded torrent: {"torrent_id":25,"info_hash":"a1a7f87d0ae885d2690d796334da490bd4e5163c"}
2024-02-07T10:37:36.015561837+00:00 [seeder][INFO] Uploading torrent #3 ...
2024-02-07T10:37:36.015694115+00:00 [seeder][INFO] Uploading torrent with uuid: 0681510f-0c90-411e-9033-73c14d6806f2 ...
2024-02-07T10:37:37.288633248+00:00 [seeder][INFO] Uploaded torrent: {"torrent_id":26,"info_hash":"bd82002960d390a4edf16c0610dcbcb0f84e9ddb"}
2024-02-07T10:37:38.288803517+00:00 [seeder][INFO] Uploading torrent #4 ...
2024-02-07T10:37:38.288898443+00:00 [seeder][INFO] Uploading torrent with uuid: 05fb849c-7f72-4e85-b935-703f4088892a ...
2024-02-07T10:37:40.024003792+00:00 [seeder][INFO] Uploaded torrent: {"torrent_id":27,"info_hash":"f502332150ef445ffb808808c77ae1fa77bc8448"}
2024-02-07T10:37:41.024183655+00:00 [seeder][INFO] Uploading torrent #5 ...
from torrust-index.
The server is quite busy today handling UDP announce requests:
In the last 12 days, the tracker has handled:
{
"torrents": 38167,
"seeders": 19020,
"completed": 166956,
"leechers": 28201,
"tcp4_connections_handled": 6520,
"tcp4_announces_handled": 6497,
"tcp4_scrapes_handled": 23,
"tcp6_connections_handled": 0,
"tcp6_announces_handled": 0,
"tcp6_scrapes_handled": 0,
"udp4_connections_handled": 416488740,
"udp4_announces_handled": 198269599,
"udp4_scrapes_handled": 7227459,
"udp6_connections_handled": 0,
"udp6_announces_handled": 0,
"udp6_scrapes_handled": 0
}
That is 191.232252122 UDP announce req/sec.
We are using the 6$/month droplet in Digital Ocean:
There are at least two problems:
Missing timeout in the seeder client
2024-02-07T16:54:30.067031801+00:00 [seeder][INFO] Uploading torrent #505 ...
2024-02-07T16:54:30.067125264+00:00 [seeder][INFO] Uploading torrent with uuid: 14b86e5c-994a-4b1a-af1d-a71857872f9b ...
It takes more than 1 second for the seeder to upload a new torrent.
Sometimes the seeder waits forever for the API response. We have to add a timeout to the API client. And retry after 5 seconds. Otherwise, you have to re-run it manually. However, we should find out why the seeder (API client) does not get any response later. Are we losing/discarding some requests?
Tracker Statistics Importer (cronjob) is too slow
The cronjob is importing 1 torrent per second with the current load:
index | 2024-02-07T16:23:05.871624145+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1611 ...
index | 2024-02-07T16:23:06.999765366+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2323 ...
index | 2024-02-07T16:23:07.549943822+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1399 ...
index | 2024-02-07T16:23:08.157716317+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 899 ...
index | 2024-02-07T16:23:08.745064543+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2169 ...
index | 2024-02-07T16:23:09.146923810+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 850 ...
index | 2024-02-07T16:23:09.982634698+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2114 ...
index | 2024-02-07T16:23:10.912697512+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1237 ...
index | 2024-02-07T16:23:11.232916642+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2031 ...
index | 2024-02-07T16:23:11.592363898+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2580 ...
index | 2024-02-07T16:23:11.889599852+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2405 ...
index | 2024-02-07T16:23:12.322081053+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2277 ...
index | 2024-02-07T16:23:12.906672020+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1948 ...
index | 2024-02-07T16:23:14.593848136+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 754 ...
index | 2024-02-07T16:23:15.760567733+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 491 ...
index | 2024-02-07T16:23:16.313846869+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 643 ...
index | 2024-02-07T16:23:16.802825537+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2168 ...
index | 2024-02-07T16:23:17.231140891+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 329 ...
index | 2024-02-07T16:23:17.620177038+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1569 ...
index | 2024-02-07T16:23:18.303492748+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 807 ...
index | 2024-02-07T16:23:18.831055766+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 551 ...
index | 2024-02-07T16:23:19.213513531+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 273 ...
index | 2024-02-07T16:23:19.735776466+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 120 ...
index | 2024-02-07T16:23:20.390742014+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 892 ...
index | 2024-02-07T16:23:20.904715668+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2170 ...
index | 2024-02-07T16:23:22.049670834+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2455 ...
index | 2024-02-07T16:23:23.632835819+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2427 ...
index | 2024-02-07T16:23:23.997367030+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 425 ...
That means we can only import 3600 torrents per hour. That's normal because it's a small droplet but the problem is sometimes it seems that the process is stopped. If I stop the seeder the cronjob continues working. If I re-start the seeder after a while the cronjob stops again. But it could be a coincidence.
See #428
I think I'm going to add more logs for HTTP Tracker and the API so that we can detect it there are:
- Some requests that are not processed.
- Some problems with the importer.
I think the problem with the importer could also be a missing timeout in the Tracker API client. I will open a new issue to track problems with timeouts as I did for the tracker.
from torrust-index.
We have 27570 torrents in the live demo.
It seems the server is always busy finally.
from torrust-index.
I'm going to stop uploading more torrents. I think we have reached the server maximum load:
There are 27570 torrents. I guess 1000 torrents is the max acceptable load with the current level of UDP requests/sec. I should isolate the tests for the UDP tracker and for the Index importing the torrents from the tracker. Using a non-used UDP tracker to serve the Index. If we want to test only the performance of the statistics importer. But I only wanted to have a rough idea of how much load the smallest server can handle.
from torrust-index.
Related Issues (20)
- Move from `log` to `tracing` crate
- Remove duplicate slash `/` in tracker URL inside to downloaded torrent file HOT 1
- Update tracker TOML configuration (breaking changes) HOT 1
- Update tracker configuration HOT 1
- Configuration overhaul: add version to the configuration
- Configuration overhaul: rename `log_level` to `threshold`
- Configuration overhaul: rename `auth::secret_key` to `auth::user_claim_token_pepper` HOT 1
- Configuration overhaul: make mail-related configuration clearer HOT 3
- Configuration overhaul: split `TrackerMode`into two flags `private` and `listed`
- Configuration overhaul: print the final configuration to console
- Update database driver in Tracker configuration
- Config overhaul: use an explicit section for metadata HOT 1
- Config overhaul: make some fields mandatory HOT 2
- Config overhaul: rename config option `verified` to `verification_required`
- Fix docker warning in `Containerfile`
- Update tracker configuration
- Allow overwriting hardcoded casbin configuration
- Config overhaul: make secrets mandatory HOT 1
- Wrong error message for `PasswordTooLong` error
- New config option: max torrent upload size
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from torrust-index.