Giter VIP home page Giter VIP logo

Comments (4)

josecelano avatar josecelano commented on September 10, 2024

I have started uploading random torrents to the live demo:

2024-02-07T10:37:29.671005513+00:00 [seeder][INFO] Trying to login with username: admin ...
2024-02-07T10:37:30.280078305+00:00 [seeder][INFO] Logged as admin with account: admin 
2024-02-07T10:37:30.280152064+00:00 [seeder][INFO] Uploading 2419200 random torrents to the Torrust Index with a 1 seconds interval...
2024-02-07T10:37:30.280172397+00:00 [seeder][INFO] Uploading torrent #1 ...
2024-02-07T10:37:30.280199768+00:00 [seeder][INFO] Uploading torrent with uuid: 538f2a1b-d048-4976-8bd8-b1a1bf73d27e ...
2024-02-07T10:37:31.896728858+00:00 [seeder][INFO] Uploaded torrent: {"torrent_id":24,"info_hash":"882b8399ba5c5b45a0e6503efd311acef13f957d"}
2024-02-07T10:37:32.897125478+00:00 [seeder][INFO] Uploading torrent #2 ...
2024-02-07T10:37:32.897204089+00:00 [seeder][INFO] Uploading torrent with uuid: 45e94cef-1ce3-4e91-b06f-36a43f50821f ...
2024-02-07T10:37:35.015353750+00:00 [seeder][INFO] Uploaded torrent: {"torrent_id":25,"info_hash":"a1a7f87d0ae885d2690d796334da490bd4e5163c"}
2024-02-07T10:37:36.015561837+00:00 [seeder][INFO] Uploading torrent #3 ...
2024-02-07T10:37:36.015694115+00:00 [seeder][INFO] Uploading torrent with uuid: 0681510f-0c90-411e-9033-73c14d6806f2 ...
2024-02-07T10:37:37.288633248+00:00 [seeder][INFO] Uploaded torrent: {"torrent_id":26,"info_hash":"bd82002960d390a4edf16c0610dcbcb0f84e9ddb"}
2024-02-07T10:37:38.288803517+00:00 [seeder][INFO] Uploading torrent #4 ...
2024-02-07T10:37:38.288898443+00:00 [seeder][INFO] Uploading torrent with uuid: 05fb849c-7f72-4e85-b935-703f4088892a ...
2024-02-07T10:37:40.024003792+00:00 [seeder][INFO] Uploaded torrent: {"torrent_id":27,"info_hash":"f502332150ef445ffb808808c77ae1fa77bc8448"}
2024-02-07T10:37:41.024183655+00:00 [seeder][INFO] Uploading torrent #5 ...

from torrust-index.

josecelano avatar josecelano commented on September 10, 2024

The server is quite busy today handling UDP announce requests:

image

In the last 12 days, the tracker has handled:

{
  "torrents": 38167,
  "seeders": 19020,
  "completed": 166956,
  "leechers": 28201,
  "tcp4_connections_handled": 6520,
  "tcp4_announces_handled": 6497,
  "tcp4_scrapes_handled": 23,
  "tcp6_connections_handled": 0,
  "tcp6_announces_handled": 0,
  "tcp6_scrapes_handled": 0,
  "udp4_connections_handled": 416488740,
  "udp4_announces_handled": 198269599,
  "udp4_scrapes_handled": 7227459,
  "udp6_connections_handled": 0,
  "udp6_announces_handled": 0,
  "udp6_scrapes_handled": 0
}

That is 191.232252122 UDP announce req/sec.

We are using the 6$/month droplet in Digital Ocean:

image

There are at least two problems:

Missing timeout in the seeder client

2024-02-07T16:54:30.067031801+00:00 [seeder][INFO] Uploading torrent #505 ...
2024-02-07T16:54:30.067125264+00:00 [seeder][INFO] Uploading torrent with uuid: 14b86e5c-994a-4b1a-af1d-a71857872f9b ...

It takes more than 1 second for the seeder to upload a new torrent.

Sometimes the seeder waits forever for the API response. We have to add a timeout to the API client. And retry after 5 seconds. Otherwise, you have to re-run it manually. However, we should find out why the seeder (API client) does not get any response later. Are we losing/discarding some requests?

Tracker Statistics Importer (cronjob) is too slow

The cronjob is importing 1 torrent per second with the current load:

index  | 2024-02-07T16:23:05.871624145+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1611 ...
index  | 2024-02-07T16:23:06.999765366+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2323 ...
index  | 2024-02-07T16:23:07.549943822+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1399 ...
index  | 2024-02-07T16:23:08.157716317+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 899 ...
index  | 2024-02-07T16:23:08.745064543+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2169 ...
index  | 2024-02-07T16:23:09.146923810+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 850 ...
index  | 2024-02-07T16:23:09.982634698+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2114 ...
index  | 2024-02-07T16:23:10.912697512+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1237 ...
index  | 2024-02-07T16:23:11.232916642+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2031 ...
index  | 2024-02-07T16:23:11.592363898+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2580 ...
index  | 2024-02-07T16:23:11.889599852+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2405 ...
index  | 2024-02-07T16:23:12.322081053+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2277 ...
index  | 2024-02-07T16:23:12.906672020+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1948 ...
index  | 2024-02-07T16:23:14.593848136+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 754 ...
index  | 2024-02-07T16:23:15.760567733+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 491 ...
index  | 2024-02-07T16:23:16.313846869+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 643 ...
index  | 2024-02-07T16:23:16.802825537+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2168 ...
index  | 2024-02-07T16:23:17.231140891+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 329 ...
index  | 2024-02-07T16:23:17.620177038+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 1569 ...
index  | 2024-02-07T16:23:18.303492748+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 807 ...
index  | 2024-02-07T16:23:18.831055766+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 551 ...
index  | 2024-02-07T16:23:19.213513531+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 273 ...
index  | 2024-02-07T16:23:19.735776466+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 120 ...
index  | 2024-02-07T16:23:20.390742014+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 892 ...
index  | 2024-02-07T16:23:20.904715668+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2170 ...
index  | 2024-02-07T16:23:22.049670834+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2455 ...
index  | 2024-02-07T16:23:23.632835819+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 2427 ...
index  | 2024-02-07T16:23:23.997367030+00:00 [torrust_index::tracker::statistics_importer][INFO] Updating torrent 425 ...

That means we can only import 3600 torrents per hour. That's normal because it's a small droplet but the problem is sometimes it seems that the process is stopped. If I stop the seeder the cronjob continues working. If I re-start the seeder after a while the cronjob stops again. But it could be a coincidence.

See #428

I think I'm going to add more logs for HTTP Tracker and the API so that we can detect it there are:

  • Some requests that are not processed.
  • Some problems with the importer.

I think the problem with the importer could also be a missing timeout in the Tracker API client. I will open a new issue to track problems with timeouts as I did for the tracker.

cc @WarmBeer @da2ce7

from torrust-index.

josecelano avatar josecelano commented on September 10, 2024

We have 27570 torrents in the live demo.

image

image

It seems the server is always busy finally.

image

cc @da2ce7 @WarmBeer

from torrust-index.

josecelano avatar josecelano commented on September 10, 2024

I'm going to stop uploading more torrents. I think we have reached the server maximum load:

image

There are 27570 torrents. I guess 1000 torrents is the max acceptable load with the current level of UDP requests/sec. I should isolate the tests for the UDP tracker and for the Index importing the torrents from the tracker. Using a non-used UDP tracker to serve the Index. If we want to test only the performance of the statistics importer. But I only wanted to have a rough idea of how much load the smallest server can handle.

from torrust-index.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.