Memory leaks on the last version (2.3)? about ngpost HOT 27 CLOSED

Hum try the latest version, I've just released it :)
It inclludes the same fix I did for the temporary version.
hum, I'll need to investigate this crash when you get lots of thread. I guess it is arriving in a dead lock.

Good news that the speed is better. I've a bit cheated, increasing to 5 the number of Articles prepared for each Connection. On your station with lots of ram, you could gain in speed playing with that parameter. But it is hardcoded at the moment and I set it back to 3.
I may make it a config in a future release.

with the command line, you can now do
ngPost --auto /Downloads/testNgPost --compress --gen_par2 --gen_name --gen_pass --rar_size 42 --disp_progress files

this will scan the folder /Downloads/testNgPost and post each file or folder individually after compressing with a random name and password and generating the password. (folder will be compressed recursively)

Please update the config file ~/.ngPost as there is a new config (cf here) where the history of the posts will be append so you can get quickly the archive name and password of your posts (plus you get the date, size and avg bandwidth)

I've also made a nice GUI that propose to do the same, have a look if you fancy, it's pretty sexy! ;)

from ngpost.

I had a look at your logs and your station is quite too fast! haha
in the two logs with 1 Thread, we can see that the MainThread that produces the Article stay always in advance and has 5 articles prepared for each connection. We never see the message "no article prepared"
In the opposite with many threads, the MainThread just get catch up really fast by the Posting threads. Line 2994. Then he will never managed to get back in advance but it is accumulating some request in its event loop to build articles.
So I guess the crash comes from an overflow of the Main thread event loop.
I'm going to think about a solution to support multi threading properly. I'm not sure we would gain in speed but maybe. Anyway, it's a good programming challenge.
One solution I see would be to have several producers, in fact, each thread should own their producers and have as many as it is having Connections.
I'll let you know when I'll have done the change.
For now, you should be safe using one thread ;)

from ngpost.

I tried the --auto argument on the latest version, it outputs 'Error syntax: Unknown option 'auto'' :s

arf... I built the wrong version... it's still the 2.3

$ ./ngPost_v3.1-x86_64.AppImage -v
                   __________               __
       ____    ____\______   \____  _______/  |_
      /    \  / ___\|     ___/  _ \/  ___/\   __\
     |   |  \/ /_/  >    |  (  <_> )___ \  |  |
     |___|  /\___  /|____|   \____/____  > |__|
          \//_____/                    \/
                          v2.3

I've pushed the real v3.1 for Linux. Please download it and give it a try, it should work!

Same thing for the history file, it was v2.3 so it didn't exist...

from ngpost.

I used the debug -d argument, nothing really useful came out

[Thread #9][NgPost::getNextArticle] no article prepared...
[avg. speed: 6.78 MB/s] >>>>> [5/138] ***.part004.rar
[Thread #2][NgPost::getNextArticle] no article prepared...
[Thread #16] we've read 716800 bytes from 1433600 (=> new pos: 2150400)
[Thread #19][NgPost::getNextArticle] no article prepared...
[Thread #12][NgPost::getNextArticle] no article prepared...
[Thread #6] we've read 716800 bytes from 2150400 (=> new pos: 2867200)
[Thread #17][NgPost::getNextArticle] no article prepared...
[Thread #4] we've read 716800 bytes from 2867200 (=> new pos: 3584000)
[Thread #1] we've read 716800 bytes from 3584000 (=> new pos: 4300800)
[Thread #16][NgPost::getNextArticle] no article prepared...
[Thread #20] we've read 716800 bytes from 4300800 (=> new pos: 5017600)
[Thread #6][NgPost::getNextArticle] no article prepared...
[Thread #15] we've read 716800 bytes from 5017600 (=> new pos: 5734400)
[Thread #4][NgPost::getNextArticle] no article prepared...
[Thread #1][NgPost::getNextArticle] no article prepared...
[Thread #10] we've read 716800 bytes from 5734400 (=> new pos: 6451200)
[Thread #14] we've read 716800 bytes from 6451200 (=> new pos: 7168000)
[Thread #20][NgPost::getNextArticle] no article prepared...
[Thread #8] we've read 716800 bytes from 7168000 (=> new pos: 7884800)
[Thread #15][NgPost::getNextArticle] no article prepared...
[Thread #18] we've read 716800 bytes from 7884800 (=> new pos: 8601600)
[MainThread] we've read 716800 bytes from 8601600 (=> new pos: 9318400)
[Thread #10][NgPost::getNextArticle] no article prepared...
[Thread #14][NgPost::getNextArticle] no article prepared...
[Thread #3] we've read 716800 bytes from 9318400 (=> new pos: 10035200)
[Thread #13] we've read 716800 bytes from 10035200 (=> new pos: 10752000)
[Thread #8][NgPost::getNextArticle] no article prepared...
[Thread #5] we've read 716800 bytes from 10752000 (=> new pos: 11468800)
[Thread #11] we've read 716800 bytes from 11468800 (=> new pos: 12185600)
[Thread #3][NgPost::getNextArticle] no article prepared...
[Thread #18][NgPost::getNextArticle] no article prepared...
[Thread #7] we've read 716800 bytes from 12185600 (=> new pos: 12902400)
[Thread #13][NgPost::getNextArticle] no article prepared...
[Thread #9] we've read 716800 bytes from 12902400 (=> new pos: 13619200)
[Thread #5][NgPost::getNextArticle] no article prepared...
[Thread #2] we've read 716800 bytes from 13619200 (=> new pos: 14336000)
[Thread #11][NgPost::getNextArticle] no article prepared...
[Thread #19] we've read 716800 bytes from 14336000 (=> new pos: 15052800)
[Thread #7][NgPost::getNextArticle] no article prepared...
[Thread #12] we've read 716800 bytes from 15052800 (=> new pos: 15769600)
[Thread #9][NgPost::getNextArticle] no article prepared...
Killed

from ngpost.

Hello,
normally, there is no change to the posting process between 2.2 and 2.3, just some HMI updates to preview the next major version.

Could you give the full log please.
Is it crashing every time? How big is the full post? each archive?

Sounds like the main thread who's supposed to be ahead of the Posting ones to prepare the Articles has been caught up by the posting threads. This is weird, not supposed to happen but anyway, I did some debugging with no article prepared and it was stable...

Could you also give me the full command please.
Why do you use so many threads? have you set it up or is it your server that has more than 20 cores?

from ngpost.

Could you give the full log please.

You mean the full output of the --debug right?

Is it crashing every time? How big is the full post? each archive?

Yep, for every >5GB post. I tried from 20MB to 500MB per archive

Why do you use so many threads? have you set it up or is it your server that has more than 20 cores?

There are over 24 CPU (576 threads in total) on the server I use. I already tried with a single thread (modified the thread value in the config file to 1), it still crashes a few second after the posting begins, so I'm pretty sure the problem is not here :s

Could you also give me the full command please.

./ngPost_v2.3-x86_64.AppImage -c ~/.ngPost -i ~/downloads/***.mkv -o ~/ngPost/***.nzb --compress --gen_name --gen_par2

After that, I tried to post the rars created during the process using

./ngPost_v2.3-x86_64.AppImage -c ~/.ngPost -i ~/tmp/*** -o ~/ngPost/***.nzb
but still crashes a few seconds after.

from ngpost.

hum interesting, I've never tried on such a powerful station...
Are you definitely sure there is no crash with the v2.2 using exactly the same parameters? that would definitely help to narrow down the potential issue.

You mean the full output of the --debug right?

yes could you run it with output redirection to a file and stderr also. somthing like this:
./ngPost_v2.3-x86_64.AppImage -d -t 1 -i ~/tmp/*** -o ~/ngPost/***.nzb > /tmp/ngPost.log 2>&1

as you can see on the command:

• no need to put -c ~/.ngPost as it is picked up automatically as a default configuration file (-c is to be used in case you wish to use an alternative config file with let's say another provider)
• use only one thread, if the crash is there it will be easier to debug as we won't have to deal with mutex synchronization (less in fact)
• as the problem doesn't come from the rar/par2 generation, please use directly a folder as input

How many Usenet severs / connections are you using? Is it happening also if you decrease it? if it is the case, it would help to debug if you could use only 1 connection. (edit the config file and set connection to 1)

You say it crashes after few seconds in case of uploading a folder.
Will it do it whatever files are inside or just if you've let's say 5GB of files?

Thanks in advance for your debugging ;)

from ngpost.

Here is the log: https://tknk.io/bY33
I tried with a 13GB file, splitted in 131 parts of 100MB each.

Are you definitely sure there is no crash with the v2.2 using exactly the same parameters?

I don't have the 2.2 anymore since I removed it, and I wasn't able to download it using the previous link. :s
But iirc, I haven't encountered any issue while posting before.

How many Usenet severs / connections are you using?

I usually use 2 servers, one with 30 connections, the other one with 10. I already tried using a single server with a single connection, still the same issue at the end.

Will it do it whatever files are inside or just if you've let's say 5GB of files?

I think it works with the >5GB files/folders because it ends up posting before the crash happens lol

from ngpost.

hum... strange, I'm going to try to reproduce it.

it sounds like the Posting thread doesn't consume the Article queue of the main thread whereas it is getting filled. My guess would be that you reach the memory limit and thus crash...
can you check the size of the memory used by ngPost (using top for exemple).
is it like just continuously increasing?
this might be a big design hole...

would be great if you could test with the older version. it's available here (in the release/old folder)
Cheers.

from ngpost.

Yep, I used top since the first crash happened, and that's how I ended up thinking this could be a memory leak. CPU usage was over 100% (around 125). I tried to limit the cpu consumption with cpulimit, even limiting it to 50% ended up in a crash.

And you were right, it's not a version issue: https://tknk.io/eY1Z
Also, I rechecked top and CPU usage was over 120% again, but the memory usage was around 0.3% (there are 258GB of RAM on the server)

from ngpost.

Ok,
that's weird, I don't understand why the article queue is not consummed.
Could you try with this version, I've added more logs around the filling/consuming of the queue.
Please send me the log.
I suppose there is no need to post many files. I'm really interested just in the beginning (first hundreds lines, lets say first 1000 lines with the extra logging)
Just post one file of 15Mo for example.
You should only see this line few times: [Thread #1][NgPost::getNextArticle] no article prepared...

from ngpost.

This was quasi instatanned, the file posted is 10.2MB: https://tknk.io/2FpL

from ngpost.

alright can you try this new release on a big post.
The crash should be fixed.
but the upload speed may suffer from the change.
Please provide me the full log and let me know what became the upload speed. is it much worst?

from ngpost.

can you play with the number of threads? Check first with the number your were using before.
then check with 1, then half.
Please provide me the log file for the 3 cases.
Cheers ;)

from ngpost.

any update? this version solve the issue no? what about the downsize on the upload speed?
I'm working on the v3.1, I'll include that fix with all the new features (posting queue and folder scanning to post each files/folders individually)

from ngpost.

Hi mbruel, sorry for the delay, I was pretty busy.
The upload speed are way better than previously (over 110MB/s).

Here are the logs: one thread (1file of 13GB, average upload speed of 115.99 MB/s): https://tknk.io/J8ze
automatic number of threads (1file of 26GB, around 20 threads, ends up killed): https://tknk.io/1EXQ
Edit: same 26GB, but limited to a single thread: worked perfectly. https://tknk.io/4nOZ

Note that before, when the program was killed, even by limiting the post a single thread still ended up
in a crash, so I think you fixed this.

all the new features (posting queue and folder scanning to post each files/folders individually)

Wow good news! So it will be possible to input a folder, and ngPost will post each file of the folder individually? Also, is it possible to post a folder in this hierarchy:

Folder1/
------------- Folder2/
---------------------------- Files

If I input -i ~/Folder1, it will create an empty archive, while I would like the Folder1 to be rared with its children

from ngpost.

Nice! Well from now I'll be only using a single thread, it's still super fast and the "Killed" is gone.

I tried the --auto argument on the latest version, it outputs 'Error syntax: Unknown option 'auto'' :s
Also, the history seems to not be working for me. I set in my configfile:
POST_HISTORY = /home/me/ngPost/history.cvs
but it never creates any file :(

from ngpost.

By the way, out of curiosity, is ngPost faster than the poster you were using before? or same speed? or slower?
which one were you using? Nyuu?

Edit: I'm quite amazed by your perfs
Upload size: 26755.7 MB in 00:03:39.370 (219 sec) => average speed: 121.97 MB/s (19 connections on 1 threads)
what's your connection upload limit? cause 121.97 is 975.76Mbps oO full gigabit!

from ngpost.

By the way, out of curiosity, is ngPost faster than the poster you were using before? or same speed? or slower?
which one were you using? Nyuu?

Yes, I was using nyuu and I found it pretty slow compared to ngPost. I also diddn't like the fact it was using node.js.

Edit: I'm quite amazed by your perfs
Upload size: 26755.7 MB in 00:03:39.370 (219 sec) => average speed: 121.97 MB/s (19 connections on 1 threads)
what's your connection upload limit? cause 121.97 is 975.76Mbps oO full gigabit!

Hehe, 10gbps server :D It's quite slow compared to what I can usually reach

from ngpost.

I was using nyuu and I found it pretty slow compared to ngPost

Do I have a figure? what was the best upload speed with Nyuu?

I also diddn't like the fact it was using node.js.

same for me, that's why I decided to write my own in C++.

Hehe, 10gbps server :D It's quite slow compared to what I can usually reach

wow nice! I suppose you can play with the number of connections also, What happen if you use more than 19?
Which Usenet provider do you use? I'm quite surprised they provide such a quality of service for the upload speed! There is work on their side too to validate the post and launch the replication.
I could try to get yencode working instead of my basic yEnc encoding method (most of it is pure C)

from ngpost.

Do I have a figure? what was the best upload speed with Nyuu?

I'm not quite sure, around 50Mb/s iirc. I don't pay that much attention about the speeds, I'm just letting the program running in the background :)

wow nice! I suppose you can play with the number of connections also, What happen if you use more than 19?

When I tried with 40 connection, that ended in a crash lol. But that's maybe because I was using a ton of threads

Which Usenet provider do you use?

Usenetfarm, Newsdemon, Vipernews & Newsgroupsdirect. I find UsenetFarm having the best speeds for posting.

Another feature that would be pretty neat is the .nfo linked in the post, like NewsUP does (-nfo nfo_file argument, so the .nfo won't be rared with the rest of the post, just like this guy suggested for newsup https://github.com/demanuel/NewsUP/issues/58).

Btw, don't you think it would be more convenient if you published the binaries & appimages in the releases section, instead of the readme page? :P

from ngpost.

There is a bug in v3.1 with the auto mode.
The password in the nzb files is wrong, they all have the one of the last post.
Please update to v3.2 asap.

Don't throw you nzb. You'll have the correct password in the history file. You'll have to check them and update them one by one... sorry for the inconvenience... I guess I should have waited some testers properly try the soft before releasing...

When I tried with 40 connection, that ended in a crash lol. But that's maybe because I was using a ton of threads

Stick to 1 thread for the moment or max 8 like we're having on regular core i7...

Usenetfarm, Newsdemon, Vipernews & Newsgroupsdirect. I find UsenetFarm having the best speeds for posting

well for me, it's my home connection the limitation... I'm just achieving 15MB/s without vpn, 12MB/s with it.
Are you using a vpn on the server or is it a direct connection? I suppose you use ssl connection right?

Another feature that would be pretty neat is the .nfo linked in the post, like NewsUP does (-nfo nfo_file argument, so the .nfo won't be rared with the rest of the post

I'll check that when I've some time. Could you open a different ticket for that so it doesn't get lost.

Btw, don't you think it would be more convenient if you published the binaries & appimages in the releases section, instead of the readme page? :P

They are of course in the release folder. I'm just adding manually links in the README to help lazy users... (to not say Windows and Mac users... lol )

from ngpost.

They are of course in the release folder. I'm just adding manually links in the README to help lazy users... (to not say Windows and Mac users... lol )

I meant in the "Releases" section of the git, next to the code source hehe https://github.com/mbruel/ngPost/releases

I'll check that when I've some time. Could you open a different ticket for that so it doesn't get lost.

Done.

well for me, it's my home connection the limitation... I'm just achieving 15MB/s without vpn, 12MB/s with it.
Are you using a vpn on the server or is it a direct connection? I suppose you use ssl connection right?

My home connection is like 400kb/s in upload.. nope I'm using a direct connection on my server, and obsiouvly with SSL :)

from ngpost.

I meant in the "Releases" section of the git, next to the code source hehe https://github.com/mbruel/ngPost/releases

I don't know how to do that. The release folder is populated each time I do a git tag. Don't know how to add specific files there...

My home connection is like 400kb/s in upload.. nope I'm using a direct connection on my server, and obsiouvly with SSL :)

So what the bandwith you reach with ngPost on the 400kbps? 30MB/s? less?
You're the only person I know that has access to a 10Gb/s connection.
If you don't mind, I'll give you several versions to try to see which one get the best upload speed.
Something easy would be to make the Posting Thread do the Article body string formatting instead of having the Producer doing it. Then I'm thinking how I can redesign the app to support properly multi-threading and thus have several producers, one on each thread.

from ngpost.

I don't know how to do that. The release folder is populated each time I do a git tag. Don't know how to add specific files there...

https://help.github.com/en/github/administering-a-repository/creating-releases Here for you mate :o)

So what the bandwith you reach with ngPost on the 400kbps? 30MB/s? less?

Nope, I have an asymetrical 28mbps at home, my upload speed can reach max at 400kb/s. That's why I have to rent some servers

If you don't mind, I'll give you several versions to try to see which one get the best upload speed.
Something easy would be to make the Posting Thread do the Article body string formatting instead of having the Producer doing it. Then I'm thinking how I can redesign the app to support properly multi-threading and thus have several producers, one on each thread.

Sure, I would love to help you in this project.

from ngpost.

cool thanks,
I'll provide you soon a version with that easy change to format the Article body in the Posting Thread.
Then a bit latter the new multi-threaded design. This one I may have to think a little bit more before starting ;)
Send me an email ([email protected]) it will be more convenient to communicate outside github.

from ngpost.

Send me an email ([email protected]) it will be more convenient to communicate outside github.

Mail sent, check out your spam folder.

from ngpost.