Comments (27)
Hum try the latest version, I've just released it :)
It inclludes the same fix I did for the temporary version.
hum, I'll need to investigate this crash when you get lots of thread. I guess it is arriving in a dead lock.
Good news that the speed is better. I've a bit cheated, increasing to 5 the number of Articles prepared for each Connection. On your station with lots of ram, you could gain in speed playing with that parameter. But it is hardcoded at the moment and I set it back to 3.
I may make it a config in a future release.
with the command line, you can now do
ngPost --auto /Downloads/testNgPost --compress --gen_par2 --gen_name --gen_pass --rar_size 42 --disp_progress files
this will scan the folder /Downloads/testNgPost and post each file or folder individually after compressing with a random name and password and generating the password. (folder will be compressed recursively)
Please update the config file ~/.ngPost as there is a new config (cf here) where the history of the posts will be append so you can get quickly the archive name and password of your posts (plus you get the date, size and avg bandwidth)
I've also made a nice GUI that propose to do the same, have a look if you fancy, it's pretty sexy! ;)
from ngpost.
I had a look at your logs and your station is quite too fast! haha
in the two logs with 1 Thread, we can see that the MainThread that produces the Article stay always in advance and has 5 articles prepared for each connection. We never see the message "no article prepared"
In the opposite with many threads, the MainThread just get catch up really fast by the Posting threads. Line 2994. Then he will never managed to get back in advance but it is accumulating some request in its event loop to build articles.
So I guess the crash comes from an overflow of the Main thread event loop.
I'm going to think about a solution to support multi threading properly. I'm not sure we would gain in speed but maybe. Anyway, it's a good programming challenge.
One solution I see would be to have several producers, in fact, each thread should own their producers and have as many as it is having Connections.
I'll let you know when I'll have done the change.
For now, you should be safe using one thread ;)
from ngpost.
I tried the --auto argument on the latest version, it outputs 'Error syntax: Unknown option 'auto'' :s
arf... I built the wrong version... it's still the 2.3
$ ./ngPost_v3.1-x86_64.AppImage -v
__________ __
____ ____\______ \____ _______/ |_
/ \ / ___\| ___/ _ \/ ___/\ __\
| | \/ /_/ > | ( <_> )___ \ | |
|___| /\___ /|____| \____/____ > |__|
\//_____/ \/
v2.3
I've pushed the real v3.1 for Linux. Please download it and give it a try, it should work!
Same thing for the history file, it was v2.3 so it didn't exist...
from ngpost.
I used the debug -d argument, nothing really useful came out
[Thread #9][NgPost::getNextArticle] no article prepared...
[avg. speed: 6.78 MB/s] >>>>> [5/138] ***.part004.rar
[Thread #2][NgPost::getNextArticle] no article prepared...
[Thread #16] we've read 716800 bytes from 1433600 (=> new pos: 2150400)
[Thread #19][NgPost::getNextArticle] no article prepared...
[Thread #12][NgPost::getNextArticle] no article prepared...
[Thread #6] we've read 716800 bytes from 2150400 (=> new pos: 2867200)
[Thread #17][NgPost::getNextArticle] no article prepared...
[Thread #4] we've read 716800 bytes from 2867200 (=> new pos: 3584000)
[Thread #1] we've read 716800 bytes from 3584000 (=> new pos: 4300800)
[Thread #16][NgPost::getNextArticle] no article prepared...
[Thread #20] we've read 716800 bytes from 4300800 (=> new pos: 5017600)
[Thread #6][NgPost::getNextArticle] no article prepared...
[Thread #15] we've read 716800 bytes from 5017600 (=> new pos: 5734400)
[Thread #4][NgPost::getNextArticle] no article prepared...
[Thread #1][NgPost::getNextArticle] no article prepared...
[Thread #10] we've read 716800 bytes from 5734400 (=> new pos: 6451200)
[Thread #14] we've read 716800 bytes from 6451200 (=> new pos: 7168000)
[Thread #20][NgPost::getNextArticle] no article prepared...
[Thread #8] we've read 716800 bytes from 7168000 (=> new pos: 7884800)
[Thread #15][NgPost::getNextArticle] no article prepared...
[Thread #18] we've read 716800 bytes from 7884800 (=> new pos: 8601600)
[MainThread] we've read 716800 bytes from 8601600 (=> new pos: 9318400)
[Thread #10][NgPost::getNextArticle] no article prepared...
[Thread #14][NgPost::getNextArticle] no article prepared...
[Thread #3] we've read 716800 bytes from 9318400 (=> new pos: 10035200)
[Thread #13] we've read 716800 bytes from 10035200 (=> new pos: 10752000)
[Thread #8][NgPost::getNextArticle] no article prepared...
[Thread #5] we've read 716800 bytes from 10752000 (=> new pos: 11468800)
[Thread #11] we've read 716800 bytes from 11468800 (=> new pos: 12185600)
[Thread #3][NgPost::getNextArticle] no article prepared...
[Thread #18][NgPost::getNextArticle] no article prepared...
[Thread #7] we've read 716800 bytes from 12185600 (=> new pos: 12902400)
[Thread #13][NgPost::getNextArticle] no article prepared...
[Thread #9] we've read 716800 bytes from 12902400 (=> new pos: 13619200)
[Thread #5][NgPost::getNextArticle] no article prepared...
[Thread #2] we've read 716800 bytes from 13619200 (=> new pos: 14336000)
[Thread #11][NgPost::getNextArticle] no article prepared...
[Thread #19] we've read 716800 bytes from 14336000 (=> new pos: 15052800)
[Thread #7][NgPost::getNextArticle] no article prepared...
[Thread #12] we've read 716800 bytes from 15052800 (=> new pos: 15769600)
[Thread #9][NgPost::getNextArticle] no article prepared...
Killed
from ngpost.
Hello,
normally, there is no change to the posting process between 2.2 and 2.3, just some HMI updates to preview the next major version.
Could you give the full log please.
Is it crashing every time? How big is the full post? each archive?
Sounds like the main thread who's supposed to be ahead of the Posting ones to prepare the Articles has been caught up by the posting threads. This is weird, not supposed to happen but anyway, I did some debugging with no article prepared and it was stable...
Could you also give me the full command please.
Why do you use so many threads? have you set it up or is it your server that has more than 20 cores?
from ngpost.
Could you give the full log please.
You mean the full output of the --debug right?
Is it crashing every time? How big is the full post? each archive?
Yep, for every >5GB post. I tried from 20MB to 500MB per archive
Why do you use so many threads? have you set it up or is it your server that has more than 20 cores?
There are over 24 CPU (576 threads in total) on the server I use. I already tried with a single thread (modified the thread value in the config file to 1), it still crashes a few second after the posting begins, so I'm pretty sure the problem is not here :s
Could you also give me the full command please.
./ngPost_v2.3-x86_64.AppImage -c ~/.ngPost -i ~/downloads/***.mkv -o ~/ngPost/***.nzb --compress --gen_name --gen_par2
After that, I tried to post the rars created during the process using
./ngPost_v2.3-x86_64.AppImage -c ~/.ngPost -i ~/tmp/*** -o ~/ngPost/***.nzb
but still crashes a few seconds after.
from ngpost.
hum interesting, I've never tried on such a powerful station...
Are you definitely sure there is no crash with the v2.2 using exactly the same parameters? that would definitely help to narrow down the potential issue.
You mean the full output of the --debug right?
yes could you run it with output redirection to a file and stderr also. somthing like this:
./ngPost_v2.3-x86_64.AppImage -d -t 1 -i ~/tmp/*** -o ~/ngPost/***.nzb > /tmp/ngPost.log 2>&1
as you can see on the command:
- no need to put -c ~/.ngPost as it is picked up automatically as a default configuration file (-c is to be used in case you wish to use an alternative config file with let's say another provider)
- use only one thread, if the crash is there it will be easier to debug as we won't have to deal with mutex synchronization (less in fact)
- as the problem doesn't come from the rar/par2 generation, please use directly a folder as input
How many Usenet severs / connections are you using? Is it happening also if you decrease it? if it is the case, it would help to debug if you could use only 1 connection. (edit the config file and set connection to 1)
You say it crashes after few seconds in case of uploading a folder.
Will it do it whatever files are inside or just if you've let's say 5GB of files?
Thanks in advance for your debugging ;)
from ngpost.
Here is the log: https://tknk.io/bY33
I tried with a 13GB file, splitted in 131 parts of 100MB each.
Are you definitely sure there is no crash with the v2.2 using exactly the same parameters?
I don't have the 2.2 anymore since I removed it, and I wasn't able to download it using the previous link. :s
But iirc, I haven't encountered any issue while posting before.
How many Usenet severs / connections are you using?
I usually use 2 servers, one with 30 connections, the other one with 10. I already tried using a single server with a single connection, still the same issue at the end.
Will it do it whatever files are inside or just if you've let's say 5GB of files?
I think it works with the >5GB files/folders because it ends up posting before the crash happens lol
from ngpost.
hum... strange, I'm going to try to reproduce it.
it sounds like the Posting thread doesn't consume the Article queue of the main thread whereas it is getting filled. My guess would be that you reach the memory limit and thus crash...
can you check the size of the memory used by ngPost (using top for exemple).
is it like just continuously increasing?
this might be a big design hole...
would be great if you could test with the older version. it's available here (in the release/old folder)
Cheers.
from ngpost.
Yep, I used top since the first crash happened, and that's how I ended up thinking this could be a memory leak. CPU usage was over 100% (around 125). I tried to limit the cpu consumption with cpulimit, even limiting it to 50% ended up in a crash.
And you were right, it's not a version issue: https://tknk.io/eY1Z
Also, I rechecked top and CPU usage was over 120% again, but the memory usage was around 0.3% (there are 258GB of RAM on the server)
from ngpost.
Ok,
that's weird, I don't understand why the article queue is not consummed.
Could you try with this version, I've added more logs around the filling/consuming of the queue.
Please send me the log.
I suppose there is no need to post many files. I'm really interested just in the beginning (first hundreds lines, lets say first 1000 lines with the extra logging)
Just post one file of 15Mo for example.
You should only see this line few times: [Thread #1][NgPost::getNextArticle] no article prepared...
from ngpost.
This was quasi instatanned, the file posted is 10.2MB: https://tknk.io/2FpL
from ngpost.
alright can you try this new release on a big post.
The crash should be fixed.
but the upload speed may suffer from the change.
Please provide me the full log and let me know what became the upload speed. is it much worst?
from ngpost.
can you play with the number of threads? Check first with the number your were using before.
then check with 1, then half.
Please provide me the log file for the 3 cases.
Cheers ;)
from ngpost.
any update? this version solve the issue no? what about the downsize on the upload speed?
I'm working on the v3.1, I'll include that fix with all the new features (posting queue and folder scanning to post each files/folders individually)
from ngpost.
Hi mbruel, sorry for the delay, I was pretty busy.
The upload speed are way better than previously (over 110MB/s).
Here are the logs: one thread (1file of 13GB, average upload speed of 115.99 MB/s): https://tknk.io/J8ze
automatic number of threads (1file of 26GB, around 20 threads, ends up killed): https://tknk.io/1EXQ
Edit: same 26GB, but limited to a single thread: worked perfectly. https://tknk.io/4nOZ
Note that before, when the program was killed, even by limiting the post a single thread still ended up
in a crash, so I think you fixed this.
all the new features (posting queue and folder scanning to post each files/folders individually)
Wow good news! So it will be possible to input a folder, and ngPost will post each file of the folder individually? Also, is it possible to post a folder in this hierarchy:
Folder1/
------------- Folder2/
---------------------------- Files
If I input -i ~/Folder1, it will create an empty archive, while I would like the Folder1 to be rared with its children
from ngpost.
Nice! Well from now I'll be only using a single thread, it's still super fast and the "Killed" is gone.
I tried the --auto argument on the latest version, it outputs 'Error syntax: Unknown option 'auto'' :s
Also, the history seems to not be working for me. I set in my configfile:
POST_HISTORY = /home/me/ngPost/history.cvs
but it never creates any file :(
from ngpost.
By the way, out of curiosity, is ngPost faster than the poster you were using before? or same speed? or slower?
which one were you using? Nyuu?
Edit: I'm quite amazed by your perfs
Upload size: 26755.7 MB in 00:03:39.370 (219 sec) => average speed: 121.97 MB/s (19 connections on 1 threads)
what's your connection upload limit? cause 121.97 is 975.76Mbps oO full gigabit!
from ngpost.
By the way, out of curiosity, is ngPost faster than the poster you were using before? or same speed? or slower?
which one were you using? Nyuu?
Yes, I was using nyuu and I found it pretty slow compared to ngPost. I also diddn't like the fact it was using node.js.
Edit: I'm quite amazed by your perfs
Upload size: 26755.7 MB in 00:03:39.370 (219 sec) => average speed: 121.97 MB/s (19 connections on 1 threads)
what's your connection upload limit? cause 121.97 is 975.76Mbps oO full gigabit!
Hehe, 10gbps server :D It's quite slow compared to what I can usually reach
from ngpost.
I was using nyuu and I found it pretty slow compared to ngPost
Do I have a figure? what was the best upload speed with Nyuu?
I also diddn't like the fact it was using node.js.
same for me, that's why I decided to write my own in C++.
Hehe, 10gbps server :D It's quite slow compared to what I can usually reach
wow nice! I suppose you can play with the number of connections also, What happen if you use more than 19?
Which Usenet provider do you use? I'm quite surprised they provide such a quality of service for the upload speed! There is work on their side too to validate the post and launch the replication.
I could try to get yencode working instead of my basic yEnc encoding method (most of it is pure C)
from ngpost.
Do I have a figure? what was the best upload speed with Nyuu?
I'm not quite sure, around 50Mb/s iirc. I don't pay that much attention about the speeds, I'm just letting the program running in the background :)
wow nice! I suppose you can play with the number of connections also, What happen if you use more than 19?
When I tried with 40 connection, that ended in a crash lol. But that's maybe because I was using a ton of threads
Which Usenet provider do you use?
Usenetfarm, Newsdemon, Vipernews & Newsgroupsdirect. I find UsenetFarm having the best speeds for posting.
Another feature that would be pretty neat is the .nfo linked in the post, like NewsUP does (-nfo nfo_file argument, so the .nfo won't be rared with the rest of the post, just like this guy suggested for newsup https://github.com/demanuel/NewsUP/issues/58).
Btw, don't you think it would be more convenient if you published the binaries & appimages in the releases section, instead of the readme page? :P
from ngpost.
There is a bug in v3.1 with the auto mode.
The password in the nzb files is wrong, they all have the one of the last post.
Please update to v3.2 asap.
Don't throw you nzb. You'll have the correct password in the history file. You'll have to check them and update them one by one... sorry for the inconvenience... I guess I should have waited some testers properly try the soft before releasing...
When I tried with 40 connection, that ended in a crash lol. But that's maybe because I was using a ton of threads
Stick to 1 thread for the moment or max 8 like we're having on regular core i7...
Usenetfarm, Newsdemon, Vipernews & Newsgroupsdirect. I find UsenetFarm having the best speeds for posting
well for me, it's my home connection the limitation... I'm just achieving 15MB/s without vpn, 12MB/s with it.
Are you using a vpn on the server or is it a direct connection? I suppose you use ssl connection right?
Another feature that would be pretty neat is the .nfo linked in the post, like NewsUP does (-nfo nfo_file argument, so the .nfo won't be rared with the rest of the post
I'll check that when I've some time. Could you open a different ticket for that so it doesn't get lost.
Btw, don't you think it would be more convenient if you published the binaries & appimages in the releases section, instead of the readme page? :P
They are of course in the release folder. I'm just adding manually links in the README to help lazy users... (to not say Windows and Mac users... lol )
from ngpost.
They are of course in the release folder. I'm just adding manually links in the README to help lazy users... (to not say Windows and Mac users... lol )
I meant in the "Releases" section of the git, next to the code source hehe https://github.com/mbruel/ngPost/releases
I'll check that when I've some time. Could you open a different ticket for that so it doesn't get lost.
Done.
well for me, it's my home connection the limitation... I'm just achieving 15MB/s without vpn, 12MB/s with it.
Are you using a vpn on the server or is it a direct connection? I suppose you use ssl connection right?
My home connection is like 400kb/s in upload.. nope I'm using a direct connection on my server, and obsiouvly with SSL :)
from ngpost.
I meant in the "Releases" section of the git, next to the code source hehe https://github.com/mbruel/ngPost/releases
I don't know how to do that. The release folder is populated each time I do a git tag. Don't know how to add specific files there...
My home connection is like 400kb/s in upload.. nope I'm using a direct connection on my server, and obsiouvly with SSL :)
So what the bandwith you reach with ngPost on the 400kbps? 30MB/s? less?
You're the only person I know that has access to a 10Gb/s connection.
If you don't mind, I'll give you several versions to try to see which one get the best upload speed.
Something easy would be to make the Posting Thread do the Article body string formatting instead of having the Producer doing it. Then I'm thinking how I can redesign the app to support properly multi-threading and thus have several producers, one on each thread.
from ngpost.
I don't know how to do that. The release folder is populated each time I do a git tag. Don't know how to add specific files there...
https://help.github.com/en/github/administering-a-repository/creating-releases Here for you mate :o)
So what the bandwith you reach with ngPost on the 400kbps? 30MB/s? less?
Nope, I have an asymetrical 28mbps at home, my upload speed can reach max at 400kb/s. That's why I have to rent some servers
If you don't mind, I'll give you several versions to try to see which one get the best upload speed.
Something easy would be to make the Posting Thread do the Article body string formatting instead of having the Producer doing it. Then I'm thinking how I can redesign the app to support properly multi-threading and thus have several producers, one on each thread.
Sure, I would love to help you in this project.
from ngpost.
cool thanks,
I'll provide you soon a version with that easy change to format the Article body in the Posting Thread.
Then a bit latter the new multi-threaded design. This one I may have to think a little bit more before starting ;)
Send me an email ([email protected]) it will be more convenient to communicate outside github.
from ngpost.
Send me an email ([email protected]) it will be more convenient to communicate outside github.
Mail sent, check out your spam folder.
from ngpost.
Related Issues (20)
- Log for the posting process? HOT 1
- in GUI monitoring and file name obfuscation checked - Endless posting loop
- Problem HOT 5
- When will the new version be released? HOT 13
- [Enhancement][Quality Of Life ''improvement''] just 2 requests. nothing huge or major.
- add command line parameter for rar compression level
- compression and encryption independently
- add in command line output nzb for every input file option
- Question about automated posting
- LinuxMint: Remote host closed the connection
- Auto compress not doing anything (GUI, Windows)
- Have some requests for new version HOT 6
- Autoposting: all files in monitored folder with same password? HOT 10
- program crashes oftenly HOT 10
- [Bug] posting failed
- [feature Request] random Poster Adress HOT 1
- [Windows 10] No par files created HOT 3
- Nfo
- Upload files from the folders and save the folder name as nzb HOT 1
- So much missing connection cause incomplete upload HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ngpost.