[feature request] performance improvements / multithreading / parallel execution about m4b-tool HOT 13 CLOSED

Ok so I was testing to see what the performance was like on my ryzen 7 1700x just to give an idea of the speed differences. This is in no way perfect testing as this is a machine used for serving files and media use but I tested about 2 times each and saw about a 2 to 10 second difference between runs.

I tested with a 3 hour and 40 minute audiobook saved as flac in mulitple tracks.

for --jobs=1 I got a merge time of 4 minutes and 30 seconds

for --jobs=4 I got a time of 1 minute and 30 seconds

for --jobs=8 I got a time of 54 seconds

for --jobs=16 i got a time of 47 seconds (this last one was the see how smt helped in this workload which it did what I expected.)

Thank you again so much for all the work put into this it appears to have decreased time by a large amount. I dont see this helping for large single files but most rips from cds are lots of smaller tracks being merged together which this will help alot for.
Have a great day

from m4b-tool.

Good news, i played around with the symfony process wrapper and found a dirty-hack-solution, that might at least work for file conversions (the most time consuming task). First experiments show huge performance improvements, if multiple files are converted (up to 90% better performance using a dual-core system while converting 2 files simultaneously). Tests with --ffmpeg-threads did not result in a significant performance boost on my system.

The --jobs parameter can be used, to define, how many files should be converted simultaneously.

Here is a (highly experimental) nightly build of m4b-tool.tar.gz.

Use e.g. --jobs=8 to run 8 instances of ffmpeg converting 8 files simultaneously. Less files will result in slower conversion... I would really appreciate feedback :-)

Hint: At the moment this does not work in combination with the --audio-profile parameter for high efficiency codecs.

from m4b-tool.

Glad I could help... was interesting to implement :-) There is still much room for improvement, but for now, I'm closing this ticket, as the biggest problem seems to be solved...

from m4b-tool.

Unfortunately PHP is very limited regarding multithreading, well, limited means it just is not supported without really dirty hacks and really extensive effort...

For the moment you can append the experimental parameter --ffmpeg-threads to control how many threads should be used for encoding...

I'll leave this feature request open and will do some research, since performance is one of the main topics in my personal todo list...

from m4b-tool.

ah ok wasn't sure how much php was designed for multi threading very well. shoot me down if im wrong I don't know php that well or to be honest really at all aside from the small tinkering I've done in such things but what about pthreads?

from m4b-tool.

ah ok wasn't sure how much php was designed for multi threading very well

Since PHP is usually running on a web server, thread safety was not the main goal of PHP, which in my opinion leads to the conclusion, that PHP is not the right tool for the job (i thought about re-implementing m4b-tool in go or rust, but for now it's not worth the time)

what about pthreads?

pthreads is integrated in PHP ZTS (thread safe), but by default, PHP install is the NTS (non thread safe). You could check it on your system:

$ php -v
PHP ... ( NTS )
// ....

BUT: Threadsafety is not the problem at all. Since what m4b-tool mainly does is calling other binaries (mostly ffmpeg and mp4v2), multithreading could be implemented by using a non-blocking call of binaries approach. This is possible, since symfony already performs non-blocking calls, but to prevent problems, the calls are implemented as blocking at the moment

The thing is, that i have to verify, if this would be a real improvement over --ffmpeg-threads, if i call multiple binaries at the same time, or if this is just MUCH effort that is hardly measurable...
Multithreading is hard - and not always a valuable improvement...

I have to finish some other tasks before releasing the 0.4.0... e.g. the --batch-pattern feature, which allows to convert a bunch of audio-books without writing an extra shell script :-) But before the release the performance improvements are on my bucket list...

from m4b-tool.

would it be possible to write them as non blocking but check when the tasks have exited? then as the tasks close open new tasks for the next files?

When you are converting from inputFile to outputFile which is a .m4b in this case why not spawn either all the tasks wait for it to be done or spawn X number of tasks wait for some to complete spawn more until all the tasks r complete.

Unsure if u can tell if u use non blocking whether a task is complete or not

from m4b-tool.

Yes, that would be possible, but much effort to do this reliable in PHP. Also i have to VERIFY, that this would really improve the performance (lets say at least 5%).

Since ffmpeg already is multithreaded and the conversion step is the most time consuming task, i'm not sure that it will help a lot. The most time consuming things are:

• Reading the exact duration of the single tracks, to generate valid chapters (either with mp4info or ffmpeg)
• Converting each of the source files to tmp m4b files before merging is possible (ffmpeg)
• Merging the resulting m4b-files (ffmpeg)
• Tagging / generating chapters (mp4tags, mp4chaps or ffmpeg)

Even more time is needed if you use the following flags:

• --adjust-for-ipod means that each input file length has to be estimated, to check, if the desired sample rate is playable on ipods
• --max-chapter-length and --desired-chapter-length (only in the nightly) will result in a very time consuming silence detection, if chapters are longer than --max-chapter-length to place the chapters marks where silences are

If you would like to check, what is the most time consuming thing in your case, you could add a:

-v --logfile=m4b-tool.log

which results in a detailed logfile with time entries for each task / command in ms... example:

NOTICE          0ms 1 match for pattern ./%g/%a/%s/%p - %n/
NOTICE          0ms ================================
NOTICE          2ms merge ./
NOTICE          3ms   => ./001.m4b
NOTICE          4ms - name: ...
NOTICE          4ms - artist: ...
NOTICE          4ms - genre: ...
NOTICE          4ms - series: ...
NOTICE          5ms - series-part: ...
NOTICE          5ms 
NOTICE          5ms ================================
NOTICE          5ms 
NOTICE          6ms ================================
NOTICE         17ms 0 matches for pattern ...
NOTICE         17ms 
NOTICE         18ms ================================
NOTICE         18ms processing ./
NOTICE         30ms reading metadata and streaminfo for file ./001.mp3
NOTICE       3560ms 
NOTICE       3630ms searching for cover in ./
NOTICE       3630ms using cover  ./cover.jpg
NOTICE       3631ms searching for description.txt in ./
NOTICE       3631ms success: found description.txt for import
NOTICE       3633ms ipod auto adjust active, getting track durations
NOTICE       3633ms load estimated duration for file ./001.mp3
....

But as I already said: There are some things to do before the performance tweaking. It's not that i don't know HOW to do it or how to do it the best way... its about IF the effort is valuable and it really improves the performance... Perhaps it is more valuable to improve the order of commands, implement a cache, remove redundant binary calls, use new ffmpeg features to combine two steps to one, etc. etc... it will take some research... patience please ;)

from m4b-tool.

sorry i wasn't trying to be impatient I was just trying to throw some ideas at the fan to see what stuck. anyways please try to have a good day.

Just saw that ffmpeg was only utilizing one of my 8 cores so I figured maybe just throw my execution cycles since it seems to already convert from say flac or mp3 to m4b then from that string the files together. I may be wrong but anyways sorry to bother you so much

from m4b-tool.

I may be wrong but anyways sorry to bother you so much

I really like your ideas and your commitment - so every idea is a good idea, every tester is a good tester, feel free to keep suggesting :-) I just would like to state that it takes some time for the release and news to this topic, I don't think that I'm able to release or analyse this in the next 4 weeks.

Just saw that ffmpeg was only utilizing one of my 8 cores

I think, that ffmpeg tries to automatically use the optimal number of cores since 2014. See this thread for more information: https://superuser.com/questions/155305/how-many-threads-does-ffmpeg-use-by-default

So it strongly depends on the codec, how many cores ffmpeg is using. Referring this thread, ffmpeg is not able to encode every codec fully in parallel, even if more cores would be available.

Did you already try the --ffmpeg-threads=8 param?

This should result in ffmpeg utilize all the cores you have - unless it is not possible for the used codec... (m4b is mp4 with aac or alac under the hood). Important is, that ffmpeg has to be compiled with pthreads, see https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu for reference.

For testing purposes, you could take a look at the docker section - i tried to optimize the docker image for performance and enable all the features needed to get the best audio quality...

from m4b-tool.

I will probably end up taking a look at the docker section thanks btw. I didn't mean for you to do it right at this minute I sorta expected it to be done either in 2 months or never.

However i am going to try compiling with pthreads for ffmpeg and seeing what happens hopefully it works well. I will let you know the result in the end. Thanks btw

EDIT:

Just tested with pthreads enabled and have no different in core utilization. ill try docker soon enough

from m4b-tool.

@popcorn9499 Ok, here is an improved version, where simultaneus merging should be fully supported. This should improve performance a lot when merging a set of files.

Download the nightly build

Provide the --jobs=8 parameter to run 8 simultaneous conversion processes...

Limits:

• Only the CONVERSION of files is possible simultaneously atm - other things like metadata reading or length detection are still limited to one call at a time
• If you have less files than cores (e.g. when you would like to merge 2 files but have 8 cores), the amount of files is the limit for the simultaneous encoding - so only 2 parallel encoding processes are started

I'll wait for testing feedback for 5 days, then I'll close this issue. Feel free do reopen it, if something is wrong with this feature.

from m4b-tool.

Sorry I took so long to get back that thing called life happened. Anyways I tested this abit and it seems to work extremely well seemed well in initial testing. I am going to send some numbers using time to see how long each command takes to execute on the same files both with some varying numbers of cores on the jobs parameter
Thank you so far its looking good

from m4b-tool.