High CPU utilization about rmlint HOT 9 CLOSED

Ok good feedback.

I moved to a new branch for this: https://github.com/SeeSpotRun/rmlint/tree/mem_miser which is branched from sahib's sqlite-hash-table branch.

I just ran a synthetic test case with about 5M (admittedly very small) files with the same command options as yours (except -vvv instead of -VVV) (64 bit Fedora 21). It's bogging down at the start of traversal but not stopping completely. It seems that reconstructing the file path each time we need to use it is quite cpu-intensive. I'll have a look at whether we can reduce the number of reconstructions required.

from rmlint.

Seems to work reasonably well now. Test case above had all 5M files in one folder and all the same size which appears to have slowed things down. New test case has 8M files spread across 1024 folders and only 8000 of each size.

Some stats on RAM usage and runtime:

• Regular rmlint 4.21GB 236 seconds
• mem_miser 3.43 GB 1020 seconds
• mem_miser with option --without-fiemap 3.19 GB 1093 seconds
• sqlite swap table 3.42 GB 1619 seconds

from rmlint.

Hey @vvs- are you able to test out the mem_miser branch on your system to see how it performs?

from rmlint.

After finding 1582327 duplicates in 655938 groups it stalled with 100% CPU utilization. I waited for two hours and killed it. No visible progress during that time was reported. No disk activity either. rmlint new -T df -pp --without-fiemap -o progressbar -o summary -VVV

Attached is a profile report. The report was created after running with 100% CPU for about two hours. It seems that the CPU utilization might be related to without-fiemap option.

from rmlint.

Ok thanks for the update. Out of interest, do you recall how many files/GB it had left to scan? I mean was if close to finished or still a long way to go?
I don't think the --without-fiemap is related but I suggest you drop that anyway, it should reduce disk head seek load.
To help us rule out anything not related to -pp, would you mind doing a run without -pp?
Edit: actually hold on that, i managed to reproduce the problem and should have a fix soon.

from rmlint.

Please try again with https://github.com/SeeSpotRun/rmlint/tree/paranoid_mem_debug (with -pp)

from rmlint.

Amazing! It worked like a charm. It used all available memory, but didn't crash and worked very fast. Some stats:

5098832 files
3045531 duplicates using 491.39 GB
1300870 originals
9h48m total run time

Yes, that's not mistake - less than ten hours! Congratulations!

BTW, with-metadata-cache is broken in this version - it's a no op.

from rmlint.

Great to hear.
Actually by default it uses 256MB for buffering file comparisons, plus the overhead for file metadata and other internal structures (which with 5 million files is quite a lot). If you have extra RAM available you can speed up the file scanning process slightly by allocating more memory to this buffer (eg --max-paranoid-mem=512MB).

from rmlint.

Changes have been merged into the main repo (https://github.com/sahib/rmlint/tree/sqlite-hash-table branch) so I'll close off this issue and we can continue the discussion if necessary over there.

from rmlint.