Bug: Runaway memory usage when run on large directories about diskonaut HOT 9 OPEN

Holy cow, that's indeed not good!
I was afraid of this, but quite honestly didn't think to test it on such large volumes. Thanks for this input.

I very much agree with your diagnosis of the problem, for sure the issue is keeping everything we scanned in memory. I like your proposed direction. How about keeping a certain "radius" of the tree in memory... a depth level most users won't need to go into (eg. 5, but maybe that's too much/too little... we'd have to play around with it to find out) and triggering a rescan of that branch if the user goes into it (or into one folder before it, ideally)?

I'm very happy to hear you like diskonaut. Your contributions are always very welcome. Will be looking forward to them (for this or other issues). Give me a shout for absolutely anything you need.

from diskonaut.

Just a thought - would an sqlite in-memory db help offload? I have scanned a huge tree of files/dirs and the db file is ~200mb. You can store path/parent/size in three columns, add indexes and then compute dir-size based on the "parent". Or even inodes. Sorry if that's exactly what your app does.

from diskonaut.

Hmm, interesting idea. I wonder how sqlite would deal with the String issue. :)
Tbh though, I feel this can be solved without introducing another dependency. Thanks for the suggestion though.

from diskonaut.

or use an on disk sqlite DB, I dont see that the in memory one helps, simply swapping one form of memory storage for another. On disk sqlite gives you ultra fast access with seemless spill onto disk.

from diskonaut.

@pm100 an on-disk dependency means the app will require to deal with DB versions and migrations and what not. My suggestion was to help with using a battle-tested piece of software that has very efficient data structure for storage and querying (as long as it is configured and used correctly).

@imsnif what String issue? The multibyte characters (#54)?

from diskonaut.

I will make a on disk sqlite version for kicks, I know sqlite v well.

from diskonaut.

@pm100 Feel free to fork this repo and do whatever you feel is best.

from diskonaut.

@imsnif what String issue? The multibyte characters (#54)?

Ah no, sorry. I was meaning what OP mentioned, where this issue likely is coming from.
What (we assume) the issue here is, is that the string names from all the files are kept in memory and for extremely large volumes (which I believe OP was talking about) this can take up quite some memory.
We may be able to solve this with sqlite. But it will surely need to solve this problem as well (same memory and same strings, after all). While I don't know how/if sqlite solves an issue like this, I assume it involves some sort of caching on the HD. I think it would serve us better to implement something similar ourselves (as I suggested in my comment above), seeing as it would allow us to specialize and even improve the app: eg. allow for an HD cache per subfolder that can also be used between runs and busted when the last scan for that particular subfolder is older than X (or upon user request - we can show "time since last scan" for each subfolder in the UI as an example).

That said, I make a lot of assumptions here. I'm of course willing to take a look at a version with sqlite, but I must honestly say that I'm not convinced about its benefits in this case.

from diskonaut.

I just had this happen too.

from diskonaut.