Interactive navigation for tree with millions of files is really slow about dua-cli HOT 5 CLOSED

Thanks for reporting.

What matters here is that lstat calls are very slow, which is run nearly unconditionally whenever an entry is entered. This is merely to detect presence, as it will highlight entries that aren't present anymore in red. This also means it won't detect type changes, i.e. dir to flie or vice-versa.

Maybe there is a fee to be paid when the lstat call goes though longer paths, which is probably why there is a difference between ls and the way this works here.

To fix it, one could probably just add a flag that prevents these lstat checks entirely. Independently of that, one could try to to speed up the procedure by doing a directory listing instead through which metadata might come back faster. But that's to be validated.

I will implement the command-line flag to disable this behaviour, but also encourage you to see if read_dir + DirEntry::metadata() is faster than a bunch of symlink_metadata() calls on the known entries in a directory. If that's the case, a contribution of that algorithm would make it faster for everyone even if that flag isn't used.

from dua-cli.

When thinking about this use-case I can also think of another improvement: How useful would it be to be able to dump the graph in some format, say JSON, so it could be made accessible by other tooling? Waiting 8 hours for something that is lost when hitting q seems like waste to be avoided.

From there I could absolutely imagine to add additional sub-commands that can handle these snapshot files, to compute changes between them for example. The simplest case would probably be to be able to use the UI to visualise such a snapshot, instead of re-scanning the underlying tree.

Maybe @piotrwach is interested in this as well :).

from dua-cli.

Mitigated in https://github.com/Byron/dua-cli/releases/tag/v2.28.0 .

from dua-cli.

@Byron Yes, I may be interested :) Depending on how it is implemented I think dua could then also be used for analysing directory trees from other sources, like for example S3,. E.g. all that would be required is to dump the object listing in the right format:

<name> <size> <mtime>

Such file then could be loaded in dua in interactive mode. What this would potentially also allow is interactively analysing file systems on other machines without actually running dua there: ssh remote-server find /directory/to/scan -t f | dua i --from-traversal

from dua-cli.

I absolutely love your thoughts! With this feature, dua i would become a front-end for viewing and organizing a graph data structure, which of course can also be created by other tools then.

ssh find… | dua i --from-traversal seems like a separate feature though as that new data structure wouldn't be involved directly. I also don't see how a simple find would produce all the data that's needed per entry, which I find critical to make this 'remote crawler that isn't dua' scenario work (after all, if dua was on the remote one could just use that). However, if that could be made to work, that would definitely be a nice feature to have as well.

For the terminal application input now is just a stream of events, and it doesn't have to care at all about their source anymore.

from dua-cli.