imsnif / diskonaut Goto Github PK

So that users can install diskonaut with brew.

I just did cargo install diskonaut on my rasp pi.

when I run it it only displays a few dirs , for example when i start it in the root of a cargo project only the target dir shows up, no src dir, no files.

Am I doing something wrong

pi@raspberrypi:~ $ cargo -V
cargo 1.46.0 (149022b1d 2020-07-17)
pi@raspberrypi:~ $ rustc -V
rustc 1.46.0 (04488afe3 2020-08-24)
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux
pi@raspberrypi:~ $ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
pi@raspberrypi:~ $

diskonaut preserves the state of the filesystem as it was when it scanned it. If the filesystem changes while the app is running (eg. someone downloads a file into the folder diskonaut previously scanned), it will not be updated.

This feature would mean adding the timestamp of the scan to the title line on the right side.

This might cause trouble with our tests (since they are snapshot tests and we can't set the timestamp in them). So when the app the tests are running, this feature should just output instead. To do this, we can use the #[cfg(not(test))] and #[cfg(test)] configurations.

Would be great to add diskonaut to all repositories who want it. I'm not listing them here because there are quite a few of them. :)

It would be nice if diskonaut supported emacs keys for navigation.

Those keys would be:
left -> Ctrl + b
right -> Ctrl + f
up -> Ctrl + p
down -> Ctrl + n

I think it's not necessary to show that those are supported in the legend as they are kinda common in TUIs and assumed to "just work" because of being supported by GNU Readline.

Otherwise it might be a little confusing...

It would be useful to have prebuild deb packages in a ppa at https://launchpad.net .
So people can install it with getting updates and without having to compile it themself.

When pressing "r", the app should prompt the user with an "Are you sure?" modal, and then rescan the entire folder. This is good in case the contents of the folder have changed since the app was open and we don't want to exit the app and enter it again.

Hey there,

I'm packaging diskonaut for Alpine Linux but it seems tests fails for those two architectures.

https://gitlab.alpinelinux.org/StacyHarper/aports/-/pipelines/114814

Is it known ?

edit: and it seems that armv7 can also fail o_O https://gitlab.alpinelinux.org/StacyHarper/aports/-/pipelines/114819

When we're still scanning, we don't want to let the user delete anything. Both for safety reasons (because things might be jumping around quite a little), and so that they can make an informed decision.

We do not give any indication for this though (only the key missing from the legend). It would be nice to have a modal appear when the user tries to delete in this mode. Something that would read "Sorry, deletion is only allowed once the scanning has completed". The modal should disappear after a few seconds.

Currently one presse Control-D to delete an entry. My first thought was to press the Delete key! I suggest making Delete also work to delete

diskonaut uses the "SIGWINCH" event to detect when it should resize itself. For some reason this does not happen inside a tmux pane.

To reproduce:

1. Start tmux
2. Split the screen (either vertically or horizontally) and try to resize.
3. See that the app stays the same size

At the moment, when a folder has files too small to render ("Small Files") they appear as white "x"s and we can't find out what's inside.

This feature would allow the user to zoom in and out by pressing + and -. Doing so should change the display of the current folder, removing larger files and giving room for smaller files to appear.

A visual indication of this should also appear in the title line. Something like ( < 5M).

EDIT: this does not have to only happen when there are small files. This could also be helpful for removing larger files from the display by zooming in past them.

Before entering a folder, the user would be able to press "p" in order to see a preview of the folder.

This preview would appear on the folder's tile itself instead of the folder name and size, and would contain a nested map of the folder itself. The user would not be able to select any of the rectangles within though - for that they would have to press "ENTER" as usual and enter it.

When diskonaut exits, it returns the user to the folder they were in when they started.

It would be nice if instead of doing that, it would change the user's working directory to the directory they are currently in with diskonaut.

Right now, if we fail to read files (eg. because we got a "permission denied" error), we only show their count globally. It would be great if we could also in addition show them for the current folder (if there are any).

Scanning large volumes with diskonaut takes some time. Not terrible (likely less than 10 minutes for really populated volumes) but this is still not the best experience.

It could be nice to experiment with scanning parallelization, as long as it does not impact performance. I feel a slow scan is better than a fast scan that makes your computer stuck and unable to do anything else.

diskonaut does not re-scan the hard-drive once it is done. If the files change, it does not know about it.

It would be cool if it could listen to the folders it scanned for changes and update itself. Whatever method we choose to implement it, let's make sure not to take up too many resources (CPU mostly) on the host machine.

The folder takes up too much space (7091231513.3G)
in fact, the folder takes up 659 KB
OS: Ubuntu 20.04.4 LTS on Windows 10 x86_64
Shell: bash 5.0.17

Here is the display using ncdu:

--- /path/to/projects/shader_brot ------------------------------------------------------------------------
                         /..
    1.1 GiB [##########] /target                                                                          
  216.0 KiB [          ] /.git
  116.0 KiB [          ]  Cargo.lock
   20.0 KiB [          ]  LICENSE
   20.0 KiB [          ] /resources
   20.0 KiB [          ] /src
    4.0 KiB [          ]  README.md
    4.0 KiB [          ]  TODO.md
    4.0 KiB [          ]  Cargo.toml
    4.0 KiB [          ]  .gitattributes
    4.0 KiB [          ]  .gitignore

                       ┌───Item info──────────────────────────────────────────────┐
                       │                                                          │
                       │  Name: target                                            │
                       │  Path: /path/to/projects/shader_brot                     │
                       │  Type: Directory                                         │
                       │                                                          │
                       │     Disk usage:   1.1 GiB (1,180,995,584 B)              │
                       │  Apparent size:   1.1 GiB (1,174,963,573 B)              │
                       │                                                          │
                       │                              Press i to hide this window │
                       └──────────────────────────────────────────────────────────┘

here is the display in diskonaut:

 Total: 1.4G (2797 files), freed: 0 | /path/to/projects/shader_brot
┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                                        │
│                                                                                                        │
│                                                                                                        │
│                                                                                                        │
│                                      target/ (+2740 descendants)                                       │
│                                                                                                        │
│                                              1.4G (100%)                                               │
│                                                                                                        │
│                                                                                                        │
│                                                                                                        │
│                                                                                                        │
└────────────────────────────────────────────────────────────────────────────────────────────────────────┘
                                                                                        (x = Small files)

I ran the numbers:

• small files take up 425984B
• target takes up 0.999639430994373 of the directory (FP errors in that)

Only going by behavior shown, I suspect the % of the directory that target/ takes up is being rounded to a flat 100%, causing the render to exclude the other files.

Running as root can be dangerous, since we have permission to delete a lot of things we might not want to. It would be nice if we had a visual indication in the title that we're "running as root'.

To reproduce:

1. Create folder /path_to_my/folder
2. Create nested_folder /path_to_my/folder/nested_folder
3. Create a file inside nested_folder /path_to_my_folder/folder/nested_folder/file.txt
4. Change the permissions of nested_folder to be read only.
5. Start diskonaut and give it /path_to_my as its path (diskonaut /path_to_my).
6. Try to delete folder.
7. See that we get an uninformative error message and are unsure what happened.

In this scenario, the message should read something like "Permission denied deleting /path_to_my/folder/nested_folder".

Hey @imsnif ! Another awesome application! Hopefully I can help out with some of those easier issues you've added to the list sometime soon!

I've run into a bit of an issue in running diskonaut on my server through: after around an hour of indexing files, diskonaut was using 22GB of RAM! This pushed a bunch of system processes to swap and brought down a couple of services I had running.

I'll have to take a peek at the code, but I'm assuming this is from tracking the name and information of every file encountered during a scan. If that's the case, then perhaps diskonaut could intelligently decided when RAM usage is getting a bit out of control and collapse sub-trees into a single total size. That would mean that, if you wanted to dive into that sub-tree, you might need to re-index it, but it's just an idea. Having the current directory fully indexed would be good to keep zooming in and out quickly, but I think it's acceptable to re-index sub-directories when you enter them.

As I understand, some other disk-usage applications (like baobab from GNOME) don't even bother tracking individual files. I think it's good that diskonaut does allow you to view those (I'm absolutely in love with the zoom feature!!), but maybe it shouldn't store information for every file all of the time.

Thanks again for the awesome app!

There are two main things I know of that we need to do to port diskonaut to work on windows:

1. Currently we rely on hard disk blocks to scan the hard-drive. As far as I know, this does not work on windows, and so we need to find another solution.
2. Termion doesn't(?) work on windows, so we would need to use another backend for TUI.

I am VERY interested in a solution for this. If you want to work on this and are unsure how to begin, or the points above are not clear, I would be very happy to discuss this with you further and clarify anything. Please do not hesitate. :)

When running diskonaut on a BTRFS filesystem with compression enabled, it shows the uncompressed space used by folders and files, not the actual disk-space used.

One folder of mine using 69.5G of storage, but if I delete this folder I would not regain 69.5G worth of disk space because that folder is being compressed instead I would only regain 50G of space which represents the actual space used on the disk.

The command [sudo compsize /path/to/folder] was able to identify the post-compression space used.

Rationale for feature:

If I am using this tool, I am likely trying to free space so that I may allocate a new file.

Suppose I want to download a 4GiB iso image.
If I have a 4.5GiB zip archive and a 5GiB text-file, diskonaut would make it appear that deleting the text-file would let me download the iso with a GiB to spare.
Unfortunately, with compression enabled, the compressible zip archive would still free up 4.5GiB while the highly compressible text-file may only free a 900MiB.
At that point I would download the iso, run out of space and then have to reopen diskonaut to free ? more GiB (and hope that compression doesn't cause more trouble).

Design Questions

• How should the compressed vs uncompressed space be represented in the UI?

  The uncompressed usage may still be useful if I plan on copying my files to a location without compression.

• On filesystems where it is relatively slow to compute the compressed disk usage, should there be a fast estimation run where placeholders (e.g. 69.5G??? would be displayed above) until the slower more accurate run occurs.

  • How should the estimation be displayed? I chose 69.5G??? earlier but maybe <69.5G> or some other formatting makes more sense.

• What configuration flags should be added?

  • Formatting
  • Enabling/Disabling feature
  • Enabling/Disabling faster estimation process.

Relevant Filesystems

This Wikipedia table of filesystem capabilities shows that the following support compression.

• BeeGFS
• APFS
• NTFS
• F2FS
• Reiser4
• NSS
• NWFS
• Fossil
• ZFS
• BTRFS
• SquashFS

This may only make sense when scanning whole mounts, but it would be nice to have a visual feedback no how much free space there is. It could start as something as simples as a special node/branch that accounts for what df would returnn on the given path. Like diskonaut /foo/bar would include a special colored "free space" with the size of df /foo/bar.

In order to quit the app, we use 'q' or 'ctrl-c'. It should also appear in the legend on the bottom line near the other controls.

Many Unix programs have the -x option for not crossing filesystems. Is this possible to implement for diskonaut?

This feature would allow the user to do a "soft delete". Meaning they would mark a file for deletion, it would disappear from the UI but will not actually be deleted from the filesystem.

When the user exits the app, the app should give a list of these files marked for deletion and ask the user if they want to delete them.

Optionally, a "delete marked files" shortcut can also be added.

Sometimes if there is only one (renderable) tile inside a folder, it can be a little annoying to manually enter all the folders until we get to a folder with some "interesting" content (eg. more than one file :) ).

If we have such a situation, it would be cool if we would just jump to the first interesting folder.

Right now, if the app panics, the stacktrace we see is very much messed up and hard to read, since the terminal is in "raw mode". It would be nice if we could catch all these errors, clear the terminal, exit raw mode and then print them to the screen.

Like du's -x option: File system mount points are not traversed.. This would be useful for scanning / and ignoring external drives.

• Sometimes I don't want to index the external HDDs I have connected
• Sometimes I want to index some files which match some pattern

Running the following commands

mkdir tmp
ls -hs ./tmp
wget --quiet https://www.gutenberg.org/files/1661/1661-0.txt --output-document=./tmp/sherlock.txt
ln ./tmp/sherlock.txt ./tmp/sherlock-link.txt
ls -hs ./tmp

Will produce a 600 KiB file sherlock.txt and a hard link sherlock-link.txt.
diskonaut currently detects these 2 as different files but deleting either file will not free any disk space (beyond a little metadata).
This is similar to the problem of issue #26 where the expectation of xG of deleted data = xG of more usable space proves incorrect.

This problem is typical of hard links (see ln's behavior) but it would be cool if diskonaut could deal with them nicely.

UI idea

Put upper and lower bounds on disk-space before running the slower process of detecting the details of hard-linked files.
By checking for files with a link-count of 1, you can identify non hard linked files.
If you assume that all hard linked files are used outside the current folder (this assumption may be modified for the root directory perhaps) than you get a lower bound of space freed upon deletion.
If you assume that all hard linked are within a directory, you get an upper bound of space freed upon deletion (this is the current behavior of diskonaut)

1G < ?G < 2.4G

When you enter a folder and there's only one subfolder, it makes sense to automatically select it.
Maybe also consider auto selecting the biggest folder if there are many subfolders.
Because currently it takes an extra button press to select anything.

On my machine and several CI builders, diskonaut tests fail. The problem is that the rust-filesize uses MetadataExt::blocks to get the number of file blocks:

https://github.com/Freaky/rust-filesize/blob/e8042c00cebd215ac9f106e8b5a20b0c072fd77d/src/lib.rs#L72

However, this method is not reliable. For example:

$ cat blksize.rs 
use std::fs;
use std::os::unix::fs::MetadataExt;
use std::io;

fn main() -> io::Result<()> {
    let meta = fs::metadata("2pow20bytes")?;
    let blocks = meta.blocks();
    let block_size = meta.blksize();
    eprintln!("blocks: {}, block size: {}", blocks, block_size);
    Ok(())
}
$ rustc blksize.rs
$ dd if=/dev/zero of=2pow20bytes bs=1024 count=1024
$ ls -l 2pow20bytes 
-rw-r--r-- 1 daniel users 1048576 Jun 25 19:48 2pow20bytes
$ ./blksize 
blocks: 1, block size: 131072

So, this will be reported as a 512 byte file(!), even though it is 1MiB. The wonders of (ZFS) filesystem compression.

$ dd if=/dev/urandom of=2pow20bytes bs=1024 count=1024
$ ./blksize 
blocks: 2065, block size: 131072

If I want to delete a lot of files, and let's pretend I can be trusted, is there anyway to 'delete without confirmation? ncdu has this feature

It would be super to have a feature which would allow me to hide selected folders from the view. Especially with Rust projects there's usually a huge target directory completely swamping out everything else.

Right now, diskonaut uses blocks in order to count file size. This means it counts how much space a file takes on the hard-drive, rather than how big that file is. These two can be different when (for example) the filesystem uses compression, and so the actual size taken on the hard-drive would be smaller than the file size.

Since differentiating them, while useful, is very much an edge-case - let's implement an --apparent-size flag rather than displaying both of them on-screen and possibly providing too much information to most users. This will also help us run tests predictably on platforms that use file-system compression (as discussed here: #50).

It would be great to have a feature which would allow us to automatically re-select the parent folder when going one level up instead of removing the selection and having to start over with the navigation.

Right now, diskonaut doesn't look very nice on white terminals. It would be nice to add some other color schemes. Ideally allowing the user to both provide a command line flag (something like --colorscheme white) and to change it while the app is running (to be able to better judge what works best for them).

Hey there, sorry to leave an issue just to say good job, but I figured it'd be worth it. 😉

Anyway, this is the best disk usage analyzer I've ever seen. I've tried out a few different ones, GUI or TUI based, and maybe I haven't searched super hard, but this one beats all the ones I've tried for one big reason: It lets you look around while the tool is working.

That's amazing for UX, and the terminal UI is very nice. Good job on this, and good job with a simple, but nice README and some issues to start with for further improvements. 🎉

The original issue was reported (along with a PR) here: #51
This fixes the issue for tiles on the board (the filenames in the rectangle grid), but we would also need to fix this:

error_box
title_telescope
bottom_line

• for folders we enter and the base folder (in the title bar): src/ui/title/title_telescope.rs (I think a change here would suffice:

  diskonaut/src/ui/title/title_telescope.rs

  Line 186 in 120058d

  current_position += truncated_cell.chars().count() as u16;

  )

• for file display in the bottom lines: src/ui/bottom_line.rs (especially in render_currently_selected and render_last_read_path)

• in deletion confirmation and error message boxes: src/ui/modals/error_box.rs and src/ui/modals/message_box.rs

@goto-bus-stop, I'm assigning this to you since you said you'd like to work on this when you have time. Good catch on these!

Here https://github.com/imsnif/diskonaut/blob/main/Cargo.toml#L15

See the Rust's failure crate: https://crates.io/crates/failure

Hi,

Currently (v0.11.0) when you have mounted (remote) filesystems, diskonaut is also accounting them in its scan.

I would love to see a flag like -xdev in the (GNU) find command or -x in the ncdu command which tells diskonaut to ignore all directories that are other filesystem mounts.

Thanks in advance for your consideration.

Previously discussed in #40

When I run the tests locally with cargo test on the main branch, I get very different results each time. Sometimes all the tests pass (which is somewhat rare), sometimes most of them fail.

Terminal screen capture:
https://asciinema.org/a/MGtehVJYHt76Pq1DTDQyRxv2J

Raw output captured by script:
output.txt

/mnt/n/AppData/Local/Bisq/runtime/include/win32/bridge                                      thread 'main' panicked at 'index out of bounds: the len is 3600 but the index is 3600', /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libcore/slice/mod.rs:2842:10

Running in WSL 2 with rustc 1.43.0 (4fb7144ed 2020-04-20).

When I ssh to a remote and I ran the utility, the graphics break down, although the functionality seem to work.

My machine is Windows, the remote is Windows.
If the remote is Linux, no problem.

Would help a lot in packaging diskonaut, thanks!

Right now, when we press "Ctrl-C" or "q" to exit, the app exits immediately. It would be nice if we could add a modal that prompts the user in case they pressed this by mistake.