Giter VIP home page Giter VIP logo

space-radar's Introduction

Space Radar Electron

SpaceRadar allows interactive visualization of disk space and memory. It currently supports Sunburst, Treemap, and Flamegraph charts.

Downloads

Download Mac and Windows at the releases page

Features

  • space visualizations using sunburst and treemap charts
  • previews visualization as disk is being scanned
  • fast (completes disk scanner faster than du)
  • cross platform (at least on Mac OS X and Windows)
  • allow drilldown of directories
  • breadcrumbs and navigation
  • opens files and directories
  • analyze disk contents from a remote server (see section Reading from a file)

Screenshots

space-radar-4

screenshot 2015-11-09 04 45 27

screenshot 2015-11-09 04 45 36

Reading from a file

To create a file to be read from use du -ak, for example:

  • du -ak /var/log /usr | gzip -c > /tmp/sizes.txt.gz
  • du -ak /opt /home /tmp > /tmp/sizes.txt

Compressed files can be read directly. To detect them, the file name has to end with .gz.

Future Enhancements

  • more target for scanning
  • color by file types
  • filter hidden files
  • moar!!
  • let me know what you think

Futher Explorations

  • More efficient memory usage
  • More efficient scanning process
  • 3D visualization

History

This project started as quick prototype for me to test drive electron (& some es6 syntax), d3.js and for me to explore the question of "what's taking up my disk space". Turns out writing a disk visualization app isn't that simple as I dwell into figuring out how to make disk scanning not block the ui thread, ipc calls go faster, smoother rendering, lesser memory usage, more sensible interactions...

Whats New

V5

  • Import from DU file
  • Upgrade electron
  • Flamegraphs (BETA)
  • Directory Listview
  • Update libs - Electron, D3

V4

  • Treemap view
  • Memory monitoring
  • Mac App look using Photon
  • Context Menus for locating + opening + deleting files / directories
  • Navigation controls (back/fwd/up)
  • Switched disk scanning jobs to invisible renderer process

Version 3

  • App icon finally! Thanks Jill for the help with this :)
  • Many Bug fixes
  • Disk scanning is moved to a webview process
  • Investigated various RPC methods. Now uses LocalStorage + FileSystem IPC message passing
  • Reduce memory usage (and Electron crashes) by not caching key paths
  • Tested on > 100GB & 2M files
  • Improvements to user interactivity esp on hover states
  • To prevent renderer process from hitting heap mem limit (1.5GB), all previous data is null, with dom elements removed to reduce memory pressure
  • Allow target selection for disk usage scanning
  • Locate path in Finder
  • Env Debug Flags

Version 2

  • Major speed up scanning directories. About 10x from version 1, and almost as fast or faster than du.
  • Runs disk scanning as a separate headless renderer process
  • Json is passed back via IPC
  • Remove Async npm dependency

Issues

Please raise on github issue tracker or contact @blurspline on twitter

Development

Run

npm run debug

or

npm run app

Check that dependencies are installed, otherwise run (this may take awhile for electron binaries)

npm run install

Thanks

  • Jill for designing the app logo
  • Jianwei for his comments on the app
  • Chee Aun for helping alpha test the app
  • WM for his talk on Electron that got me started
  • Contributors

space-radar's People

Contributors

afbjorklund avatar joerg avatar kant avatar peterdavehello avatar zz85 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

space-radar's Issues

Blank window

Space radar starts scanning my disk and periodically refreshing the on-screen image while saying "Scanning, try grabbing a drink". When done, everything disappears and I'm faced with a blank window, only the window controls (close, minimise, maximise) show.

Mac OS X 10.11.3
256GB SSD

Windows feedback

First off, I want to say that I really like the UI, it's actually pretty similar to the disk scanner tool I currently use. So when I found this project and saw in the README that you were looking for feedback from Windows users, I thought I'd give you my first impressions.
Note this analysis only pertains to v3.3.0, as there currently isn't a windows version of the pre-release .

  • As you can see in the screenshot below, the text that is supposed to be in the middle is a little off-centered.
  • When compared to the images in the README, it also seems like the graph colors are appearing differently.
  • To analyze my whole root directory the program took about 3.5 minutes, which is a bit slow, but it appears that you plan to address performance issues in an upcoming release.
  • I would prefer if the program could also display the remaining empty portion of the drive as a part of the graph.
  • It would also be nice to have the graph scale with the size of its containing window, because when I first launched the application the window only displayed about 80% of the overall graph and needed to be adjusted.

2017-04-15_02h04_26

Overall, I think this project has a lot of potential, so feel free to let me know if any of my points need clarification or you need someone to test future builds. ๐Ÿ‘

Can't build/run current master

I'm interested in using space-radar to view data from a remote server. That feature does not appear to be in any of the release versions, so I've tried to build from master. I know next to nothing about electron or the node ecosystem....

I'm on a Mac running OS X 10.11.2

I cloned the repo, ran npm install electron-packager --save-dev, then npm run build. The resulting .app does this when I run it:

screen shot 2016-10-27 at 8 56 56 am

npm install electron and then npm run app does this:

hartzelg-UXG8WL:space-radar hartzelg$ npm run app

> [email protected] app /Users/hartzelg/tmp/space-radar
> electron .

App threw an error during load
Error: Cannot find module 'app'
    at Module._resolveFilename (module.js:455:15)
    at Function.Module._resolveFilename (/Users/hartzelg/tmp/space-radar/node_modules/electron/dist/Electron.app/Contents/Resources/electron.asar/common/reset-search-paths.js:35:12)
    at Function.Module._load (module.js:403:25)
    at Module.require (module.js:483:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/Users/hartzelg/tmp/space-radar/main.js:4:11)
    at Module._compile (module.js:556:32)
    at Object.Module._extensions..js (module.js:565:10)
    at Module.load (module.js:473:32)
    at tryModuleLoad (module.js:432:12)
^C

and after npm install standard, running npm run test complains vociferously.

Can you share instructions on how to make this work?

Or, can you release (preview release?) a version that includes the "read data from file" functionality?

Thanks!

Read and UI Performance

After various testing on speeding up scanning the disk, I'm settling with simply using the node.js async FS methods. However, running the disk intensive methods seems to starve the libuv event loop a little way that even setIntervals seems to be heavily delayed. What's worse the experience of running electron's renderer process while readDirs would seem like the UI has became unresponsive.

My current approach (#5) to that is to run disk scanning in separate (headless) renderer process (either spawn from browser or remote). However, for the target UI process to receive data, the disk scanning processing would need to send messages as an electron IPC method. The scan disk process would send the browser (main.js) an IPC message, and the browser would send the UI process another IPC method. At the expense of more responsive UI overall, it seems the overheads are the price to pay for message passing, and the UI would still seem to freeze marshalling a large json.

One of my favourite hacks for doing message passing on the web is abusing localStorage. LocalStorage is usually pretty performant and allows multiple pages to listen for localStorage change events and that allows me to write a simple localStorage IPC mechanism. Without any benchmarks, this would be faster than electron's IPC in theory (since it would be serialize once instead of twice), and it feels faster than with electron's IPC (yes, feeling is subjective and bad for actually measuring performance).

There's a catch with localStorage, because there're limits associated with how large a string you could write to it. "Uncaught QuotaExceededError: Failed to set the 'lsipc' property on 'Storage': Setting the value of 'lsipc' exceeded the quota."

A simple workaround for now is simply to fallback to electron's IPC calls when localStorage throws me an error (see #7). However even electron IPC has it limits when I start getting [50582:1023/134245:ERROR:ipc_channel_reader.cc(65)] IPC message is too big in my logs.

What other choices do we have? Right out at the top of my mind is the following

  1. IndexedDB
  2. Web Workers
  3. Service Workers
  4. JSON Streaming
  5. Message Chunks

IndexedDB seems like a natural upgrade for localStorage. I haven't use that, but it seems like the next natural path to take. Web Workers could be a lighter form of spawning a headless process, but doing require('fs') in a Web Worker seems like a no-no for now. Which then the next better thing would be to try service workers.

Apart from trying to incase the amount of data that can be transferred, the existing size limitation could be overcome is messages were sent in chunks. One way is perhaps to send diffs for updating in a way a streaming JSONParser could recreate the JSON message entirely.

Lastly, the simplest UI hack would be actually how I think most scanning applications does that. Simply not do anything or show a progress bar until the entire scanning is done (then the disk scanning could be done in the same process).

Project should have a LICENSE file

I see from packages.json that this project is MIT licensed. It's not immediately apparent what the license is when first looking at the project, and not everyone will know to check in packages.json. Having a LICENSE file in the repo that contains the full text of the license would make it much easier to find out how the project is licensed.

Thanks much, and apologies for being such a nit-picky nerd :)

delete files directly from space radar

It would be cool to be able to delete files directly from space radar's interface. It's a good tool to quickly locate files that could free up space if deleted. Adding that possibility would speed up that workflow.

App icon

Requires a vibrant app icon to distinguish itself from other apps, especially when switching apps ๐Ÿ˜›

Backlog

TODO

Visualizations

  • Flame Graphs (Added v5)
  • Color coding
  • Display list of items (Added v5)

Optimizations

  • Canvas Graph Donut Rendering
  • Webgl TreeMaps
  • More efficient memory structure
  • Collapse too small items

Functionality

  • Export DU file
  • Watch directory structure

Releases

  • Bump electron builds (Added v5)
  • Windows builds (Added v5)
  • Linux builds (Added v5)
  • Fix Automatic Mac Builds
  • Try Electron builder

  • Animations for Treemaps
  • Kill Processes
  • Filter / Search files
  • Tweak Graph Colors
  • Free space and estimate scanning
  • Incremental Updates
  • Side panel
  • Treemaps WebGL Rendering
  • Decide whether to use FlatBuffers, CaptnProto, custom data structure or implement some streaming interfaces

DU file import

  • Read-only Mode
  • Allow STDIN / allow loading as a parameter
  • Remove button and add Menu Item
  • Add Linux builds

Size != %space

Hi,
In some cases I find that the size of a child ring is very disproportionate to its relative amount of space.

For instance, I have a folder "Run16" of size 33.50 Gb (figure 1) and it has many sub-directories. One of them is called "vpd_slew" with a size of 23.88 Gb (figure 2). Even though this is 71% of the space (reported as 36%) it shows up as a tiny almost invisible sliver.

Let me know if I can provide more useful info. SpaceRadar is great, thanks for good work.

figure_1
figure_2

Improve styling for Box-View

I love that you have included the Box-View now, but it could be improved style-wise. Right now it looks to gray. I would like some styling similar to Grand Perspective.

Grand Perspective Screenshot

Release 4.4.0 is only for Mac OS

I wanted to try the tool but vesion 4.4.0 is not for Windows. Only for mac. Version 3.3.0 is vor Windows.
Please make it available for Windows too.

Thx

Permission Denied (publickey)

I'm pretty this is my issue but after looking through readme and release notes I'm nost sure how to resolve it and I'm using Windows 10 64x Home Edition.
I downloaded, unzipped and ran npm install on space-radar-4.4.0.zip.
During the npm install I received this error:
Cloning into bare repository 'path\git-github-com-experiments-photon-git-{guid}
Warning: Permanently add 'github, {IP}' (RSA) to the list of known hosts
Permission denied (publickey)
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

Improve Performance with Database

One of the known issues with keeping file system objects in memory is that that v8 heap memory grows towards a limit (~2GB max) with increased CPU usage as garbage collection becomes more expensive.

I've started looking into using different approaches of solving the problem, including using native libs (eg. hashset) and embed databases (sqlite, leveldb, lmdb, ejdb), and with each approach comes its own set of advantages and drawbacks.

Co-incidentally, there's has been some interesting discussion on disk visualization tools on hacker news and led me to find out about other work with regards to dealing with performance and huge number of files. In particularly,

http://duc.zevv.nl/
https://github.com/jarun/nnn/wiki/performance-factors
zevv/duc#161

open a directory

I'm using space-radar on a Mac. When I right-click on a directory, the "open" option is grayed out. I suggest that this menu option should work the same way as "open" on a command line - that it should launch Finder on the directory. That's the interface most familiar to a new user. (I suppose the corresponding tool under Windows would be the Windows Explorer).

I would also like the option to open a terminal window in the directory, since for an experienced user there are many actions that are easier from the command line.

Treemap

Seems to be a popular feature request from my friends (including @cheeaun)

color by type

I suggest assigning colors as follows:
For files, use the same colors as GNU ls by respecting the environment variable LS_COLORS. If that variable is not set, then use the defaults printed by "dircolors -p".
For directories:
For .git, foo.git, .svn, and other configuration management directories: use the same color as for a .tar file.
For .../bin, /Applications, etc: same as an executable, or a .exe file.
For ~/Pictures: same as a .jpg file.
Otherwise, a directory should inherit the color of the contents, with votes weighted by space. I.e., add up the space for all the green files & subdirectories, all the red files & subdirectories, etc. Whichever color accounts for the most space is assigned to the directory.

Coloring by content would make it easier to spot duplicate collections of files, because those directories would be assigned the same color. To make this even easier, one could construct a directory color by averaging over the color of its contents. However, I suggest this be an option rather than the default.

Refreshing forever on Mac Sierra

Looks nice on the videos, but as far as I can tell this does not work on Mac Sierra (10.12.6). If I scan the disk or a folder or add a du file it always just displays "Refreshing..." (tried two separate Macs). During the scan I can see the "Scanning..." window and it seems to get stuck on a file. I tried earlier versions too (5, 4.4, etc).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.