Giter VIP home page Giter VIP logo

randomfractals / tabular-data-viewer Goto Github PK

View Code? Open in Web Editor NEW
10.0 3.0 0.0 16.28 MB

Tabular Data Viewer πŸ€„ VSCode extension for viewing very large local and remote CSV and TSV data files with Tabulator Table, Perspective Datagrid and D3FC Chart Views πŸ“ŠπŸ“ˆ

Home Page: https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.tabular-data-viewer

License: Apache License 2.0

TypeScript 67.19% JavaScript 30.21% CSS 2.60%
vscode tabular data viewer flat-data csv tsv dsv large-data tabulator perspective view d3fc charts remote-data data-packages datapackage

tabular-data-viewer's Introduction

tabular-data-viewer

Apache-2.0 License Version Installs Downloads https://ko-fi.com/dataPixy


Tabular Data Viewer δΈ­ for Visual Studio Code

See Data Preview 🈸 vscode extension for advanced Data Grid Viewer with support of many common data formats and Save options for smaller datasets.

Tabular Data Viewer δΈ­

Tabular Data Viewer provides fast DSV data loading and custom Table Views πŸ€„ for very large (1-8 GB) local and remote .csv, .tsv and .tab data files with table, View, D3FC Chart Views πŸ“ŠπŸ“ˆ, Papa Parse CSV parser, NodeJS File Streams and Worker threads:

Tabular Data View δΈ­

Features

  • View large CSV and TSV data files in a custom Tabulator Table or Perspective Data View πŸ€„
  • Open local file:/// or remote https:// Data File Table View πŸ€„ with quick Data File URL input prompt
  • Explore over 750 public datasets from the curated list of https://frictionlessdata.io/ Data Package πŸ“¦ JSON configs with Tabular Data Resources, and Table Views πŸ€„ to search, view, and graph in Chart Views πŸ“Š πŸ“ˆ
  • Load large datasets with millions of rows and over 1GB of tabular data using Papa Parse CSV parser in a worker thread with data streaming and incremental loading into Table View πŸ€„
  • View data loading progress, local data file size, colulmns, rows and load time stats display in vscode status bar
  • Sort displayed table data by multiple columns
  • Resize table columns in Tabulator Table View
  • Rearrange table columns in Tabulator Table and Perspective Data Grid views
  • Set default Data Page Size to 1K, 10K, or 100K (default) rows per page to speed up large datasets parsing and loading into Table View πŸ€„
  • Auto-save table columns layout, sort order, and Perspective View configs
  • Restore customized Table View πŸ€„ configuration on vscode reload
  • Load saved table configuration options from auto-generated *.table.json config file when reopening previously customized Table View πŸ€„
  • Generate and view Frictionless Table *.schema.json configration file created with tableschema-js.infer()
  • Copy displayed table data to Clipboard in TSV format from Tabulator Table View or in CSV format from Perspective View
  • Save displayed table data in CSV (, or ; delimited), TSV, HTML table, or JSON data array format from Tabulator Table View
  • Save displayed table data in Apache Arrow β‹™ data format from Perspective View:

Data Perspective View δΈ­

Limitations

Large CSV/TSV data files streaming and display is very alpha and has the following limitations for now:

  • Large datasets are paged and loaded into view for display with default page size set to 100K rows per page
  • Columns sort only works for the paged data loaded and displayed in Table View πŸ€„ and doesn't sort data for the whole dataset
  • Save data is also limited to max 100k rows for the currently displayed tabular data in Table View πŸ€„

Tabular Data View δΈ­ Large Dataset

Installation

Install Tabular Data Viewer δΈ­ from VSCode Extensions tab (ctrl+shift+x) by searching for tabular data in Extensions: Marketplace sidebar view, or install it directly from VS Code Marketplace Tabular Data Viewer δΈ­ extension page in a browser.

Tabular Data Viewer δΈ­ Installation

Deployments

Tabular Data Viewer δΈ­ is designed to work with VSCode Desktop IDE version and also works with GitHub Codespaces in a browser:

Tabular Data Viewer δΈ­ in Github Codespaces

Tabular Data Viewer δΈ­ works in VSCodium too. However, VSCodium and other vscode extension compatible IDE flavors are not officially supported as they require additional testing in those IDE variants.

You can use tabular-data-viewer-x.y.z.vsix extension packages attached in Assets section with every published release of this extension on github. Follow install from .vsix instructions in your vscode extensions compatible IDE or online container service to install it.

If you prefer to use other IDEs that support VSCode extensions published in .vsix format and download your extensions from open-vsx.org, we recommend you try older Data Preview version of this extension.

Currently, there are no plans to publish this extension to Open VSX Registry due to poor publishing experience in that vscode extensions marketplace and low usage by majority of developers.

Contributions

Tabular Data Viewer δΈ­ Settings, Commands, Languages, Custom Editors and Activation Events contributed to VSCode IDE:

Tabular Data Viewer δΈ­ Feature Contributions

Settings

Create User or Workspace Settings in vscode to change Tabular Data Viewer δΈ­ extension Settings listed below. All Tabular Data Viewer Settings start with tabular.data. prefix, which is omottied in the Settings table below for better display in VSCode marketplace.

Setting tabular.data. Type Default Value Description
createTableSchemaConfig boolean true Create table *.schema.json configuration file alongside data file when parsing CSV data.
createTableViewConfig boolean true Create *.table.json configuration file alongside data file for saving and restoring customized Table View πŸ€„ columns, sort settings, Perspective View and D3FC Charts πŸ“ŠπŸ“ˆ config options.
pageSize integer 100000 Default Data Page Size to use for incremental loading and display of large datasets.
parser.dynamicTyping boolean true Enable type conversions for numeric and boolean data fields when parsing CSV data. Set this option to false to speed up large datasets parsing and loading.
ui.theme string light Default data view UI theme to use for tabular data display. Current options include light and dark UI themes for Tabulator and Perspective data views.
view string Tabulator Default Tabular Data View type to use for displaying data. Current options include Tabulator and Perspective.

You can set your Tabular Data Viewer δΈ­ Configuration Settings by adding them to ./vscode/settings.json in your project workspace. The following workspace configuration settings will default Table Views πŸ€„ to use Perspective Data Grid and Dark UI theme for tabular data and basic D3FC Charts πŸ“ŠπŸ“ˆ display:

{
	// Tabular Data Viewer settings for this workspace
	"tabular.data.view": "Perspective",
	"tabular.data.ui.theme": "dark"
}

You can also view Tabular Data Viewer δΈ­ Configuration Settings via standard vscode Preferences -> Settings -> Extensions -> Tabular Data Viewer Settings view or use custom Tabular Data: View Settings command from View -> Command Palette... menu prompt by typing Tabular in it.

Tabular Data Viewer δΈ­ Configuration Settings

Hit Reload β†Ί button in an open Table View πŸ€„ after you toggle Tabular Data Settings to activate your configuration changes.

Note: tabular.data.view type default and tabular.data.ui.theme Setting changes might require you to close and reopen Tabular Data Views for those settings to take effect.

Commands

Tabular Data Viewer δΈ­ provides a number of global and context specific commands to load and view tabular data files and Data Packages πŸ“¦.

You can access custom Tabular Data Commands from View -> Command Palette... menu (ctrl+shift+p) by typing Tabular in the command search box:

Tabular Data δΈ­ Commands

All Tabular Data Commands start with Tabular Data: prefix, and exposed public Tabular Data Viewer δΈ­ command Ids use tabular.data. prefix to enable other extensions to use Table View πŸ€„ and tabular data list commands.

Some of the Tabular Data Commands like View Table and List Data Resources are enabled in vscode file explorer and in code editors for the supported data file types and formats via the corresponding context menues in vscode IDE.

Currently supported Tabular Data δΈ­ Commands include:

Command tabular.data. Name Description
listDataPackages List Data Packages Lists over 750 public Data GitHub Repositories with datapackage.json configuration files that ship with Tabular Data Viewer δΈ­ v1.8.0 and above for exploring public datasets and Tabular Data Resources. Curated list of those Data Packages πŸ“¦ is configured in our packages.json configuration file.
listDataResources List Data Resources Lists Tabular Data δΈ­ Resources for an open datapackage.json file in an active vscode editor or a public dataset GitHub repository selected from the Data Package πŸ“¦ Quick Pick List above.
openDataFile Open Data File Prompts for a local file:/// or remote https:// Tabular Data Resource or a Data Package πŸ“¦ Url to load and view the Data Resource List or a Table View πŸ€„ for data files.
openTextDocument Open Text Document Opens local Tabular Data Resource in built-in vscode Text Editor, or opens it in a browser for GitHub hosted data sources and other data files loaded via remote https:// Urls.
viewTable View Table Displays Table View πŸ€„ for an open Tabular Data file in vsode Editor or from context menus in built-in File Explorer.
viewDataFileOnGitHub View Data File on GitHub Opens public GitHub Repository Data File or a Data Package in a browser.
viewSettings View Settings Displays Tabular Data Viewer δΈ­ Configuration Settings.

Tabular Data Viewer δΈ­ v1.8.0 and above provides a custom Uri Handler for launching Table Views πŸ€„ and other tabular.data commands from a Url in a browser. All of the listed commands above can be launched from a browser using the following vscode:// Url format:

vscode://RandomFractalsInc.tabular-data-viewer/<command>[#<dataUrl>|?dataUrl=<dataUrl>]

which instructs a browser to open VSCode IDE Desktop application, activate Tabular Data Viewer δΈ­ extension, and execute one of the tabular.data. commands via system-wide Urls. Commands that require a data file or data package Url accept that parameter via fragment/anchor (#) portion of the Url, or via dataUrl query parameter.

For example, the following vscode:// Url will display Table View πŸ€„ for the public https://github.com/datasets/world-cities data repository. Copy and paste this Url in your browser to try it:

vscode://RandomFractalsInc.tabular-data-viewer/viewTable#https://raw.githubusercontent.com/datasets/world-cities/master/data/world-cities.csv

Data Packages

Tabular Data Viewer δΈ­ v1.8.0 and above ships with alpha support of https://frictionlessdata.io/ Data Package πŸ“¦ JSON format and over 750 built-in curated public datasets with Tabular Data δΈ­ Resources for you to search and explore:

Tabular Data δΈ­ Packages πŸ“¦

Public GitHub repositories have over 12K datapackage.json configurations you can browse using this query: https://github.com/search?p=4&q=filename%3Adatapackage.json+language%3AJSON+language%3AJSON&type=Code

You can load those Data Package πŸ“¦ configurations to view Tabular Data δΈ­ Resources via Tabular Data: Open Data File command. For example, here is how to open Periodic Table datapackage.json example: https://github.com/frictionlessdata/examples/blob/main/periodic-table/datapackage.json

Tabular Data δΈ­ Open Data Package πŸ“¦

Observable JS Notebooks

You can try Tabular Data Viewer πŸ€„ Observable JS Notebook πŸ““ with public datasets and supported data formats in a browser:

Tabular Data Viewer δΈ­ Observable Notebook πŸ““

You can also experiment with some public Data Packages πŸ“¦ and datapackage-js library in our Data Package πŸ“¦ Observable JS Notebook πŸ““ in a browser:

Data Package πŸ“¦ Observable JS Notebook πŸ““

Recommended Extensions

Recommended custom dataViz extensions produced by Random Fractals Inc. and other 3rd party vscode extension authors for working with data 🈸 charts πŸ“ˆ geo πŸ—ΊοΈ data formats and Interactive Notebooks πŸ“š in VSCode:

Extension Description
Rainbow CSV 🌈 Highlight CSV and TSV files, Run SQL-like queries
Data Preview 🈸 Data Preview 🈸 extension for importing πŸ“€ viewing πŸ”Ž slicing πŸ”ͺ dicing 🎲 charting πŸ“Š & exporting πŸ“₯ large JSON array/config, YAML, Apache Arrow, Avro & Excel data files
Geo Data Viewer πŸ—ΊοΈ kepler.gl Geo Data Analytics tool to gen. some snazzy πŸ—ΊοΈs w/0 Py 🐍 pyWidgets βš™οΈ pandas 🐼 or react βš›οΈ
Vega Viewer πŸ“ˆ Provides Interactive Preview of Vega & Vega-Lite maps πŸ—ΊοΈ & graphs πŸ“ˆ
Observable JS Observable JS compiler with Observable js and md code outline and previews.
JS Notebook πŸ““ Inspector πŸ•΅οΈ Provides Interactive Preview of Observable JS Notebooks πŸ“š, Notebook πŸ““ nodes βŽ‡ & cells βŒ— source code
Data Table 🈸 Data Table 🈸 renderer for Notebook πŸ““ cell βŒ— data outputs
Leaflet Map 🌿 πŸ—ΊοΈ Leaflet Map πŸ—ΊοΈ for Notebook πŸ““ cell βŒ— data outputs

Dev Log

See #TabularDataViewer πŸ€„ tag on Twitter for the latest and greatest updates on this vscode extension and what's in store next.

Dev Build

$ git clone https://github.com/RandomFractals/tabular-data-viewer
$ cd tabular-data-viewer
$ npm install
$ npm run compile
$ code .

Press F5 to launch Tabular Data Viewer πŸ€„ extension debug session, or run the following command to generate tabular-data-viewer-x.x.x.vsix package with vsce from our latest for local dev install from .vsix in vscode.

tabular-data-viewer>vsce package

Contributing

Any and all test, code or feedback contributions are welcome. Open an issue or submit a pull request to make this Tabular Data Viewer πŸ€„ extension work better for all.

Backers

support me on ko-fi.com

tabular-data-viewer's People

Contributors

randomfractals avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

tabular-data-viewer's Issues

Adopt frictionless data standards

Change Tabular Data Text Editor default

Set custom Tabular Data Text editor provider added in #12 to use priority: "option" instead of priority: "default", so that CSV and TSV data files can be opened in the built-in vscode Text editor by default and viewed with Tabular data viewer via Open With ... file menu option:

image

Document CUDA drivers and native node modules setup, extraction, and integration in this vscode extension

Add notes from preliminary discussion with Paul Taylor (@trxcllnt) about their work on CUDA compute libs and apis for loading data via native nodejs libraries, and map out how they will be integrated in this vscode extension via platform specific extension packaging and publishing mechanisms: https://code.visualstudio.com/api/working-with-extensions/publishing-extension#platformspecific-extensions

See CUDA node rapidsai repo for more info for now: https://github.com/rapidsai/node

Expose Tabular Data Viewer extension api for other extensions to load datasets in our Table View via standard vscode commands api

so that other extension authors can use table view for their data display.

Example:

const rows = await db.query(statement);
vscode.commands.executeCommand(`tabular.data.viewTable`, rows);

This tabular data viewer api should povide a simple load/add data and filter methods for other 3rd party extensions that have query panels or get their data in other ways ...

query-example-UI

Most likely how we'll add rapidsai cuDF data integration for their large data toolkit described in #6.

See example in vscode docs for setting up public api exposed by an extension via its activation method:

https://code.visualstudio.com/api/references/vscode-api#extensions

Resolve reopening table view for a data file after tab close

Currently, Tabular data viewer fails on reopening table view for a data file after that view was created, closed and disposed.

Potentially related to our custom table view management, tracking and dispose code added in #10.

Actual error when that happens:

image

Consider moving table views lookup to a separate view manager. See example setup in geo data viewer: https://github.com/RandomFractals/geo-data-viewer/blob/master/src/view.manager.ts

Persist table view config settings for restore on vscode reload or after table view tab close

Tabulator table persistence added in #22 doesn't work with localStorage in vscode electron app for some reason.

In order to save and load Tabulator table columns, sort, and filter changes, consider saving them per data file in extension context storage. See: https://code.visualstudio.com/api/extension-capabilities/common-capabilities#data-storage

Also, perhaps combining those data view settings in a single <dataFileName>.table.json could make it more manageable for the table view config save and restore on vscode reload, and opening that config for manual JSON table config edits and tabular data view reload.

Add Columns visibility dropdown to the Table View toolbar

Current table view setup provides an option to hide a column via column header context menu added in #23. However, there is no way to restore a hidden column.

Add Columns dropdown to the custom table view toolbar with checkboxes to toggle columns visibility.

See this removed column header UI commit and attempt to do it in the column header conext menu: 7db9dcf

Add basic project info to docs

Provide links to other data table extensions without async data reads, etc. and why we want to create new extension for loading and viewing large data files, i.e CSVs and arrow data files that are over 1Gbytes in size on disk to start with.

Add Save as CSV or JSON data UI options

Use Tabulator Download API for this feature implementation:

http://tabulator.info/docs/5.0/download

and Data Preview Save UI setup: https://github.com/RandomFractals/vscode-data-preview/blob/master/web/data.view.html#L26

 <select id="save-file-type-selector" 
          title="Save Data View .config or Data" 
          onChange="saveData()">
          <option value="">πŸ“₯&nbsp;Save</option>
          <option value=".config">βš™οΈ &nbsp;config</option>
          <option value=".arrow">β‹™ arrow</option>
          <option value=".csv">,,, csv</option>
          <option value=".json">{} &nbsp;json</option>
...

with the saveData JS: https://github.com/RandomFractals/vscode-data-preview/blob/master/web/scripts/data.view.js#L402

function saveData() {
	const dataFileType = saveFileTypeSelector.value;
	switch (dataFileType) {
...

and saveData TS from Data Preview extension: https://github.com/RandomFractals/vscode-data-preview/blob/master/src/data.preview.ts#L712

Add Save as TSV data UI option

See #27 for example of a similar custom CSV data save UI setup.

Currently, clicking on the Tabulator table and using CTRL + C actually copies TSV data to the vscode IDE clipboard.

Might as well add save as TSV data option.

Resolve webview UI toolkit bundling and loading

Looks like webview UI toolkit added in #8 is not being properly bundled and loaded in this extension for the table view toolbar UI controls setup:

image

It does work in tabular data viewer debug (see reload, and goto last and first table row buttons in table view toolbar):

image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.