Giter VIP home page Giter VIP logo

appbaseio / dejavu Goto Github PK

View Code? Open in Web Editor NEW
8.3K 130.0 508.0 94.26 MB

The Missing Web UI for Elasticsearch: Import, browse and edit data with rich filters and query views, create search UIs visually.

Home Page: https://dejavu.reactivesearch.io

License: MIT License

JavaScript 99.35% HTML 0.47% Dockerfile 0.18%
elasticsearch-plugin chrome-extension dejavu appbaseio elasticsearch database-gui realtime-updates elasticsearch-gui react reactjs

dejavu's Introduction

dejavu: The missing Web UI for Elasticsearch

GitHub License React Version Docker Pulls

  1. Dejavu: Intro
  2. Features
    a. Easily Connect to Indices
    b. Visual Filters
    c. Modern UI Elements
    d. Import JSON or CSV Data
    e. Build search UIs
  3. Comparison
  4. Roadmap
  5. Build Locally / Contributing
  6. Get Dejavu
    a. Docker Installation
    b. Hosted Alternatives

1. Dejavu Intro

dejavu is the missing web UI for Elasticsearch and OpenSearch. Existing web UIs leave much to be desired or are built with server-side page rendering techniques that make it less responsive and bulkier to run (I am looking at you, Kibana).

We started building dejavu with the goal of creating a modern Web UI (no page reloads, infinite scroll, filtered views, realtime updates, search UI builder) for Elasticsearch with 100% client-side rendering so one can easily run it as a hosted app on github pages, or as a docker image.

Starting v1.0, dejavu is the only Elasticsearch web UI that supports importing data via JSON and CSV files, as well as defining field mappings from the GUI.

Starting with v1.5, we support the ability of creating custom headers so you can easily pass different authentication headers, provide enhanced filtering and bulk updating of data via Elasticsearch's Query DSL.

Starting with v2.0, we support the ability to build faceted search UIs to test relevancy. You can also export the generated code to a codesandbox.

Starting with v3.0, we support the ability to connect to multiple indexes. You can also globally search across your indexes using global search bar.

2. Features

Easily Connect and Remember Indices

Connect to an Index

Dejavu allows you to connect to any of the indexes present in your cluster and also caches each connected index locally so they are easily accessible when browsing again.

Visual Filters

Filter Views

Sort through the data, find information visually, hide irrelevant data and make sense of all. With all the native data types we have . Global searchbar allows you to perform text search across your dataset.

Moreover, any filtered view can be exported as a JSON or CSV file.

Modern UI elements

Pagination

It's not uncommon to have thousands of documents in your index. Dejavu supports paginated view which also allows you to change page size.

Dejavu also supports browsing data from multiple indexes and types, updating data either individually or via queries in bulk. Deletions are also supported.

Import JSON or CSV Data

Import JSON or CSV files

Importer view allows importing CSV or JSON data directly into Elasticsearch through a guided data mappings configuration.

Build Search UIs

Build search UIs

With Search Preview, you can now build visual search UIs, test search relevancy and export code to a codesandbox.


3. Comparison with other data browsers

Features dejavu ES-head ES-kopf ES-browser Kibana
Installation Docker image, Hosted app Elasticsearch plugin, static page Elasticsearch plugin, static page Elasticsearch plugin (doesn't work with 2.0+) Elasticsearch plugin
Modern UI React 16.6. jQuery 1.6.1, slightly stodgy Angular 1.x ExtJs, a bit stodgy Node.JS, Hapi, Jade
Browser features CRUD, data filters Read data, full-text search Data view for a single type Read view, visualizations, charting
Data import/export ✔️ JSON, CSV Only export, no CSV
Search preview Visually build and test search UI
License MIT Apache 2.0 MIT Apache 2.0 Apache 2.0

4. Roadmap

Here's a rough roadmap of things to come in the version 1.0.0 release.

🎆 We just hit the 1.0.0 roadmap:

  • Battle-testing with different datasets
  • Feature support for advanced filtering Offline detection and reconnection for realtime updates
  • Performance improvements while scrolling
  • Support for importing and exporting data
  • Support for a continuous query view
  • Available as a docker image

🍾 We just hit the 2.0.0 release:

  • An intuitive data editing experience in tabular mode (v/s JSON edit mode)
  • View data types from within the data browser view
  • A more streamlined import process
  • Refactor codebase to improve hackability (Migrate to React 16+, ES6 syntax)
  • Ability to build (and test) search visually

✨ We just hit the 3.0.0 release:

  • Rewrite dejavu browser for high performance when browsing large datasets
  • Add support for browsing multiple indexes
  • Powerful filtering of data with field level facet based filters and a global search
  • Built on React 16.6 and future compatible with React 17
  • A more intuitive data editing experience (in addition to the raw JSON, we now show a relevant UI field with validations)

🔜 Roadmap beyond 3.0.0:

  • Improve test coverage
  • Mobile friendly viewing and editing experience
  • Additional filtering support (add range filters, date filters)

5. Build Locally

See the contributing guidelines.


6. Get Dejavu

Docker Installation

docker run -p 1358:1358 -d appbaseio/dejavu
open http://localhost:1358/

You can also run a specific version of dejavu by specifying a tag. For example, version 3.6.0 can be used by specifying the docker run -p 1358:1358 appbaseio/dejavu:3.6.0 command.

Cross-origin resource sharing (CORS)

To make sure you enable CORS settings for your Elasticsearch instance, add the following lines in the elasticsearch.yml configuration file.

http.port: 9200
http.cors.allow-origin: 'http://localhost:1358'
http.cors.enabled: true
http.cors.allow-headers: X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
http.cors.allow-credentials: true

If you are running your Elasticsearch with docker-compose, you can refer to the example reference here.

If you are running your Elasticsearch with docker, you can use the following flags to pass the custom CORS configuration:

OpenSearch 1.x
docker run --name opensearch --rm -d -p 9200:9200 -e http.port=9200 -e discovery.type=single-node -e http.max_content_length=10MB -e http.cors.enabled=true -e http.cors.allow-origin=\* -e http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization -e http.cors.allow-credentials=true -e plugins.security.disabled=true opensearchproject/opensearch:1.2.4

You can run both Opensearch and Dejavu together with:

docker-compose up -d

Elasticsearch 8.x
docker run -d --rm --name elasticsearch -p 127.0.0.1:9200:9200 -e http.port=9200 -e discovery.type=single-node -e http.max_content_length=10MB -e http.cors.enabled=true -e http.cors.allow-origin=\* -e http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization -e http.cors.allow-credentials=true -e network.publish_host=localhost -e xpack.security.enabled=false docker.elastic.co/elasticsearch/elasticsearch:8.1.0

You can run both Elasticsearch 8.1.0 and Dejavu together with:

docker-compose -f docker-compose-v8.yml up -d

Elasticsearch 7.x
docker run -d --rm --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "http.cors.enabled=true" -e "http.cors.allow-origin=*" -e "http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization" -e "http.cors.allow-credentials=true" docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2

You can run both Elasticsearch 7.10.2 and Dejavu together with:

docker-compose -f docker-compose-v7.yml up -d

Hosted Alternatives

dejavu can also be run as a hosted app at https://dejavu.appbase.io.


dejavu's People

Contributors

anik-ghosh-au7 avatar bietkul avatar dependabot[bot] avatar divyanshu013 avatar hasnat avatar heinrichreimer avatar jyash97 avatar l0l3r avatar lakhansamani avatar mohdashraf010897 avatar mrexodia avatar savvyshah avatar shahanuj2610 avatar siddharthlatest avatar tigran10 avatar venttola avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dejavu's Issues

Error on case-sensitive platforms

Hi,

I tried to deploy this application locally on a Linux environment, and it failed to launch with an error:

[09:13:26] Using gulpfile /tmp/dejaVu/gulpfile.js
[09:13:26] Starting 'browserify'...
[09:13:26] Starting 'connect'...
[09:13:26] Finished 'connect' after 6.67 ms
[09:13:26] Server started http://localhost:1358
[09:13:26] LiveReload started on port 35729
events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: Cannot find module './homePage.jsx' from '/tmp/dejaVu/_site/src/js'
    at /tmp/dejaVu/node_modules/resolve/lib/async.js:55:21
    at load (/tmp/dejaVu/node_modules/resolve/lib/async.js:69:43)
    at onex (/tmp/dejaVu/node_modules/resolve/lib/async.js:92:31)
    at /tmp/dejaVu/node_modules/resolve/lib/async.js:22:47
    at FSReqWrap.oncomplete (fs.js:82:15)

Checking the src/js folder, I can see that it contains a file "HomePage.jsx"

For some preliminary checks so that it at least starts up - I renamed (locally):

Old Name New Name
features/FeatureComponent.jsx features/featureComponent.jsx
HomePage.jsx homePage.jsx
table/fieldCheckbox.jsx table/FieldCheckbox.jsx
TypeTable.jsx typeTable.jsx

However, I'd recommend a more permanent fix of standardizing the filenames to be a specific case, and modify either the code referencing it, or the file name to suit whichever this standard is.

Can 'input_state' potentially save an actual query as well as the client, index and types contexts?

Thanks for this project and its sister mirage. Both serve an important gap in the ES GUI space.

I've been playing around with a local installation against my data and trying to get a better feel for how 'input_state' is working, in terms of it somehow encoding a 'state' ... ie. client-connection info, index selection and perhaps type selection. I don't think it's base64 encoded ... but I can see you are using cryptojs in some way.

Are there any spec documents as to how this is working ... or perhaps an example of an encoded/decoded payload.

I was also wondering if it's possible to include a query that gets applied to the given 'state', and pass that as part of the url ... in either encoded or plain JSON form?

Thanks.

Autofill app (aka index) URLs

Saving an app's URL connection configuration would be nice to implement an autofill functionality based on the app name. Autofilling connection configurations would help users multitask b/w different apps easily.

Autofill would work based on the appname field. To add some spice, it can also have an autocomplete suggestion functionality.

Note: For security reasons, this should only be implemented in the ES plugin and chrome app (and not on the gh-pages URL) branches.

* in CORS is failing

I have an ES dev server with Access-Control-Allow-Origin set to * (and does not have any auth credentials). Dejavu refuses to connect - I see the following in browser console log

XMLHttpRequest cannot load http://example.com:9200/foobar/_search?search_type=query_then_fetch. Response to preflight request doesn't pass access control check: A wildcard '*' cannot be used in the 'Access-Control-Allow-Origin' header when the credentials flag is true. Origin 'http://opensource.appbase.io' is therefore not allowed access. The credentials mode of an XMLHttpRequest is controlled by the withCredentials attribute.

Looks dejavu should recognize that there no credentials supplied and withCredentials flag accordingly.

I'm using the hosted version of dejavu: http://opensource.appbase.io/dejavu/live on Chrome 53.

support for column resizing

Support column width resizing to better view data.

  • Long ids are hard to view as the current UI truncates values longer than 20 characters, see #11.
  • Columns with smaller fields can be compacted and columns with long text data can be expanded to better utilize the screen estate.

The ideal solution should be similar to excel / google sheet's resizing behavior, contributions are welcome :-).

Filter option does not appear for strings and keywords fields

I use elasticsearch 5.0.0 with this index:

curl -XPUT http://localhost:9200/cnaes -d '
{
 "mappings" : {
  "_default_" : {
   "properties" : {
    "code" :          {"type": "keyword" },
    "descripcion" :   {"type": "string" },
    "value":          {"type": "integer"}
   }
  }
 }
}
';

When I access to Dejavu and inspect index, filter option does not appear for strings and keywords fields.

dejavu_screenshot

Any help?

plaintext password in UI

Hello, in the UI, the password is entred in plaintext as part of the connection url (http(s)://user:pass@server:port).
After connection, le url - with the plaintext password - is still displayed.
=> Anyone who looks at my screen can see my credentials.

Could you update your UI to hide the password please ?

Single Tool for ElasticSearch

Thanks for the great product. I love all the three products such as GEM, Mirage and Dejavu.

I am heavily using them for my local testing. I am facing the problem that I have to frequently switch between tabs. It is really disturbing. Atleast Mirage and Dejavu can be single app so developer can visualise and then test different query from same page.

feature: optionally expand the width of the ID column

If your type + id strings are longer than about 20 characters, it gets truncated in the id column (which is expected, to an extent). It would be nice if there was a way to expand the width of the ID column to make a larger portion of IDs visible. This would make it a lot easier to browse data; which is pretty difficult right now.

Additionally, the {...} show json button takes up a lot of screen real estate.

"No data to view" shown while loading

A "No data to view!" message is shown in the data browser even if there's data and it's loading; the message goes away after data is ready. Perhaps something like "Loading data..." would be better.

Deja Vu does not work without port o the url

Google Chrome Web console:

/_plugin/dejaVu/src/js/helper/feed.js
---------------------------------------------
        filterUrl: function(url) {
            if (url) {
                var obj = {
                    USERNAME: 'test',
                    PASSWORD: 'test',
                    URL: url
                };
                var urlsplit = url.split(':');
                --> var pwsplit = urlsplit[2].split('@');

Exception:

Uncaught TypeError: Cannot read property 'split' of undefined

Adding a button "Select all / Unselect all"

Hi,

Thank you so much for your Chrome extensions. Really helpful !

Can you add a button to (un)select all the results, please ?

My use case is to select all the result in order to delete them through your extension !

Thanks,
Rasmey

Refactor code

Refactor the code, sync with other branches should be smooth.

dejavu link change

dejavu URL links now should be using a lowercase 'v' in share links to https://appbaseio.github.io/dejavu because of the repository rename from dejaVu to dejavu.

We need to change this in all the relevant branches and create a new release.

Code Refactor and Improvements

We have divergent branches that require some major code changes for every release cycle. It would be nice if we can separate code concerns that are specific to a branch (for example, in the chrome app - we have to use an async storage API instead of localstorage; we can move the storage code and it's dependencies in a single file).

We can also create an automated merge script to simplify the release creation process.

dejavu as a kibana plugin

With Elasticsearch 5.0 being GA, the site plugins have been removed. Instead, Elastic encourages Kibana plugins.

While dejavu can be still usable with the github hosted app and chrome app (which is going away next year), we should consider supporting dejavu as a Kibana plugin.

site directory rename ?

Elasticsearch 2.3.3 fails to start with the dejaVu plugin:

[2016-06-14 09:44:41,077][ERROR][bootstrap ] Exception
java.lang.IllegalArgumentException: Plugin [dejaVu] is a site plugin but has no '_site/' directory

dejaVu currently has a site directory. Changing this to _site lets elasticsearch start

Receiving support?

Hi there! This isn't a issue but I didn't know where to ask!
Are you accepting donations or supports? Because may you get some extra support for dejaVu.
I don't trying to spam you, but I want to invite you to beerpay.io, a simple platform to monetize open source projects, like hamburgers.
I know that the open source is free, but it's okay if someone want to contribute in other way.

Thanks for this work man!

plugin installation failed - plugin-descriptor.properties missing in release tarball

Installing the 0.7.1 release tarball as a plugin fails with

[2016-06-14 09:40:28,120][ERROR][bootstrap ] Exception
java.lang.IllegalStateException: Could not load plugin descriptor for existing plugin [dejaVu]. Was the plugin built before 2.0?

Installing plugin-descriptor.properties to .../elasticsearch/plugins/dejaVu fixes this.

Please add plugin-descriptor.properties to the release tarball on github.

Update utility doesn't work

Update data window is not working anymore, I used this before and it worked properly.
Now 'Update' button from Update Data window doesn't do anything

can't get it to work/connect

  • elastic search is running and I can browse data using sense.
  • installed dejavu as a chrome extension
  • no instruction as to what I should enter for appname or host, tried appname as anything and host as 'localhost' and 'localhost:9200', all of which result in dejavu blocking any user input and continual blue loader at top
  • tried host as 'http://localhost:9200' and 'http://127.0.0.1:9200' which also don't work, but at least get an 'authentication' error
  • tried with above hosts and using my index name for app name, still the same authentication error.

Looks like a great tool, just what I've been looking for, but needs a getting started/how to use section in the read me.

Keeping state within dejaVu URL

One of the key ideas for dejaVu is that it's a client side app and can be used within a browser or as an Elasticsearch plugin.

The idea is to capture dejaVu's state within it's URL: Things like ES host, index name, selected types, applied filters. That way, passing the state is as easy as sharing the dejaVu URL. Potential benefits are:

  1. An easier way to debug while developing,
  2. Sharing public data sets in Elasticsearch would be super easy,

Of course, localhost URLs won't be accessible and there's a potential pitfall security concern wise (but if you were securing ES by obscurity, you have bigger problems).

limit which indexes can be edited manually

It would be great if there was a config option to make everything readonly, unless an index/type was whitelisted as write enabled. This doesn't exist already does it?

Select and copy a record content

Using ElasticSearch plugin via Chrome app I cannot select the content of a record. When I click on the brackets icon {...} I can see the content but I cannot select and copy it

inconsistent install behavior

I've got weird behavior installing dejavu as an ES plugin, and running it stand-alone. This is all using chrome.

as a plugin:

  1. works OK locally
  2. 404s on main.min.js in another environment when loading splash page

built + run locally (npm install, bower install, gulp watch):

  1. splash page loads. bombs out on something about chrome localstorage when I try to connect. chrome.storage is undefined (in chrome). "url" == e.name ? chrome.storage.local.set({

Am I missing something?

Connection Issue Reporting

Is there a way to know what's happening with a connection? I'm trying to connect to our instance and get no feedback on what the connection issue is.

Find a Chrome App alternative

Chrome Apps will be deprecated in 2017 and disabled shortly after on all platforms except ChromeOS. In this light, we should publish a native version of dejaVu. A Mac / PC app can certainly be a starting point.

websocket connection failure

dejaVu installed as an ES plugin.

WebSocket connection to 'ws://xxxx:9200/meta' failed: Error during WebSocket handshake: Unexpected response code: 200

Is there an extra plugin dependency for the live-reload to work? Or is this a feature only supported within appbase?

Chrome extension installation steps

I tried installing the chrome extension and connects to localhost:9200. It kept loading.

Finally I got it working by adding the following to config/elasticsearch.yml:

 http.port: 9200
 http.cors.allow-origin: "chrome-extension://jopjeaiilkcibeohjdmejhoifenbnmlh"
 http.cors.enabled: true
 http.cors.allow-headers: "X-Requested-With,X-Auth-Token,Content-Type, Content-Length, Authorization"
 http.cors.allow-credentials: true

I think there should be error messages that is more informative, and the need for configuring allow-origin to chrome-extension should be addressed in README.

could not run the plugin

I had installed dejaVu successful. And config as below:
`http.port: 9200

http.cors.allow-origin: "http://127.0.0.1:9200"

http.cors.enabled: true

http.cors.allow-headers : "X-Requested-With,X-Auth-Token,Content-Type, Content-Length, Authorization"

http.cors.allow-credentials: true`

but when I open http://localhost:9200/_plugin/dejaVu/, I got error:
`feed.js:19 Uncaught TypeError: Cannot read property 'split' of null(anonymous function) @ feed.js:19

main.min.js:12 Download the React DevTools for a better development experience: https://fb.me/react-devtools

main.min.js:2 Uncaught TypeError: Cannot read property 'length' of undefined`

I am using ES 2.3.2

dejaVu is now used by 1,000+ developers

Since we announced dejavu's open-source launch on Product Hunt in March, it has found home with 1,000+ developers. I would like to thank everyone here for giving the project an early visibility 😃 . It means a great deal for a side project being built by a small team!

Id selection ambiguity

In the data view, when selecting a row with a specific id, other rows that have the same id but different types also get selected. That shouldn't be the case.

Chrome extension: clicks don't work on certain elements

For instance, the JSON data is not selectable. And the "watch video" and "Create your own ElasticSearch in cloud with appbase.io" hyperlinks don't open in a browser.

Both could be separate issues, nonetheless, something we should look into.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.