Giter VIP home page Giter VIP logo

dremio-kdb-plugin's Introduction

Dremio kdb+ connector

Build Status

Intro

This is a community maintained connector to enable Dremio to talk to a kdb+ timeseries database. The connector translates ANSII SQL into q-sql on the fly and submits to the kdb+ rpc mechansim. This connector is not an official Dremio product and support and improvements is the responsibility of the community

Status

This is very beta software and is only able to handle a subset of q-sql. The Dremio engine will return an error when it can't properly parse a SQL statement so there should be minimal amount of crashes. However it has not extensively been tested.

Features

The engine can currently do the following:

  • all data types, including symbols and arrays
  • basic project, aggregate and filter
  • a subset of functions for date manipulation and mathematical functions
  • basic handling of partitioned on disk data and where clause ordering
  • fast and efficient conversion from the kdb ipc format to arrow buffers

It does not support:

  • some aggregates (most notably avg and stddev)
  • imperfect handling of the differences inherit in symbols and char arrays
  • some complex and nested SQL
  • joins are not pushed down
  • no support for higher level q functionality and timeseries operations

The best way to use this plugin is either to use it to do simple table access on kdb tables, for example incremental raw reflections on tables or to query kdb+ where all the heavy lifting is done inside of q and a result table is made available.

Building and Installation

  1. In root directory with the pom.xml file run mvn clean install
  2. Take the resulting .jar file in the target folder and put it in the \dremio\jars folder in Dremio
  3. Restart Dremio

Contributing

The project is actively looking for maintainers and contributors. Please feel free to dive in and improve this project.

dremio-kdb-plugin's People

Contributors

dependabot[bot] avatar kxconor avatar rymurr avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

mattbaird

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.