Giter VIP home page Giter VIP logo

oxidsql's Introduction

OxidSQL (Toy) SQL Database in Rust

OxidSQL (Toy) SQL Database in Rust

I'm interested in databases and in Rust so i thought why not combine it and learn about databases and Rust by writing a SQL database in Rust. There's a working (although very limited and kinda hacked together) database on the v1 branch which is based on the embedded Sled KV DB as a storage engine.

The goals with v2 are:

  • Write my own storage engine
  • support all CRUD operations (maybe ALTER TABLE too)
  • be multi user and transactional (i'm thinking likely MVCC)
  • write a proper cost based based optimizer with a simple cost model (likely DPccp/DPhyp)
  • make it an actual database server (maybe support postgres wire protocol)
  • actually test it too

I will likely implement create table, insert and select first again and leave the rest for later. All the multi user, transactional and server stuff will likely come last.

Design principles

The main goal is to write a disk based system that can start out with a rather traditional architecture and then be used as a play-/testing-ground for me to implement all kinds of cool new techniques (JIT compilation using cranelift, fancy new buffer managers, ...). Although the system is supposed to be disk-based, we live in a world where RAM in the tens or even hundreds of GBs is ubiquitous. Therefore we can often just assume that stuff fits into main memory. Especially for System- and Metadata we will mostly not bother having the option of writing them to disk other than for persistence. For execution we will for now also assume that we won't have to spool to disk (otherwise we could still rely on OS swapping for this purpose for now).

What currently works:

  • Simple CREATE TABLE
  • Simple INSERT (INSERT INTO x VALUES (a,b,c))
  • Simple select from where including simple predicates with <,>,>=,<=,= and equi-joins in "attribute = attribute" style (currently only "AND" supported)
  • Analysis, planning and optimization (DPccp for now, will do DPhyp later)
  • Execution through a simple single threaded volcano style interpreted engine printing the results to stdout (for now)
  • crashing and burning (no, but it panics) when someone enters a query that results in a cross product

oxidsql's People

Contributors

mzinsmeister avatar

Stargazers

Jonas mg avatar Emil Jaszczuk avatar  avatar tianqi sun avatar Mert Sidal avatar Tim Struthoff avatar Roman Nestertsov avatar Maximilian Berger avatar Pavel Ivanov avatar Federico Carrone avatar Gosha Spark avatar Changsu Jiang avatar zetsukhun avatar Yong avatar Jatin Jindal avatar Graeme McCutcheon avatar Alan P Currie avatar pia avatar Saulius Grigaliunas avatar ZF avatar Agastya Darma avatar Zach avatar Wayan jimmy avatar Manish Gill avatar Ivan Polyakov avatar V. Can Keklik avatar IdiotNe avatar  avatar Jaffrey avatar Ahmed Tadde avatar prest avatar André Heringer avatar Stefan Rummer avatar Jigao Luo avatar Rafael Thalhofer avatar Pavan Keerthi avatar Leon Windheuser avatar Simon Huang avatar Valentin Metz avatar Bennett Clement avatar Zoom.Quiet avatar Andrew Byers avatar Matěj Koubík avatar Kevin Duff avatar Adel Attia avatar ZJPzjp avatar Willi Kappler avatar Jacob Lerche avatar  avatar Chris Kaminski avatar  avatar  avatar Daniel Grittner avatar

Watchers

 avatar  avatar

oxidsql's Issues

Write Planner/Optimizer

There's already a somewhat hacky and untested implementation of DPccp in the src/optimizer directory. Long term DPhyp should be implemented together with somewhat adaptive optimization. For now OxidSQL won't support any non-inner joins so for now DPccp is enough. The Planner should take a representation of the query and transform it to an optimized query plan that can then be executed by the ExecutionEngine.

  • Check/Refactor/Test DPccp implementation
  • Create/use input representation for incoming queries
  • Create/use output representation for executable plan

Integrate SQL Parser + analyzer

V1 already has an SQL parser that could be ported over. Alternatively a new one could be written.

  • Integrate old or new SQL parser
  • If integrating old one, refactor and test it

Write a basic traditional storage manager

For the first version OxidSQL should have a basic traditional disk based storage manager based on a traditional hashtable based buffer manager, slotted pages and b-trees. All of this should be thread safe and at least somewhat scalable. I will also try to introduce as many abstractions as possible to enable swapping of components later on.

Basic ToDos (considered done once i have a somewhat useable version done and tested somewhat well):

  • Buffer manager
  • Disk manager
  • Page replacement strategy
  • Free Space Inventory
  • Slotted Page Segment
  • Iterators for Scans over Slotted-Pages
  • B-Tree Segment
  • Iterators for Scans over B-Trees

Execution Engine

Write a basic volcano style engine. Maybe a push based tuple-at-a-time engine could also be considered. The APIs should ideally allow swapping for other vectorized/compiled/push/pull based engines in the future.

  • Develop API for Execution Engine
  • Create Operators to execute all plans generated by planner/optimizer

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.