Giter VIP home page Giter VIP logo

skytable / skytable Goto Github PK

View Code? Open in Web Editor NEW
2.3K 2.3K 81.0 5.92 MB

Skytable is a modern scalable NoSQL database with BlueQL, designed for performance, scalability and flexibility. Skytable gives you spaces, models, data types, complex collections and more to build powerful experiences

Home Page: https://skytable.io

License: GNU Affero General Public License v3.0

Rust 99.61% Shell 0.28% Dockerfile 0.03% Makefile 0.02% PowerShell 0.06%
beginner-friendly column-store contributions-welcome database database-engine database-server databases dbms distributed-database document-database json key-value-store multi-model nosql nosql-database rust skybase skytable sql terrabasedb

skytable's Introduction

Skytable Skytable Logo

A modern NoSQL database, powered by BlueQL.

GitHub release (with filter) GitHub Workflow Status (with event) Discord Docs Static Badge

What is Skytable?

Skytable is a modern NoSQL database that focuses on performance, flexibility and scalability. Skytable is a primarily in-memory, wide-column based database with support for additional data models* that uses its own storage engine (structured like n-ary records with optimized delayed-durability transactions) and enables querying with its own query language BlueQL that builds on top of SQL to improve security and flexibility.

Skytable is best-suited for applications that need to store large-scale data, need high-performance and low latencies.

You can read more about Skytable's architecture, including information on the clustering and HA implementation that we're currently working on, and limitations on this page.

Features

  • Spaces, models and more: For flexible data definition
  • Powerful querying with BlueQL: A modern query language based on SQL
  • Rich data modeling: Use models to define data with complex types, collections and more
  • Performant: Heavily multithreaded, optimized write batching and more
  • Secure: BlueQL is designed to strongly deter query injection pathways
  • Enforces best practices: If you're building with Skytable today, the practices you'll learn here will let you easily take on the job of building performant systems, even outside Skytable

Learn more about Skytable's features here.

Getting started

  1. Set up Skytable on your machine: You'll need to download a bundled release file from the releases page. Unzip the files and you're ready to go.
  2. Start the database server: ./skyd --auth-root-password <password> with your choice of a password for the root account. The root account is just like a root account on Unix based systems that has control over everything.
  3. Start the interactive client REPL: ./skysh and then enter your password.
  4. You're ready to run queries!

For a more detailed guide on installation and deployment, follow the guide here.

Using Skytable

Skytable has SPACEs instead of DATABASEs due to signficant operational differences (and because SPACEs store a lot more than tabular data).

With the REPL started, follow this guide:

  1. Create a space and switch to it:
    CREATE SPACE myspace
    USE myspace
  2. Create a model:
    CREATE MODEL myspace.mymodel(username: string, password: string, notes: list { type: string })
    The rough representation for this in Rust would be:
    pub struct MyModel {
     username: String,
     password: String,
     notes: Vec<String>,
    }
  3. INSERT some data:
    INSERT INTO mymodel('sayan', 'pass123', [])
  4. UPDATE some data:
    UPDATE mymodel SET notes += "my first note" WHERE username = 'sayan'
  5. SELECT some data
    SELECT * FROM mymodel WHERE username = 'sayan'
  6. Poke around! And then make sure you read the documentation learn BlueQL.

For a complete guide on Skytable, it's architecture, BlueQL, queries and more we strongly recommend you to read the documentation here.

While you're seeing strings and other values being used here, this is so because the REPL client smartly parameterizes queries behind the scenes. BlueQL has mandatory parameterization. (See below to see how the Rust client handles this)

Find a client driver

You need a client driver to use Skytable in your programs. Officially, we maintain a regularly updated Rust client driver which is liberally license under the Apache-2.0 license so that you can use it anywhere.

Using the Rust client driver, it's very straightforward to run queries thanks to Rust's powerful type system and macros:

use skytable::{Config, query};

fn main() {
    let mut db = Config::new_default("username", "password").connect().unwrap();
    let query = query!("select username, password from myspace.mymodel where username = ?", "sayan");
    let (username, password): (String, Vec<u8>) = db.query_parse(&query).unwrap();
    // do something with it!
}

You can find more information on client drivers on this page. If you want to help write a client driver for your language of choice, we're here to support your work. Please reach out to: [email protected] or leave a message on our Discord server!

Getting help

We exclusively use Discord for most real-time communications โ€” you can chat with developers, maintainers, and our amazing users! Outside that, we recommend that you use our GitHub Discussions page for any questions or open a new issue if you think you've found a bug.

We're here to help!

Contributing

Please read the contributing guide here.

Acknowledgements

Please read the acknowledgements document.

License

Skytable is distributed under the AGPL-3.0 License. You may not use Skytable's logo for other projects.

skytable's People

Contributors

aaron1011 avatar dependabot[bot] avatar glydr avatar jse1t5 avatar no3371 avatar ohsayan avatar sanjayts avatar tanzkalmar35 avatar zhangyuannie avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

skytable's Issues

Feature: Strong Actions

Description
There are cases when you want to run an action, only if all the other operations succeed. For an example, say you want to delete three keys, only if all three of them exist. For these scenarios, we can have strong actions. Do note that these aren't the same as ACID transactions. To maintain consistency with our action naming scheme, we will have these three variants:

  • SSET
  • SDEL
  • SUPDATE
  • Tests

We'll look into improving the performance of these actions once we have an implementation ready

[ci] Workaround for caching errors on macOS runners

After months of bearing with slow builds on macOS, I'm fed up -- like really fed up. We've had extensive discussions here, here and across several other threads.

The problem
macOS using BSD tar causes problems with caching (as used by actions/cache). For some reason (yeah, a concrete explanation is yet to be found) BSD's tar is corrupting a part of the cache (especially the serde-*.dylib file) which causes a 'crate not found error' for serde.

The solution
Thanks to this post, the workaround for using GNU Tar seems promising. We'll be attempting to do this.

Feature: Support for multiple namespaces

Description
We should support multiple namespaces like most of the other databases out there. This will enable the user to have multiple tables or keyspaces for storing data. However, we still have some unresolved questions.

Unresolved questions

  • Q1: Differentiation between keyspaces and tables
    Since we have to support keyspaces and tables, we'll have to make sure what exactly we mean by a keyspace. As we'll be moving into a distributed architecture in the coming year, we'll have to keep ideas like 'keyspace-per-node' in mind.
  • Q2: Types of keyspaces
    Replication strategy? Sharding?

Implementing this from a 'single node' view is what we'll try and achieve initially, but we'll need to move that into a 'distributed view' when we finalize our distributed architecture.

Update: Please refer to this comment

Tracking issue: v0.7.0

With 0.7.0 being an extremely large release, we need an issue to track what we have left to work on:

  • Multiple keyspaces and tables (see #189)
  • Volatile tables (see #190)
  • Binary data type (see #157)
  • Tests
  • Migration tool (tracked in #200)
  • #197
  • #194
  • Table-based accesses for actions (already implemented for lskeys, flushdb and dbsize) This will be done in a later release
  • Rename actions (done: POP/MPOP)
    If you have any more questions to raise, please leave a comment here.

Server: Persistence is broken since tag v0.5.2

Description of bug
Persistence is broken since the last tag. A more concrete reproduction is yet to be produced.

Steps to reproduce
Steps to reproduce the behavior:

  1. Run skyd
  2. Launch skysh and run SET somekey somevalue
  3. Close skysh and then close skyd
  4. Launch skyd
  5. Launch skysh and run GET somekey
  6. You'll get a (Nil) printed on the screen

Expected behavior
"somevalue" should have been printed

Meta

  • Release tag: v0.5.2
  • Branch: NA
  • Commit ID: NA
  • Operating system: Indepedent

Additional context
None

Tasks

  • Backport fix
  • Tag fix into release channel
  • Merge fix into next

Feature: Support multiple keys in actions

Description
Support multiple keys in actions.

Preliminary tasks

  • Multiple SET
  • Multiple GET
  • Multiple EXIST
  • Multiple DEL
  • Multiple UPDATE
  • Upgrade existing interfaces
  • Test

Optimization

  • Refactor into kvengine
  • DB ops should return primitives
  • Use the ExceptFor type

Final tasks

  • Implement Display for ExceptFor
  • Benchmark

Also, from now on, we should be supporting multiple keys in the actions, wherever applicable.

The terrapipe impl is not sufficient

The merge of #2 has, unfortunately, not provided the kind of implementation that we need for reading an incoming stream (TCP).
We need individual methods for parsing the meta frame and the data frame. These changes however will not be worked on in a separate branch, but instead the required changes will be pushed to the server-0.1.0 branch

Roadmap: 2020

This issue tracks how far we are from our roadmap, published here. The initials goals set out in the post linked above, were quite indistinct โ€” which is why this issue exists to track our progress.

Data structures ๐Ÿ—„๏ธ:
Atomic Counters*

Actions ๐ŸŽฌ:

  • Strong actions (#23)
  • DBSIZE
  • FLUSHDB
  • USET: UPDATE or SET
  • KEYLEN

Reliability โ›ฑ๏ธ:

  • BGSAVE (#21)
  • Snapshots (#25)

Security ๐Ÿ”’:
SSL*

This roadmap is likely to change โ€” and we'll most likely add more features to the list. If you've got ideas: create a separate issue describing it and we'll be right with you!
*see this

Security: Merge v0.5-hotfix.1 into next

Introduction
As we notified via this tweet, a security vulnerability has been identified in v0.5.0 and a corresponding patch has been released.

Patch link: This patch can be downloaded by users for their platform from this link: https://dl.skybasedb.com/v0.5-hotfix.1/.

Affected versions
This vulnerability only affects v0.5.0 of the database server and all such deployments are immediately requested to deploy this hotfix.

Pending actions
We will be releasing a full length security and versioning policy soon. At the same time, we'll be releasing the security advisory and disclose the bug on the embargo date of 17 Feb 2020, 0630 UTC.

This issue exists to track the status of the successful merging of this hotfix into the primary branch.

Tracking issue for implementation of the Terrapipe protocol

This is the tracking issue for the implementation of the Terrapipe protocol.
Tasks:

  • Basic parsing of a TP Query
  • Basic generation of a TP Result
  • Basic generation of a TP Query
  • Formal implementation of a TP Query
  • Formal implementation of a TP Result
  • Testing successful

Implement native file locking mechanisms

As noted in #123 and attempted in #124 -- it is a good "practice of compliance" to lock the data.bin (or the data file) when the database server is running to avoid other processes from overwriting it. We tried to implement it using POSIX based advisory locks, but instead we'll be doing native implementations for Windows and Unix-based systems.

  • For Windows, implement exclusive locks using the fileapi
  • For Unix-based systems, use BSD-style flock and LOCK_EX as the operation

Limitations

The use of BSD-style flocks is only supported in Linux kernel versions >= 2.0 but I'm sure that isn't a big issue per se -- so we can ignore this limitation

Tasks:

  • Basic implementation
  • Tests
  • Document
  • Enable locking

Server: Connection reset on client disconnection

Description of bug

Whenever any client program/driver disconnects from the database server, a connection reset message is shown which indicates that the connection wasn't terminated correctly.

Steps to reproduce
Steps to reproduce the behavior:

  1. Run tdb
  2. Run tsh
  3. Close tsh with CTRLC

Expected behavior

The connection should've terminated successfully when a client disconnected

Meta

  • Release tag: v0.3.0
  • Branch: next
  • Commit ID: None
  • Operating system: Ubuntu 20.04

Additional context

None

Server: The server fails to flush the in-memory table if the data file is locked by another process

Description of bug
When the server starts, it only reads from the data.bin file to restore the previously stored data. In the current implementation, if another process locks the file (for some reason, not something predictable) while the server is running, the server quits without flushing the data - with an error message.

There are two solutions to this:

  1. Lock the file (record-level) OR
  2. Wait for the other process to release the lock

For (1): This is fairly easy to implement and is better than (2) since we already have a lock on the file. Also, in the event that a quick shutdown is needed, this would be the most reliable option.
For (2): This is the easiest thing to implement - but there are doubts about how reliable this can be. The lock on the file may never be released and we may never be able to write to it. Also, if a quick shutdown is required, this solution is not favorable at all.

Steps to reproduce
NA

Expected behavior
NA

Meta

  • Release tag: None
  • Branch: None
  • Commit ID: None
  • Operating system: None

Additional context
None

Feature: Multiple keyspaces and tables

Background

Almost every database provides the ability to create multiple tables and multiple databases. Of course, in the NoSQL world, things will have different physical meanings.

Definitions

  • A keyspace determines the replication strategy along with several other properties unique to a keyspace.
  • A keyspace is the SQL equivalent of a database
  • A keyspace holds multiple tables
  • A keyspace has multiple properties itself, besides holding the actual data

Implementation

  • In-memory table implementation
  • A new disk storage format
    • More the keyspaces, more the data. Simple serialization is not going to work
    • Our in-house format Snapstore is definitely the way ahead and this time with a threaded implementation
  • Make a dummy struct and safely swap the keyspace in real-time
  • Change the way actions are parsed
  • Add table types (for now this will just be a keyspace type that supports the storage of key-value pairs)
  • Implement table types (add the binary type #157)
  • Upgrade all interfaces to use multiple-keyspaces
  • Add the keyspace substitute header magic to enable swapping keyspaces in real-time
  • Add tests for the entire implementation

Questions

  • โœ… Altering keyspaces in real time: is that feasible?
    Outcome: Yes, we can achieve this

  • ๐ŸŸ  Writing our own hashtable implementation? We wrote our own hashtable some time back, but due to performance gaps in iterators, we dropped its use. We currently use a modified version of std's hashtable โ€” but is it time that we move away for good? Drawbacks of std's hashtable:

    • Cannot be frozen in a performance friendly way (since RwLocking an entire table and all records is hazardous to performance, we don't ever make its use). We require the ability to explicitly freeze a table and all records in a thread-friendly way, that is not starving readers/writers

    Outcome: This will be worked on later

Native support for Apple Silicon (arm64-v8)

This issue is an 'umbrella' issue that is used to track the state of Skytable's native support for Apple Silicon aka the Apple M1 (arm64-v8).

  • T2 Support (native support via rustup; see rust-lang/rust#73908)
  • Build passes
  • OpenSSL builds with static linking
  • Test suite passes
  • CI Runner Developed/Ported
    • Secure environment (admins only) [refer to #130]
    • Registry cache
    • Artifact cache
    • Invalidate unused dependencies
  • Automated release of bundles
    - [ ] T1 Support (rustc)

This issue will be updated regularly to track the current level of support

Terrabasedb as a graph database

A multi model fast and resource efficient database written in Rust is a great idea.

The world is moving towards graph databases,I particularly like Terminusdb https://github.com/terminusdb/terminusdb-server specially their data modeling and querying language WOQL https://terminusdb.com/blog/2020/08/07/terminusdb-graph-basics/ but currently Terminusdb is not suitable for web/mobile applications their focus is more on version control and data science.

Is there any plan to support graph queries with a query language like WOQL.

Build: Enable build automation

Our build system (CI) currently has several flaws which either causes the CI to run when it doesn't need to, preventing the runners from doing jobs which actually need to be run.

Tasks:

  • Skip builds iff markdown files are modified
  • Automatically build docs if the actions doc is modified
  • Fix CI failures on macOS (regression from #20)

Server: Failed to flush data to disk on Windows upon termination

Description of bug
Whenever the skyd.exe process is terminated by pressing Ctrl+C it will fail to flush the data to disk:
bild

It seems like if you close the process after this and try to start skyd.exe again it will fail like this:
bild

Steps to reproduce
Steps to reproduce the behavior:

  1. Run skyd.exe
  2. Press Ctrl+C
  3. Run skyd.exe again

Expected behavior
Flushing the data to disk correctly and being able to start skyd again.

Meta

  • Release tag:
    None
  • Branch:
  • Commit ID:
  • Operating system: Windows

Additional context
N/A

Feature: Volatile tables

Description
It would be really useful to have tables that act as a volatile cache, meaning that they are not persistent and will not be written to disk.

Example:

  1. Start skyd
  2. Add a volatile table (Option flag? Separate function from regular Create table?)
  3. Set/update data in the volatile table. Use it as you wish (kind of like a cache).
  4. Shut down skyd
  5. Start skyd
  6. Table should be empty/gone(?) (I think empty makes sense...)

Add advisory locking for POSIX based systems

The most important file for us during the lifecycle of the database server is the data.bin file that holds the serialized equivalent of the data in the in-memory table. Whenever BGSAVE kicks in, it writes to this file and then on the next run, it writes again and this goes on until the daemon is killed. However, we aren't doing something that we're supposed to โžก๏ธ setting a lock on the file.

This issue aims to have this implemented for Unix based systems and not Windows systems just yet (this will be achieved in a future PR/commit). We'll be using locking as per the advisory record locking scheme in POSIX.

Tasks:

  • Basic implementation
  • Document
  • Tests

Related discussions: #14

Tracking issue for stabilization of the Terrapipe Protocol

Description
If you've been closely following development, you'd have noticed that we have made three iterations of the protocol, with two revisions this month alone. Not only has this led to confusion but it is also preventing us from implementing new features. This issue tracks how far we are from stabilizing Terrapipe.

  • Rewrite docs (skytable/docs#2)
  • Rewrite the parser
  • Remove libtdb deps from server
  • Clarify response types (Okay, Nil, Response Codes, ...)
  • Implement Writable traits. (We had already discussed this)
  • Upgrade the k/v engine (#13)
  • Write tests across all modules (#16) Left to #16
  • Create new Query/Response builders
  • New Display impl for CLI
  • Stabilize

Unsolved problems:

  • Feasibility of the Except type
    Solved: The except type can get very problematic when handling large number of keys, we will just return the number of keys set for mset
  • Have a dedicated MSET/MGET action? Solved: we now have MGET and MSET actions

Feature: Automated background saving of data

Description
Reliability is one of the most important things for databases - this issue tracks the implementation of BGSAVE, short for 'background save', which enables the database server to automatically flush stored data onto disk without user intervention.
Tasks:

  • Preliminary implementation
  • Test
  • Verify performance implications
  • Enable custom BGSAVE durations
  • Enable BGSAVE to be disabled

Feature: Add `lskeys` to list all keys

Description
This issue aims to track the implementation of a lskeys action that will enable users to inspect the keys present in the database. By default, we'll limit lskeys to return 10 keys while the user can pass another argument to indicate how many keys they'd like to see. Do note that a single fetch will be atomic, i.e it will give a window into the keys present in the underlying bucket vector at that moment, but the keys can change after that query. We should add this as a note.

Steps:

  • Basic implementation
  • Actiondoc
  • Add tests

Testing: Tests on Windows fail spuriously

Description
Tests on Windows have, are and maybe will fail without any reason. On all the other platforms, tests run without any issues - but it's just Windows. Since I don't have a Windows machine with me, it's going to be quite a problem to reproduce this locally. But from what I can see on the CI logs, the background server is not being terminated. We should maybe have an Abortable Future to stop the server?

Build: Optimize CI builds

We can optimize our CI builds with:

  • Caching
  • Reduced build steps (#40)

These are necessary to reduce testing time and resource usage at the same time.

Feature: musl support

Description

Kindly consider supporting musl libc as it will prove benefitial for lots of embedded projects out there.

Testing: Add tests across all modules

Description
With the last commit, we slipped in a bug and our automated testing suite failed to identify it. This is because our code coverage is very poor at this moment. We need to include tests for almost all the components to ensure that such a thing doesn't happen again. We also need to make sure that these tests are non-deterministic with timeouts.

Tasks:

  • GET
  • SET
  • UPDATE
  • DEL
  • EXISTS
  • MGET
  • MSET
  • MUPDATE

Stability Audit (2020)

This issue aims to check the current stability of the database server, the command line client, the benchmarking tool and dependencies.


Expect this issue to be updated frequently

Feature: JSON output for `tdb-bench`

Description
We should be able to provide JSON output for the tool tdb-bench, as we might be needing it in the future when we have automated benches and require a structured output.

Use a `Bytes` structure to store data

Currently we're using a String object to store data which has it's own drawbacks. We should instead use a memory-efficient and thread-friendly structure which would greatly improve the time taken for reading/writing data.

Feature: Support configuration files

Description
As of now, we just support binding to port 2003 on the loopback address. However, this should be configurable. For this, we'll make use of a configuration file. This also provides future extensibility, when we need to provide tunable options.

Tasks:

  • IP Address
  • Port
  • noart for secure environments
  • Enable it

Feature: Supporting array queries/responses

Description
Since we plan to support more data structures (and more data models), beyond just keys and values - we need to support arrays in queries and/or responses.

Tasks

  • New QueryBuilder impl
  • New ResponseBuilder impl
  • New query packet parser
  • New response packet parser
  • New Display impl for array responses
  • Deprecate old interfaces and start using new interface
  • Test

Edit: The new protocol's specification can be found here

Question: Whether we should support both TLS/non-TLS ports at the same time

As we're headed to implementing TLS, I couldn't reach a conclusion as to whether we should have both TLS and non-TLS ports at the same time.

For example,

  • we could have a TLS port on port n and a non-TLS port on port n+1. Is that what we want?

  • Another possibility is that we have a TLS-only or non-TLS only port at a time. Would this be something we want?

Server: Handle systemctl stop when running as a systemd service

Description of bug
When running skyd as a systemd service, calling systemctl stop will send a SIGTERM and abruptly kill the skyd process leaving .sky_pid behind. (Not sure if any risk of data loss or if it will flush to disk, have not tested that yet.)

Steps to reproduce

  1. Run skyd as a systemd service (systemctl start <service_name>)
  2. Stop skyd service (systemctl stop <service_name>)

Expected behavior
Terminate gracefully like a Ctrl+C i.e. flush data to disk and clean up the .sky_pid file.

Meta

  • Release tag: None
  • Branch:
  • Commit ID:
  • Operating system: Ubuntu 20.04.2 for Raspberry PI

Bug: BGSAVE and memory usage

So I've been closely tracking the memory usage of the database server (that is to say profiling) to explore a doubt that I've had for a long while: memory usage and BGSAVE and boy, I was right!

If you see this line:
https://github.com/terrabasedb/terrabasedb/blob/771bf46e276cb199319fa7c84fb3936d06de3b90/server/src/diskstore/mod.rs#L80..L83

We aren't doing something that is very clean in a memory perspective.

For example, see this graph which illustrates the memory usage of the database server:
db-mem-profile

Why those bumps? Those bumps happen whenever BGSAVE kicks in, since we are cloning everything! Good Lord! I managed to track this by reducing the --saveduration interval. This needs to be fixed soon.

PS: I'm evaluating a disk storage format that was initially proposed for 2020, but it'll possibly be a goal for 2021.

Tracking issue: Stabilize Skyhash 1.0

Skyhash is the future protocol that Skytable will use. It saves bandwidth, simplifies client implementation and supports complex structures like nested arrays that can hold multiple types. Skytable 0.5.2 will be the last version to use the Terrapipe protocol and reaches EOL today. This issue tracks the migration from Terrapipe to Skyhash.
Tasks:

  • Implement
  • Test
  • Document
  • Migrate dbnet
  • Migrate resp
  • Migrate kvengine
  • Migrate queryengine
  • Implement client (see skytable/client-rust#3)
  • Upgrade test macros (waiting on merge of skytable/client-rust#4)
  • Let CLI use client
  • Merge

This is a long and tedious procedure, but let's make it happen!

Make it possible to use it as an embedded database

Description
I like the idea of document and key-value databases as they are easier to set up and use. What about making it possible to use TDB as an embedded database? The use case here is using it in Android and iOS apps as a replacement for SQLite, as well as using it in web and Desktop apps. As the trends show the margin between mobile, web and Desktop apps keep narrowing, this being the case a fast database written in a language that supports easy interfacing with other languages places TDB at a very potential position!

Server: Running two skyd in same directory can lead to loss of data

Description of bug
Currently it is possible to start two skyd processes from the same directory. This will lead to data being overwritten by the other process.

Steps to reproduce
Steps to reproduce the behavior:
For example:

  1. Run skyd
  2. Run skyd again
  3. Store some data on both servers
  4. When both skydare closed you will end up with data for only the last server closed

Expected behavior
Stop any subsequent skyd process from starting up if there is already one running in that directory.

Meta

  • Release tag:
    None
  • Branch:
  • Commit ID:
  • Operating system:

The current way of query parsing is highly inefficient

The current query parsing strategy does several inefficient things:

  1. There is no pre-allocated buffer (we keep allocating them on the way) here and here
  2. UTF-8 parsing isn't mandatory here

The most expensive thing is UTF-8 checking and this should be removed immediately as this makes query handling painfully slow

[bug] Odd issue with the termination of connections on Windows

Windows users have reportedly been having 'weird' and unexpected errors while terminating a connection to Skytable (i.e hitting ctrl+c on skysh). These errors are of the 'connection forcibly closed...' genre and there's one thing that I can make out of it: we may not be terminating the connection correctly, on Windows. As per the impl of std's TcpListener, we'd expect tokio to have the same Drop impl, that is terminating the connection if we receive a connection close (i.e RST). Now, this might not be happening on Windows; vaguely put, this might be happening for some reason.

If you give a read through the reported problems, it is clear that this hasn't been documented by MS at all.

If you look at our understanding of RSTs:
https://github.com/skytable/skytable/blob/next/server/src/protocol/mod.rs#L107..L125

We're doing it right; so no doubt it is a platform-specific problem. Since RSTs are not 'dangerous' per se, and simply indicate that the client just shutdown the connection and we get a 'Connection Reset' error because we're still trying to write to a non-existent stream -- could we just suppress this on Windows? I am yet to reproduce it once I get my hands on a Windows machine.

Feature: Implement snapshots

Description
This issue tracks the implementation progress for snapshots. For more info, see the roadmap(#24)

Tasks

  • Implement snapshotting
  • Enable snapshot configuration via config file
  • Enable automated snapshots
  • Test
  • MKSNAP action
  • Tests for MKSNAP

Deps: Do something about Dependabot or disable it

Without a doubt, Dependabot is a fantastic utility and there's nothing bad about it, except for one thing โ€” multiple commits. We want to keep our commit log linear and creating multiple (10 to 15 to 20) commits for dep updates don't exactly support this goal. So, we have two options:

  • Disable Depandabot entirely (-manual updates, +clean commit log)
  • Rebase on Dependabot pushes (+clean commit-log, +auto updates, -more complex CI script)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.