hedisdb / hedis Goto Github PK

View Code? Open in Web Editor NEW

98.0 6.0 7.0 19.97 MB

Hedis can retrieve data from **ANY** database directly via Redis

Home Page: http://hedis.io

License: BSD 3-Clause "New" or "Revised" License

Makefile 0.29% Shell 0.58% C 81.03% C++ 0.29% Ruby 2.13% Tcl 15.64% Smarty 0.04%

hedis redis database

hedis's Introduction

Hedis

Hedis stands for "Hyper Redis". It can retrieve data from ANY database directly and stores to itself.

What is Hedis?

Traditionally, application server retrieves hot data from in-memory database (like Redis) to reduce unnecessary paths. If in-memory database doesn't have specific data, application server gets back to retrieve data from original database.

Hedis can retrieve data from ANY database directly. Application server retrieves hot data from Hedis server, If Hedis server doesn't have specific data, Hedis retrieves data from original database via ANY connector.

Benchmark

Latency (lower is better)

Requests (higher is better)

Hedis configuration file

cdh1: # connector name (REQUIRED)
  type: hbase # connector type (REQUIRED)
  zookeeper: localhost:2181 # other configuration (depends on its type)
  env: # environment variables (OPTIONAL)
    CLASSPATH: $CLASSPATH:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/async-1.4.0.jar:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/asynchbase-1.5.0-libhbase-20140311.193218-1.jar:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/commons-configuration-1.6.jar:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/commons-lang-2.5.jar:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/commons-logging-1.1.1.jar:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/hadoop-core-1.0.4.jar:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/hbase-0.94.17.jar:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/libhbase-1.0-SNAPSHOT.jar:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/log4j-1.2.17.jar:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/netty-3.8.0.Final.jar:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/protobuf-java-2.5.0.jar:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/slf4j-api-1.7.5.jar:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/slf4j-log4j12-1.7.5.jar:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/zookeeper-3.4.5.jar
    HBASE_LIB_DIR: /home/kewang/hbase/lib
    HBASE_CONF_DIR: /home/kewang/hbase/conf
    LD_LIBRARY_PATH: $LD_LIBRARY_PATH:/usr/lib/jvm/java-7-oracle/jre/lib/amd64/server
mysqltest:
  type: mysql
  username: root
  password: MY_PASSWORD
  host: localhost
  database: hedistest
  env:
    LD_LIBRARY_PATH: $LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu
otherdb:
  type: gooddb

Connectors

How to make

Before you make the hedis, please clean the dependencies first in order to rebuild dependencies. "make" does not rebuild dependencies automatically, even if something in the source code of dependencies is changes.

make distclean

After clean the dependencies, build the hedis using "make" command.

make

How to run

redis-server --hedis hedis.yml

Commands

GET

GET [CONNECTOR-NAME]://(INVALIDATE-SIGN)[CONNECTOR-COMMAND]

Example

# HBase example:
#
# get "kewang" rowkey at "user" table on "cdh1" connector
GET "cdh1://user@kewang"

# HBase example:
#
# get "kewang" rowkey at "user" table on "cdh1" connector, again
GET "cdh1://!user@kewang"

# MySQL example:
#
# Query one record at "user" table on "mysqltest" connector
GET "mysqltest://select * from user limit 1"

SCAN, and so on.

TODO

Related with Redis

This is a modified version of Redis, maintaining by Kewang now.

hedis's People

Contributors

Stargazers

Watchers

Forkers

gitter-badger devopsberlin rayleyva zhengshuxin ameanshinobu vtmux peterparkette

hedis's Issues

Move parse GET command to connector

Add docker

Let users easy to use hedis

Add git support

When write hedis configuration file, just use git-like protocol and will auto build connectors

cdh1:
  type: hbase # https://github.com/hedisdb/hedis-connector-hbase
  zookeeper: localhost:2181
mysqltest:
  type: kewangtw/hedis-connector-mysql # https://github.com/kewangtw/hedis-connector-mysql
  username: root
  password: MY_PASSWORD
  host: localhost
  database: hedistest
otherdb:
  type: https://bitbucket.org/good/gooddb # https://bitbucket.org/good/gooddb

Improve all warning when compile

like malloc deprecated messages.

Completion of error handling

Would you like to add more error handling for return values from functions like the following?

fclose ⇒ parse_hedis_config
malloc ⇒ build_connector

Environment variables support to configuration file

Like below

cdh1:
  type: hbase
  zookeeper: localhost:2181
  env: # Environment variables
    CLASSPATH: $CLASSPATH:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/async-1.4.0.jar:/home/kewang/git/libhbase/target/libhbase-1.0-SNAPSHOT/lib/asynchbase-1.5.0-libhbase-20140311.193218-1.jar
    HBASE_LIB_DIR: /home/kewang/hbase/lib
mysqltest:
  type: mysql
  username: root
  password: MY_PASSWORD
  host: localhost
  database: hedistest
otherdb:
  type: gooddb

Add enviroment variables when use env section

Support multiple Redis version

Hedis is based on Redis unstable branch. I think that should support old Redis version.

Can't load the same library more times

If I load same type twice or more, always become latest variables. Like cdh1 and cdh2:

cdh1:
  type: hbase
  zookeeper: localhost:2181
  extra: aaa
cdh2:
  type: hbase
  zookeeper: localhost:2181
  extra: bbb
cdh3:
  type: template

mysqltest:
type: mysql
username: root
password: MY_PASSWORD
host: localhost
database: hedistest
env:
LD_LIBRARY_PATH: $LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu

I add some debug messages and show as follow.

root@8baeaa89a5cf:/home/amean/hedis/src# ./redis-server --hedis ../hedis.yml
Close config file, and ready to add connector
Finish zmalloc sizeof(hedisConnector), line_count = 9
Switch type 0, at i = 0
Switch type 1, at i = 1
Switch type 2, at i = 5
case 2 first step, finish zmalloc, entry_count=4
finish case 2, j= 4
Segmentation fault (core dumped)

If I use original hedis.yml file to test, there is no such problem. So I think there are some bugs in parse_yaml_entry function that cause segmention fault at initial stage of the program execute.

Remove YAML dependency

It should develop myself YAML parser version

Fix memory leak

When I ran benchmark over 10000 current connections, it maybe has memory leak situation.