Giter VIP home page Giter VIP logo

arana's Introduction

Arana

Arana is a Cloud Native Database Proxy. It can be deployed as a Database mesh sidecar. It provides transparent data access capabilities, when using Arana, user doesn't need to care about the sharding details of database, they can use it just like a single MySQL database.

Overview

License codecov Go Report Card Release Docker Pulls

Stargazers Over Time Contributors Over Time
Stargazers over time Contributor over time

Introduction | 中文

First, Arana is a Cloud Native Database Proxy. It provides transparent data access capabilities, when using Arana, user doesn't need to care about the sharding details of database, they can use it just like a single MySQL database. Arana also provide abilities of Multi Tenant, Distributed transaction, Shadow database, SQL Audit , Data encrypt / decrypt and so on. Through simple config, user can use these abilities provided by Arana directly.

Second, Arana can also be deployed as a Database mesh sidecar. As a Database mesh sidecar, arana switches data access from client mode to proxy mode, which greatly optimizes the startup speed of applications. It provides the ability to manage database traffic, it takes up very little container resources, doesn't affect the performance of application services in the container, but provides all the capabilities of proxy.

Architecture

Features

Feature Complete
Single DB Proxy
Read Write Splitting
Sharding
Multi Tenant
Distributed Primary Key
Shadow Table
Tracing / Metrics
Distributed Transaction WIP
Database Mesh WIP
SQL Audit WIP
Data encrypt / decrypt Roadmap
SQL LIMITER Roadmap

Getting started

Please reference this link Getting Started

arana start -c ${configFilePath}

Prerequisites

  • Go 1.20+
  • MySQL Server 5.7+

Design and implementation

Roadmap

Built With

  • TiDB - The SQL parser used

Contact

Arana Chinese Community Meeting Time: Every Saturday At 9:00PM GMT+8

Contributing

Thanks for your help improving the project! We are so happy to have you! We have a contributing guide to help you get involved in the Arana project.

Developer

Thanks to all developers!

License

Arana software is licenced under the Apache License Version 2.0. See the LICENSE file for details.

arana's People

Contributors

alexstocks avatar baerwang avatar binbin0325 avatar charlie17li avatar chuntaojun avatar cjphaha avatar csynineyang avatar demanwei avatar dk-lockdown avatar dobet avatar dongzl avatar georgehao avatar gongna-au avatar gufengwyx8 avatar jasonzhang95 avatar jjeffcaii avatar luky116 avatar lvnszn avatar maronghe avatar mengchuang123 avatar mulavar avatar pangxing avatar penglq avatar raspberry-hu avatar raymario avatar robbinhan avatar tydhot avatar viviel avatar wang1309 avatar zlber avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

arana's Issues

Default tag not work when parse config

What happened:
default tag not work when parse config
What you expected to happen:
default tag should work when var is empty
How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Implement 'SHOW TABLES' statement

What would you like to be added:
The statement of SHOW TABLES should only list those logical tables and non-sharding tables.
Please see the official manual, https://dev.mysql.com/doc/refman/5.7/en/show-tables.html

Why is this needed:
The tables in ARANA means 'logical tables' instead of 'physical tables', we need to handle it. For example, we have student_0000..student0031, the actual result should be student.

Support table metadata

What would you like to be added:
Support table metadata for star expand, in case:
select * from person
->
select id, name from person

Why is this needed:

When execute "select * from student" mysql/server.go:161 mysql_server caught panic:

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Start server refer this PR comment: #90 .

execute:select * from student.

2022-03-25T11:43:14.767+0800    DEBUG   executor/redirect.go:135        ComQuery: select connection_id()
2022-03-25T11:43:14.773+0800    DEBUG   executor/redirect.go:135        ComQuery: select * from student
2022-03-25T11:43:14.773+0800    DEBUG   optimize/optimizer.go:149       compute shards: result=["*"], isFullScan=false
2022-03-25T11:43:14.773+0800    ERROR   mysql/server.go:161     mysql_server caught panic:
implement me
github.com/arana-db/arana/pkg/mysql.(*Listener).handle.func1
        /Users/dongzonglei/source_code/Github/arana/pkg/mysql/server.go:161
runtime.gopanic
        /usr/local/go/src/runtime/panic.go:965
github.com/arana-db/arana/pkg/runtime/plan.UnionPlan.ExecIn
        /Users/dongzonglei/source_code/Github/arana/pkg/runtime/plan/union.go:39
github.com/arana-db/arana/pkg/runtime.(*defaultRuntime).Execute
        /Users/dongzonglei/source_code/Github/arana/pkg/runtime/runtime.go:545
github.com/arana-db/arana/pkg/executor.(*RedirectExecutor).ExecutorComQuery
        /Users/dongzonglei/source_code/Github/arana/pkg/executor/redirect.go:180
github.com/arana-db/arana/pkg/mysql.(*Listener).ExecuteCommand.func1
        /Users/dongzonglei/source_code/Github/arana/pkg/mysql/server.go:558
github.com/arana-db/arana/pkg/mysql.(*Listener).ExecuteCommand
        /Users/dongzonglei/source_code/Github/arana/pkg/mysql/server.go:589
github.com/arana-db/arana/pkg/mysql.(*Listener).handle
        /Users/dongzonglei/source_code/Github/arana/pkg/mysql/server.go:206
2022-03-25T11:43:14.775+0800    DEBUG   executor/redirect.go:135        ComQuery: select connection_id()

表中有值,也给客户端返回结果了,但是客户端没解析出来

var v = &struct {
	EmpNo     int       `gorm:"emp_no"`
	BirthDate time.Time `gorm:"birth_date"`
	FirstName string    `gorm:"first_name"`
	LastName  string    `gorm:"last_name"`
	Gender    string    `gorm:"gender"`
	HireDate  time.Time `gorm:"hire_date"`
}{}

db, _ := sql.Open("mysql", "dksl:123456@tcp(127.0.0.1:13306)/employees?timeout=1s&readTimeout=1s&writeTimeout=1s&parseTime=true&loc=Local&charset=utf8mb4,utf8") //连接数据库
defer db.Close()

row := db.QueryRow(`SELECT emp_no FROM employees`)
if err := row.Scan(&v.EmpNo); err != nil {
	panic(err)
}
fmt.Println(v.EmpNo)

运行结果:
panic: sql: no rows in result set

image

What happened:
表中有值,也给客户端返回结果了,但是客户端没解析出来

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Cannot connect with mycli

Cannot connect the arana server over mycli

The mycli is a very popular command-line tool, we should support it.

What happened:

  1. start server
  2. connect with mycli -h127.0.0.1 -udksl employees -p123456 -P13306

What you expected to happen: Should connect successfully

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Fixes project file License header format.

Now, the project file License header format is:

//
// Licensed to Apache Software Foundation (ASF) under one or more contributor
// license agreements. See the NOTICE file distributed with
// this work for additional information regarding copyright
// ownership. Apache Software Foundation (ASF) licenses this file to you under
// the Apache License, Version 2.0 (the "License"); you may
// not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.
//

we want to fix it:

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

启动加载yaml配置文件,role、type字段支持类型判断

func (t *DataSourceType) UnmarshalText(text []byte) error {
	if t == nil {
		return errors.New("can't unmarshal a nil *DataSourceRole")
	}
	if t.unmarshalText(bytes.ToLower(text)) {
		return fmt.Errorf("unrecognized protocal type: %q", text)
	}
	return nil
}

What happened:
支持的角色类型、协议类型,返回了error的错误
What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

if t.unmarshalText(bytes.ToLower(text)) {
		return fmt.Errorf("unrecognized protocal type: %q", text)
	}

改为

if !t.unmarshalText(bytes.ToLower(text)) {
		return fmt.Errorf("unrecognized protocal type: %q", text)
	}

Anything else we need to know?:

user list

Welcome to use arana. To know who is using it, pls append your org info as follow,

  • Organization:Alibaba (Required)
  • Location: Hangzhou, Zhejiang, China(Required)
  • Contact: [email protected] (Optional)
  • Purpose:used as our fundamental network transport lib (Required)
  • Registry: zookeeper(Required)
  • Version: 1.5.5(Required)
  • Com/Org Logo:(Optional)
    Thanks again for your participation!

欢迎使用 arana,首先感谢你的使用,其次您可以参考下面的样例来提供您的信息以收集下使用场景:

* 组织:阿里巴巴(Required)
* 地点:**杭州(Required)
* 联系方式:[email protected](Optional)
* 场景:阿里内部服务化改造(Required)
* 注册中心: zookeeper(Optional)
* 版本: 1.5.5(Optional)
* 公司图标:(Optional)

再次感谢你的参与!!!

Feature:etcd as configuration center

将etcd 作为配置中心:
配置获取策略:
默认拉取etcd中的数据,当etcd 中不存在数据时,将配置文件的数据同步到etcd当中。
已实现功能:
etcd作为配置存储,更新,初始化配置文件到etcd 中。
还需要实现功能点:
存储到etcd中的key value 键值对的粒度根据情况再划分小一些;
接入etcd 的watch 机制 和加入对应的 key value 的version 进行相关版本匹配;
将此方案的配置替换现有实现中的配置加载策略

Handling login without database

What would you like to be added:
Handling login without database, for example:

$ mycli -h127.0.0.1 -P13306 -uarana -p123456

Why is this needed:

跑集成测试报错了

What happened:

当执行集成测试查询时,aranan会报错:
mysql_server caught panic:
trying to call recycleReadPacket while currentEphemeralPolicy is 0

image

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
启动main函数,然后执行 test 包下面的 integration_test.go 文件的 TestSelect方法,即可复现问题

Anything else we need to know?:

filter mechanism

常规的请求应答模型,filter 机制只需要在代理层实现一个 filter chain,类似于高速公路上的收费站,每条通路上设置一个收费站,实现对请求的拦截。由于 DB 路由有 sharding 的需求,它的 filter 机制应该类似于水站,水源经过水站拦截后,分散成无数的水管,每个水管上安装有水阀。
db proxy 001
需要两种类型的 filter,一种是 executor filter,一种是 connection filter。

Enhance observability

What would you like to be added:

  • Use Prometheus go client to add more metrics.
  • Use OpenTelemetry to add more trace.
  • Add config.

Use Prometheus client-go and OpenTelemetry is because they are the most used now. :) Correct me if I am wrong, or if you have better choice, just comment.

Why is this needed:
Observability is very important for basic software architecture.There are three sections in this field.
image

一些编程细节

What happened:

dubbogo 积累的一些历史旧账

What you expected to happen:

多注意细节,多看看历史问题,多翻翻旧账,照亮来时路,少犯错。

'unknown error: no such logical database \x02employees'

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

  1. Start server refer this PR comment: #90 .
  2. Connect the server with mycli: mycli -h127.0.0.1 -P13306 -udksl employees -p123456
  3. show databases; is OK.
  4. use employees; is OK.
  5. show tables; is not OK. error message: (1105, 'unknown error: no such logical database \x02employees')

image

Support simple DELETE operation.

What would you like to be added:
implement simple DELETE statement, for example: delete from student where uid = 1 limit 1.

Why is this needed:
It is a basic feature.

readme

What happened:

no readme

What you expected to happen:

readme

Anything else we need to know?:

tell us what it is and how to use it.

Start arana server, execute TestSelect method in integration_test.go will case panic.

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

2021-12-30T16:22:00.085+0800    DEBUG   executor/redirect.go:137        ComQuery: SET NAMES utf8mb4
2021-12-30T16:30:24.726+0800    DEBUG   executor/redirect.go:137        ComQuery: SET NAMES utf8mb4
^[[20~2021-12-30T16:30:37.444+0800      DEBUG   executor/redirect.go:137        ComQuery: SET NAMES utf8mb4
2021-12-30T16:30:46.986+0800    DEBUG   executor/redirect.go:137        ComQuery: SET NAMES utf8mb4
2021-12-30T16:31:01.774+0800    DEBUG   executor/redirect.go:137        ComQuery: SET NAMES utf8mb4
2021-12-30T16:31:04.551+0800    DEBUG   executor/redirect.go:137        ComQuery: SET NAMES utf8mb4
2021-12-30T16:31:04.553+0800    INFO    executor/redirect.go:193        SELECT emp_no, birth_date, first_name, last_name, gender, hire_date FROM employees 
                WHERE emp_no = 100001
2021-12-30T16:31:04.554+0800    ERROR   mysql/server.go:156     mysql_server caught panic:
trying to call recycleReadPacket while currentEphemeralPolicy is 0
github.com/dubbogo/arana/pkg/mysql.(*Listener).handle.func1
        /Users/dongzonglei/source_code/Github/arana/pkg/mysql/server.go:156
runtime.gopanic
        /usr/local/go/src/runtime/panic.go:971
github.com/dubbogo/arana/pkg/mysql.(*Conn).recycleReadPacket
        /Users/dongzonglei/source_code/Github/arana/pkg/mysql/conn.go:345
github.com/dubbogo/arana/pkg/mysql.(*Listener).handle
        /Users/dongzonglei/source_code/Github/arana/pkg/mysql/server.go:186

Simple sequence implementation.

What would you like to be added:
Generate pk when executing INSERT:

  1. register entrance
  2. impls: snowflake/sequence table based/...

Why is this needed:

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.