Giter VIP home page Giter VIP logo

meituan-dianping / zebra Goto Github PK

View Code? Open in Web Editor NEW
2.7K 164.0 713.0 8.61 MB

美团点评集团统一使用的MySQL数据库访问层的中间件。主要提供对业务开发透明、读写分库、分库分表能力,并提供了端到端SQL监控的集成方案。

License: Apache License 2.0

Java 93.70% JavaScript 1.36% HTML 0.09% Vue 4.43% CSS 0.42% TSQL 0.01%
mysql dynamic-datasource read-write-separation zebra database-sharding mybatis

zebra's Introduction

Zebra

Introduction

Zebra是一个基于JDBC API协议上开发出的高可用、高性能的数据库访问层解决方案,是美团点评内部使用的数据库访问层中间件。具有以下的功能点:

  • 配置集中管理,动态刷新
  • 支持读写分离、分库分表
  • 丰富的监控信息在CAT上展现
  • 异步化数据库请求,多数据源支持

Core Value

  • 简化了读写分离、分库分表的开发工作,使得业务方在分库分库、读写分离的情况下,依然可以像操作单个库那样去操作,屏蔽底层实现的复杂性,对业务透明。 提供了从读写分离到分库分表全生命周期的技术支持。
  • 完善的监控体系帮助开发掌控数据库请求的整个链路,快速定位问题。
  • dao层扩展功能

Modules

  • zebra-client(核心) : 除了监控外,几乎zebra所有核心功能,如读写分离、分库分表、就近路由、流量控制
  • zebra-cat-client(可选): 提供端到端的监控,将监控信息上报到CAT监控平台
  • zebra-dao(可选):对mybatis的轻量级封装,兼容mybatis原有的功能,并额外提供了异步化接口、分页插件、多数据源等功能
  • zebra-admin-web:zebra配置管理平台 用于管理zk和保存在zk中的zebra配置
  • zebra-sample: zebra客户端使用的demo

Quick Start

Project Design

License

Company


接入Zebra的公司欢迎在此接入公司留下联系方式, 谢谢。

Contact us

zebra's People

Contributors

8wy1985659 avatar ainilife avatar dependabot[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

zebra's Issues

Zebra相比shardingJDBC 解决了什么问题

问一个不成熟的问题,shardingJDBC已经算是久经考验的中间件了,18年也进了Apache,发展的挺好。那zebra是为了解决一些什么痛点问题而发开的呢,还是更多为了适配大众点评自己的业务?另,zebra-dao提供的RowBoundsWithCallback和RowBoundsWithFuture,可以理解成是异步的JDBC么

打的zebra-client的包druid的有疑问

打出来zebra-client的包里面包含了druid的内容,但是没有修改包名,这样的话没有太大的作用
而且分库分表本身就会使用druid的语法分析等,而且新老版本兼容问题,该处还是改写包名,依赖一个固定版本的druid,另druid的版本太低,有些bug都修复了,但是依赖新的druid对分库分表有影响
image

动态表

请问下我有一个date_time时间字段:begin_time,我想已begin_time为纬度,进行分表,表我会提前生成,每天一个表:table_yyyyMMdd,这种方式支持吗?

zebra如何支持mysql的迁移、扩容方案

我有两个问题:
迁移:

  1. zebra如何支持mysql的主库、从库的迁移。

扩容:
在分库分表的方案中,zebra是如何支持扩容的,这里面可能有两个方面

  1. 不更改库表的个数,迁移库,如何支持在线 更改路由
  2. 如果库、表的个数初始估计不足,后期如何扩容、迁移数据。

为什么组件的ShardDatabaseMetaData实现是空的

在结合hibernate使用该组件的时候,发现开启hibernate的自动生成表功能后,报出空指针异常,具体看下来是因为组件的ShardDatabaseMetaData实现为空,这里是使用错误吗?有解决方案吗?

jdk版本

请问下zebra的jdk版本是不是1.6,那1.7支持吗?

异步化的时候返回类型是void,mybaitis是无法构造出对象的,这个问题怎么解决

Caused by: org.apache.ibatis.executor.ExecutorException: No constructor found in void matching [java.lang.Integer, java.lang.String, java.lang.Integer] at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.createByConstructorSignature(DefaultResultSetHandler.java:668) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.createResultObject(DefaultResultSetHandler.java:621) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.createResultObject(DefaultResultSetHandler.java:594) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.getRowValue(DefaultResultSetHandler.java:396) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValuesForSimpleResultMap(DefaultResultSetHandler.java:355) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValues(DefaultResultSetHandler.java:330) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSet(DefaultResultSetHandler.java:303) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSets(DefaultResultSetHandler.java:196) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.executor.ReuseExecutor.doQuery(ReuseExecutor.java:60) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:326) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:136) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:77) ~[mybatis-3.4.6.jar:3.4.6] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_191] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_191] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_191] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191] at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433) ~[mybatis-spring-1.3.1.jar:1.3.1] at com.sun.proxy.$Proxy85.selectOne(Unknown Source) ~[?:?] at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:166) ~[mybatis-spring-1.3.1.jar:1.3.1] at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:83) ~[mybatis-3.4.6.jar:3.4.6] at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:59) ~[mybatis-3.4.6.jar:3.4.6] at com.sun.proxy.$Proxy89.findUserById2(Unknown Source) ~[?:?] at com.maoyan.service.impl.UserServiceImpl.asynGetUser(UserServiceImpl.java:24) ~[classes/:?] at com.maoyan.service.movie.walle.biz.com.maoyan.biz.impl.UserBizImpl.asynGetUserInfo(UserBizImpl.java:25) ~[classes/:?] at com.maoyan.service.movie.walle.provider.controller.MyController.asynUserInfo(MyController.java:28) ~[classes/:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_191] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_191] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_191] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) ~[spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) ~[spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) ~[spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.13.RELEASE.jar:4.3.13.RELEASE] ... 51 more

我的mapper接口,我采用的是纯注解开发mapper,没有使用配置文件
`@Mapper
public interface IUserDAO {
@select("select * from user where id=#{id}")
User findUserById(Integer id);

@Select("select * from user where id=#{id}")
@TargetMethod(name = "findUserById")
void findUserById2(@Param("id") Integer id, AsyncDaoCallback<User> callback);

}`

异步方法void的话,那么resultType是void就没有构造函数,无法构造出对象
是我姿势错了吗?

分库分表时RouterResult不是SingleTarget的情况下希望也打印路由出来的db和sql debug日志,方便调试

com.dianping.zebra.shard.jdbc.ShardStatement#executeQueryByOriginal
此方法中isSingleTarget(routerTarget)情况下打印了db,sql日志;希望else情况下也将db、sql日志打印出来。谢谢!

if (LOGGER.isDebugEnabled()) {
						LOGGER.debug("db:" + targetedSql.getDatabaseName());
						LOGGER.debug("sql:" + executableSql);
					}
if (isSingleTarget(routerTarget)) {
			// if has only one sql,then serial execute it
			for (RouterTarget targetedSql : routerTarget.getSqls()) {
				for (String executableSql : targetedSql.getSqls()) {
					if (LOGGER.isDebugEnabled()) {
						LOGGER.debug("db:" + targetedSql.getDatabaseName());
						LOGGER.debug("sql:" + executableSql);
					}

					Connection conn = connection.getRealConnection(targetedSql.getDatabaseName(), autoCommit);
					Statement stmt = createStatement(conn);
					actualStatements.add(stmt);

					resultList.add(stmt.executeQuery(executableSql));
				}
			}
		}

辅维度不支持写操作

辅维度不支持写操作,如果辅维度和主维度是一套物理表的话就可以执行写操作。

XXE漏洞

您好:
我是360代码卫士的工作人员,在我们的开源项目代码审计过程中,发现Zebra多处处理xml数据时没有禁用外部实体。详细信息如下:
在Zebra-master\zebra-admin-web\src\main\java\com\dianping\zebra\administrator\util\JaxbUtils.java文件中的jaxbReadXml方法中
default
该方法是将xml转化为java,在JdbcrefController.java文件的findJdbcrefDetil方法中调用了该函数
default
groupValue变量是从数据库中取的,用户可以通过插入任意值来进行xxe攻击(由于项目我也没搭建起来,不敢百分百确认,这个需要你们确认下)。

同样的问题也存在于
Zebra-master\zebra-client\src\main\java\com\dianping\zebra\util\JaxbUtils.java 43行
Zebra-master\zebra-client\src\main\java\com\dianping\zebra\group\config\datasource\transform\DefaultSaxParser.java 81行
Zebra-master\zebra-client\src\main\java\com\dianping\zebra\group\config\system\transform\DefaultSaxParser.java 85行

本地配置,打成jar包后运行,报空指针异常。远程配置也无法读取zookeepe.properties的内容

public static File toFile(URL url) {
logger.info("*********url:"+url.getFile()+""+url.getProtocol());

    if (url == null || !url.getProtocol().equals("file")) {
        return null;
    } else {
        String filename = url.getFile().replace('/', File.separatorChar);
        int pos = 0;
        while ((pos = filename.indexOf('%', pos)) >= 0) {
            if (pos + 2 < filename.length()) {
                String hexStr = filename.substring(pos + 1, pos + 3);
                char ch = (char) Integer.parseInt(hexStr, 16);
                filename = filename.substring(0, pos) + ch + filename.substring(pos + 3);
            }
        }
        return new File(filename);
    }
}

打成jar后,url.getProtocol()=“jar”,然后报错,应该怎么解决呢

前端没法启动

node:v10.16.0
npm:6.13.1

$ npm run dev

[email protected] dev /Users/calvin/prj/github/Zebra/zebra-admin-web/src/main/webapp/app
webpack-dev-server --inline --progress --config build/webpack.dev.conf.js

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] dev: webpack-dev-server --inline --progress --config build/webpack.dev.conf.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

10 silly lifecycle [email protected]dev: Args: [ '-c',
10 silly lifecycle 'webpack-dev-server --inline --progress --config build/webpack.dev.conf.js' ]
11 silly lifecycle [email protected]
dev: Returned: code: 1 signal: null
12 info lifecycle [email protected]~dev: Failed to exec dev script
13 verbose stack Error: [email protected] dev: webpack-dev-server --inline --progress --config build/webpack.dev.conf.js
13 verbose stack Exit status 1
13 verbose stack at EventEmitter. (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:332:16)
13 verbose stack at EventEmitter.emit (events.js:198:13)
13 verbose stack at ChildProcess. (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack at ChildProcess.emit (events.js:198:13)
13 verbose stack at maybeClose (internal/child_process.js:982:16)
13 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
14 verbose pkgid [email protected]
1

与mysql8以及c3p0的适配问题

目前在使用中发现当与mysql8版本的数据库连接时如果使用c3p0的连接池则会报错,即使把c3p0的版本换成0.9.5版本依然会有错,请知悉。

web端保存数据到zookeeper有问题,一直卡在那里

public class ZooKeeperConnection {

private Logger logger = LoggerFactory.getLogger(this.getClass().getName());

private ZooKeeper zooKeeper;

private int sessionTimeout = 50000;

private final CountDownLatch connectedSignal = new CountDownLatch(1);

**public ZooKeeper connect(String host) throws IOException, InterruptedException {
    **zooKeeper = new ZooKeeper(host, sessionTimeout, new Watcher() {
        @Override
        public void process(WatchedEvent watchedEvent) {
           if(watchedEvent.getState() == Event.KeeperState.SyncConnected) {
               connectedSignal.countDown();
           }
        }
    });
    connectedSignal.await();
    return zooKeeper;**
}**

public void close() {
    try {
        zooKeeper.close();
    } catch (InterruptedException e) {
        logger.error("zookeeper connection close fail!", e);
    }
}

}

文档说自己很久没更新,当当网的更好。。。

目前的实现方案有:阿里巴巴开源的tddl,大众点评开源的zebra,当当网开源的sharding-jdbc。需要注意的是tddl的开源版本只有读写分离功能,没有分库分表,且开源版本已经不再维护。大众点评的zebra开源版本代码已经很久更新,基本上处于停滞的状态。当当网的sharding-jdbc目前算是做的比较好的,代码时有更新,文档资料比较全。

文档补充

hello
有没有架构图和压测结果文档?
SQL解析请问下是用druid开源的,还是自己实现的?
另能否有个QQ等交流工具,方便交流 :)

关于config下的几个问题

这个主要应用在什么场景下?
个人感觉在项目中改了配置文件重新部署,重新加载就ok?还是在不重新部署,只修改properties文件?

Zebra管理平台 -> zookeeper配置导致《Write dataSource is currently in the maintaining stage》

Zebra管理平台 ->读写分离命名,JdbcRef 如果配置了下划线会导致无法区分主从,如命名 :zebra_demo,会报
java.sql.SQLException: Write dataSource is currently in the maintaining stage. at com.dianping.zebra.util.JDBCUtils.throwWrappedSQLException(JDBCUtils.java:62) at com.dianping.zebra.group.jdbc.GroupPreparedStatement.executeUpdate(GroupPreparedStatement.java:220) at com.dianping.zebra.sample.jdbc.GroupDataSourceSample.insertValue(GroupDataSourceSample.java:72) at com.dianping.zebra.sample.jdbc.GroupDataSourceSample.groupTest(GroupDataSourceSample.java:14) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

如果命名为 :zebra-demo就会没事。

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.