Giter VIP home page Giter VIP logo

yearning's Introduction

Yearning
Simple, Efficient and MYSQL-Like

Seamlessly integrates SQL detection and query auditing, tailored for the use of DBAs and developers.
A locally deployed, privacy-focused, simple and efficient for MYSQL audit platform.

OSCS Status

LICENSE Github Stars Releases

English | 简体中文

Feature

  • SQL Audit — Support the creation of SQL audit tickets with approval workflows and automated syntax checkers to validate submitted SQL statements for correctness, security, and compliance. Automatically generate rollback statements corresponding to the submitted DDL/DML operations for easy recovery when needed. Maintain a comprehensive history log of all SQL audit operations for traceability and auditing purposes.
  • Query Audit — Our solution supports auditing of user query statements, including restrictions on data sources and databases, as well as anonymization of sensitive fields. Query records are also saved for future reference.
  • Check Rules — The automated syntax checker supports dozens of check rules, catering to most of the automatic checking scenarios.
  • Privacy focussed - Locally deployable and open-source solution ensures the security of your database and SQL statements. In addition to providing control over the infrastructure, the solution also includes encryption mechanisms to protect sensitive data before storing it in your database. This ensures that even if there is unauthorized access to the database, the encrypted data remains secure and unreadable. By combining local deployment, open-source transparency, and data encryption, we prioritize the privacy and security of your database and SQL statements.
  • RBAC - In our platform, you can create and manage different roles and assign specific permissions to each role. This allows you to restrict users' access to query work orders, auditing functions, and other sensitive operations based on their assigned roles.

Docs

Yearning Docs only Chinese

Install

Download the latest release and extract it.

First make sure you have configured ./config.toml

Manual

## init database
./Yearning install

## start
./Yearning run

## help
./Yearning --help

Yes, it's that simple

Docker

## init database
docker run --rm -it -p8000:8000 -e SECRET_KEY=$SECRET_KEY -e MYSQL_USER=$MYSQL_USER -e MYSQL_ADDR=$MYSQL_ADDR -e MYSQL_PASSWORD=$MYSQL_PASSWORD -e MYSQL_DB=$Yearning_DB yeelabs/yearning "/opt/Yearning install"
## You must initialize your database in the startup container
docker run -d -it -p8000:8000 -e SECRET_KEY=$SECRET_KEY -e MYSQL_USER=$MYSQL_USER -e MYSQL_ADDR=$MYSQL_ADDR -e MYSQL_PASSWORD=$MYSQL_PASSWORD -e MYSQL_DB=$Yearning_DB yeelabs/yearning

Recommend

Spug - 开源轻量自动化运维平台

Automatic SQL Checker

The SQL statement detection function tests against predefined rules and syntax
we can set predefined rules to check whether the SQL statement conforms to specific coding standards, best practices or security requirements.


SQL syntax highlighting and Auto-completion

SQL syntax highlighting and auto-completion features to enhance the user experience and improve query writing efficiency.
SQL syntax highlighting helps users visually distinguish different parts of the SQL query, such as keywords, table names, column names, and operators. This makes it easier to read and understand the query structure.



Order/Query record

Supports auditing of user order/query statements
Through the auditing feature, you can track and record all query operations, including the data source, database, and handling of sensitive fields. This ensures that query operations comply with regulations and allows for tracing query history.


Ecosystem

Gemini Yearning front-end Project

Yee Yearning web framework

Contact Us

E-mail: [email protected]

License

See LICENSE for details.

2023 © Henry Yee

yearning's People

Contributors

chaiyd avatar chenlingmin avatar cookiey avatar cookieye avatar dependabot[bot] avatar fangpengbo avatar gitads avatar lvillis avatar nexpro avatar vlorc avatar zze326 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

yearning's Issues

iptables和firewalld的问题

您好!
1、请问启动容器的时候是会对操作系统内存中的ipatbles规则进行覆写吗?启动容器之后,原来内存中加载的规则都没有了。
2、我把firewalld服务disable和mask掉了,容器重启就不成功了,是有依赖关系吗?

谢谢~

查看回滚语句提示 (此工单没有备份或语句执行失败)

1、工单提交的时候已经点选备份。
2、数据库连接正常
[mysql]
db = yearning
address = 127.0.0.1
port = 3306
username = yearning
password = 12312300

[host]
ipaddress = 10.170.195.166:8080

[Inception]
ip = 127.0.0.1
port = 6669
user = root
password =
backupdb = 127.0.0.1
backupport = 3306
backupuser = admin
backuppassword = 12312300
......

查询结果导出格式错误

hi,各位大神:
首先感谢大家提供这样一个优美的平台。

SQL查询界面-导出数据,如下查询
image

字段含有“,”时,导出结果如下
image

Inception的问题是否有修复计划

在使用过程中遇到如:
inception中如TEXT/BLOB类型的列不支持NOT NULL、备份语句长度超过限制、变更字段类型时的默认值问题等问题,网上提供修改sql/sql_parse.cc的方案(没试是否有效),担心稳定性。请问官方是否有修复计划

Who is using Yearning ?

欢迎 正在使用Yearning/Yearning-go的小伙伴 留下你们的公司名~~

企业名将会加入到readme当中,相关公司使用者的需求 将会被优先考虑!

使用的企业有(包括但不限于以下公司):

1.Closeli
2.大搜车
3.盛大游戏DNA工作室
4.蔷薇控股
5.深圳活力天汇科技
6.深圳轻松到家科技有限公司
7.一药网
8.同程金服
9.迅雷
10.杭州码泰

Yearning v1.2.0 正式发布

1.查询支持字段脱敏

2.支持多级审核

3.工单定时执行

4.新增设置页面,减少启动配置项

5.修复待办事项点击完成后显示的异常问题

6.修复横向导航标签 首页标签会变更坐标的异常问题

7.新增一键关闭所有横向导航标签按钮

8.修复一键生成字典下拉框样式异常的问题

9.优化翻页

10.添加执行失败显示

11.取消跨域限制

12.允许自定义机房

13.修复查询bigint字段精度丢失的问题

14.修复查询blob字段报错的问题

15.修改前端数据保存方式

16.修改查询时左侧表名过长的样式问题

17.优化查询及执行工单时错误提示语句

18.新增查询审核开关

19.修复ddl及索引提交空sql后审核检测时错误提示不清晰的问题

20.修复由于mysql 连接超时时间设置过短而导致查询时限失效的问题

21.修复sql过多导致的样式问题

22.新增执行结果自动刷新,不再需要手动刷新

23.修复ldap用户更改密码后登陆异常的问题

24.修复了其他影响用户体验的bug

邮箱推送失败

首先,设置里邮箱是没有启用ssl端口的,在设置页面测试了是可以发送测试邮件的。
使用人提交工单后,管理员没有收到邮件通知。
管理员审核工单后,工单转移到执行人,执行人可以收到邮件通知。
执行人驳回、执行工单,使用人都收不到邮件通知。

查看容器里的日志/mnt/src/log/error.log,有记录下面错误:
2018-11-14 10:09:56,754 [Thread-21:140058122672872] [Yearning.core.views:318] [ERROR]- TypeError: init() got an unexpected keyword argument 'ssl'
2018-11-14 10:23:34,430 [Thread-21:140058122660584] [Yearning.core.views:309] [ERROR]- TypeError--邮箱推送失败: init() got an unexpected keyword argument 'ssl'
2018-11-14 10:54:04,711 [Thread-24:140058121612008] [Yearning.core.views:195] [ERROR]- TypeError--邮箱推送失败: init() got an unexpected keyword argument 'ssl'
2018-11-14 10:54:31,000 [Thread-25:140058122672872] [Yearning.core.views:250] [ERROR]- TypeError--邮箱推送失败: init() got an unexpected keyword argument 'ssl'
2018-11-14 10:54:47,521 [Thread-25:140058121337576] [Yearning.core.views:195] [ERROR]- TypeError--邮箱推送失败: init() got an unexpected keyword argument 'ssl'
2018-11-14 11:05:35,234 [Thread-26:140058122672872] [Yearning.core.views:309] [ERROR]- TypeError--邮箱推送失败: init() got an unexpected keyword argument 'ssl'
2018-11-14 11:07:42,205 [Thread-26:140058122398440] [Yearning.core.views:250] [ERROR]- TypeError--邮箱推送失败: init() got an unexpected keyword argument 'ssl'
2018-11-14 11:08:49,158 [Thread-27:140058122672872] [Yearning.core.views:309] [ERROR]- TypeError--邮箱推送失败: init() got an unexpected keyword argument 'ssl'
2018-11-14 11:09:07,690 [Thread-28:140058121337576] [Yearning.core.views:195] [ERROR]- TypeError--邮箱推送失败: init() got an unexpected keyword argument 'ssl'

jwt用户认证的方式有漏洞

通过修改用户名的方式,即可切换到别的用户上。
当前版本是1.3.0,不过看最新的changelog,并没发现修复该漏洞。
image

我吐槽一下,

代码审核,就返回1.或者2, 别人怎么知道错在哪里,这显然不人性化啊,好比你去政府办事,就和你说证件不够,缺啥都不告诉你,这用户体验简直了。
_20180517091148

报错连不上Inception

本机配置就是127.0.0.1 没有配置密码,本机可以登陆inception 在yearning报错

image

新版提交和驳回无法发送邮件

新版提交和驳回无法发送邮件
原因,task.py 中 实例化对下,传入两个参数,但send_email init方法没有接收ssl参数 :
put_mess = send_email.send_email(to_addr=mail.email, ssl=tag.message['ssl']
class send_email(object):
def init(self, to_addr=None):
临时修改了send_email.py代码解决:
class send_email(object):
def init(self, to_addr=None,ssl="False"):

npm 和nginx不成功

(yearning) wangchao@wangchao-lyan:~/PycharmProjects/Yearning/webpage$ npm run dev

[email protected] dev /home/wangchao/PycharmProjects/Yearning/webpage
node build/dev-server.js

module.js:540
throw err;
^

Error: Cannot find module '/home/wangchao/PycharmProjects/Yearning/webpage/build/dev-server.js'
at Function.Module._resolveFilename (module.js:538:15)
at Function.Module._load (module.js:468:25)
at Function.Module.runMain (module.js:684:10)
at startup (bootstrap_node.js:187:16)
at bootstrap_node.js:608:3
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] dev: node build/dev-server.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/wangchao/.npm/_logs/2018-01-08T05_41_27_716Z-debug.log

1,npm启动静态资源服务器时候一直报错(node 6.10和node 8.9版本都试了)

2,for nginx,webpage/dist/没有这个文件夹啊

Yearning v1.3.0 正式发布

功能更新如下:

1.修改了报错提示

2.新增细粒化权限组,废除之前的个人权限划分

3.连接名webook合并至配置页面webhook

4.删除查询时限,新增管理员结束用户查询权限

4.查询页面支持自动补全库名及表名

5.新增版本号接口 (登陆后/#/version)

6.修复权限审核页面,数据库管理页面低版本chrome或其他浏览器401报错的问题

7.修复查询审核时查询工单提交没有消息推送的问题

执行影响多条的sql,回滚语句只有一条

我执行一个sql,类似 update set name='lalala' where id in (1,2) ,影响了两条。
但是在查看回滚语句时,只显示了一条。
我认为是这行代码的问题:(Yearning/src/core/api/auditorder.py 215行)
_h = sorted([i[0][0] for i in sql])

查看sql 执行时value为:
sql = [ (
1、 (update xxxxx),
2、 (update xxxxx),
),
(
3、 (update xxxxx),
4、 (update xxxxx),
)]
上面_h 应该是只取了1、和3、吧。

感觉延迟执行的功能设计的不是特别科学

延迟执行的时间是提交审核申请时填写的, 但是延迟时间是从审核通过的时间开始计算的, 这样就没有办法达到让一条SQL在特定时间执行的效果了, 除非提交审核后立刻审批通过, 但在真实的场景中审核通过的时间还是比较难控制, 不知道目前这样的设计是不是有什么其他的考虑

1.2版本

1.2版本的使用普通用户进行查询,执行查询报错,报错Error: Request failed with status code 500
yangning

docker 安装方式登录失败

Failed to load http://1x8.2x.43.2x4:8000/api-token-auth/: Response to preflight 
request doesn't pass access control check: No 'Access-Control-Allow-Origin' header
 is present on the requested resource. Origin 'http://1x8.2x.4x.2x4:86' is therefore 
not allowed access.

image

unicode is an undefined name in Python 3

unicode was removed from Python 3 because all strs are Unicode.

flake8 testing of https://github.com/cookieY/Yearning on Python 3.6.3

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./install/cursors.py:354:36: F821 undefined name 'unicode'
                if isinstance(msg, unicode):
                                   ^
1     F821 undefined name 'unicode'

生成数据字典出错

生成数据字典出错的时候报了个错误
[Yearning.core.views:698] [ERROR]- DataError: (1406, "Data too long for column 'Type' at row 1")

权限申请和添加字段默认值报错

问题1: 在个人信息中修改用户,如,admin权限申请报错如下:

python3 -V

Python 3.6.4

2018-04-16 13:50:17,874 [Thread-20:140082289620736] [django.request:118] [ERROR]- Internal Server Error: /api/v1/apply_grained/
Traceback (most recent call last):
File "/usr/local/python3/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
response = get_response(request)
File "/usr/local/python3/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/python3/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/python3/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/local/python3/lib/python3.6/site-packages/django/views/generic/base.py", line 69, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/python3/lib/python3.6/site-packages/rest_framework/views.py", line 494, in dispatch
response = self.handle_exception(exc)
File "/usr/local/python3/lib/python3.6/site-packages/rest_framework/views.py", line 454, in handle_exception
self.raise_uncaught_exception(exc)
File "/usr/local/python3/lib/python3.6/site-packages/rest_framework/views.py", line 491, in dispatch
response = handler(request, *args, **kwargs)
File "/usr/local/yearning/src/core/api/applygrained.py", line 77, in post
thread = threading.Thread(target=push_message, args=({'to_user': request.user, 'workid': work_id}, 2, request.user, mail.email, work_id, '已提交'))
AttributeError: 'NoneType' object has no attribute 'email'
2018-04-16 13:50:17,904 [Thread-20:140082289620736] [django.server:124] [ERROR]- "POST /api/v1/apply_grained/ HTTP/1.1" 500 16085

/usr/local/yearning/src/core/api/applygrained.py:
class apply_grained(baseview.BaseView):

def post(self, request, args: str = None):

    grained_list = json.loads(request.data['grained_list'])
    work_id = util.workId()
    applygrained.objects.get_or_create(work_id=work_id, username=request.user, permissions=grained_list, status=2)
    mail = Account.objects.filter(id=1).first()
    thread = threading.Thread(target=push_message, args=({'to_user': request.user, 'workid': work_id}, 2, request.user, mail.email, work_id, '已提交'))
    thread.start()
    return Response('权限申请已提交!')

问题2: 在工单提交--DDL中添加字段,如果选择不为空,默认值设置为空字符串''时就会多加一对单引号,如下:
如果默认值为",就会变成'"'
SQL> show create table t2\G
*************************** 1. row ***************************
Table: t2
Create Table: CREATE TABLE t2 (
id int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT 'PK',
name2 varchar(5) NOT NULL DEFAULT '' COMMENT 'name',
group2 varchar(5) NOT NULL DEFAULT '''' COMMENT 'group2',
PRIMARY KEY (id)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=utf8 COMMENT='t2'
1 row in set (0.00 sec)

请作者帮忙解惑,谢谢

docker方式运行,有跨域问题

run docker run -it -d -p 80:80 -p 8000:8000 -e 192.168.1.109 registry.cn-hangzhou.aliyuncs.com/cookie/yearning:v0.0.3

登录点击后,控制台报:
Failed to load http://192.168.1.109:8000/api-token-auth/: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://192.168.1.109' is therefore not allowed access.

DML检测不通过

报错Invalid source infomation。用过上一个版本搭过完全没问题,配置没变,现在新版这个样子,测了insert,delete都会报这个错

1 NONE 2 None Invalid source infomation. Global environment 0

查询-数据库字典中数据库连接下拉框没有数据

我是用docker运行的1.3.1
DDL和DML中都可以先选择机房后会有数据库连接,但在查询-数据库字典中,没有机房选择,数据库连接下拉框也没有值,权限组中 这里连接名也没有值可选择。DDL和DML那里是有连接名选择的

数据字典导出权限: 是 否
连接名:
全选

执行安装命令无效

下载Yearning,进入install/yearning-docker-compose目录

执行 docker-compose up -d 这个报 -bash: docker-compose: 未找到命令,主要以及安装最新版本docker服务了。

8000端口如何修改

因为我们的8000端口已经被占用,并且我看到“3306和8000端口不可更改!”字样,如果我必须修改8000端口的话,可以修改那个文件?

是否支持自定义检查规则

看inception已经将近3年没有维护了,在inception基础上开发是否底座有风险?
是否支持mysql自定义检查规则?

开源吗

请问你这个平台开源吗?

容器化的站内信是被关掉了吗?

Yearning功能很强大,初次使用交互做的很不错,但测试过程中出现了一些问题

版本:1.3.3
1、提交的ddl审核在审核人这没有收到站内信息或邮件;
2、主页右上角的站内信息提示标志不存在:

_20181106194418

首页登录问题

登录页输入默认账号和密码 没有反映,web日志在哪里?如何排错,谢谢。

Yearning v1.3.1发布

1.修复未设置limit数据查询报错问题
2.修复提交sql中使用use语句导致无法正常显示备份信息的问题
3.添加用户注册功能(限制邮箱后缀名方式)
4.添加邮箱检查功能
5.添加数据库过滤功能
6.更改审核工单sql为表格样式,更加美观整洁
7.更改工单详细信息表格样式,更加美观整洁
8.删除站内信功能
9.驳回信息将显示在驳回详细信息中
10.支持ssl邮箱
11.前端模块升级
12.核心功能 执行 幂等实现
13.修复ddl不选择表名点击连接假死的问题
14.修复我的工单详情页面 执行失败的工单看不到失败信息的问题
14.修改权限展示页面样式,更加美观
15.新增real_name字段 展示用户真实姓名
16.ldap修复多dc下无法登陆成功的问题

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.