tal-tech / conan Goto Github PK

View Code? Open in Web Editor NEW

758.0 27.0 163.0 2.08 MB

柯南平台开源版本，为用户提供流量回放全流程解决方案

License: MIT License

Java 57.80% JavaScript 6.04% Batchfile 0.07% HTML 0.24% Vue 27.84% SCSS 0.80% PLpgSQL 7.04% Shell 0.17%

traffic-playback traffic-collection conan-platform multi-rule-traffic

conan's Introduction

conan

柯南流量回放平台

English | 简体中文

After continuous polishing and iteration in online education business, Conan platform is finally open source, aiming to provide more professional and stable quality-effectiveness assurance scheme for more quality-effectiveness assurance teams in the industry. With the continuous changes of business and technology architecture, the quality assurance work of the service end becomes more and more complex. In recent years, the traffic playback has taken root in the industry, but most of them are tools based and the cost of use and secondary development are relatively high. Conan platform emerged.

Objectives

Based on the recording and playback capability and result verification capability of online real user traffic, the solution is provided for smoke test, integrated regression test, online verification and online inspection.

Core functions

Flow collection

The traffic recording and acquisition based on ES log source, platform configuration access, low cost of use, and detailed traffic collection data are provided.

Traffic playback

The distributed back-end architecture improves the execution efficiency for traffic playback, supports service authentication configuration, and the playback based on HTTP protocol conforms to the real business scenario.

Result verification

The routine verification mode of traffic playback is mainly diff of traffic results, but a large number of traffic noise (time stamp, self increasing data...) has always affected the accuracy of the results. Conan platform performs the first layer verification based on the configured jsonschema in playback, and then carries out the second layer verification of traffic diff based on the self-developed noise reduction comparison service, thus ensuring the accuracy of the result verification , which greatly improves the reliability of the traffic playback results.

Platform advantages and application scenarios

Advantages

-Solve the problem of low coverage and high maintenance cost of traditional automation

-Multi rule traffic result Assertion Verification

-Support for multi rule traffic result comparison

-Flow data can be used for automated testing and performance testing

-Simple interaction, configuration access

-Open source co construction, continuous optimization

Application scenarios

-Lift and measure the quality card point

-CI/CD pipeline quality card point

-Service line monitoring and inspection

Platform business architecture

Business architecture

Platform technical architecture

Server architecture

Platform capability and function

Instructions for use

-Traffic collection: ES log;

-Playback protocol: http protocol;

-For specific environment, please refer to the open source detailed technical documents

##Write at the end

The improvement of quality and efficiency may not be achieved through a single platform, and the combination of technology and human can bring more breakthroughs. Being good at using technological innovation, we can calmly face more and more frequent demands and more complex business. The technical solution of Conan platform is produced by the class business of online school and gradually popularized. The platform is now open source, and hope more excellent people or teams will participate in it to provide more solutions for quality and efficiency assurance.

Detailed use of documents

https://dengkunnanmayun.gitee.io/conan-docs/#/use/README

More

https://mp.weixin.qq.com/s/1Cvi5kkqfF9y1rBi97qLwg

Project leader - Li Ning

Project member - Liu Jinsong huyaoguo dengkunnan Ji Ying

Official QQ group of Conan

conan's People

Contributors

Stargazers

Watchers

Forkers

dazzlingn sengeiou zhanghongzhi99 jyingge dengkunnan aqiang520 wujunze bestjex saviorghost funjackyone bzvs1992 geekcodingx dreamyeah g00176450 tuheishan sdgdsffdsfff ortontd xml2008 themycode luchao0111 dangkaio keyman9848 heliusa yzx-66 laulunsi huanzui kakukeme chensheng0619 ytong01 zhangke4042602 zuodh metertop wjygithub xiejiajun pactortester xzyxyz lzmrex mengtinghe huaying12345 datafactorys xujianming2017 xiaonaiping servicefoundation finley08 xiongxiaoqing0614 linuer laimf1995 chongchong01 xuzhitong89 lqnasa lihengjing youjia0721 yalehu jinke-joyce kakawaa zipengbo zgq346712481 yiding17 nanbole aaas19920513 yd955213 louis874 liuyongs1 zhancongc dystudio naonao1027 zhangping294279 vikinglee ifulltest aslion-li fllulu bluesky129 dizhaung chxiaowu bobyhub yzl2323g11 tayanzhuifeng lforigin liuqunqun215 solacq leeleez onehalfboy fpxtest liberjiang heming1990 ivory88 john1688 fanyongjun lol666666 willingsasi onlyone11 pingyan2019 muyimo sheldonfly yccock xdf020168 verminyy asciima faxianleo wangyuan168

conan's Issues

录制异常，异常code:Unable to parse response body

2022-01-27 10:24:39.615 [流量回放线程-8] INFO  c.t.w.conan.agent.service.impl.RecordServiceImpl - scroll_id=FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAA==
2022-01-27 10:24:39.615 [流量回放线程-8] INFO  c.t.w.conan.agent.service.impl.RecordServiceImpl - 查询到总流量0条
2022-01-27 10:24:39.615 [流量回放线程-8] INFO  c.t.w.conan.agent.service.impl.RecordServiceImpl - 查询到总流量 length0
ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [DELETE], host [http://10.33.40.22:9200], URI [/_search/scroll], status line [HTTP/1.1 404 Not Found]
{"succeeded":true,"num_freed":0}];
        at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1872)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626)
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
        at org.elasticsearch.client.RestHighLevelClient.clearScroll(RestHighLevelClient.java:1225)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl.getFlowByEsClientAndSaveFLow(RecordServiceImpl.java:379)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl.record(RecordServiceImpl.java:262)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl.startRecord(RecordServiceImpl.java:208)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl$$FastClassBySpringCGLIB$$12c31d87.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at com.alibaba.druid.support.spring.stat.DruidStatInterceptor.invoke(DruidStatInterceptor.java:73)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl$$EnhancerBySpringCGLIB$$c9e46460.startRecord(<generated>)
        at com.tal.wangxiao.conan.agent.config.KafkaConsumerListener.record(KafkaConsumerListener.java:99)
        at sun.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:774)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: ParsingException[Failed to parse object: expecting field with name [error] but found [succeeded]]
                at org.elasticsearch.common.xcontent.XContentParserUtils.ensureFieldName(XContentParserUtils.java:50)
                at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:592)
                at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:179)
                at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892)
                at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869)
                ... 31 more
Caused by: org.elasticsearch.client.ResponseException: method [DELETE], host [http://10.33.40.22:9200], URI [/_search/scroll], status line [HTTP/1.1 404 Not Found]
{"succeeded":true,"num_freed":0}
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
        ... 30 more
2022-01-27 10:24:39.621 [流量回放线程-8] ERROR c.t.w.conan.agent.config.KafkaConsumerListener - 录制异常，异常code:Unable to parse response body

启动ConanAdminApplication后报“Table 'conandb.bss_task_schedule' doesn't exist”

sql脚本里的表名和代码里的表名是不匹配的，比如脚本里的表是'diff_detail'，代码里的表是'bss_diff_detail'

录制不到数据

执行结果显示录制成功，域名查询条件也配置了，但是实际录制条数为0

针对域名的鉴权逻辑支持类型不足

当前的Web系统的认证类型较多，当前的系统仅仅支持采用 header里使用 Cookie的类型的。
比如我的产品使用了AKSK认证。是在header中增加了Authorization参数。且认证逻辑比较复杂。

[AKSK认证信息]参考 https://support.huaweicloud.com/devg-apisign/api-sign-provide-aksk.html

怎么接入阿里的sls日志服务呢？

ES 查询条件配置的 ”ES中_source内接口名称“ ，进行流量获取时过滤条件要求包含 http method 不大合理

特指此配置项：

不合理原因：
在录制时，com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl#getFlowByEsClientAndSaveFLow 和此参数有关的
过滤条件为 + 空格 + ：

BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery()
                .must(QueryBuilders.wildcardQuery(url + ".keyword", method + " " + requestQuery.getApi() + "*"))
                .must(shouldQuery);

但同样在录制逻辑中，实际入库完整 request 内容的 com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl#saveFlowInDb ，查询条件变为了纯 api path ，相关逻辑：

...
String apiKey = esConditionSetting.getApi();
...
String apiName = RegexUtils.getMsgByRegex(esFlowMap.get(apiKey) + "", esConditionSetting.getApiRegex());
...
List<Api> apiList = apiRepository.findByNameAndMethod(apiName, HttpMethodConstants.valueOf(method).getValue());

这样会导致如果没有配置 "接口正则表达式" ，且保证可以过滤掉 http method ，就会导致获取到的 apiName 为类似 GET /api/case/list 的值，导致无法匹配数据库中已定义的接口，进而出现录制是成功的（条数是对的），但实际没有任何数据入库到 bss_record_result 表（因为没有匹配到任何已定义的接口），没法回放

是否可以调整为，条数的查询条件里，去掉 http method 合并到 url 的做法，而是单独多开一个查询条件，专门查 http method ？本身 es 配置里也有要求配置单独对应 http method 的字段了，应该是可以适用的

怎么录制回放带有 queryString 的 get 接口？

在接口管理里配置的接口名称：/api/case/list

在域名管理中的 es 配置：

ES中_source内接口名称对应的原始值：GET /api/case/list?pageSize=10&pageNum=1&productLineId=1&caseType=0&channel=1&bizId=83b0bcf4

接口正则表达式：/.*(?=\?)

录制完毕后，回放时，提示如下错误：

2021-06-22 05:24:39 [INFO]  --开始执行回放, 回放ID：replay_id=12,task_execution_id=32,record_id=32\n2021-06-22 05:24:44 [INFO]   --回放url=http://127.0.0.1:8094/api/case/list,body=,response={code:10400,msg:Required Integer parameter 'channel' is not present,data:null}
2021-06-22 05:24:44 [INFO]   --回放url=http://127.0.0.1:8094/api/case/list,body=,response={code:10400,msg:Required Integer parameter 'channel' is not present,data:null}
...

错误原因为 url 里缺失了 queryString 部分。查看源码发现，回放时接口请求完整url拼接逻辑为：

com.tal.wangxiao.conan.agent.service.impl.ReplayServiceImpl#doReplay

...
Optional<Api> apiOptional = apiRepository.findById(apiId);
...
String domainName = domainOptional.get().getName();
            String method = EnumUtil.getByField(HttpMethodConstants.class, "getValue", String.valueOf(api.getMethod())).getLabel();
            String urlStr = "http://" + domainName + api.getName();
...
response = replayByHttp(urlStr, method, body, headerMap);
                log.info("回放url={},method={},body={},response={}", urlStr, method, body, response);

逻辑里面，url 用的是 api.getName() ，即数据库中的接口名称。这个名称是不会带有 queryString 的，所以导致回放时缺失这部分。

是我使用姿势不对吗？还是目前不支持带有 QueryString 的 Get 接口录制回放？