tal-tech / conan Goto Github PK

View Code? Open in Web Editor NEW

760.0 27.0 164.0 2.08 MB

柯南平台开源版本，为用户提供流量回放全流程解决方案

License: MIT License

Java 57.80% JavaScript 6.04% Batchfile 0.07% HTML 0.24% Vue 27.84% SCSS 0.80% PLpgSQL 7.04% Shell 0.17%

traffic-playback traffic-collection conan-platform multi-rule-traffic

conan's Issues

文档地址打不开了

怎么接入阿里的sls日志服务呢？

怎么录制回放带有 queryString 的 get 接口？

在接口管理里配置的接口名称：/api/case/list

在域名管理中的 es 配置：

ES中_source内接口名称对应的原始值：GET /api/case/list?pageSize=10&pageNum=1&productLineId=1&caseType=0&channel=1&bizId=83b0bcf4

接口正则表达式：/.*(?=\?)

录制完毕后，回放时，提示如下错误：

2021-06-22 05:24:39 [INFO]  --开始执行回放, 回放ID：replay_id=12,task_execution_id=32,record_id=32\n2021-06-22 05:24:44 [INFO]   --回放url=http://127.0.0.1:8094/api/case/list,body=,response={code:10400,msg:Required Integer parameter 'channel' is not present,data:null}
2021-06-22 05:24:44 [INFO]   --回放url=http://127.0.0.1:8094/api/case/list,body=,response={code:10400,msg:Required Integer parameter 'channel' is not present,data:null}
...

错误原因为 url 里缺失了 queryString 部分。查看源码发现，回放时接口请求完整url拼接逻辑为：

com.tal.wangxiao.conan.agent.service.impl.ReplayServiceImpl#doReplay

...
Optional<Api> apiOptional = apiRepository.findById(apiId);
...
String domainName = domainOptional.get().getName();
            String method = EnumUtil.getByField(HttpMethodConstants.class, "getValue", String.valueOf(api.getMethod())).getLabel();
            String urlStr = "http://" + domainName + api.getName();
...
response = replayByHttp(urlStr, method, body, headerMap);
                log.info("回放url={},method={},body={},response={}", urlStr, method, body, response);

逻辑里面，url 用的是 api.getName() ，即数据库中的接口名称。这个名称是不会带有 queryString 的，所以导致回放时缺失这部分。

是我使用姿势不对吗？还是目前不支持带有 QueryString 的 Get 接口录制回放？

启动ConanAdminApplication后报“Table 'conandb.bss_task_schedule' doesn't exist”

sql脚本里的表名和代码里的表名是不匹配的，比如脚本里的表是'diff_detail'，代码里的表是'bss_diff_detail'

针对域名的鉴权逻辑支持类型不足

当前的Web系统的认证类型较多，当前的系统仅仅支持采用 header里使用 Cookie的类型的。
比如我的产品使用了AKSK认证。是在header中增加了Authorization参数。且认证逻辑比较复杂。

[AKSK认证信息]参考 https://support.huaweicloud.com/devg-apisign/api-sign-provide-aksk.html

录制异常，异常code:Unable to parse response body

2022-01-27 10:24:39.615 [流量回放线程-8] INFO  c.t.w.conan.agent.service.impl.RecordServiceImpl - scroll_id=FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAA==
2022-01-27 10:24:39.615 [流量回放线程-8] INFO  c.t.w.conan.agent.service.impl.RecordServiceImpl - 查询到总流量0条
2022-01-27 10:24:39.615 [流量回放线程-8] INFO  c.t.w.conan.agent.service.impl.RecordServiceImpl - 查询到总流量 length0
ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [DELETE], host [http://10.33.40.22:9200], URI [/_search/scroll], status line [HTTP/1.1 404 Not Found]
{"succeeded":true,"num_freed":0}];
        at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1872)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626)
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
        at org.elasticsearch.client.RestHighLevelClient.clearScroll(RestHighLevelClient.java:1225)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl.getFlowByEsClientAndSaveFLow(RecordServiceImpl.java:379)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl.record(RecordServiceImpl.java:262)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl.startRecord(RecordServiceImpl.java:208)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl$$FastClassBySpringCGLIB$$12c31d87.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at com.alibaba.druid.support.spring.stat.DruidStatInterceptor.invoke(DruidStatInterceptor.java:73)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl$$EnhancerBySpringCGLIB$$c9e46460.startRecord(<generated>)
        at com.tal.wangxiao.conan.agent.config.KafkaConsumerListener.record(KafkaConsumerListener.java:99)
        at sun.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:774)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: ParsingException[Failed to parse object: expecting field with name [error] but found [succeeded]]
                at org.elasticsearch.common.xcontent.XContentParserUtils.ensureFieldName(XContentParserUtils.java:50)
                at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:592)
                at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:179)
                at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892)
                at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869)
                ... 31 more
Caused by: org.elasticsearch.client.ResponseException: method [DELETE], host [http://10.33.40.22:9200], URI [/_search/scroll], status line [HTTP/1.1 404 Not Found]
{"succeeded":true,"num_freed":0}
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
        ... 30 more
2022-01-27 10:24:39.621 [流量回放线程-8] ERROR c.t.w.conan.agent.config.KafkaConsumerListener - 录制异常，异常code:Unable to parse response body

录制不到数据

执行结果显示录制成功，域名查询条件也配置了，但是实际录制条数为0

ES 查询条件配置的 ”ES中_source内接口名称“ ，进行流量获取时过滤条件要求包含 http method 不大合理

特指此配置项：

不合理原因：
在录制时，com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl#getFlowByEsClientAndSaveFLow 和此参数有关的
过滤条件为 + 空格 + ：

BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery()
                .must(QueryBuilders.wildcardQuery(url + ".keyword", method + " " + requestQuery.getApi() + "*"))
                .must(shouldQuery);

但同样在录制逻辑中，实际入库完整 request 内容的 com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl#saveFlowInDb ，查询条件变为了纯 api path ，相关逻辑：

...
String apiKey = esConditionSetting.getApi();
...
String apiName = RegexUtils.getMsgByRegex(esFlowMap.get(apiKey) + "", esConditionSetting.getApiRegex());
...
List<Api> apiList = apiRepository.findByNameAndMethod(apiName, HttpMethodConstants.valueOf(method).getValue());

这样会导致如果没有配置 "接口正则表达式" ，且保证可以过滤掉 http method ，就会导致获取到的 apiName 为类似 GET /api/case/list 的值，导致无法匹配数据库中已定义的接口，进而出现录制是成功的（条数是对的），但实际没有任何数据入库到 bss_record_result 表（因为没有匹配到任何已定义的接口），没法回放

是否可以调整为，条数的查询条件里，去掉 http method 合并到 url 的做法，而是单独多开一个查询条件，专门查 http method ？本身 es 配置里也有要求配置单独对应 http method 的字段了，应该是可以适用的

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.