tapdata / tapdata Goto Github PK

Please use emoji to up vote or down vote this feature.

这是内容

在windows环境或者ide中如何运行，有没有开源web端部分功能的计划呢？

希望可以支持做目标, 这个很常用

我在连接管理里面测试源数据库的连接情况，显示没有任何问题，目标数据库测试也没问题，但在执行同步任务的时候连接失败
Connect to source mysql failed, config: {snapshot.locking.mode=none, restRetryTime=3, connector.class=io.debezium.connector.mysql.MySqlConnector, dataFlow.id=634d7038c0c94167270a2910, max.queue.size=8000, job.id=634d703bc162f518b15885f2, table.whitelist=demo.student2,demo.student1, database.history.sourceConnId=6345927d8eb5d14b4f74047e, accessCode=1e611290a0134b4dacd4da56bdffcb8af8aaf012f6aa4a0da887b066db8346c3, poll.interval.ms=500, database.history.baseURL=https://cloud.tapdata.net/console/tm/api/, database.history.skip.unparseable.ddl=true, job.name=测试_1, database.whitelist=demo, database.history.job.id=634d703bc162f518b15885f2, binlogStartPoint=true, database.user=test, database.binlog.position={"filename":"master1-bin.000001","position":2335,"gtidSet":null}, database.history.dataFlowId=634d7038c0c94167270a2910, offset.storage=io.tapdata.common.MongoOffsetBackingStore, database.server.id=1181781442, roleId=0, database.server.name=634d703bc162f518b15885f2, userId=63458e7ec162f518b171e383, database.port=3307, database.history.roleId=0, threadName=Connector runner-测试_1-[634d703bc162f518b15885f2], offset.flush.interval.ms=60000, database.history.accessCode=1e611290a0134b4dacd4da56bdffcb8af8aaf012f6aa4a0da887b066db8346c3, baseURL=https://cloud.tapdata.net/console/tm/api/, database.hostname=my ip addr, database.password=********, database.history.userId=63458e7ec162f518b171e383, name=634d703bc162f518b15885f2, database.history.store.only.monitored.tables.ddl=true, max.batch.size=1000, database.history.restRetryTime=3, database.history=io.tapdata.common.MongoDatabaseHistoryBackingStore, snapshot.mode=schema_only}, err: Error reading MySQL variables: Access denied for user 'test'@'my ip addr' (using password: YES), stacks: org.apache.kafka.connect.errors.ConnectException: Error reading MySQL variables: Access denied for user 'test'@'my ip addr' (using password: YES) at io.debezium.connector.mysql.MySqlJdbcContext.querySystemVariables(MySqlJdbcContext.java:329) at io.debezium.connector.mysql.MySqlJdbcContext.readMySqlSystemVariables(MySqlJdbcContext.java:309) at io.debezium.connector.mysql.MySqlTaskContext.<init>(MySqlTaskContext.java:81) at io.debezium.connector.mysql.MySqlTaskContext.<init>(MySqlTaskContext.java:54) at io.debezium.connector.mysql.MySqlConnectorTask.createAndStartTaskContext(MySqlConnectorTask.java:428) at io.debezium.connector.mysql.MySqlConnectorTask.initTaskContext(MySqlConnectorTask.java:359) at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:169) at io.debezium.tapdata.connector.common.BaseSourceTask.start(BaseSourceTask.java:105) at io.debezium.tapdata.embedded.EmbeddedEngine.run(EmbeddedEngine.java:815) at io.tapdata.manager.ConnectorJobManager.lambda$createMySqlEngine$5(ConnectorJobManager.java:921) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.sql.SQLException: Access denied for user 'test'@'my ip addr' (using password: YES) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:833) at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:453) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198) at com.tapdata.constant.JdbcUtil.retryByMaxConnect(JdbcUtil.java:2083) at io.debezium.tapdata.jdbc.JdbcConnection.lambda$patternBasedFactory$1(JdbcConnection.java:174) at io.debezium.tapdata.jdbc.JdbcConnection.connection(JdbcConnection.java:786) at io.debezium.tapdata.jdbc.JdbcConnection.connection(JdbcConnection.java:781) at io.debezium.tapdata.jdbc.JdbcConnection.connect(JdbcConnection.java:320) at io.debezium.connector.mysql.MySqlJdbcContext.querySystemVariables(MySqlJdbcContext.java:316) ... 12 more

Hope to add the function of synchronizing data to hive,Support Kerberos authentication。
hive 1.1.0
cdh 5.16.1

在Oracle转换数据到MySQL时，MySQL不存在相应的数据表时创建数据表因为没有相应的编码导致创建失败。
[ERROR] 2022-08-02 14:17:28 [pool-34-thread-5] io.tapdata.manager.TransformerJobManager
Failed to create jdbc table [***], sql: CREATE TABLE
...省略
default charset=ZHS16GBK , cause: Unknown character set: 'ZHS16GBK'; Will stop job
是否可以设置数据库的编码格式，或是对数据库的编码格式进行查询，对数据转换自动进行编码格式的匹配。

Implement TDEngine source and target

能否实现API以及WebService类型数据的同步，涉及用户的登录，数据循环读取等

金蝶精斗云

cli.py发起的http request（如：DataSource的save函数）后，manager工程中响应的对象为：com.tapdata.tm.base.dto.ResponseMessage。其代表error message的属性为message。但cli.py中使用的是msg，导致http请求出现error的时候，在shell中不能正确的显示error message。

# 如cli.py line 3169
res.json()["msg"] 修改为 res.json()["message"]

com.tapdata.tm.base.dto.ResponseMessage

public class ResponseMessage<T>{

	public static final String OK = "ok";

	private String reqId;

	private long ts = new Date().getTime();

	/**
	 * 请求处理的代码
	 */
	protected String code = OK;

	/**
	 * 请求处理失败时的错误消息
	 */
	protected String message;

	/**
	 * 请求处理成功的数据
	 */
	protected T data;

tapshell cli.py

@help_decorate("save a connection in idaas system")
    def save(self):
        api = system_server_conf["api"] + "/Connections" + system_server_conf["auth_param"]
        data = self.to_dict()
        res = requests.post(api, json=data)
        show_connections(quiet=True)
        if res.status_code == 200 and res.json()["code"] == "ok":
            self.id = res.json()["data"]["id"]
            self.c = DataSource.get(self.id)
            self.validate(quiet=False)
            return True
        else:
            logger.warn("save Connection fail, err is: {}", res.json())
            logger.warn("save Connection fail, err is: {}", res.json()["msg"])
        return False

Does it support kafka connection ?

我们有这种场景，需要将线上clickhouse数据同步到另外一个clickhouse数据，希望可以支持一下！！！感谢

In current version, udf can not change schema, changing schema in udf is very common and useful, eg, add a last_update timestamp in processor, target table should auto get it and create table with last_update column

xxx

refactor:

• log module
• request module
• param verify, include check.py(param check class) and rules.py(define param rule)

fix:

• pipeline.config

test:

• add end2end test module (tapdata/tapshell/test)

Currently to get into tapshell prompt, the only way is to run "bash build/quick-use.sh".

tapshell is the primary user interface, we should provide a dedicated command to start/launch tapshell

Provide a directly downloadable binary for non-docker users

Date formats are not synchronized. It is recommended to create separate format instances for each thread. If multiple threads access a format concurrently, it must be synchronized externally.

希望能够在 spec_clickhouse.json 中支持 clickhouse 的 date 类型的转换。感谢

You may Up or Down Vote this feature here.

如题

建议 ConnectorBase 优化接口形式

1. onStart， onStop， 增加数据源类型（源 or 目标），方便根据不同类型做不同初始化动作。onStart(TapConnectionContext connectionContext, DataSourceType type)
2. Consumer 类改为符合语义的自定义回调接口（临时兼容方案），比如 connectionTest方法可以改为 ConnectResult.submitResult

The original intention of Tapdata community is to grow together with developers and create unlimited possibilities. Here, we can speak freely and help each other solve problems.

In the world of hackers, when you throw out a technical question, whether you can finally get a useful answer often depends on the way you ask and ask. The wisdom of asking questions teaches you how to ask questions correctly to get your satisfactory answers. Here are some excerpts to share with you.

【Before asking questions】
Before you are ready to ask technical questions through groups or forums, please try to search the answers by various means, and try not to often ask some questions that can be solved by search. Including but not limited to:

• Tapdata Technical Manual
• Tapdata GitHub problem search
• Search on Baidu, Google and related technology communities

If the above behavior still does not solve your problem, please indicate that you have made the above efforts when asking others for advice and questions; This will help establish that you are not a questioner who gets something for nothing and wastes other people's time. Asking questions or initiating discussions on relevant technical topics in the community is encouraged, but asking questions is also an art that needs to be grasped by the questioner.

【When asking questions】

1. Maintain a friendly and polite attitude
   No one is obliged to answer your question, especially the answer to this question may take the other party's time / energy, so we need to be polite when asking questions. It is suggested that when asking questions, we first introduce ourselves and politely express our problems. No matter whether the problems are really solved or not, we should thank each other to show our proper bearing and demeanor.
2. When asking questions, try to describe your questions in detail, clearly, correctly and concisely; The person being asked will be more likely to give a more comprehensive answer to your question.
3. Try to use specific words and sentences that others can understand when asking questions.

【after questioning】

1. Express gratitude to the responders.
2. If your problem has not been solved, please continue to ask and feed back the problems you encounter. It is best to attach operation tips or screenshots.

【Say no to the following actions】
In fact, participating in community interaction is also a social activity, which requires individuals to grasp social propriety and abide by basic social etiquette. Unacceptable participant behaviors include but are not limited to:

1. Discussion issues rise to personal attacks, provocative, insulting or derogatory comments;
2. Public or private harassment;
3. Release private information of others without permission;
4. Publish advertisements or other bad information without permission;
5. Brush the screen and post without reason, thus occupying public resources, etc.

Reference source for community code development:
Original website of the wisdom of asking questions: http://www.catb.org/~esr/faqs/smart-questions.html
The English version is copyrighted by Eric S. Raymond, Rick Moen.

我使用tapdata cloud进行从mysql到sqlserver的全量加增量的数据同步
在全量同步完成后，在增量同步的过程中，我去本地的mysql添加了一行数据，但是在sqlserver中没有相应的行的添加（附件是运行日志，从22点9分左右到22点21分多的运行日志，我是在22点13分左右在MySQL中添加了一行，日志.txt）
里程碑是这样显示的（我怀疑是可能是因为增量读取模式在22点10分就结束了，所以增量传输不了？）：

全量是正常的：

所以我搞不清我这些操作问题出在了那里，该怎样解决？求解答

Github docker pull too slow, suggest use Docker Hub or add a local mirror(Aliyun)

目前只能通过shell来启动tapdata任务，后续web界面会开源吗

这是内容

[PDK TDD] add QueryByFilterTest

support doris as source/sink

TDEngine 很常用, 希望支持做目标