Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

canal同步数据到doris,主键只能是叫“id”吗?主键不叫“id”回报错 #5354

Open
2 tasks
lyj-refuel opened this issue Dec 14, 2024 · 1 comment

Comments

@lyj-refuel
Copy link

  • I have searched the issues of this repository and believe that this is not a duplicate.
  • I have checked the FAQ of this repository and believe that this is not a duplicate.

environment

  • canal version 1.1.8
  • mysql version 5.5
    doris version 2.1.7

Issue Description

canal-adapter同步到doris表,binlog日志有,测试只支持主键是“id”表?主键不叫“id”回报错:java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Target column: id not matched

主键是“id”
dbMapping:
database: v3-all
table: ebay_description
targetTable: ebay_description
targetPk:
id: id
mapAll: true

日志
2024-12-14 15:07:28.090 [pool-4-thread-1] DEBUG c.a.o.canal.client.adapter.rdb.service.RdbSyncService - DML: {"data":{"id":6175712,"sku":"YD648151_52","dcid":"","smtid":"","lazadaCateid":"","shopeeCateid":"","description":"","textDescription":"","shortDescription":"","packageContent":"","notes":"","type":1,"status":1,"page":1},"database":"v3-all","destination":"example","old":null,"table":"ebay_description","type":"INSERT"}

正常写入doris

主键不是“id”
dbMapping:
database: v3-all
table: ebay_order
targetDatabase: test
targetTable: ebay_order
targetPk:
id: ebay_id
mapAll: true

日志
2024-12-14 14:56:53.766 [pool-8-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"ebay_id":392595015,"ebay_ordersn":"cn1537622598zyjp5486052111905078","ebay_orderqk":null,"ebay_paystatus":"0","recordnumber":"5486052111905078","ebay_tid":"5486052111905078","ebay_ptid":"","ebay_orderid":"","ebay_createdtime":0,"ebay_paidtime":1734141492,"ebay_userid":"Кошевой Дмитрий Павлович","ebay_username":"Кошевой Дмитрий Павлович","ebay_usermail":"","ebay_street":"г. Благовещенск, ул. 50 лет Октября, 15","ebay_street1":"г. Благовещенск, ул. 50 лет Октября, 15","ebay_city":"Благовещенск г","ebay_state":"Амурская обл","ebay_couny":"RU","ebay_countryname":"俄罗斯","ebay_postcode":"675004","ebay_phone":"9638141314","ebay_currency":"USD","ebay_total":1.7,"ebay_status":1723,"ebay_user":"system","ebay_addtime":1734145359,"ebay_shipfee":"0.790000","ebay_combine":"0","market":"韦建秋","ebay_account":"cn1537622598zyjp","ebay_note":"","ebay_noteb":":2025-02-27 11:02:39","is_reg":0,"ordertype":"aliexpress","status":"0","mailstatus":0,"templateid":"0","postive":"0","ebay_carrier":"速卖通线上无忧-简易带电","ebay_carrierstyle":"","ebay_warehouse":"234","ebay_markettime":0,"ebay_tracknumber":"","ebay_site":"","location":null,"eBayPaymentStatus":null,"PayPalEmailAddress":"","ShippedTime":"0","RefundAmount":0.0,"resendreason":null,"refundreason":null,"resendtime":null,"refundtime":null,"canceltime":null,"cancelreason":null,"ebay_feedback":0,"ebay_sdsn":null,"isprint":0,"ebay_ordertype":"0","profitstatus":0,"orderweight":0.024,"orderweight2":0.0,"ordershipfee":0.0,"ordercopst":1.0,"scantime":0,"accountid":22196,"ishide":1822,"packingtype":"0","packinguser":null,"packagingstaff":null,"order_no":"0","ebay_phone1":"+7-9638141314","ismodifive":0,"totalprofit":0.0,"ebay_case":0,"paypal_case":0,"moneyback":0,"moneyback_total":0.0,"orders_pcase_time":0,"orders_ecase_time":-1,"erp_op_id":1,"Custom_Paypalemailaddress":"{"firmpack":0,"firmpack_assist":0}","pxorderid":"","pxordertime":null,"cky_orderid":null,"cky_item":null,"ebay_company":null,"zencartid":"ru1763105224yyxae","updateprofittime":0,"smturl":null}],"database":"v3-all","destination":"example","es":1734147071000,"groupId":"g1","isDdl":false,"old":[{"ebay_status":1722}],"pkNames":["ebay_id"],"sql":"","table":"ebay_order","ts":1734159413763,"type":"UPDATE"}
2024-12-14 14:56:53.771 [pool-8-thread-1] ERROR c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Target column: id not matched
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Target column: id not matched

不能正常写入doris

Steps to reproduce

Expected behaviour

Actual behaviour

If there is an exception, please attach the exception trace:

Just put your stack trace here!
@lyj-refuel
Copy link
Author

修改了部分代码后可以正常的update和delete
代码位置:
client-adapter
rdb
package com.alibaba.otter.canal.client.adapter.rdb.service;
RdbSyncService.java
private void appendCondition(MappingConfig.DbMapping dbMapping, StringBuilder sql, Map<String, Integer> ctype,
List<Map<String, ?>> values, Map<String, Object> d, Map<String, Object> o) {
String backtick = SyncUtil.getBacktickByDbType(dataSource.getDbType());
// 拼接主键
for (Map.Entry<String, String> entry : dbMapping.getTargetPk().entrySet()) {
String targetColumnName = entry.getKey();
String srcColumnName = entry.getValue();
if (srcColumnName == null) {
srcColumnName = Util.cleanColumn(targetColumnName);
}
// sql.append(backtick).append(targetColumnName).append(backtick).append("=? AND ");
// Integer type = ctype.get(Util.cleanColumn(targetColumnName).toLowerCase());
sql.append(backtick).append(srcColumnName).append(backtick).append("=? AND ");
Integer type = ctype.get(Util.cleanColumn(srcColumnName).toLowerCase());
if (type == null) {
// throw new RuntimeException(" Target column: " + targetColumnName + " not matched");
throw new RuntimeException(" Target column: " + srcColumnName + " not matched");
}
// 如果有修改主键的情况
if (o != null && o.containsKey(srcColumnName)) {
BatchExecutor.setValue(values, type, o.get(srcColumnName));
} else {
BatchExecutor.setValue(values, type, d.get(srcColumnName));
}
}
int len = sql.length();
sql.delete(len - 4, len);
}

方法尽提供参考,需要自己验证

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant