Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

adapter 报错，kafka中也看不到消费者 #5371

Open
dmanbu opened this issue Dec 30, 2024 · 0 comments
Open

adapter 报错，kafka中也看不到消费者 #5371

dmanbu opened this issue Dec 30, 2024 · 0 comments

Comments

@dmanbu
Copy link

dmanbu commented Dec 30, 2024 •
edited
Loading

环境

  • canal version: 1.1.7
  • mysql version: 5.7

adapter 日志

2024-12-30 17:11:24.183 [main-SendThread(10.10.0.101:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.10.0.101/10.10.0.101:2181. Will not attempt to authenticate using SASL (unknown error)
2024-12-30 17:11:24.183 [main-SendThread(10.10.0.101:2181)] WARN  org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 10090ms for sessionid 0x0
2024-12-30 17:11:24.183 [main-SendThread(10.10.0.101:2181)] INFO  org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 10090ms for sessionid 0x0, closing socket connection and attempting reconnect
2024-12-30 17:11:24.284 [main] ERROR org.apache.curator.framework.imps.CuratorFrameworkImpl - Ensure path threw exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /canal-adapter
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:2021)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:2049)
	at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:274)
	at org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:108)
	at org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:594)
	at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:151)
	at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:39)
	at com.alibaba.otter.canal.adapter.launcher.common.SyncSwitch.initMutex(SyncSwitch.java:104)
	at com.alibaba.otter.canal.adapter.launcher.common.SyncSwitch.init(SyncSwitch.java:53)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1778)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:602)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveBeanByName(AbstractAutowireCapableBeanFactory.java:461)
	at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.autowireResource(CommonAnnotationBeanPostProcessor.java:527)
	at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.getResource(CommonAnnotationBeanPostProcessor.java:497)
	at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor$ResourceElement.getResourceToInject(CommonAnnotationBeanPostProcessor.java:650)
	at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:228)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)
	at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessProperties(CommonAnnotationBeanPostProcessor.java:318)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1413)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:601)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:338)
	at com.alibaba.otter.canal.adapter.launcher.CanalAdapterApplication.main(CanalAdapterApplication.java:22)
2024-12-30 17:11:27.286 [main] WARN  org.apache.curator.ConnectionState - Connection attempt unsuccessful after 14372 (greater than max timeout of 6000). Resetting connection and trying again with a new connection.
2024-12-30 17:11:34.398 [main-SendThread(10.10.0.102:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.10.0.102/10.10.0.102:2181. Will not attempt to authenticate using SASL (unknown error)
2024-12-30 17:11:34.398 [main-SendThread(10.10.0.102:2181)] WARN  org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 10115ms for sessionid 0x0
2024-12-30 17:11:34.499 [main] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x0 closed
2024-12-30 17:11:34.499 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x0
2024-12-30 17:11:34.499 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=10.10.0.101:2181,10.10.0.102:2181,10.10.0.103:2181 sessionTimeout=6000 watcher=org.apache.curator.ConnectionState@7e1f584d
2024-12-30 17:11:34.499 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
2024-12-30 17:11:34.499 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
2024-12-30 17:11:44.587 [main-SendThread(10.10.0.102:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.10.0.102/10.10.0.102:2181. Will not attempt to authenticate using SASL (unknown error)
2024-12-30 17:11:44.587 [main-SendThread(10.10.0.102:2181)] WARN  org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 10088ms for sessionid 0x0
2024-12-30 17:11:44.587 [main-SendThread(10.10.0.102:2181)] INFO  org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 10088ms for sessionid 0x0, closing socket connection and attempting reconnect
2024-12-30 17:11:44.753 [main-SendThread(10.10.0.103:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.10.0.103/10.10.0.103:2181. Will not attempt to authenticate using SASL (unknown error)
2024-12-30 17:11:44.754 [main-SendThread(10.10.0.103:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.10.0.200:54580, server: 10.10.0.103/10.10.0.103:2181
2024-12-30 17:11:44.761 [main-SendThread(10.10.0.103:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server 10.10.0.103/10.10.0.103:2181, sessionid = 0x3072ccfdab7000f, negotiated timeout = 6000
2024-12-30 17:11:44.764 [main-EventThread] INFO  o.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2024-12-30 17:11:46.018 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
2024-12-30 17:11:46.028 [main] INFO  o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
2024-12-30 17:11:46.032 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## syncSwitch refreshed.
2024-12-30 17:11:46.033 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters.
2024-12-30 17:11:46.034 [main] INFO  c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: /opt/canal-adapter/plugin
2024-12-30 17:11:46.063 [main] INFO  c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Start loading rdb mapping config ... 
2024-12-30 17:11:46.071 [main] INFO  c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Rdb mapping config loaded
2024-12-30 17:11:46.257 [main] INFO  com.alibaba.druid.pool.DruidDataSource - {dataSource-2} inited
2024-12-30 17:11:46.261 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: rdb succeed
2024-12-30 17:11:46.267 [main] INFO  c.alibaba.otter.canal.connector.core.spi.ExtensionLoader - extension classpath dir: /opt/canal-adapter/plugin
2024-12-30 17:11:46.277 [main] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## something goes wrong when starting up the canal client adapters:
java.lang.NullPointerException: null
	at java.util.Hashtable.put(Hashtable.java:461)
	at com.alibaba.otter.canal.adapter.launcher.loader.AdapterProcessor.<init>(AdapterProcessor.java:72)
	at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.lambda$init$0(CanalAdapterLoader.java:85)
	at java.util.HashMap.computeIfAbsent(HashMap.java:1128)
	at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.init(CanalAdapterLoader.java:81)
	at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterService.init(CanalAdapterService.java:60)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1778)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:602)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:374)
	at org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:376)
	at org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:179)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:371)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1154)
	at org.springframework.cloud.context.scope.refresh.RefreshScope.eagerlyInitialize(RefreshScope.java:125)
	at org.springframework.cloud.context.scope.refresh.RefreshScope.start(RefreshScope.java:117)
	at org.springframework.cloud.context.scope.refresh.RefreshScope.onApplicationEvent(RefreshScope.java:112)
	at org.springframework.cloud.context.scope.refresh.RefreshScope.onApplicationEvent(RefreshScope.java:67)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143)
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421)
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378)
	at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:938)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:338)
	at com.alibaba.otter.canal.adapter.launcher.CanalAdapterApplication.main(CanalAdapterApplication.java:22)
2024-12-30 17:11:46.284 [main] INFO  c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 85.62 seconds (JVM running for 86.242)

adapter 配置

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: kafka
  flatMessage: true
  zookeeperHosts: 10.10.0.101:2181,10.10.0.102:2181,10.10.0.103:2181
  batchSize: 500
  syncBatchSize: 1000
  retries: -1
  timeout: 3000
  consumerProperties:
    # kafka consumer
    kafka.bootstrap.servers: 10.10.0.101:9092
    kafka.enable.auto.commit: false
    kafka.auto.commit.interval.ms: 1000
    kafka.auto.offset.reset: latest
    kafka.request.timeout.ms: 40000
    kafka.session.timeout.ms: 30000
    kafka.isolation.level: read_committed
    kafka.max.poll.records: 1000
  srcDataSources:
    defaultDS:
      url: jdbc:mysql://127.0.0.1:3307/canal_testing_01?useUnicode=true&characterEncoding=utf8&autoReconnect=true&useSSL=false
      username: root
      password: __root_passwd__
  canalAdapters:
  - instance: dev-mysql57-01
    groups:
    - groupId: g01
      outerAdapters:
      - name: rdb
        key: mysql_01
        properties:
          jdbc.driverClassName: com.mysql.jdbc.Driver
          jdbc.url: jdbc:mysql://127.0.0.1:3307/canal_testing_02?useUnicode=true&characterEncoding=utf8&autoReconnect=true&useSSL=false
          jdbc.username: root
          jdbc.password: __root_passwd__

Kafka 无法消费

image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@dmanbu