Can etcd cluster nodes have different timezones?

[Chinmaybhat388]

I am suspecting that the failover happened because of the different timezones. Following are my ETCD hosts :

etcdctl endpoint status --cluster -w table
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://172.29.43.189:2379 | 3d98445d3c61dfb1 | 3.5.4 | 20 kB | true | false | 11 | 160126301 | 160126301 | |
| http://172.26.39.29:2379 | 786357f730e66f90 | 3.5.4 | 20 kB | false | false | 11 | 160126301 | 160126301 | |
| http://172.26.39.28:2379 | 97201570c7b48a23 | 3.5.4 | 20 kB | false | false | 11 | 160126301 | 160126301 | |
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
Following is my patroni cluster :

• Cluster: lxpgskcdeprd-14.4-patroni ---+---------+-----------+-----+-----------+
  | Member | Host | Role | State | TL | Lag in MB |
  +-----------------+---------------------+---------+-----------+-----+-----------+
  | lxpgds01 | 172.29.43.189:50001 | Leader | running | 126 | |
  | lxpgds02 | 172.29.43.190:50001 | Replica | streaming | 126 | 0 |
  | lxpgdw01 | 172.26.39.28:50001 | Replica | streaming | 126 | 0 |
  | lxpgdw02 | 172.26.39.29:50001 | Replica | streaming | 126 | 0 |
  | lxpgdw03 | 172.26.39.30:50001 | Replica | streaming | 126 | 0 |
  +-----------------+---------------------+---------+-----------+-----+-----------+
  The patroni logs show the following during the failover :

2024-11-30 01:04:52,802 DEBUG: http://172.26.39.28:2379 "PUT /v2/keys/service/lxpgskcdeprd-14.4-patroni/leader HTTP/1.1" 412 109
2024-11-30 01:04:52,803 ERROR:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/patroni/dcs/etcd.py", line 523, in _run_and_handle_exceptions
return retry(method, *args, **kwargs) if retry else method(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/patroni/dcs/etcd.py", line 795, in _do_update_leader
prevValue=self._name, ttl=self._ttl) is not None
File "/usr/lib/python3.6/site-packages/patroni/dcs/etcd.py", line 495, in retry
return retry(method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/patroni/utils.py", line 589, in call
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/etcd/client.py", line 500, in write
response = self.api_execute(path, method, params=params)
File "/usr/lib/python3.6/site-packages/patroni/dcs/etcd.py", line 283, in api_execute
return self._handle_server_response(response)
File "/usr/lib/python3.6/site-packages/etcd/client.py", line 987, in _handle_server_response
etcd.EtcdError.handle(r)
File "/usr/lib/python3.6/site-packages/etcd/init.py", line 306, in handle
raise exc(msg, payload)
etcd.EtcdCompareFailed: Compare failed : [lxpgskcdeprdw01 != lxpgskcdeprds02]
2024-11-30 01:04:52,803 ERROR: failed to update leader lock
2024-11-30 01:04:52,803 INFO: Demoting self (immediate-nolock)
2024-11-30 01:04:52,811 WARNING: Exception happened during processing of request from 172.29.43.183:16310
2024-11-30 01:04:52,811 WARNING: Traceback (most recent call last):
File "/usr/lib64/python3.6/socketserver.py", line 654, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib64/python3.6/socketserver.py", line 364, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.6/site-packages/patroni/api.py", line 91, in init
super(RestApiHandler, self).init(request, client_address, server)
File "/usr/lib64/python3.6/socketserver.py", line 724, in init
self.handle()
File "/usr/lib64/python3.6/http/server.py", line 425, in handle
self.handle_one_request()
I am suspecting it is because of different timezones in the cluster nodes.

Address: 172.26.39.28

timedatectl
Local time: Sat 2024-11-30 02:25:05 EST
Universal time: Sat 2024-11-30 07:25:05 UTC
RTC time: Sat 2024-11-30 07:25:05
Time zone: America/New_York (EST, -0500)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
Address: 172.26.39.29

timedatectl
Local time: Sat 2024-11-30 02:26:08 EST
Universal time: Sat 2024-11-30 07:26:08 UTC
RTC time: Sat 2024-11-30 07:26:08
Time zone: America/New_York (EST, -0500)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
Address: 172.29.43.189

timedatectl
Local time: Sat 2024-11-30 01:26:51 CST
Universal time: Sat 2024-11-30 07:26:51 UTC
RTC time: Sat 2024-11-30 02:26:51
Time zone: America/Chicago (CST, -0600)
System clock synchronized: yes
NTP service: active
RTC in local TZ: yes

Warning: The system is configured to read the RTC time in the local time zone.
This mode cannot be fully supported. It will create various problems
with time zone changes and daylight saving time adjustments. The RTC
time is never updated, it relies on external facilities to maintain it.
If at all possible, use RTC in UTC by calling
'timedatectl set-local-rtc 0'.
I could not find much details on the web for "etcd.EtcdCompareFailed: Compare failed"