[server/kv]fix bug for deleting remote kv dir #245
Conversation
Vipamp
changed the title
fluss-server/src/main/java/com/alibaba/fluss/server/kv/KvManager.java
Outdated
Show resolved
Hide resolved
fluss-server/src/main/java/com/alibaba/fluss/server/kv/KvManager.java
Outdated
Show resolved
Hide resolved
| KvTablet dropKvTablet = | ||
| inLock(tabletCreationOrDeletionLock, () -> currentKvs.remove(tableBucket)); | ||
|
|
||
| if (dropKvTablet != null) { | ||
| TablePath tablePath = dropKvTablet.getTablePath(); | ||
| try { | ||
| dropKvTablet.drop(); | ||
| if (deleteRemote) { | ||
| dropRemoteKvSnapshot(physicalTablePath, tableBucket); |
There was a problem hiding this comment.
This removal is very similar to remote log removal. We can move this invoking KvManager.dropRemoteKvSnapshot into ReplicaManager#stopReplica after remoteLogManager.stopReplica. Ideally, we should delete the remote files (log&kv) asynchronously to not block the worker thread. We can refactor them in the near future.
There was a problem hiding this comment.
Yes,i have notice that，I find RemoteLogManager to manager remote files of LogTable without RemoteKvManager to manager snapshot files of KvTable，is there any plan to unify those for managering remote files? I will continue to pay attention about it and look forward to participating it.
Thank you for your guidance，I will add it. |
|
@wuchong PTAL，thinks |
| + "skip this one and schedule the next one in {} seconds", | ||
| tableBucket, | ||
| periodicSnapshotDelay / 1000); | ||
| if (started && !periodicExecutor.isShutdown()) { |
There was a problem hiding this comment.
Why add if (started && !periodicExecutor.isShutdown()) in here?
There was a problem hiding this comment.
For periodicExecutor summitting snapshot trigger with delay，after bucket deleted and started change to false, the snapshot trigger will execute again for last time and generate an empty snapshot, the remote bucket dir will be created again，so i add this code for avoiding this case.
There was a problem hiding this comment.
Thanks for the explanation...Make sense to me..But we won't need to check !periodicExecutor.isShutdown(), it periodicExecutor.isShutdown(), the task won't be scheduled..
| @@ -1239,6 +1239,10 @@ private StopReplicaResultForBucket stopReplica( | |||
| } | |||
|
|
|||
| remoteLogManager.stopReplica(replicaToDelete, delete && replicaToDelete.isLeader()); | |||
| if (delete) { | |||
There was a problem hiding this comment.
Only replica leader need to drop remote kv snapshot....
=> if (delete && replicaToDelete.isLeader())
| @@ -85,6 +93,7 @@ void testKvSnapshot() throws Exception { | |||
| } | |||
| } | |||
|
|
|||
| Set<File> bucketDirs = new HashSet<>(); | |||
There was a problem hiding this comment.
nit: bucketDirs -> bucketKvSnapshotDirs?
| @@ -270,4 +281,17 @@ public KvTablet loadKv(File tabletDir) throws Exception { | |||
| this.currentKvs.put(tableBucket, kvTablet); | |||
| return kvTablet; | |||
| } | |||
|
|
|||
| public void dropRemoteKvSnapshot(PhysicalTablePath physicalTablePath, TableBucket tableBucket) { | |||
There was a problem hiding this comment.
nit:
dropRemoteKvSnapshot -> deleteRemoteKvSnapshot
as we also use delete for log....
Purpose
Linked issue: close #121
this pr is aims to solve the bug mentioned in #121
Tests
API and Format
Documentation