Skip to content

Fix DedicatedClusterSnapshotRestoreIT.testSnapshotWithStuckNode that is failing with reproducible seed #17389

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 9 commits into from
Closed

Fix DedicatedClusterSnapshotRestoreIT.testSnapshotWithStuckNode that is failing with reproducible seed #17389

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
de2f7a2
Fix DedicatedClusterSnapshotRestoreIT.testSnapshotWithStuckNode that …
cwperks Feb 19, 2025
4c0302c
Merge branch 'main' into fix-dedicated-cluster
cwperks Feb 26, 2025
fe9d1e6
Ensure that generation is non-null when snapshot is aborted to assure…
cwperks Feb 28, 2025
aa260e6
Merge branch 'main' into fix-dedicated-cluster
cwperks Feb 28, 2025
96d866c
Merge branch 'main' into fix-dedicated-cluster
cwperks Feb 28, 2025
d123f8c
Merge branch 'main' into fix-dedicated-cluster
cwperks Feb 28, 2025
30f2b90
Merge branch 'main' into fix-dedicated-cluster
cwperks Mar 17, 2025
110f901
Merge branch 'main' into fix-dedicated-cluster
cwperks Apr 17, 2025
5af80bd
Re-use method
cwperks Apr 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions server/src/main/java/org/opensearch/snapshots/SnapshotShardsService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ private void startNewSnapshots(SnapshotsInProgress snapshotsInProgress) {
// due to CS batching we might have missed the INIT state and straight went into ABORTED
// notify cluster-manager that abort has completed by moving to FAILED
if (shard.getValue().state() == ShardState.ABORTED && localNodeId.equals(shard.getValue().nodeId())) {
notifyFailedSnapshotShard(snapshot, shard.getKey(), shard.getValue().reason());
notifyFailedSnapshotShard(snapshot, shard.getKey(), shard.getValue().reason(), shard.getValue().generation());
}
} else {
snapshotStatus.abortIfNotCompleted("snapshot has been aborted");
Expand Down Expand Up @@ -314,7 +314,7 @@ public void onFailure(Exception e) {
logger.warn(() -> new ParameterizedMessage("[{}][{}] failed to snapshot shard", shardId, snapshot), e);
}
snapshotStatus.moveToFailed(threadPool.absoluteTimeInMillis(), failure);
notifyFailedSnapshotShard(snapshot, shardId, failure);
notifyFailedSnapshotShard(snapshot, shardId, failure, snapshotStatus.generation());
}
}
);
Expand Down Expand Up @@ -594,10 +594,15 @@ private void notifySuccessfulSnapshotShard(final Snapshot snapshot, final ShardI

/** Notify the cluster-manager node that the given shard failed to be snapshotted **/
private void notifyFailedSnapshotShard(final Snapshot snapshot, final ShardId shardId, final String failure) {
notifyFailedSnapshotShard(snapshot, shardId, failure, null);
}

/** Notify the cluster-manager node that the given shard failed to be snapshotted **/
private void notifyFailedSnapshotShard(final Snapshot snapshot, final ShardId shardId, final String failure, String generation) {
sendSnapshotShardUpdate(
snapshot,
shardId,
new ShardSnapshotStatus(clusterService.localNode().getId(), ShardState.FAILED, failure, null)
new ShardSnapshotStatus(clusterService.localNode().getId(), ShardState.FAILED, failure, generation)
);
}

Expand Down
Loading