Implemented computation of segment replication stats at shard level #17055
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The method implemented here computes the segment replication stats at the shard level, instead of relying on the primary shard to compute stats based on reports from its replicas. Signed-off-by: Vinay Krishna Pudyodu <[email protected]>
vinaykpud
requested review from
anasalkouz,
andrross,
ashking94,
Bukhtawar,
CEHENKLE,
dblock,
dbwiddis,
gbbafna,
jed326,
kotwanikunal,
mch2,
msfroh,
nknize,
owaiskazi19,
reta,
Rishikesh1159,
sachinpkale,
saratvemulapalli,
shwetathareja,
sohami,
VachaShah,
jainankitk and
linuxpi
as code owners
github-actions
bot
added
enhancement
Signed-off-by: Vinay Krishna Pudyodu <[email protected]>
Signed-off-by: Vinay Krishna Pudyodu <[email protected]>
Signed-off-by: Vinay Krishna Pudyodu <[email protected]>
Signed-off-by: Vinay Krishna Pudyodu <[email protected]>
Signed-off-by: Vinay Krishna Pudyodu <[email protected]>
|
❌ Gradle check result for 04ba008: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Vinay Krishna Pudyodu <[email protected]>
mch2
reviewed
Feb 26, 2025
There was a problem hiding this comment.
@vinaykpud Thanks for pushing on this, I think this is really close.
server/src/main/java/org/opensearch/indices/replication/SegmentReplicator.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/opensearch/indices/replication/SegmentReplicatorTests.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/indices/replication/SegmentReplicator.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/indices/replication/SegmentReplicator.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Vinay Krishna Pudyodu <[email protected]>
vinaykpud
force-pushed
the
node-stats
branch
from
7ff35b8 to
d89fdce
Compare
vinaykpud
force-pushed
the
node-stats
branch
from
d89fdce to
fcc57bf
Compare
Signed-off-by: Vinay Krishna Pudyodu <[email protected]>
|
❌ Gradle check result for 25fd006: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
mch2
approved these changes
Feb 26, 2025
Signed-off-by: Vinay Krishna Pudyodu <[email protected]>
|
❌ Gradle check result for d8585f7: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❕ Gradle check result for d8585f7: UNSTABLE Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
mch2
approved these changes
Feb 27, 2025
|
The backport to To backport manually, run these commands in your terminal: # Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-17055-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ee7fbbd226b2be81128eaafe19aad0a39244368c
# Push it to GitHub
git push --set-upstream origin backport/backport-17055-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.xThen, create a pull request where the |
|
ignoring backport here, this is a 3.0 only change. |
vinaykpud
added a commit
to vinaykpud/OpenSearch
that referenced
this pull request
Mar 18, 2025
…pensearch-project#17055) * Implemented computation of segment replication stats at shard level The method implemented here computes the segment replication stats at the shard level, instead of relying on the primary shard to compute stats based on reports from its replicas. Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Updated style checks in the test Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Updated changelog Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * fixed style issues Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Fix the failing integration test Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Fix stylecheck Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Fixed the comments for the initial revision Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Updated to use System.nanoTime() for lag calculation Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Fixed the integration test for node stats Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Modified the version in the ReplicationCheckpoint for backward compatibility Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Added precomputation logic for the stats calculation Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Removed unwanted lines Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Clean up the maps when index closed Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Added a null check for the indexshard checkpoint Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * fix style checks Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Updated version and added bwc for RemoteSegmentMetadata Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Upated the javadoc comments Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Address comments PR Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Removed the latestReceivedCheckpoint map from SegmentReplicationTargetService Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Added granular locks for the concurrency of stats methods Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Style check fixes Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Changes to maintain atomicity Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * spotlessApply Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * removed querying the remotestore when replication is in progress Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * spotlessApply Signed-off-by: Vinay Krishna Pudyodu <[email protected]> --------- Signed-off-by: Vinay Krishna Pudyodu <[email protected]>
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The method implemented here computes the segment replication stats at the shard level, instead of relying on the primary shard to compute stats based on reports from its replicas.
Method implemented in this PR serves the segment replication stats for following core APIs:
/_nodes/stats)/_cluster/stats)/_statsor/{index}/_stats)Related Issues
Resolves #16801
Related to #15306
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.