Add hybrid query and score/rank based normalization processor stats #1326

q-andy · 2025-05-15T21:33:15Z

PR to add initial stats for hybrid query, normalization processor, and RRF processor

Normalization processor event stats:

normalization_processor_executions
comb_arithmetic_executions
comb_geometric_executions
comb_harmonic_executions
comb_rrf_executions
norm_l2_executions
norm_minmax_executions
norm_zscore_executions

Normalization processor info stats:

normalization_processors
comb_arithmetic_processors
comb_geometric_processors
comb_harmonic_processors
norm_l2_processors
norm_minmax_processors
norm_zscore_processors

RRF processor event stats:

rank_based_normalization_processor_executions

RRF processor info stats:

rank_based_normalization_processors
comb_rrf_processors

Hybrid query event stats

hybrid_query_requests
hybrid_query_with_filter_requests
hybrid_query_with_inner_hits_requests
hybrid_query_with_pagination_requests

Example response:

	"info": {
		"cluster_version": "3.1.0",
		"processors": {
			"search": {
				"hybrid": {
					"comb_geometric_processors": 0,
					"comb_rrf_processors": 0,
					"norm_l2_processors": 0,
					"norm_minmax_processors": 0,
					"comb_harmonic_processors": 0,
					"comb_arithmetic_processors": 0,
					"norm_zscore_processors": 0,
					"rank_based_normalization_processors": 0,
					"normalization_processors": 0
				}
			},
			"ingest": {
				"text_chunking_delimiter_processors": 0,
				"text_embedding_processors_in_pipelines": 0,
				"text_chunking_fixed_length_processors": 0,
				"text_chunking_processors": 0
			}
		}
	},
	"all_nodes": {
		"query": {
			"hybrid": {
				"hybrid_query_with_pagination_requests": 0,
				"hybrid_query_with_filter_requests": 0,
				"hybrid_query_with_inner_hits_requests": 0,
				"hybrid_query_requests": 0
			}
		},
		"semantic_highlighting": {
			"semantic_highlighting_request_count": 0
		},
		"processors": {
			"search": {
				"hybrid": {
					"comb_harmonic_executions": 0,
					"norm_zscore_executions": 0,
					"norm_l2_executions": 0,
					"comb_rrf_executions": 0,
					"rank_based_normalization_processor_executions": 0,
					"comb_arithmetic_executions": 0,
					"normalization_processor_executions": 0,
					"comb_geometric_executions": 0,
					"norm_minmax_executions": 0
				}
			},
			"ingest": {
				"text_chunking_executions": 0,
				"text_embedding_executions": 0,
				"text_chunking_fixed_length_executions": 0,
				"text_chunking_delimiter_executions": 0
			}
		}
	},
....

Related Issues

Resolves #1146

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created. <- Since we are backfilling stats for previous features, we will add them all to the spec and documentation near 3.1 release
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created. <- Since we are backfilling stats for previous features, we will add them all to the spec and documentation near 3.1 release

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

martin-gaievski · 2025-05-18T18:42:30Z

src/main/java/org/opensearch/neuralsearch/processor/RRFProcessor.java

@@ -70,6 +72,7 @@ <Result extends SearchPhaseResult> void hybridizeScores(
        Optional<FetchSearchResult> fetchSearchResult = getFetchSearchResults(searchPhaseResult);
        boolean explain = Objects.nonNull(searchPhaseContext.getRequest().source().explain())
            && searchPhaseContext.getRequest().source().explain();
+        EventStatsManager.increment(EventStatName.RRF_PROCESSOR_EXECUTIONS);


for rank based normalization you also need to increment counter for combination technique calls, currently we do support "rrf" https://docs.opensearch.org/docs/latest/search-plugins/search-pipelines/score-ranker-processor/

Do we plan to add more combination techniques for rank based normalization in the future? We could also wait until then to add more granular breakdowns, otherwise the stats will just be duplicated right? I added anyways for now

no in near future, my worry was more about consistent result format, because for RRF we have "rrf" as both normalization technique (hidden), and combination technique.

src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessor.java

src/main/java/org/opensearch/neuralsearch/stats/events/EventStatName.java

src/main/java/org/opensearch/neuralsearch/stats/info/InfoStatName.java

CHANGELOG.md

src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessor.java

will-hwang · 2025-05-20T19:51:18Z

src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessor.java

@@ -40,6 +49,24 @@ public class NormalizationProcessor extends AbstractScoreHybridizationProcessor
    private final ScoreCombinationTechnique combinationTechnique;
    private final NormalizationProcessorWorkflow normalizationWorkflow;

+    private final Map<String, Runnable> normTechniqueIncrementers = Map.of(


looks like we have switch/case to increment for some processors, and runnable as map values for others. We should have one standard approach

+1. I am more inclined towards using map.

Sure, can refactor the others as maps.

this map can be static, same for the second map combTechniqueIncrementers

will-hwang · 2025-05-20T19:55:43Z

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

        for (QueryBuilder query : queries) {
            if (filter == null) {
                compoundQueryBuilder.add(query);
            } else {
                compoundQueryBuilder.add(query.filter(filter));
            }
+
+            // Check if children have inner hits for stats
+            if (hasInnerHits == false) {


is this the right comparison?

if children have inner hits, shouldn't this be true?

will-hwang · 2025-05-20T19:57:42Z

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

+
+            // Check if children have inner hits for stats
+            if (hasInnerHits == false) {
+                Map<String, InnerHitContextBuilder> innerHits = new HashMap<>();


shouldn't we emit stats for the inner hits? seems like we're checking for inner hits without doing anything to them. hasInnerHits can be flipped between true/false in the for loop, but we're only emitting once after the loop finishes in line 295. Is this valid?

The thought here is that we care about the stats at the level of the hybrid query. If at least one child query has inner hits then that means the hybrid query is an inner hits query. So we check each child, if at least one has inner hits, then we set hasInnerHits = true and we don't have to keep checking the rest.

vibrantvarun · 2025-05-20T21:26:27Z

src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessor.java

@@ -40,6 +49,24 @@ public class NormalizationProcessor extends AbstractScoreHybridizationProcessor
    private final ScoreCombinationTechnique combinationTechnique;
    private final NormalizationProcessorWorkflow normalizationWorkflow;

+    private final Map<String, Runnable> normTechniqueIncrementers = Map.of(


+1. I am more inclined towards using map.

vibrantvarun · 2025-05-20T21:28:00Z

src/main/java/org/opensearch/neuralsearch/processor/RRFProcessor.java

+
+    private void recordStats(ScoreCombinationTechnique combinationTechnique) {
+        EventStatsManager.increment(EventStatName.RRF_PROCESSOR_EXECUTIONS);
+        Optional.of(combTechniqueIncrementers.get(combinationTechnique.techniqueName())).ifPresent(Runnable::run);


RRF processor internally creates RRFNormalizationProcessor also. Check https://github.com/opensearch-project/neural-search/blob/main/src/main/java/org/opensearch/neuralsearch/processor/factory/RRFProcessorFactory.java#L51

I added this since @martin-gaievski requested, but my opinion is if currently RRF processor only works works with RRF combination technique and RRF normalization technique, then functionally the stats will be identical, and I don't think there's a point to including a breakdown, it will just be duplicated. Combination is configurable with a single option, but normalization technique isn't a configurable in processor config: https://docs.opensearch.org/docs/latest/search-plugins/search-pipelines/score-ranker-processor/

If in the future there are more normalizaiton/score techniques added we can add this granularity then.

Having RRF as processor and as technique gives more dimensions for reporting. For instance, with original version you can generate report with breakdown "by processor type". But report with breakdown on something like "by combination technique" will be harder because for RRF you raw data will have combination technique metric with empty value.

I see your point, my concern is having duplicated information in the stats API makes it less readable and less performant. Especially as we add more features, we are concerned with possible response bloat, similar the issues core is facing with the size of _nodes/info and _nodes/stats responses causing slowdowns on large clusters. And adding stats is one-way door in the sense that it's difficult to justify removing them since it's a breaking change. Ideally we save granularity for when it is needed and don't add more stats unless necessary

From a report perspective, I'm thinking in terms of what kind of insight a breakdown would give you: If you are trying to determine which score combination techniques are seeing more usage, perhaps proportion of RRF to zscore or minmax isn't comparable since they're categorically different, e.g. used in different processors in different contexts. And if needed, the information is available implicitly from looking at RRF processor stats directly.

vibrantvarun · 2025-05-20T21:29:28Z

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

        for (QueryBuilder query : queries) {
            if (filter == null) {
                compoundQueryBuilder.add(query);
            } else {
                compoundQueryBuilder.add(query.filter(filter));
            }
+
+            // Check if children have inner hits for stats
+            if (hasInnerHits == false) {


Where are we changing this parameter value?

It's set inside the loop, see my other comment for explanation. Basically my thought is we want to increment the stat once if at least one child query has inner hits, which means the hybrid query as a whole is an inner hits hybrid query.

martin-gaievski · 2025-05-22T00:51:00Z

src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessor.java

@@ -40,6 +49,24 @@ public class NormalizationProcessor extends AbstractScoreHybridizationProcessor
    private final ScoreCombinationTechnique combinationTechnique;
    private final NormalizationProcessorWorkflow normalizationWorkflow;

+    private final Map<String, Runnable> normTechniqueIncrementers = Map.of(


this map can be static, same for the second map combTechniqueIncrementers

martin-gaievski · 2025-05-22T00:56:05Z

src/main/java/org/opensearch/neuralsearch/stats/info/InfoStatsManager.java

+                            break;
+                        case RRFProcessor.TYPE:
+                            countRRFProcessorStats(stats, processorConfig);
+                            break;


please add default case and throw one of runtime exceptions in case we reach it

Not sure if that makes sense here? Here we are iterating through all processors in the pipeline, which may contain processors from core or other plugins. If we encounter an MLInferenceRequestProcessor for example, we would not want to record stats for it, and that would trigger the default case and throw an exception, breaking the API.

martin-gaievski

Looks good to me, thanks!

vibrantvarun · 2025-06-04T22:46:00Z

LGTM

Signed-off-by: Andy Qin <[email protected]> # Conflicts: # CHANGELOG.md # src/main/java/org/opensearch/neuralsearch/stats/events/EventStatName.java # Conflicts: # CHANGELOG.md # src/main/java/org/opensearch/neuralsearch/stats/events/EventStatName.java # src/main/java/org/opensearch/neuralsearch/stats/info/InfoStatName.java

Signed-off-by: Andy Qin <[email protected]> # Conflicts: # CHANGELOG.md

Signed-off-by: Andy Qin <[email protected]>

codecov · 2025-06-05T02:25:18Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (2928d83) to head (788edd7).
Report is 2 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #1326       +/-   ##
============================================
- Coverage     82.71%       0   -82.72%     
============================================
  Files           149       0      -149     
  Lines          7383       0     -7383     
  Branches       1192       0     -1192     
============================================
- Hits           6107       0     -6107     
+ Misses          821       0      -821     
+ Partials        455       0      -455

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

q-andy force-pushed the stats-normalization-processor branch 4 times, most recently from 966d125 to 6797f1d Compare May 16, 2025 23:32

martin-gaievski reviewed May 18, 2025

View reviewed changes

q-andy force-pushed the stats-normalization-processor branch 5 times, most recently from b9765c8 to c6f1ace Compare May 19, 2025 22:21

q-andy marked this pull request as ready for review May 19, 2025 22:23

q-andy requested review from heemin32, navneet1v, VijayanB, vamshin, jmazanec15, naveentatikonda, junqiu-lei, sean-zheng-amazon, model-collapse, zane-neo, vibrantvarun, zhichao-aws, yuye-aws and minalsha as code owners May 19, 2025 22:23

q-andy changed the title ~~Add normalization processor and RRF processor stats~~ Add hybrid query and score/rank based normalization processor stats May 20, 2025

q-andy mentioned this pull request May 8, 2025

[FEATURE] Update neural-search stats API spec with new stats added in 3.1 opensearch-project/opensearch-api-specification#890

Open

7 tasks

q-andy force-pushed the stats-normalization-processor branch from c6f1ace to c7ea569 Compare May 20, 2025 00:23

github-actions bot added enhancement hybrid search labels May 20, 2025

q-andy mentioned this pull request May 20, 2025

[DOC] Update neural-search stats API docs with new stats added in 3.1 opensearch-project/documentation-website#9943

Closed

11 tasks

will-hwang reviewed May 20, 2025

View reviewed changes

vibrantvarun reviewed May 20, 2025

View reviewed changes

martin-gaievski reviewed May 22, 2025

View reviewed changes

q-andy force-pushed the stats-normalization-processor branch 3 times, most recently from ebd6ef6 to 0eac607 Compare May 22, 2025 22:14

q-andy force-pushed the stats-normalization-processor branch from c7f39e9 to bfedad1 Compare June 2, 2025 23:06

q-andy mentioned this pull request Jun 2, 2025

Add stats tracking for semantic field #1362

Merged

5 tasks

martin-gaievski approved these changes Jun 3, 2025

View reviewed changes

vibrantvarun approved these changes Jun 4, 2025

View reviewed changes

q-andy added 5 commits June 4, 2025 15:49

Use functional interface map, Add RRF combination technique stats

88a69e7

Signed-off-by: Andy Qin <[email protected]> # Conflicts: # CHANGELOG.md

Add hybrid query stats

886f1b7

Signed-off-by: Andy Qin <[email protected]>

Refactor TextChunkingProcessor from switch case to map

7859422

Signed-off-by: Andy Qin <[email protected]>

Rename hybrid query stats

788edd7

Signed-off-by: Andy Qin <[email protected]>

q-andy force-pushed the stats-normalization-processor branch from bfedad1 to 788edd7 Compare June 4, 2025 22:50

heemin32 merged commit f6236b7 into opensearch-project:main Jun 5, 2025
82 of 91 checks passed

Add hybrid query and score/rank based normalization processor stats #1326

Add hybrid query and score/rank based normalization processor stats #1326

Uh oh!

Conversation

q-andy commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Normalization processor event stats:

Normalization processor info stats:

RRF processor event stats:

RRF processor info stats:

Hybrid query event stats

Related Issues

Check List

Uh oh!

Choose a reason for hiding this comment

Uh oh!

q-andy May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

q-andy May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martin-gaievski left a comment

Choose a reason for hiding this comment

Uh oh!

vibrantvarun commented Jun 4, 2025

Uh oh!

Uh oh!

codecov bot commented Jun 5, 2025

Codecov Report

Uh oh!

Uh oh!

q-andy commented May 15, 2025 •

edited

Loading

q-andy May 19, 2025 •

edited

Loading

q-andy May 22, 2025 •

edited

Loading