[BUG] [Hybrid Search] Non-determinsitic NullPointerException bug when using hybrid search with a single shard

[ekeric13]

Currently on version 2.19.X

What is the bug?

A clear and concise description of the bug.

Null Pointer Exception here:
https://github.com/opensearch-project/neural-search/blob/2.19.2.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L324

This code seems to be gated if you have multiple shards, as this code seems to imply that if there are multiple shards fetchSearchResultOptional is empty and we have an early return:
https://github.com/opensearch-project/neural-search/blob/2.19.2.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L282-L283
https://github.com/opensearch-project/neural-search/blob/2.19.2.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L111-L114

Not exactly sure how the issue occurs but you can tell the docMap created from hit documents:
https://github.com/opensearch-project/neural-search/blob/2.19.2.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L300-L308

Doesn't have the document that was scored:
https://github.com/opensearch-project/neural-search/blob/2.19.2.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L320-L322

I am new to hybrid search, and am not running the code, but giving it a good reading this is my guess on what is going on:

1. we create unProcessedDocs from the topDocs
   https://github.com/opensearch-project/neural-search/blob/2.19.2.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L359-L366
2. We construct a map of docs, the key being from unProcessedDocs and the value being from the hit docs.
   https://github.com/opensearch-project/neural-search/blob/2.19.2.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L298C38-L308
3. Similar to how we constructed unProcessedDocs, we go over the topDocs for each hit recorded. Importantly we don't get the docs from docIds but from querySearchResult:
   https://github.com/opensearch-project/neural-search/blob/2.19.2.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L310-L324
4. the unProccedDocs was created early on:
   https://github.com/opensearch-project/neural-search/blob/2.19.2.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L62-L65
5. Before we call updateOriginalFetchResults we call updateOriginalQueryResults and in updateOriginalQueryResults we mutate the queryResults object with a different topDocs value
   https://github.com/opensearch-project/neural-search/blob/2.19.2.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L97C9-L103
   https://github.com/opensearch-project/neural-search/blob/2.19.2.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L212
6. So now i think in updateOriginalFetchResults, we are coming back with with different topDocs that are used in the trimmedLengthOfSearchHits for loop than the unProcessedDocs ones we used to make the map.
   https://github.com/opensearch-project/neural-search/blob/2.19.2.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L310C47-L311

So maybe the fix is to not pass through unprocessedDocIds and then update updateOriginalFetchResults to just use the new re-ranked docs throughout the whole function? Assuming my diagnosis was/is correct. i am new to opensearch so not exactly sure how it is suppose to work.

How can one reproduce the bug?

Steps to reproduce the behavior.

Create an instance with a single shard, add 3 documents, use a hybrid search query that returns 2 documents. Sometimes it passes but most of the time i get the NPE. Hence why i mentioned it is non-determinsitic.

What is the expected behavior?

A clear and concise description of what you expected to happen.

Return the hit documents and not throw an NPE. We see this behavior when we used 3 shards.

What is your host/environment?

Operating system, version.

Opensearch 2.19. I believe it is aws opensearch. A teammate of mine actually owns the opensearch service so do not have all the details here.

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

Add any other context about the problem.

[heemin32]

@ekeric13 Thanks for reporting the bug. Could you try to see if the issue is happening in 3.1 version as well?

[ekeric13]

Unfortunately i don't own what version of opensearch we use.

Looking at the code the issue of modifying top docs
https://github.com/opensearch-project/neural-search/blob/3.1.0.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L245

then calling updateOriginalFetchResults with unprocessedDocIds
https://github.com/opensearch-project/neural-search/blob/3.1.0.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L100-L106

and then getting the 0th index top docs
https://github.com/opensearch-project/neural-search/blob/3.1.0.0/src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java#L322-L323

remains the same

[navneet1v]

@heemin32 can someone from your team look into this.

[heemin32]

@martin-gaievski Could you take a look?

[owaiskazi19]

@ekeric13 Thanks for reporting this. Not able to reproduce the issue from our end. Tested on latest 3.1 and 2.19

1. Created an index with single shard.
2. Added 3 docs
3. Created a normalization search pipeline
4. Made a search request call with matching 2 documents. Tried in a loop of 20

Can you share the index mapping and your search request so that we can try to reproduce on our end?

[ekeric13]

yeah it might be hard to reproduce since it requires the docs to be re-ranked/re-sorted. which depends not just on the query but the underlining data.

I will ask my teammate to share the index mapping. I don't know what that is.

Here is an example query:

{
    "query": {
      "template": {
        "hybrid": {
          "pagination_depth": 6,
          "queries": [
            {
              "bool": {
                "should": [
                  {
                    "term": {
                      "attributes.widget1": "redactedValue"
                    }
                  }
                ],
                "filter": {
                  "bool": {
                    "must": [
                      {
                        "term": {
                          "attributes.widget2": "redactedValue"
                        }
                      },
                      {
                        "term": {
                          "attributes.widget3": "redactedValue"
                        }
                      },
                      {
                        "term": {
                          "attributes.widget4": "redactedValue"
                        }
                      },
                      {
                        "term": {
                          "attributes.widget5": "redactedValue"
                        }
                      }
                    ]
                  }
                }
              }
            },
            {
              "knn": {
                "attributes.embeddings": {
                  "vector": "${ext.ml_inference.params.vector}",
                  "k": 6,
                  "filter": {
                    "bool": {
                      "must": [
                        {
                          "term": {
                            "attributes.widget2": "redactedValue"
                          }
                        },
                        {
                          "term": {
                            "attributes.widget3": "redactedValue"
                          }
                        },
                        {
                          "term": {
                            "attributes.widget4": "redactedValue"
                          }
                        },
                        {
                          "term": {
                            "attributes.widget5": "redactedValue"
                          }
                        }
                      ]
                    }
                  }
                }
              }
            }
          ]
        }
      }
    },
    "ext": {
      "ml_inference": {
        "params": {
          "vector": "REDACTED_VALUE"
        }
      }
    },
    "size": 3
}

and i had documents with an assortment of different combinations of widgets (properties) and redactedValue (values).

[msmith93]

The index mapping:

{
  "routingcontext-v2": {
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "domainAttributes": {
          "properties": {
            "appName": {
              "type": "keyword"
            },
            "customerIntent": {
              "type": "keyword"
            },
            "derivedAttributes": {
              "type": "object"
            },
            "entitlements": {
              "type": "keyword"
            },
            "entitlementsHash": {
              "type": "keyword"
            },
            "experience": {
              "type": "keyword"
            },
            "intent": {
              "type": "keyword"
            },
            "ipsRelationalUUID": {
              "type": "keyword"
            },
            "issueResolution": {
              "type": "boolean"
            },
            "language": {
              "type": "keyword"
            },
            "locale": {
              "type": "keyword"
            },
            "productName": {
              "type": "keyword"
            },
            "query": {
              "type": "keyword"
            },
            "queue": {
              "type": "keyword"
            },
            "routing_context_embeddings": {
              "type": "knn_vector",
              "dimension": 768,
              "method": {
                "engine": "faiss",
                "space_type": "l2",
                "name": "hnsw",
                "parameters": {
                  "ef_construction": 100,
                  "m": 16
                }
              }
            }
          }
        },
        "eventType": {
          "type": "keyword"
        },
        "event_header": {
          "type": "object",
          "dynamic": "false",
          "enabled": false
        },
        "isEntity": {
          "type": "boolean"
        },
        "systemInfo": {
          "properties": {
            "alternateIdentifier": {
              "type": "keyword"
            },
            "attrEncKeyName": {
              "type": "keyword"
            },
            "createdBy": {
              "type": "keyword"
            },
            "createdOn": {
              "type": "date",
              "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis||date_optional_time"
            },
            "deletedOn": {
              "type": "date",
              "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis||date_optional_time"
            },
            "entityTags": {
              "type": "nested",
              "dynamic": "strict",
              "properties": {
                "tags": {
                  "type": "nested",
                  "dynamic": "strict",
                  "properties": {
                    "tagName": {
                      "type": "keyword"
                    },
                    "tagValue": {
                      "type": "keyword"
                    }
                  }
                }
              }
            },
            "identifier": {
              "type": "keyword"
            },
            "lastUpdatedBy": {
              "type": "keyword"
            },
            "lastUpdatedOn": {
              "type": "date",
              "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis||date_optional_time"
            },
            "ownedBy": {
              "type": "keyword"
            },
            "ownedByType": {
              "type": "keyword"
            },
            "payload": {
              "type": "nested",
              "dynamic": "strict",
              "properties": {
                "bucketName": {
                  "type": "keyword"
                },
                "encKeyName": {
                  "type": "keyword"
                },
                "objName": {
                  "type": "keyword"
                }
              }
            },
            "policy": {
              "type": "nested",
              "dynamic": "strict",
              "properties": {
                "tags": {
                  "type": "nested",
                  "dynamic": "strict",
                  "properties": {
                    "tagName": {
                      "type": "keyword"
                    },
                    "tagValue": {
                      "type": "keyword"
                    }
                  }
                }
              }
            },
            "version": {
              "type": "long"
            }
          }
        },
        "type": {
          "type": "keyword"
        }
      }
    }
  }
}