[BUG] Hybrid Search not working with Document Level Security

[PaulRCampbell]

Hybrid Search queries fail when authenticating with an internal user that has Document Level Security enabled.

Steps to reproduce the behaviour:

this is a rather contrived example that I have used for testing purposes

create a role with a dls filter:

{"term": { "divisionId": "448385216"}}

authenticate with an internal user assigned to that role and then run a hybrid search query

GET my_test_index/_search?search_pipeline=hybrid-search-pipeline
{
  "query": {
    "hybrid": {
      "queries": [
        {
          "bool": {
            "filter": [
              {
                "term": {
                  "organisationId": 1
                }
              }
            ]
          }
        },
        {
          "neural": {
            "label_embedding.paraphrase-multilingual-MiniLM-L12-v2": {
              "query_text": "document",
              "model_id": "Cf9VfZYBPcPsA6UT332K",
              "k": "5",
              "filter": {
                "term": {
                  "organisationId": 1
                }
              }
            }
          }
        }
      ]
    }
  },
  "_source": ["label", "organisationId", "esType", "status", "divisionId"]
}

returns error

{
  "error": {
    "root_cause": [
      {
        "type": "class_cast_exception",
        "reason": "class org.apache.lucene.search.ConstantScoreQuery cannot be cast to class org.opensearch.neuralsearch.query.HybridQuery (org.apache.lucene.search.ConstantScoreQuery is in unnamed module of loader 'app'; org.opensearch.neuralsearch.query.HybridQuery is in unnamed module of loader java.net.FactoryURLClassLoader @57435801)"
      },
      {
        "type": "illegal_argument_exception",
        "reason": "hybrid query must be a top level query and cannot be wrapped into other queries"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "my_test_index",
        "node": "j0xoYCCsTh2Jjoe_SNqmdQ",
        "reason": {
          "type": "class_cast_exception",
          "reason": "class org.apache.lucene.search.ConstantScoreQuery cannot be cast to class org.opensearch.neuralsearch.query.HybridQuery (org.apache.lucene.search.ConstantScoreQuery is in unnamed module of loader 'app'; org.opensearch.neuralsearch.query.HybridQuery is in unnamed module of loader java.net.FactoryURLClassLoader @57435801)"
        }
      },
      {
        "shard": 4,
        "index": "my_test_index",
        "node": "j0xoYCCsTh2Jjoe_SNqmdQ",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "hybrid query must be a top level query and cannot be wrapped into other queries"
        }
      }
    ],
    "caused_by": {
      "type": "class_cast_exception",
      "reason": "class org.apache.lucene.search.ConstantScoreQuery cannot be cast to class org.opensearch.neuralsearch.query.HybridQuery (org.apache.lucene.search.ConstantScoreQuery is in unnamed module of loader 'app'; org.opensearch.neuralsearch.query.HybridQuery is in unnamed module of loader java.net.FactoryURLClassLoader @57435801)",
      "caused_by": {
        "type": "class_cast_exception",
        "reason": "class org.apache.lucene.search.ConstantScoreQuery cannot be cast to class org.opensearch.neuralsearch.query.HybridQuery (org.apache.lucene.search.ConstantScoreQuery is in unnamed module of loader 'app'; org.opensearch.neuralsearch.query.HybridQuery is in unnamed module of loader java.net.FactoryURLClassLoader @57435801)"
      }
    }
  },
  "status": 500
}

Curiously, if I add a suggest block to the query I get the desired results (and the dsl filter is also being applied).

GET my_test_index/_search?search_pipeline=hybrid-search-pipeline
{
  "query": {
    "hybrid": {
      "queries": [
        {
          "bool": {
            "filter": [
              {
                "term": {
                  "organisationId": 1
                }
              }
            ]
          }
        },
        {
          "neural": {
            "label_embedding.paraphrase-multilingual-MiniLM-L12-v2": {
              "query_text": "document",
              "model_id": "Cf9VfZYBPcPsA6UT332K",
              "k": "5",
              "filter": {
                "term": {
                  "organisationId": 1
                }
              }
            }
          }
        }
      ]
    }
  },
    "suggest": {
    "text": "document",
    "label-suggest": {
      "term": {
        "field": "label"
      }
    }
  },
  "_source": ["label", "organisationId", "esType", "status", "divisionId"]
}

returns results

{
  "took": 1176,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 0.2,
    "hits": [
      {
        "_index": "my_test_index",
        "_id": "1_325861482",
        "_score": 0.2,
        "_routing": "1",
        "_source": {
          "organisationId": 1,
          "divisionId": 448385216,
          "label": "English Document",
          "status": "active"
        }
      }
    ]
  },
  "suggest": {
    "label-suggest": [
      {
        "text": "document",
        "offset": 0,
        "length": 8,
        "options": []
      }
    ]
  }
}

I have tested this on our AWS Opensearch managed service which is on version 2.17.1 and also running 2.19.1 locally with the following docker configuration. (note I have a seperate script to register and deploy the embedding model, and create ingest and search pipelines etc).

version: "3.7"
services:
  opensearch:
    build: .
    container_name: opensearch
    environment:
      - discovery.type=single-node
      - plugins.security.disabled=false
      - plugins.security.ssl.http.enabled=false
      - plugins.security.ssl.transport.enforce_hostname_verification=false
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=somestrongpassword
      - OPENSEARCH_JAVA_OPTS=-Xms6g -Xmx6g
    ports:
      - "9200:9200" # OpenSearch HTTP
      - "9600:9600" # Performance Analyzer
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 8g
    healthcheck:
      test: ["CMD-SHELL", "curl -s http://localhost:9200 || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5

  dashboards:
    image: opensearchproject/opensearch-dashboards:2.19.1
    container_name: dashboards
    environment:
      - OPENSEARCH_HOSTS=["http://opensearch:9200"]
      - OPENSEARCH_USERNAME=admin
      - OPENSEARCH_PASSWORD= somestrongpassword
      - SERVER_SSL_ENABLED=false
    ports:
      - "5601:5601"
    depends_on:
      opensearch:
        condition: service_healthy

[pawelw1]

Just FYI. This was initially reported, investigated and confirmed in OpenSearch forum.

https://forum.opensearch.org/t/hybrid-search-not-working-with-document-level-security/24239

[owaiskazi19]

@opensearch-project/admin can you move this issue to neural search repo?

[owaiskazi19]

Able to reproduced the error

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "hybrid query must be a top level query and cannot be wrapped into other queries"
            }
        ],
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [
            {
                "shard": 0,
                "index": "my-nlp-index",
                "node": "mnXXebOBTieEpZN9LfULLg",
                "reason": {
                    "type": "illegal_argument_exception",
                    "reason": "hybrid query must be a top level query and cannot be wrapped into other queries"
                }
            }
        ],

Little deep dived showed, with DSL enabled the query type for searchWith is

neural-search/src/main/java/org/opensearch/neuralsearch/search/query/HybridQueryPhaseSearcher.java

Line 54 in 0bb7053

final Query query,

BooleanQuery, without DSL the query type received is HybridQuery to searchWith.
That in returns validate the top most query as BooleanQuery

neural-search/src/main/java/org/opensearch/neuralsearch/search/query/HybridQueryPhaseSearcher.java

Line 109 in 0bb7053

private void validateQuery(final SearchContext searchContext, final Query query) {

and finally throwing an exception at

neural-search/src/main/java/org/opensearch/neuralsearch/search/query/HybridQueryPhaseSearcher.java

Line 124 in 0bb7053

throw new IllegalArgumentException("hybrid query must be a top level query and cannot be wrapped into other queries");

@cwperks is the ParsedQuery in DLS is not evaluating query as HybridQuery but as BoolQuery here?

[cwperks]

I see that the security plugin wraps the query in ConstantScoreQuery here. atm I'm not exactly sure why this is the case and will see if anything can be done to avoid the ClassCastException.

CC: @nibix @jochen-kressin

[nibix]

If DLS is active, queries might be wrapped in BooleanQuery objects which are again wrapped in ConstantScoreQuery objects in order to enforce DLS restrictions.

The neural-search code has already logic which tries to unwrap these queries again to get access to the HybridQuery object. But it seems like the code is not able to handle the wrapping performed for DLS correctly.

Possibly this logic could be refined to handle also queries wrapped for DLS?

The code in question is this:

neural-search/src/main/java/org/opensearch/neuralsearch/search/query/HybridCollectorManager.java

Lines 506 to 517 in 0bb7053

	private static HybridQuery unwrapHybridQuery(final SearchContext searchContext) {
	HybridQuery hybridQuery;
	Query query = searchContext.query();
	// In case of nested fields and alias filter, hybrid query is wrapped under bool query and lies in the first clause.
	if (isHybridQueryWrappedInBooleanQuery(searchContext, searchContext.query())) {
	BooleanQuery booleanQuery = (BooleanQuery) query;
	hybridQuery = (HybridQuery) booleanQuery.clauses().get(0).query();
	} else {
	hybridQuery = (HybridQuery) query;
	}
	return hybridQuery;
	}

neural-search/src/main/java/org/opensearch/neuralsearch/search/query/HybridQueryPhaseSearcher.java

Lines 78 to 90 in 0bb7053

	protected Query extractHybridQuery(final SearchContext searchContext, final Query query) {
	if (isHybridQueryWrappedInBooleanQuery(searchContext, query)) {
	List<BooleanClause> booleanClauses = ((BooleanQuery) query).clauses();
	if (!(booleanClauses.get(0).query() instanceof HybridQuery)) {
	throw new IllegalArgumentException("hybrid query must be a top level query and cannot be wrapped into other queries");
	}
	HybridQuery hybridQuery = (HybridQuery) booleanClauses.get(0).query();
	List<BooleanClause> filterQueries = booleanClauses.stream().skip(1).collect(Collectors.toList());
	HybridQuery hybridQueryWithFilter = new HybridQuery(hybridQuery.getSubQueries(), hybridQuery.getQueryContext(), filterQueries);
	return hybridQueryWithFilter;
	}
	return query;
	}

[owaiskazi19]

@nibix @cwperks thanks for the deep dive. Looked more into it, with DSL enable the search flow follows:

neural-search/src/main/java/org/opensearch/neuralsearch/search/query/HybridQueryPhaseSearcher.java

Line 51 in 0bb7053

public boolean searchWith(

Without DSL, it routes to the HybridCollectorManager.
When DSL is enabled, We could check and call unwrapHybridQuery to fetch the wrapped Hybrid Query. However, the DSL query gets wrapped in BooleanQuery and then in ConstantScore. Due to this nested wrapping, the query on the DSL side completely loses the hybrid query clause

The resulting query looks like this:

[1C35EKsxRxeLURAuUr5kFA][my-nlp-index][0] query=[+ConstantScore(passage_text:hi passage_text:planet) +((passage_text:hi passage_text:planet) | (passage_text:hi passage_text:planet) | (passage_text:hi passage_text:planet))]

The key issue is that since the hybrid query clause is lost during DSL wrapping, unwrapping on neural search side would be difficult.