Dynamic max boolean clause counts

[malpani]

Is your feature request related to a problem? Please describe.
As query shapes evolve, there have been multiple requests by users ato dynamically change limits on max clauses allowed for a boolean query. Today, this is a static setting indices.query.bool.max_clause_count that defaults to 1024. Any change requires a yml change and rolling restart of the cluster which is not convenient

On hitting an error like TooManyClauses: maxClauseCount is set to 1024, while a re-look at query is always recommended, but in scenarios where larger clauses are required, we should allow supporting those

Describe the solution you'd like
Make indices.query.bool.max_clause_count a dynamic setting which can then be dynamically updated via settings update like

PUT /_cluster/settings --data '{
    "transient" : {
        "indices.query.bool.max_clause_count" : 2048
    }
}'

[malpani]

Summarizing the review of #1527 with @reta and @nknize

1. Push back on making this a dynamic setting as it would provide users an easy way out to raise this limit instead of relooking at their query.
2. A query utilizing large number of clauses will cause interference with well behaved queries
3. If making dynamic, feedback is to provide as a per index setting (instead of cluster wide) - however exposing per index limit on clauses is something that can be looked at in 2.x only as it will involve building on top of https://issues.apache.org/jira/browse/LUCENE-8811 (lucene 9.x)

I dont plan to work on this in the short term and if someone wants to drive this to closure, feel free to pick it up.

[AmiStrn]

This is actually a useful tool for multitenant clusters.
Restarting just to change this is a bit of a pain. Maybe we could make a possible override on the index level.

[reta]

More context on the issue:

• IndexSearcher has maxClauseCount limit, which is what BooleanQuery is using in 9.x (see please https://issues.apache.org/jira/browse/LUCENE-8811), this property is also static
• the idea of having boolean query clause limit configurable per-query is effectively buried (see please https://issues.apache.org/jira/browse/LUCENE-7880)
• introducing something like indices.query.max_clause_count (and deprecating indices.query.bool.max_clause_count since respective BooleanQuery property is scheduled for removal in Apache Lucene 10) with dynamic scope (changeable at runtime) could be useful:

PUT /_cluster/settings --data '{
    "transient" : {
        "indices.query.max_clause_count" : 2048
    }
}'

(in 3.x, we would still support indices.query.bool.max_clause_count with deprecation warning). The major concern here is that IndexSearcher::maxClauseCount is not volatile (nor guarded by synchronization primitives), so changing it may not be visible to all threads.

@nknize does it make sense? if yes, I could pick it up, thanks!

[kkhatua]

@reta / @nknize
Is this going forward? A dynamic setting would certainly help in enabling and tuning in production clusters.

[reta]

@kkhatua yeah, I know, haven't gotten any feedback on the proposal :(

[anand3493]

Users in our org are facing this prob once we migrated to Opensearch from Elasticsearch.
This feature will really help my organization.

[jainankitk]

@reta - After going through various concerns from the lucene community and @nknize on PR #1527, I am wondering if having per query parameter makes sense? That ensures every (potentially) not so well written query fails atleast once on the default limit of 1024, and customer can deliberately choose higher limit at the cost of performance if their workload really needs it?

[reta]

@jainankitk you mean resuming discussion on [1] & [2]? I think it would make a lot of sense, because on OpenSearch side we cannot enforce this limit in isolation since Lucene applies static limits across the board.

[1] apache/lucene#8931
[2] https://issues.apache.org/jira/browse/LUCENE-7880