Star Tree Search changes related to new Aggregations supported #9163
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -26,7 +26,7 @@ A star-tree index can be used to perform faster aggregations. Consider the follo | |
|
|
||
| Star-tree indexes have the following limitations: | ||
|
|
||
| - A star-tree index should only be enabled on indexes whose data is not updated or deleted because updates and deletions are not accounted for in a star-tree index. To enforce this policy and use star-tree indexes, set the `index.append_only.enabled` setting to `true`. | ||
| - A star-tree index can be used for aggregation queries only if the queried fields are a subset of the star-tree's dimensions and the aggregated fields are a subset of the star-tree's metrics. | ||
| - After a star-tree index is enabled, it cannot be disabled. In order to disable a star-tree index, the data in the index must be reindexed without the star-tree mapping. Furthermore, changing a star-tree configuration will also require a reindex operation. | ||
| - [Multi-values/array values]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/index/#arrays) are not supported. | ||
|
|
@@ -68,6 +68,7 @@ To use a star-tree index, modify the following settings: | |
| - Set the feature flag `opensearch.experimental.feature.composite_index.star_tree.enabled` to `true`. For more information about enabling and disabling feature flags, see [Enabling experimental features]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/experimental/). | ||
| - Set the `indices.composite_index.star_tree.enabled` setting to `true`. For instructions on how to configure OpenSearch, see [Configuring settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/#static-settings). | ||
| - Set the `index.composite_index` index setting to `true` during index creation. | ||
| - Set the `index.append_only.enabled` index setting to `true` during index creation. | ||
| - Ensure that the `doc_values` parameter is enabled for the `dimensions` and `metrics` fields used in your star-tree mapping. | ||
|
|
||
|
|
||
|
|
@@ -81,7 +82,8 @@ PUT logs | |
| "settings": { | ||
| "index.number_of_shards": 1, | ||
| "index.number_of_replicas": 0, | ||
| "index.composite_index": true, | ||
| "index.append_only.enabled": true | ||
| }, | ||
| "mappings": { | ||
| "composite": { | ||
|
|
@@ -94,6 +96,9 @@ PUT logs | |
| }, | ||
| { | ||
| "name": "port" | ||
| }, | ||
| { | ||
| "name": "method" | ||
| } | ||
| ], | ||
| "metrics": [ | ||
|
|
@@ -123,6 +128,9 @@ PUT logs | |
| "size": { | ||
| "type": "integer" | ||
| }, | ||
| "method" : { | ||
| "type": "keyword" | ||
| }, | ||
| "latency": { | ||
| "type": "scaled_float", | ||
| "scaling_factor": 10 | ||
|
|
@@ -131,6 +139,7 @@ PUT logs | |
| } | ||
| } | ||
| ``` | ||
| {% include copy.html %} | ||
|
|
||
| For detailed information about star-tree index mappings and parameters, see [Star-tree field type]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/star-tree/). | ||
|
|
||
|
|
@@ -140,14 +149,20 @@ Star-tree indexes can be used to optimize queries and aggregations. | |
|
|
||
| ### Supported queries | ||
|
|
||
| The following queries are supported as of OpenSearch 2.19: | ||
sandeshkr419 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| - [Term query]({{site.url}}{{site.baseurl}}/query-dsl/term/term/) | ||
| - [Terms query]({{site.url}}{{site.baseurl}}/query-dsl/term/terms/) | ||
| - [Match all docs query]({{site.url}}{{site.baseurl}}/query-dsl/match-all/) | ||
| - [Range query]({{site.url}}{{site.baseurl}}/query-dsl/term/range/) | ||
sandeshkr419 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| To use a query with a star-tree index, the query's fields must be present in the `ordered_dimensions` section of the star-tree configuration. Queries must also be paired with a supported aggregation. Queries without aggregations cannot be used with a star-tree index. Currently, queries on `date` fields are not supported and will be added in later versions. | ||
|
|
||
| ### Supported aggregations | ||
|
|
||
Naarcha-AWS marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| The following aggregations are supported by star-tree indexes. | ||
|
|
||
| #### Metric aggregations | ||
|
|
||
| The following metric aggregations are supported as of OpenSearch 2.18: | ||
| - [Sum]({{site.url}}{{site.baseurl}}/aggregations/metric/sum/) | ||
|
|
@@ -156,13 +171,11 @@ The following metric aggregations are supported as of OpenSearch 2.18: | |
| - [Value count]({{site.url}}{{site.baseurl}}/aggregations/metric/value-count/) | ||
| - [Average]({{site.url}}{{site.baseurl}}/aggregations/metric/average/) | ||
|
|
||
| To use searchable aggregations with a star-tree index, make sure you fulfill the following prerequisites: | ||
|
|
||
| - The fields must be present in the `metrics` section of the star-tree configuration. | ||
| - The metric aggregation type must be part of the `stats` parameter. | ||
|
|
||
| The following example gets the sum of all the values in the `size` field for all error logs with `status=500`, using the [example mapping](#example-mapping): | ||
|
|
||
| ```json | ||
|
|
@@ -182,9 +195,56 @@ POST /logs/_search | |
| } | ||
| } | ||
| ``` | ||
| {% include copy.html %} | ||
|
|
||
| Using a star-tree index, the result will be retrieved from a single aggregated document as it traverses the `status=500` node, as opposed to scanning through all of the matching documents. This results in lower query latency. | ||
|
|
||
| #### Date histograms with metric aggregations | ||
|
|
||
| You can use [date histograms]({{site.url}}{{site.baseurl}}/aggregations/bucket/date-histogram/) on calendar intervals with metric sub-aggregations. | ||
|
|
||
| To use date histogram aggregations and make them searchable in a star-tree index, remember the following requirements: | ||
|
|
||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - The calendar intervals in a star-tree mapping configuration can use either the request's calendar field or a field of lower granularity than the request field. For example, if an aggregation uses the `month` field, the star-tree search can still use lower-granularity fields such as `day`. | ||
| - A metric sub-aggregation must be part of the aggregation request. | ||
|
|
||
| The following example gets the sum of all the values in the `size` field with a range query, aggregated for each calendar month, for all error logs containing `method:get`: | ||
|
|
||
| ```json | ||
| POST /logs/_search | ||
| { | ||
| { | ||
| "query": { | ||
|
There was a problem hiding this comment. Is this a valid query ? @sandeshkr419 just double checking. I don't see term/terms etc There was a problem hiding this comment. @bharath-techie: I updated the query to add terms while keeping the method. There was a problem hiding this comment. This is still not a valid query , we don't support range on timestamp. Lets reword this @Naarcha-AWS There was a problem hiding this comment. @bharath-techie: Can you suggest a valid query? There was a problem hiding this comment. This is a valid query which we can use @Naarcha-AWS |
||
| "range": { | ||
| "created": { | ||
| "gte": "2019/01/01", | ||
| "lte": "2019/12/31" | ||
| }, | ||
| "method": { | ||
| "status": "get" | ||
| } | ||
| }, | ||
| "size": 0, | ||
| "aggs": { | ||
| "by_hour": { | ||
| "date_histogram": { | ||
| "field": "@timestamp", | ||
| "calendar_interval": "month" | ||
| }, | ||
| "aggs": { | ||
| "sum_size": { | ||
| "sum": { | ||
| "field": "size" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
natebower marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ## Using queries without a star-tree index | ||
|
|
||
| Set the `indices.composite_index.star_tree.enabled` setting to `false` to run queries without using a star-tree index. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do we mean by "of blank"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The fields names are what defined inside "
". Extra clarification is just for information. Usually month, year etc are understood easily, but certain fields likeweek` needed extra clarification like (week of week based year)