[Query Insights] Capture query-level resource usage metrics

[ansjcy]

Is your feature request related to a problem? Please describe

The resource tracking framework (#1179) tracks task-level resource usage, such as CPU and memory utilization. However there's a gap to infer query-level resource usage from the resource tracking framework. We need to come up a solution for it since it would one of the most important metrics for query insights (#11429) features like top n queries (#11186) and also cost estimations (#12390).

Describe the solution you'd like

The most challenging task here is how to propagate the task-level resource usage information to the coordinator node for calculating query-level resource usage. The most straightforward solution is to piggyback the resource usage data as part of the SearchPhaseResult node response and use SearchRequestOperationsListener::onPhaseEnd to extract this information from the phase results and forward it to the query insights framework. However, this approach has limitations as the obtained resource usage data may not be entirely accurate. The reason is explained below.

Here's the workflow of a search request and resource tracking: The coordinator node sends requests to data nodes and the data nodes will create tasks to do search on shards. On a data node,

1. The data node creates tasks and starts request tracking using the resource tracking framework;
2. The data node sends back the SearchPhaseResult to the coordinator node;
3. The data node stops thes task and stops request tracking, and records the final resource utilization in resource tracking framework.

If we want to piggyback the resource utilization data in SearchPhaseResult, we must retrieve this data before the task is considered "finished." Through some experiments and analysis, I found reading the resource utilization data before the second step would result in up to ~10% lower CPU and Memory utilization compared to the final actual usage. If the actual results are not accurate at all, the data would be of no use except for roughly analyzing the overall usage trend and "relative" resource usage comparasion between 2 queries - We won't be able to use this data to make reliable query cost estimations.

Related component

Search:Query Insights

Describe alternatives you've considered

Another approach is to implement an asynchronous post-processor as part of the query insights data consumption pipeline. This post-processor would periodically gather data from data nodes and correlate it with queries to calculate the final resource usage accurately. While this method ensures the most accurate resource usage data, it comes with the overhead of introducing a periodic job running in the background to collect and share the data between nodes. We need to consider the trade-offs when deciding on the best approach for capturing query-level resource usage.

Additional context

• Query Insights Meta issue: [META] Generic Query Insights Framework #11522
• Top N Queries RFC: [RFC] Real-time Insights into Top N Queries by Latency and Resource Usage #11186

[peternied]

[Triage - attendees 1 2 3 4 5]
@ansjcy Thanks for filing

[ansjcy]

Draft for the proposed solution: #12449

• Instrument resource usages before a task finishes on a data node, more specifically, get resource usages at the last step (serialization) before a phase response is sent from query/fetch/.. phase.
• Piggyback the resource usages data with phase results.
• Gather resource usages for all shard search tasks on coordinator node (in query insights plugin) to get the query-level resource usage.

[deshsidd]

@ansjcy Thanks for the above solution and the proposed alternative.

Did we get some numbers regarding how much percentage lower cpu and memory we can expect with the proposed approach?

In the proposed approach, seems like if we stop the task and resource tracking before sending the result back to the co-ordinator node we will end up increasing the overall search latency which is undesirable.

For the alternative solution, can we piggyback any other background jobs that periodically share data to other nodes?
I am interested in the alternative approach if the overhead is not too drastic.