Add TextChunkingProcessor stats

[q-andy]

Description

Add stats for TextChunkingProcessor. Also adjust stat name unit tests to check for terminal paths.

Event stats:

• text_chunking_executions
• text_chunking_delimiter_executions
• text_chunking_fixed_length_executions

Info stats:

• text_chunking_processors
• text_chunking_fixed_length_processors
• text_chunking_delimiter_processors

{
	"_nodes": {
		"total": 1,
		"successful": 1,
		"failed": 0
	},
	"cluster_name": "integTest",
	"info": {
		"cluster_version": "3.1.0",
		"processors": {
			"ingest": {
				"text_chunking_delimiter_processors": 0,
				"text_chunking_fixed_length_processors": 0,
				"text_embedding_processors_in_pipelines": 0,
				"text_chunking_processors": 0
			}
		}
	},
	"all_nodes": {
		"processors": {
			"ingest": {
				"text_chunking_delimiter_executions": 0,
				"text_chunking_executions": 0,
				"text_embedding_executions": 0,
				"text_chunking_fixed_length_executions": 0
			}
		}
	},
	...
}

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created. <- Since we are backfilling stats for previous features, we will add them all to the spec and documentation near 3.1 release
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created. <- Since we are backfilling stats for previous features, we will add them all to the spec and documentation near 3.1 release

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[will-hwang]

same comment as above

[yuye-aws]

You are exposing your file path here

[yuye-aws]

BWC tests for text chunking processor IT are needed

[q-andy]

Could you explain what cases you want BWC to cover? Stats API will not work as intended during cluster restarts or rolling upgrades since the stat info is stored in memory and lost on restart. This is accepted by design, since the goal is to track long term trends by polling the API regularly.

Existing BWC tests for API are here: https://github.com/opensearch-project/neural-search/blob/main/qa/rolling-upgrade/src/test/java/org/opensearch/neuralsearch/bwc/rolling/RestNeuralStatsActionIT.java

[yuye-aws]

Could you explain what cases you want BWC to cover? Stats API will not work as intended during cluster restarts or rolling upgrades since the stat info is stored in memory and lost on restart. This is accepted by design, since the goal is to track long term trends by polling the API regularly.

Existing BWC tests for API are here: https://github.com/opensearch-project/neural-search/blob/main/qa/rolling-upgrade/src/test/java/org/opensearch/neuralsearch/bwc/rolling/RestNeuralStatsActionIT.java

@vibrantvarun Can you provide more context in the BWC tests?

[bzhangam]

Can we add a default behavior to enforce handling the stats for new algorithm properly? Currently if someone adds a new algorithm and forget to change this function we will silently not record the stats for the new algorithm which is not good.

[q-andy]

What kind of default behavior are you thinking? I think it's okay that we don't enforce stat collection since it's a choice by the feature developer or plugin owners on what stats we choose to collect. Not collecting stats isn't breaking so I don't think it makes sense to throw an exception. And we can't do it automatically since creating a new stat requires creating a new enum which can't be done at runtime.

[yuye-aws]

LGTM

[yuye-aws]

@q-andy Can you resolve the conflicts?

[yuye-aws]

Seems there are some flakey tests. Can @bzhangam also help verify the CI results?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (39559c1) to head (b87c0d7).
Report is 1 commits behind head on main.

Additional details and impacted files

@@     Coverage Diff      @@
##   main   #1308   +/-   ##
============================
============================

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.