[Feature Request] Add granular logging for ingest pipeline processor failure

[shiv0408]

Is your feature request related to a problem? Please describe

Currently, within the OpenSearch project, the logging mechanism for failures in the ingest pipeline processors lacks granularity, we only have failure metric count. As a result, troubleshooting and debugging issues related to failed processors become cumbersome and inefficient for developers and administrators.

OpenSearch/server/src/main/java/org/opensearch/ingest/CompoundProcessor.java

Lines 164 to 186 in 3056b67

	processor.execute(ingestDocument, (result, e) -> {
	long ingestTimeInMillis = TimeUnit.NANOSECONDS.toMillis(relativeTimeProvider.getAsLong() - startTimeInNanos);
	metric.after(ingestTimeInMillis);
	if (e != null) {
	metric.failed();
	if (ignoreFailure) {
	innerExecute(currentProcessor + 1, ingestDocument, handler);
	} else {
	IngestProcessorException compoundProcessorException = newCompoundProcessorException(e, processor, ingestDocument);
	if (onFailureProcessors.isEmpty()) {
	handler.accept(null, compoundProcessorException);
	} else {
	executeOnFailureAsync(0, ingestDocument, compoundProcessorException, handler);
	}
	}
	} else {
	if (result != null) {
	innerExecute(currentProcessor + 1, result, handler);
	} else {
	handler.accept(null, null);
	}
	}

Describe the solution you'd like

We propose adding a debug level logging feature dedicated to capturing detailed information about failures occurring within ingest pipeline processors.

Related component

Indexing

[andrross]

[Triage - attendees 1 2 3]
@shiv0408 Thanks for filing this issue. Sounds like a good improvement.

[nicdard]

@andrross I would like to work on this if it is still available

[shiv0408]

@nicdard Sure, please go ahead.

[andrross]

@nicdard I've assigned this to you if you're still interested in working on it

[nicdard]

@andrross thanks yes I am still interested, could you however please specify better what data you were thinking to log here?

[andrross]

@shiv0408 can you answer @nicdard's question?

[shiv0408]

@nicdard I am thinking that we should just log the name of failed process along with the exception but if ignoreFailure is true maybe add that we are going to move to next processor.