Skip to content

[Feature Request] Add README in example system pipeline plugin for system ingest pipeline extension points #18496

Open
@q-andy

Description

@q-andy

Is your feature request related to a problem? Please describe

System ingest pipelines added in #17817 are a new pipeline type which allow plugins to dynamically inject and execute processors based on the index mapping or other conditional checks. In essence, system pipelines can allow plugin developers to implement features that completely abstract pipelines/processor executions from the user.

Currently, system ingest pipelines are used to support semantic field (RFC), where we chunk and generate embeddings for semantic field text without the user having to manually configure pipelines themselves.

While system pipelines were developed with semantic field in mind, the extension points are flexible and may be used to develop future field types implemented by other plugins as well.

Describe the solution you'd like

We should document in detail nuances of how system pipelines work outside of the code, such as

Since this documentation is targeted at developers, I believe this should go in a readme in the code base itself, rather than in the documentation website which is for users of OpenSearch

Related component

Indexing

Describe alternatives you've considered

not having explicit documentation, letting future developers read the code

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    IndexingIndexing, Bulk Indexing and anything related to indexingenhancementEnhancement or improvement to existing feature or requestuntriaged

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions