Skip to content

Move filtered index creation totally to Airflow #3240

Open
@krysal

Description

@krysal

Problem

Currently, the DAGs for the creation of filtered indexes (for image and audio) depend on the Ingestion Server. There is no reason we can not leave all that work to Airflow and it would be preferable to have fewer moving parts so it's also easier to debug when things go wrong.

Description

Move the create_and_populate_filtered_index function out of the Ingestion Server to the create filtered index DAG in the Catalog.

def create_and_populate_filtered_index(
self,
model_name: str,
origin_index_suffix: str | None = None,
destination_index_suffix: str | None = None,
**_,
):

Additional context

This will be required down the line for other DAGs in the Search relevancy sandbox project.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status

    ⛔ Blocked

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions