[BUG] Text chunking ingest processor fails over 10,000 tokens when using index alias for ingestion

[marcus-bcl]

What is the bug?

When ingesting documents using an index alias, the text chunking ingest processor with the "fixed_token_length" algorithm and "standard" analyzer fails to tokenize text longer than 10,000 tokens.

How can one reproduce the bug?

1. Create an ingest pipeline with the text_chunking processor
2. Create an index with index.analyze.max_token_count set to a value above 10,000
3. Create an alias for the index
4. Post a document with over 10,000 tokens to the alias.

$ curl -XPUT localhost:8080/_ingest/pipeline/my_pipeline --json '{
  "processors": [{
    "text_chunking": {
      "algorithm": {
        "fixed_token_length": {
          "token_limit": 32,
          "max_chunk_limit": -1,
          "max_token_limit": 15000
        }
      },
      "field_map": { "text": "text_chunks" }
    }
  }]
}'
{"acknowledged":true}

$ curl -XPUT localhost:8080/my_index --json '{
  "settings": {
    "default_pipeline": "my_pipeline",
    "index.analyze.max_token_count" : 15000
  }
}'
{"acknowledged":true,"shards_acknowledged":true,"index":"my_index"}

$ curl localhost:8080/_aliases --json '{"actions": [{"add": {"index": "my_index", "alias": "my_alias"}}]}'
{"acknowledged":true}

$ curl localhost:8080/my_alias/_doc/1 --json '{"text": "'"$(printf 'token %.0s' {1..10001})"'"}'         
{"error":{"root_cause":[{"type":"illegal_state_exception","reason":"analyzer standard throws exception: The number of tokens produced by calling _analyze has exceeded the allowed maximum of [10000]. This limit can be set by changing the [index.analyze.max_token_count] index level setting."}],"type":"illegal_state_exception","reason":"analyzer standard throws exception: The number of tokens produced by calling _analyze has exceeded the allowed maximum of [10000]. This limit can be set by changing the [index.analyze.max_token_count] index level setting.","caused_by":{"type":"illegal_state_exception","reason":"The number of tokens produced by calling _analyze has exceeded the allowed maximum of [10000]. This limit can be set by changing the [index.analyze.max_token_count] index level setting."}},"status":500}

Response:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "analyzer standard throws exception: The number of tokens produced by calling _analyze has exceeded the allowed maximum of [10000]. This limit can be set by changing the [index.analyze.max_token_count] index level setting."
      }
    ],
    "type": "illegal_state_exception",
    "reason": "analyzer standard throws exception: The number of tokens produced by calling _analyze has exceeded the allowed maximum of [10000]. This limit can be set by changing the [index.analyze.max_token_count] index level setting.",
    "caused_by": {
      "type": "illegal_state_exception",
      "reason": "The number of tokens produced by calling _analyze has exceeded the allowed maximum of [10000]. This limit can be set by changing the [index.analyze.max_token_count] index level setting."
    }
  },
  "status": 500
}

What is the expected behavior?

Text is split into chunks, respecting the max token count / max token limit.

What is your host/environment?

OpenSearch 2.19 (AWS)

Do you have any additional context?

Originally raised here, before realising this was specific to index aliases: #1321

The workaround is to use the index name explicitly, in which the token limit config works as expected.

[yuye-aws]

May I know the reason why you are ingesting with the index alias, instead of the concrete index name?

[yuye-aws]

This problem is quite tricky because there could actually exists an index named my_alias

[marcus-bcl]

May I know the reason why you are ingesting with the index alias, instead of the concrete index name?

I have a bulk ingestion job that runs regularly. It effectively deletes the index, and then spends a day or so re-ingesting the source data. However, while this process runs, I want to ensure user searches are uninterrupted. So I have two indexes "a" and "b", and two aliases "primary" and "standby". Search queries always target the primary alias, and the bulk ingestion always targets the standby alias. When the bulk ingestion finishes, the aliases are swapped around. This ensures the primary alias always points to a fully-loaded index. I've implemented a workaround to look up the index name from the alias at the start of the job, and to use the index name instead - which is working fine.

I appreciate this may be a slightly niche use-case, and I'm happy we have a workaround for it, so would understand if this isn't a priority to fix.

[yuye-aws]

@marcus-bcl Thanks for providing detailed information. This is a valid use case. A quick solution is to introduce a new parameter named max_token_limit into the fixed token length chunking algorithm. Are you OK with this solution? If not, feel free to propose any alternatives.

[marcus-bcl]

Yes, that would work. Alternatively, would it be possible to look up the index.analyze.max_token_count from the aliased index, if no index exists with the exact name?

[yuye-aws]

@marcus-bcl This, of course, is a better solution. I find there exists two types of request GetAliasesRequest and GetIndexRequest. I am working on other features, so will deep dive later when I have time. If you need the feature now, you can also work on this issue and create an PR. I am always happy to review the code.

[yuye-aws]

@marcus-bcl Here is some sample code. You can try to verify whether it works:

GetAliasesRequest

GetAliasesRequest request = new GetAliasesRequest("your_alias_name");
ActionFuture<GetAliasesResponse> future = client.admin().indices().getAliases(request);
GetAliasesResponse response = future.actionGet();

Please note that please use action handler instead of actionGet.