Support for IndexSort in Nested Fields

[vishdivs]

Description

Since Lucene now supports IndexWriterConfig#setParentField for creating an internal field to enable index sorting, OpenSearch needs to integrate this functionality by setting the Parent Field in appropriate workflows with proper validations.
#17026 (comment)

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#17026

Testing Details:

Testing case 1: Before the Changes

curl -XPUT "localhost:9200/test_nested_sort" -H 'Content-Type: application/json' -d '{      
  "settings": {
    "index": { 
      "sort.field": ["users.age", "users.name.keyword"],
      "sort.order": ["desc", "asc"]
    }                  
  },   
  "mappings": {             
    "properties": {
      "users": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "age": {
            "type": "integer"
          }
        }
      }
    }
  }
}'

Result for this:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"cannot have nested fields when index sort is activated"}],"type":"illegal_argument_exception","reason":"cannot have nested fields when index sort is activated"},"status":400}%

Testing case 2: Testing when Index Sort field is inside Nested Field (With Validation)

curl -XPUT "localhost:9200/test_nested_sort" -H 'Content-Type: application/json' -d '{
  "settings": {
    "index": {
      "sort.field": ["users.age", "users.name.keyword"],
      "sort.order": ["desc", "asc"]
    }
  },
  "mappings": {
    "properties": {
      "users": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "age": {
            "type": "integer"
          }
        }
      }
    }
  }
}'

Result:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"index sorting on nested fields is not supported: found nested sort field [users.age] in [test_nested_sort]"}],"type":"illegal_argument_exception","reason":"index sorting on nested fields is not supported: found nested sort field [users.age] in [test_nested_sort]"},"status":400}%

Test case 3:

curl -XPUT "localhost:9200/test_nested_sort" -H 'Content-Type: application/json' -d '{
  "settings": {
    "index": {
      "sort.field": ["foo"],
      "sort.order": ["desc"]
    }
  },
  "mappings": {
    "properties": {
      "foo": {
        "type": "integer"
      },
      "users": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "age": {
            "type": "integer"
          }
        }
      }
    }
  }
}'
{"acknowledged":true,"shards_acknowledged":true,"index":"test_nested_sort"}

Tested with the search query and result:

curl -XGET "localhost:9200/test_nested_sort/_search" -H 'Content-Type: application/json' -d '{
  "query": {
    "nested": {                      
      "path": "users",
      "query": {                                        
        "bool": {                  
          "should": [  
            {"match": {"users.name": "Alice"}},
            {"range": {"users.age": {"gte": 20}}}
          ]        
        }
      }
    }                         
  },            
  "sort": [       
    {"foo": {"order": "desc"}},
    {
      "users.age": {
        "order": "desc",
        "nested": {
          "path": "users"
        }
      }
    }
  ]
}'
{"took":619,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":3,"relation":"eq"},"max_score":null,"hits":[{"_index":"test_nested_sort","_id":"uArTfJcBNvxL7QBM7V8f","_score":null,"_source":{
  "foo": 40,
  "users": [                         
    {          
      "name": "Bob Johnson",                            
      "age": 35                    
    },                 
    {           
      "name": "Jane Wilson",
      "age": 50    
    },
    {
      "name": "armane Wilson",
      "age": 50 
      }           
  ]                      
},"sort":[40,50]},{"_index":"test_nested_sort","_id":"tgrTfJcBNvxL7QBMcV9g","_score":null,"_source":{
  "foo": 30,
  "users": [
    {
      "name": "Alice Smith",
      "age": 25
    },
    {
      "name": "John Doe",
      "age": 30
    }
  ]
},"sort":[30,30]},{"_index":"test_nested_sort","_id":"ugrYfJcBNvxL7QBMBl_t","_score":null,"_source":{
  "foo": 25,
  "users": [                         
    {          
      "name": "Bob Johnson",                            
      "age": 35                    
    },                 
    {           
      "name": "Jane Wilson",
      "age": 50    
    },
    {
      "name": "armane Wilson",
      "age": 50 
      }           
  ]                      
},"sort":[25,50]}]}}%

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for b6fadef: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[vibrantvarun]

@vishdivs Can you add a bwc test for the same ?

[vishdivs]

@vishdivs Can you add a bwc test for the same ?

@vibrantvarun, I have already added validation check related to Index Version here
Do we need any other BWC tests besides this? Please let me know if there are any specific tests required.

[github-actions]

❕ Gradle check result for 2fcc7ab: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.