Skip to content

Add semantic field mapper. #1225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 17, 2025
Merged

Conversation

bzhangam
Copy link
Collaborator

@bzhangam bzhangam commented Mar 11, 2025

Description

Add semantic field mapper to support the semantic field type.

Also remove the incorrect FF NEURAL_SEARCH_HYBRID_SEARCH_DISABLED.

Related Issues

Resolves #[803]
Resolves part of the issue #1226

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link

codecov bot commented Apr 16, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (b0b8b29) to head (c678c25).
Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #1225       +/-   ##
============================================
- Coverage     80.63%       0   -80.64%     
============================================
  Files           127       0      -127     
  Lines          5722       0     -5722     
  Branches        932       0      -932     
============================================
- Hits           4614       0     -4614     
+ Misses          753       0      -753     
+ Partials        355       0      -355     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment on lines +15 to +21
public static final String MODEL_ID = "model_id";

/**
* Name of the search model id parameter. We use this key to define the id of the ML model that we will use to
* inference the query text during the search. If this parameter is not defined we will use the model_id instead.
*/
public static final String SEARCH_MODEL_ID = "search_model_id";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why we need 2 model_id fields? Can we only keep 1 model_id field? Why we need separate model_ids for inferencing for index and search?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's needed to support the sparse model use case.

@@ -214,17 +225,6 @@ public Map<String, Processor.Factory> getProcessors(Processor.Parameters paramet

@Override
public Optional<QueryPhaseSearcher> getQueryPhaseSearcher() {
// we're using "is_disabled" flag as there are no proper implementation of FeatureFlags.isDisabled(). Both
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why removing NEURAL_SEARCH_HYBRID_SEARCH_DISABLED and related lines is included in this cr?

@bzhangam bzhangam force-pushed the main branch 3 times, most recently from c7890a2 to 3147dcb Compare April 17, 2025 18:55
@junqiu-lei junqiu-lei merged commit 39b4892 into opensearch-project:main Apr 17, 2025
50 checks passed
YeonghyeonKO pushed a commit to YeonghyeonKO/neural-search that referenced this pull request May 30, 2025
YeonghyeonKO pushed a commit to YeonghyeonKO/neural-search that referenced this pull request May 30, 2025
YeonghyeonKO pushed a commit to YeonghyeonKO/neural-search that referenced this pull request May 30, 2025
YeonghyeonKO pushed a commit to YeonghyeonKO/neural-search that referenced this pull request May 30, 2025
YeonghyeonKO pushed a commit to YeonghyeonKO/neural-search that referenced this pull request May 30, 2025
yuye-aws pushed a commit that referenced this pull request Jun 10, 2025
#1342)

* Implement Optimized embedding generation in text embedding processor (#1238)

* implement single document update scenario for text embedding processor (#1191)

Signed-off-by: Will Hwang <[email protected]>

* implement batch document update scenario for text embedding processor (#1217)

Signed-off-by: Will Hwang <[email protected]>

---------

Signed-off-by: Will Hwang <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Going from alpha1 to beta1 for 3.0 release (#1245)

Signed-off-by: yeonghyeonKo <[email protected]>

* Implement Optimized embedding generation in sparse encoding processor (#1246)

Signed-off-by: Will Hwang <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Implement Optimized embedding generation in text and image embedding processor (#1249)

Signed-off-by: will-hwang <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Inner hits support with hybrid query (#1253)

* Inner Hits in Hybrid query

Signed-off-by: Varun Jain <[email protected]>

* Inner hits support with hybrid query

Signed-off-by: Varun Jain <[email protected]>

* Add changelog

Signed-off-by: Varun Jain <[email protected]>

* fix integ tests

Signed-off-by: Varun Jain <[email protected]>

* Modify comment

Signed-off-by: Varun Jain <[email protected]>

* Explain test case

Signed-off-by: Varun Jain <[email protected]>

* Optimize inner hits count calculation method

Signed-off-by: Varun Jain <[email protected]>

---------

Signed-off-by: Varun Jain <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Support custom tags in semantic highlighter (#1254)

Signed-off-by: yeonghyeonKo <[email protected]>

* Add neural stats API (#1256)

* Add neural stats API

Signed-off-by: Andy Qin <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Added release notes for 3.0 beta1 (#1252)

* Added release notes for 3.0 beta1

Signed-off-by: Martin Gaievski <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Update semantic highlighter test model (#1259)

Signed-off-by: Junqiu Lei <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Fix the edge case when the value of a fieldMap key in ingestDocument is empty string (#1257)

Signed-off-by: Chloe Gao <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Hybrid query should call rewrite before creating weight (#1268)

* Hybrid query should call rewrite before creating weight

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

* Awaits fix

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

* Rewrite with searcher

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

* Feature flag issue

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

---------

Signed-off-by: Harsha Vamsi Kalluri <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Support phasing off SecurityManager usage in favor of Java Agent (#1265)

Signed-off-by: Gulshan <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Fix multi node transport issue on NeuralKNNQueryBuilder originalQueryText (#1272)

Signed-off-by: Junqiu Lei <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Add semantic field mapper. (#1225)

Signed-off-by: Bo Zhang <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Increment version to 3.0.0-SNAPSHOT (#1286)

Signed-off-by: opensearch-ci-bot <[email protected]>
Co-authored-by: opensearch-ci-bot <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Remove beta1 qualifier (#1292)

Signed-off-by: Peter Zhu <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Fix for merging scoreDocs when totalHits are greater than 1 and fieldDocs are 0 (#1295) (#1296)

(cherry picked from commit 6f3aabb)

Co-authored-by: Varun Jain <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* add release notes for 3.0 (#1287)

Signed-off-by: will-hwang <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Allow maven to publish to all versions (#1300) (#1301)

Signed-off-by: Peter Zhu <[email protected]>
(cherry picked from commit c5625db)

Co-authored-by: Peter Zhu <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* [FEAT] introduce new FixedStringLengthChunker

Signed-off-by: yeonghyeonKo <[email protected]>

* [TEST] initial test cases for FixedStringLengthChunker

Signed-off-by: yeonghyeonKo <[email protected]>

* [FIX] gradlew spotlessApply

Signed-off-by: yeonghyeonKo <[email protected]>

* [REFACTOR] remove unnecessary comments

Signed-off-by: yeonghyeonKo <[email protected]>

* [Performance Improvement] Add custom bulk scorer for hybrid query (2-3x faster) (#1289)

Signed-off-by: Martin Gaievski <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Add TextChunkingProcessor stats (#1308)

* Add TextChunkingProcessor stats

Signed-off-by: Andy Qin <[email protected]>

# Conflicts:
#	CHANGELOG.md

* Update unit and integ tests

Signed-off-by: Andy Qin <[email protected]>

---------

Signed-off-by: Andy Qin <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Update Lucene dependencies (#1336)

* Update Lucene dependencies

Signed-off-by: Ryan Bogan <[email protected]>

* Add changelog entry

Signed-off-by: Ryan Bogan <[email protected]>

* Update model request body for bwc and integ tests

Signed-off-by: Ryan Bogan <[email protected]>

---------

Signed-off-by: Ryan Bogan <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* [REFACTOR] modify algorithm name and related parts

Signed-off-by: yeonghyeonKo <[email protected]>

* [REFACTOR] update test codes along with the change in CharacterLengthChunker

Signed-off-by: yeonghyeonKo <[email protected]>

* [REFACTOR] remove defensive check to prevent adding redundant code lines

Signed-off-by: yeonghyeonKo <[email protected]>

* Update CharacterLengthChunker to FixedCharLengthChunker

Signed-off-by: Marcel Yeonghyeon Ko <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Update ChunkerFactory

Signed-off-by: Marcel Yeonghyeon Ko <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Update CharacterLengthChunkerTests to FixedCharLengthChunkerTests

Signed-off-by: Marcel Yeonghyeon Ko <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* [FIX] handle a corner case where the content is shorter than charLimit

Signed-off-by: yeonghyeonKo <[email protected]>

* [TEST] Add integration test codes for fixed_char_length chunking algorithm

Signed-off-by: yeonghyeonKo <[email protected]>

* [TEST] integration test code for cascaded pipeline

Signed-off-by: yeonghyeonKo <[email protected]>

* Support analyzer-based neural sparse query (#1088)

* merge main; add analyzer impl

Signed-off-by: zhichao-aws <[email protected]>

* two phase adaption

Signed-off-by: zhichao-aws <[email protected]>

* two phase adaption

Signed-off-by: zhichao-aws <[email protected]>

* remove analysis

Signed-off-by: zhichao-aws <[email protected]>

* lint

Signed-off-by: zhichao-aws <[email protected]>

* update

Signed-off-by: zhichao-aws <[email protected]>

* address comments

Signed-off-by: zhichao-aws <[email protected]>

* tests

Signed-off-by: zhichao-aws <[email protected]>

* modify plugin security policy

Signed-off-by: zhichao-aws <[email protected]>

* change log

Signed-off-by: zhichao-aws <[email protected]>

* address comments

Signed-off-by: zhichao-aws <[email protected]>

* modify to package-private

Signed-off-by: zhichao-aws <[email protected]>

---------

Signed-off-by: zhichao-aws <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Fixed score value as null for single shard for sorting (#1277)

* Fixed score value as null for single shard for sorting

Signed-off-by: Owais <[email protected]>

* Addressed comment

Signed-off-by: Owais <[email protected]>

* Addressed more comments

Signed-off-by: Owais <[email protected]>

* Added UT

Signed-off-by: Owais <[email protected]>

---------

Signed-off-by: Owais <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Add IT for neural sparse query + bert-uncased mbert-uncased analyzer (#1279)

* add it

Signed-off-by: zhichao-aws <[email protected]>

* change log

Signed-off-by: zhichao-aws <[email protected]>

---------

Signed-off-by: zhichao-aws <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Add WithFieldName implementation to QueryBuilders (#1285)

Signed-off-by: Owais <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* [AUTO] Increment version to 3.1.0-SNAPSHOT (#1288)

* Increment version to 3.1.0-SNAPSHOT

Signed-off-by: opensearch-ci-bot <[email protected]>

* Update build.gradle

Signed-off-by: Peter Zhu <[email protected]>

---------

Signed-off-by: opensearch-ci-bot <[email protected]>
Signed-off-by: Peter Zhu <[email protected]>
Co-authored-by: opensearch-ci-bot <[email protected]>
Co-authored-by: Peter Zhu <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* add release notes for 3.0 (#1298)

Signed-off-by: will-hwang <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Return bad request for invalid stat parameters in stats API (#1291)

Signed-off-by: Andy Qin <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Add semantic mapping transformer. (#1276)

Signed-off-by: Bo Zhang <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Add semantic ingest processor. (#1309)

Signed-off-by: Bo Zhang <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* [Performance Improvement] Add custom bulk scorer for hybrid query (2-3x faster) (#1289)

Signed-off-by: Martin Gaievski <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Implement the query logic for the semantic field. (#1315)

Signed-off-by: Bo Zhang <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Support custom weights params in RRF (#1322)

* Support Weights params in RRF

Signed-off-by: Varun Jain <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* add validation for invalid nested hybrid query (#1305)

* add validation for nested hybrid query

Signed-off-by: will-hwang <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Add stats tracking for semantic highlighting (#1327)

* Add stats tracking for semantic highlighting

Signed-off-by: Junqiu Lei <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Update Lucene dependencies (#1336)

* Update Lucene dependencies

Signed-off-by: Ryan Bogan <[email protected]>

* Add changelog entry

Signed-off-by: Ryan Bogan <[email protected]>

* Update model request body for bwc and integ tests

Signed-off-by: Ryan Bogan <[email protected]>

---------

Signed-off-by: Ryan Bogan <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Enhance semantic field to allow to enable/disable chunking. (#1337)

* Implement the query logic for the semantic field.

Signed-off-by: Bo Zhang <[email protected]>

* Enhance semantic field to allow to enable/disable chunking.

Signed-off-by: Bo Zhang <[email protected]>

---------

Signed-off-by: Bo Zhang <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* [REFACTOR] modify algorithm name and related parts

Signed-off-by: yeonghyeonKo <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Marcel Yeonghyeon Ko <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* [FEAT] Add fixed_char_length chunking algorithm to STAT manager

Signed-off-by: yeonghyeonKo <[email protected]>

* [TEST] Add integration test codes for fixed_char_length chunking algorithm

Signed-off-by: yeonghyeonKo <[email protected]>

* [TEST] integration test code for cascaded pipeline

Signed-off-by: yeonghyeonKo <[email protected]>

* Going from alpha1 to beta1 for 3.0 release (#1245)

Signed-off-by: yeonghyeonKo <[email protected]>

* Fix multi node transport issue on NeuralKNNQueryBuilder originalQueryText (#1272)

Signed-off-by: Junqiu Lei <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Add semantic field mapper. (#1225)

Signed-off-by: Bo Zhang <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Add semantic mapping transformer. (#1276)

Signed-off-by: Bo Zhang <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>

* Fix multi node transport issue on NeuralKNNQueryBuilder originalQueryText (#1272)

Signed-off-by: Junqiu Lei <[email protected]>

* Add semantic field mapper. (#1225)

Signed-off-by: Bo Zhang <[email protected]>

* Add semantic mapping transformer. (#1276)

Signed-off-by: Bo Zhang <[email protected]>

* [FIX] minor typo

Signed-off-by: yeonghyeonKo <[email protected]>

* [REFACTOR] adopt FixedTokenLengthChunker's loop strategy for robust final chunking

Signed-off-by: yeonghyeonKo <[email protected]>

* [TEST] sum the number of processors and their executions correctly in TextChunkingProcessorIT

Signed-off-by: yeonghyeonKo <[email protected]>

* [REFACTOR] gradlew spotlessApply

Signed-off-by: yeonghyeonKo <[email protected]>

---------

Signed-off-by: Will Hwang <[email protected]>
Signed-off-by: yeonghyeonKo <[email protected]>
Signed-off-by: will-hwang <[email protected]>
Signed-off-by: Varun Jain <[email protected]>
Signed-off-by: Andy Qin <[email protected]>
Signed-off-by: Martin Gaievski <[email protected]>
Signed-off-by: Junqiu Lei <[email protected]>
Signed-off-by: Chloe Gao <[email protected]>
Signed-off-by: Harsha Vamsi Kalluri <[email protected]>
Signed-off-by: Gulshan <[email protected]>
Signed-off-by: Bo Zhang <[email protected]>
Signed-off-by: opensearch-ci-bot <[email protected]>
Signed-off-by: Peter Zhu <[email protected]>
Signed-off-by: Ryan Bogan <[email protected]>
Signed-off-by: Marcel Yeonghyeon Ko <[email protected]>
Signed-off-by: zhichao-aws <[email protected]>
Signed-off-by: Owais <[email protected]>
Co-authored-by: Will Hwang <[email protected]>
Co-authored-by: Martin Gaievski <[email protected]>
Co-authored-by: Varun Jain <[email protected]>
Co-authored-by: Junqiu Lei <[email protected]>
Co-authored-by: Andy <[email protected]>
Co-authored-by: Chloe Gao <[email protected]>
Co-authored-by: Harsha Vamsi Kalluri <[email protected]>
Co-authored-by: Gulshan <[email protected]>
Co-authored-by: Bo Zhang <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: opensearch-ci-bot <[email protected]>
Co-authored-by: Peter Zhu <[email protected]>
Co-authored-by: Ryan Bogan <[email protected]>
Co-authored-by: zhichao-aws <[email protected]>
Co-authored-by: Owais Kazi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancements Increases software capabilities beyond original client specifications neural-search
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[PROPOSAL] Neural Search field type
6 participants