Add Lucene improvements for HNSW merging #129046

carlosdelest · 2025-06-06T11:42:04Z

apache/lucene#14527 improved the heap usage for HNSW merging.

As this change still hasn't made any Lucene release, we'd like to incorporate this into Elasticsearch.

We have copied over the Lucene changes into Elasticsearch code and create a new Elasticsearch codec that will use the copy of the Lucene Lucene99HnswVectorsWriter, renamed to ES910HnswReducedHeapVectorsWriter.

Changes done:

Rename Elasticsearch900Lucene101Codec to Elasticsearch910Lucene102Codec, to signal the codec is for ES 9.10 and Lucene 10.2
Create a new ES910HnswVectorsFormat that will provide the entry point for the writer vector changes.
- The new format is used to replace the Lucene99HnswVectorsFormat usages, as it is compatible with it - same file formats
Create a new ES910HnswVectorsWriter that uses the copied over classes from Lucene. HNSW improvements will be on the merge side, so we're only focused on the Writer aspect of the format.
Three packages are created to hold the copied code from Lucene:
- org.elasticsearch.index.codec.vectors.es910.hnsw
- org.elasticsearch.index.codec.vectors.es910.internal.hppc
- org.elasticsearch.index.codec.vectors.es910.util

We can remove these changes when Lucene 10.3 is created and merged into Elasticsearch.

elasticsearchmachine · 2025-06-06T11:43:16Z

Hi @carlosdelest, I've created a changelog YAML for you.

…ne99 vector writers

…reduce-heap' into feature/dense-vector-hnsw-reduce-heap

benwtrent · 2025-06-06T16:20:30Z

...rc/main/java/org/elasticsearch/index/codec/vectors/es910/hnsw/FilteredHnswGraphSearcher.java

+ *       additional candidates is predicated on the original candidate's filtered percentage.
+ * </ul>
+ */
+public class FilteredHnswGraphSearcher extends HnswGraphSearcher {


I don't think we need this? Only used on read, never during graph building.

I see, removed in 378111f. Thanks!

benwtrent · 2025-06-06T16:21:11Z

server/src/main/java/org/elasticsearch/index/codec/vectors/es910/hnsw/HnswGraph.java

+ * thread-safe. The search method optionally takes a set of "accepted nodes", which can be used to
+ * exclude deleted documents.
+ */
+public abstract class HnswGraph {


if these are public in Lucene, can we just rely on them or are they not exposed in the module?

benwtrent · 2025-06-06T17:23:30Z

server/src/main/java/org/elasticsearch/index/codec/vectors/es910/ES910HnswVectorsFormat.java

+        if (numMergeWorkers == 1 && mergeExec != null) {
+            throw new IllegalArgumentException("No executor service is needed as we'll use single thread to merge");
+        }
+        this.numMergeWorkers = numMergeWorkers;
+        if (mergeExec != null) {
+            this.mergeExec = new TaskExecutor(mergeExec);
+        } else {
+            this.mergeExec = null;
+        }


we don't need any of this merge worker stuff, we don't use it.

Right - removed in a6db01a

benwtrent · 2025-06-06T17:25:35Z

server/src/main/java/org/elasticsearch/index/codec/vectors/es910/ES910HnswVectorsWriter.java

+        if (parallelMergeTaskExecutor != null && numParallelMergeWorkers > 1) {
+            return new ConcurrentHnswMerger(fieldInfo, scorerSupplier, M, beamWidth, parallelMergeTaskExecutor, numParallelMergeWorkers);
+        }
+        return new IncrementalHnswGraphMerger(fieldInfo, scorerSupplier, M, beamWidth);


we only need IncrementalHnswGraphMerger we never use the task exec stuff nor do concurrent merges.

Removed as part of a6db01a

benwtrent · 2025-06-06T17:26:04Z

server/src/main/java/org/elasticsearch/index/codec/vectors/es910/hnsw/ConcurrentHnswMerger.java

+import static org.apache.lucene.search.DocIdSetIterator.NO_MORE_DOCS;
+
+/** This merger merges graph in a concurrent manner, by using {@link HnswConcurrentMergeBuilder} */
+public class ConcurrentHnswMerger extends IncrementalHnswGraphMerger {


don't need this one.

Removed in a6db01a

benwtrent · 2025-06-06T17:29:54Z

.../src/main/java/org/elasticsearch/index/codec/vectors/es910/hnsw/SeededHnswGraphSearcher.java

+ *
+ * @lucene.experimental
+ */
+final class SeededHnswGraphSearcher extends AbstractHnswGraphSearcher {


don't need this

benwtrent · 2025-06-06T17:30:27Z

.../src/main/java/org/elasticsearch/index/codec/vectors/es910/internal/hppc/HashContainers.java

+ *
+ * @lucene.internal
+ */
+class HashContainers {


I wonder if we can just extract what we want and directly place it into the fixed sized array thigns?

benwtrent · 2025-06-06T17:31:23Z

server/src/main/java/org/elasticsearch/index/codec/vectors/es910/util/ArrayUtil.java

+
+public class ArrayUtil {
+
+    public static float[] growInRange(float[] array, int minLength, int maxLength) {


I think ES has access growInRange, I am not sure if any of the changes for growInRange are important?

benwtrent · 2025-06-06T17:32:17Z

server/src/main/java/org/elasticsearch/index/codec/vectors/es910/hnsw/NeighborQueue.java

+ * #insertWithOverflow(int, float)} and {@link #add(int, float)}, and provides MIN and MAX heap
+ * subclasses.
+ */
+public class NeighborQueue {


not sure we need to copy this? I thought this was available outside of lucene since its a public class. Maybe Lucene doesn't export the module?

…List

benwtrent · 2025-06-09T13:56:31Z

server/src/main/java/org/elasticsearch/index/codec/vectors/es910/hnsw/HnswUtil.java

+    private HnswUtil() {}
+
+    // Finds orphaned components on the graph level.
+    static List<Component> components(HnswGraph hnsw, int level, FixedBitSet notFullyConnected, int maxConn) throws IOException {


I don't think any of this is used now, maybe we can remove it.

That's correct - removed in 457e9a4

benwtrent · 2025-06-09T13:58:01Z

server/src/main/java/org/elasticsearch/index/codec/vectors/es910/hnsw/NeighborArray.java

+
+    @Override
+    public long ramBytesUsed() {
+        return BASE_RAM_BYTES_USED + nodes.ramBytesUsed() + scores.ramBytesUsed();


We should update the way Lucene does this (and consequently this) if we merge this. I think this will have a real performance impact as the way its done now in the builder is that it will iterate every array (e.g. every node) and do a calculation. That makes no sense IMO.

…hnsw-reduce-heap

carlosdelest · 2025-06-10T07:46:57Z

@benwtrent @ChrisHegarty I've removed the non-needed classes and tried to prune the changes down to the minimum, including the removal of a new codec name.

Do you think there's anything else we can do to reduce the size of this change?

It's still a 3K LOC change. Is this something we should add to ES 9.1/8.19?

carlosdelest · 2025-06-12T15:30:35Z

Closing as this is too big a change - let's wait for a Lucene release that contains the code

carlosdelest added 6 commits June 5, 2025 18:44

Copy over Lucene classes

4823d30

Create vector format class that references Lucene changes

6f6097d

Checkstyle

01a5be0

Change version to 910

9f6852e

Some renaming and refactoring

402f480

Change tests to reference the latest codec version

Loading
Loading status checks…

acbbb13

carlosdelest added Team:Search Relevance v9.1.0 :Search Relevance/Vectors :Search Relevance/Search >enhancement labels Jun 6, 2025

carlosdelest and others added 13 commits June 6, 2025 13:43

Update docs/changelog/129046.yaml

Loading
Loading status checks…

c0e1b91

[CI] Auto commit changes from spotless

Loading
Loading status checks…

2d18be6

Fix changelog

c1bd0ff

Add test, fix vectors format name and add to module-info

Loading
Loading status checks…

433ac23

[CI] Auto commit changes from spotless

Loading
Loading status checks…

11088f9

Change KnnVectorsWriter to use in HNSW vector formats, replacing Luce…

934a066

…ne99 vector writers

Fix test name

19bc676

Merge remote-tracking branch 'carlosdelest/feature/dense-vector-hnsw-…

33583da

…reduce-heap' into feature/dense-vector-hnsw-reduce-heap

Spotless

c786d14

Renaming

Loading
Loading status checks…

98234a0

[CI] Auto commit changes from spotless

Loading
Loading status checks…

86ff8f1

Fix javadoc

49b666f

Merge remote-tracking branch 'carlosdelest/feature/dense-vector-hnsw-…

Loading
Loading status checks…

97d993c

…reduce-heap' into feature/dense-vector-hnsw-reduce-heap

benwtrent reviewed Jun 6, 2025

View reviewed changes

carlosdelest added 7 commits June 9, 2025 12:46

Remove ConcurrentHnswMerger and related classes / params

a6db01a

Remove FilteredHnswGraphSearcher

378111f

Remove HnswGraph and related classes

0489e7f

Remove HnswGraphMerger, clean classes

8a0a968

Remove NeighborQueue

6373949

Remove unneeded code from MaxSizedFloatArrayList and MaxSizedIntArray…

a954936

…List

Remove HnswBuilder interface

Loading
Loading status checks…

6a0826e

benwtrent reviewed Jun 9, 2025

View reviewed changes

carlosdelest added 5 commits June 9, 2025 16:38

Remove HnswUtil

Loading
Loading status checks…

457e9a4

Use same codec name, change just writer implementation

2d8f30c

Merge remote-tracking branch 'origin/main' into feature/dense-vector-…

Loading
Loading status checks…

2d8329d

…hnsw-reduce-heap

Revert server/module-info

Loading
Loading status checks…

a595100

Revert server/module-info

Loading
Loading status checks…

d4c8037

Fix vector format name

Loading
Loading status checks…

5009508

carlosdelest closed this Jun 12, 2025


		public class ArrayUtil {

		public static float[] growInRange(float[] array, int minLength, int maxLength) {

Add Lucene improvements for HNSW merging #129046

Add Lucene improvements for HNSW merging #129046

Conversation

carlosdelest commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benwtrent Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

carlosdelest commented Jun 10, 2025

Uh oh!

Uh oh!

carlosdelest commented Jun 12, 2025

Uh oh!

carlosdelest commented Jun 6, 2025 •

edited

Loading

benwtrent Jun 9, 2025 •

edited

Loading