Lucene 10 in eXist-db, vector search support

[daliboris]

Notes on lucene-update branch of eXist-db.

XML version of the query and options

• eXist-db supports
  • query in plain text using Lucene format
    • format is the same from Lucene 4.2.0, available in the latest version of eXist-db, to 10.2.0
  • query in XML using "eXist-db" format
    • XML Schemas are available in my repository
• Lucene had its own XML Query syntax since version 4.0.0
  • examples of XML queries are in the test code
  • Lucene XML is different than in eXist-db

Lucence	eXist-db
<BooleanQuery fieldName="contents"> <Clause occurs="should"> <TermQuery>merger</TermQuery> </Clause> <Clause occurs="mustnot"> <TermQuery>sumitomo</TermQuery> </Clause> <Clause occurs="must"> <TermQuery>bank</TermQuery> </Clause> </BooleanQuery>	<query> <bool field="contents"> <term occur="sould">merger</term> <term occur="not">sumitomo</term> <term occur="must">bank</term> </bool> </query>

Should eXist-db support XML query syntax used by Lucene? If so, we can try XSLT transform from one syntax to another and vice versa.

Vector queries

Lucene 9 and higher supports vector queries. That means

• during indexing for each field (with textual content)
  • create embedded vector using LLM
• for the query
  • during querying create vector using the same LLM
  • compare it with stored vectors to find the best match(es)
• intensive use of LLMs during indexing and querying

Some questions:

• what if the LLM is not available?
• how to ensure the same LLM is used for indexing and querying?

From the user/developer perspective

How to define

• in the collection.xconf
• with new element <model> for LLM models used for transforming textual content to vector
• with new attribute @vector="true" in the field element
• with new attribute @model in the field element; the value corresponds with @id attribute of the <model> element

<index>
  <lucene>
    <model class="org.exist.indexing.lucene.model.ChatGPT"/>
    <model id="bert" class="org.exist.indexing.lucene.model.BERT"/>
    <text qname="TEI">
      <field name="title" expression="//tei:title" vector="true" />
      <field name="text" expression="//tei:text" vector="true" model="bert" />
    </text>
  </lucene>
<index>

How to query

Classes in Lucene 10

• KnnFloatVectorQuery
  • also allows for performing a kNN search subject to a filter. In this case, it first executes the filter for each leaf, then chooses a strategy dynamically
    • If the filter cost is less than k, just execute an exact search
    • Otherwise run a kNN search subject to the filter
    • If the kNN search visits too many vectors without completing, stop and run an exact search
• KnnByteVectorQuery
  • also allows for performing a kNN search subject to a filter. In this case, it first executes the filter for each leaf, then chooses a strategy dynamically
    • If the filter cost is less than k, just execute an exact search
    • Otherwise run a kNN search subject to the filter
    • If the kNN search visits too many vectors without completing, stop and run an exact search
• PatienceKnnVectorQuery
  • experimental and might change in incompatible ways in the next release
  • exits early when HNSW queue saturates over a #saturationThreshold for more than #patience times
• SeededKnnVectorQuery
  • experimental and might change in incompatible ways in the next release
  • provides a query seed to initiate the vector search. NOTE: The underlying format is free to ignore the provided seed

List of fields (used for vector querying):

Field	Type	Description
field	String	a field that has been indexed as a KnnVectorField
filter	Query	a filter applied before the vector search
target	float[]	the target of the search
k	int	the number of documents to find
searchStrategy	KnnSearchStrategy	strategy for kNN search, providing additional search strategy configuration

Parameters for query constructors, for example

• fromFloatQuery
• fromByteQuery
• fromSeededQuery

Parameter	Type	Description
saturationThreshold	double	the early exit saturation threshold
patience	int	the patience oarameter

Compare with org.apache.lucene.search.KnnVectorQuery​ in Lucence 9

• KnnVectorQuery (String field, float[] target, int k)
• Find the k nearest documents to the target vector according to the vectors in the given field.

Parameter	Notes
field	a field that has been indexed as a KnnVectorField
target	the target of the search
k	the number of documents to find

Notes:

• vector search can be combined with full-text search (KnnFloatVectorQuery, KnnByteVectorQuery)
• Lucene's vector query doesn't accept plain text, but vector representation of the text content

Possible solutions in eXist-db

New function vs:query

• vs for vector search, like ft for full-text (search)

• why not use ft:query

  • vector search doesn't have "Lucene" syntax
  • options are different then for full-text queries

• vs:query signature

  • vs:query($nodes as node()*, $query as item()?, $options as item()?)
  • returns: node()* — all nodes from the collection, which match input node set matching the query.

  Parameter	Description
  $nodes*	The node set to search using a Lucene vector index which is defined on those nodes
  $query?	The query to search for, provided either as a string or text
  $options?	An XML fragment containing options to be passed to Lucene's query parser (see below).

Cons: another method (function) must be created to combine vs:query with ft:query.

Overloaded version of ft:query

• with new parameter and new content of existing parameter
• signature: ft:query($nodes as node()*, $query as item()?, $options as item()?, $vector-query as item()?)
• $vector-query: contains text to be searched by vector query (i.e. must be converted to the target parameter)
• $options: contains new set of options (elements) used for defining vector query, for example
  • <query-model-id>: id of model used for generating embedded vector from the query
  • <field>: field name with vector index to be queried
  • <documents>: number of returned nodes for each query (i.e. k parameter)
  • <saturation-threshold>: saturationThreshold
  • <patience>: patience