feat(native): Add index source plan conversion with connector extensibility (#27596)

[zacw7]

Summary:

Add a virtual method toVeloxTableHandleForIndexSource to PrestoToVeloxConnector that allows connectors to customize how an IndexSourceNode is converted into a Velox ConnectorTableHandle. The default implementation delegates to toVeloxTableHandle, ignoring the IndexHandle. Connectors that support index lookup can override this to incorporate the IndexHandle into the resulting table handle, enabling index-aware table scan creation.

Without this extensibility point, the IndexHandle (carrying index column names resolved by the coordinator) would be silently dropped during worker-side plan conversion, and the worker would have no way to know which columns are indexed.

Also handles the case where IndexJoinOptimizer inserts a FilterNode between IndexJoinNode and IndexSourceNode for non-domain-expressible predicates. The FilterNode is unwrapped and its filter is merged into the join's remaining filter.

Changes:

• PrestoToVeloxConnector.h: Add toVeloxTableHandleForIndexSource() virtual method with default delegation to toVeloxTableHandle().
• PrestoToVeloxQueryPlan.cpp: Add toConnectorTableHandleForIndexSource() free function, update IndexSourceNode conversion to use it, add VELOX_CHECK_NOT_NULL guard, and add FilterNode unwrapping for IndexJoinNode.
• PlanConverterTest.cpp: Add indexSource test covering IndexSourceNode to TableScanNode conversion.

Release Notes

== NO RELEASE NOTE ==

[sourcery-ai]

Reviewer's Guide

Adds an extensibility point for connector-specific IndexSourceNode-to-ConnectorTableHandle conversion and wires IndexSourceNode plan conversion to use it, plus a regression test ensuring default behavior remains equivalent to a normal table scan producing a HiveTableHandle.

Sequence diagram for IndexSourceNode to ConnectorTableHandle conversion with index handle

sequenceDiagram
  participant PlanConverter as VeloxQueryPlanConverterBase
  participant IndexNode as protocol_IndexSourceNode
  participant Helper as toConnectorTableHandleForIndexSource
  participant ConnectorRegistry as PrestoToVeloxConnectorRegistry
  participant Connector as PrestoToVeloxConnector
  participant TableScan as core_TableScanNode

  PlanConverter->>PlanConverter: VELOX_CHECK_NOT_NULL(node)
  PlanConverter->>IndexNode: read outputVariables, assignments, tableHandle, indexHandle
  PlanConverter->>Helper: toConnectorTableHandleForIndexSource(tableHandle, indexHandle, exprConverter, typeParser)
  Helper->>ConnectorRegistry: getPrestoToVeloxConnector(tableHandle.connectorHandle._type)
  ConnectorRegistry-->>Helper: connector
  Helper->>Connector: toVeloxTableHandleForIndexSource(tableHandle, indexHandle, exprConverter, typeParser)
  alt default_implementation
    Connector-->>Helper: toVeloxTableHandle(tableHandle, exprConverter, typeParser)
  else index_aware_connector
    Connector-->>Helper: index_aware ConnectorTableHandle
  end
  Helper-->>PlanConverter: connectorTableHandle
  PlanConverter->>TableScan: create core_TableScanNode(id, rowType, connectorTableHandle, assignments)
  TableScan-->>PlanConverter: TableScanNode instance

Loading

Updated class diagram for PrestoToVeloxConnector and IndexSourceNode plan conversion

classDiagram
  class PrestoToVeloxConnector {
    <<interface>>
    +toVeloxTableHandle(tableHandle: protocol_TableHandle, exprConverter: VeloxExprConverter, typeParser: TypeParser) ConnectorTableHandle
    +toVeloxTableHandleForIndexSource(tableHandle: protocol_TableHandle, indexHandle: protocol_IndexHandle, exprConverter: VeloxExprConverter, typeParser: TypeParser) ConnectorTableHandle
    +toVeloxInsertTableHandle(tableHandle: protocol_TableHandle, exprConverter: VeloxExprConverter, typeParser: TypeParser) ConnectorInsertTableHandle
  }

  class VeloxQueryPlanConverterBase {
    -exprConverter_: VeloxExprConverter
    -typeParser_: TypeParser
    +toVeloxQueryPlan(node: protocol_IndexSourceNode, tableWriteInfo: TableWriteInfo, taskId: TaskId) core_PlanNode
  }

  class PlanConversionHelpers {
    +toConnectorTableHandle(tableHandle: protocol_TableHandle, exprConverter: VeloxExprConverter, typeParser: TypeParser) ConnectorTableHandle
    +toConnectorTableHandleForIndexSource(tableHandle: protocol_TableHandle, indexHandle: protocol_IndexHandle, exprConverter: VeloxExprConverter, typeParser: TypeParser) ConnectorTableHandle
  }

  class protocol_IndexSourceNode {
    +id: string
    +outputVariables: VariableList
    +assignments: AssignmentMap
    +tableHandle: protocol_TableHandle
    +indexHandle: protocol_IndexHandle
  }

  class protocol_TableHandle {
    +connectorHandle: ConnectorHandle
  }

  class protocol_IndexHandle {
    +indexColumns: ColumnList
  }

  class core_TableScanNode {
    +id: string
    +rowType: RowType
    +tableHandle: ConnectorTableHandle
    +assignments: ColumnHandleMap
  }

  class ConnectorTableHandle
  class ConnectorInsertTableHandle
  class VeloxExprConverter
  class TypeParser
  class TableWriteInfo
  class TaskId
  class core_PlanNode

  class ConcreteConnector {
    +toVeloxTableHandleForIndexSource(tableHandle: protocol_TableHandle, indexHandle: protocol_IndexHandle, exprConverter: VeloxExprConverter, typeParser: TypeParser) ConnectorTableHandle
  }

  VeloxQueryPlanConverterBase --> PlanConversionHelpers : calls
  PlanConversionHelpers --> PrestoToVeloxConnector : calls
  protocol_IndexSourceNode --> protocol_TableHandle : has
  protocol_IndexSourceNode --> protocol_IndexHandle : has
  core_TableScanNode --> ConnectorTableHandle : has
  PrestoToVeloxConnector <|.. ConcreteConnector

Loading

File-Level Changes

Change	Details	Files
Introduce connector hook for IndexSourceNode-specific table handle creation and default behavior.	Add virtual toVeloxTableHandleForIndexSource() to PrestoToVeloxConnector, taking a TableHandle and IndexHandle along with existing converter and parser utilities. Provide default implementation that delegates to existing toVeloxTableHandle(), ignoring the IndexHandle so current connectors keep existing behavior unless overridden. Document that connectors supporting index lookup should override this method to incorporate IndexHandle into the resulting ConnectorTableHandle.	presto-native-execution/presto_cpp/main/connectors/PrestoToVeloxConnector.h
Wire IndexSourceNode conversion to use the new connector hook and ensure node non-nullity.	Add free function toConnectorTableHandleForIndexSource() that resolves the connector from the table handle’s connector type and calls its toVeloxTableHandleForIndexSource(). Update VeloxQueryPlanConverterBase::toVeloxQueryPlan(IndexSourceNode) to null-check the node with VELOX_CHECK_NOT_NULL. Change IndexSourceNode conversion to call toConnectorTableHandleForIndexSource() with the node’s TableHandle and IndexHandle instead of the generic toConnectorTableHandle().	presto-native-execution/presto_cpp/main/types/PrestoToVeloxQueryPlan.cpp
Add regression test and test data validating default IndexSourceNode conversion behavior.	Add PlanConverterTest.indexSource test that converts IndexSource.json and asserts the resulting plan contains a TableScanNode with expected id, output schema, and column assignments. Verify that the resulting table handle is a HiveTableHandle pointing at tpch.nation, confirming default delegation behavior. Introduce IndexSource.json test data describing an IndexSourceNode used by the new test.	presto-native-execution/presto_cpp/main/types/tests/PlanConverterTest.cpp presto-native-execution/presto_cpp/main/types/tests/data/IndexSource.json

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've reviewed your changes and they look great!

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[tanjialiang]

LGTM % nit