Skip to content

Commit b9db9e3

Browse files
BryanCutlerpdabre12
andcommitted
Add FlightShim module for connector federation. This includes an Apache
Arrow Flight server that can load Presto Java connectors, and provide a record batch stream from the given connector split. The FlightShim server is designed to work with a native arrow connector, that will use a Flight client to forward the connector split to the server and process the record batch stream. See related PR at #26404 RFC: prestodb/rfcs#46 Co-authored-by: Pratik Joseph Dabre <pdabre12@gmail.com>
1 parent 8e27030 commit b9db9e3

31 files changed

+4558
-0
lines changed

.github/workflows/arrow-flight-tests.yml

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,45 @@ jobs:
8787
if: needs.changes.outputs.codechange == 'true'
8888
run: ./mvnw test ${MAVEN_TEST} -pl ${{ matrix.modules }} -Dtest="*,!TestArrowFlightNativeQueries*"
8989

90+
arrowflight-shim-tests:
91+
runs-on: ubuntu-latest
92+
strategy:
93+
fail-fast: false
94+
matrix:
95+
java: [17.0.15]
96+
modules:
97+
- :presto-flight-shim # Only run tests for the `presto-flight-shim` module
98+
99+
timeout-minutes: 80
100+
concurrency:
101+
group: ${{ github.workflow }}-test-${{ matrix.modules }}-${{ github.event.pull_request.number }}-${{ matrix.java }}
102+
cancel-in-progress: true
103+
104+
steps:
105+
# Checkout the code only if there are changes in the relevant files
106+
- uses: actions/checkout@v4
107+
with:
108+
show-progress: false
109+
persist-credentials: false
110+
111+
# Set up Java and dependencies for the build environment
112+
- uses: actions/setup-java@v4
113+
with:
114+
distribution: temurin
115+
java-version: ${{ matrix.java }}
116+
cache: maven
117+
- name: Download nodejs to maven cache
118+
run: .github/bin/download_nodejs
119+
120+
# Install dependencies for the target module
121+
- name: Maven Install
122+
run: |
123+
export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
124+
./mvnw install ${MAVEN_FAST_INSTALL} -e -am -pl ${{ matrix.modules }}
125+
# Run Maven tests for the target module, excluding native tests
126+
- name: Maven Tests
127+
run: ./mvnw test ${MAVEN_TEST} -pl ${{ matrix.modules }} -Dtest="*,!TestArrowFederationNativeQueries*"
128+
90129
prestocpp-linux-build-for-test:
91130
runs-on: ubuntu-22.04
92131
needs: changes

pom.xml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,7 @@
241241
<module>presto-sql-helpers/presto-sql-invoked-functions-plugin</module>
242242
<module>presto-sql-helpers/presto-native-sql-invoked-functions-plugin</module>
243243
<module>presto-lance</module>
244+
<module>presto-flight-shim</module>
244245
</modules>
245246

246247
<dependencyManagement>
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
flight-shim.server=localhost
2+
flight-shim.server.port=9999
3+
flight-shim.server-ssl-certificate-file=src/test/resources/certs/server.crt
4+
flight-shim.server-ssl-key-file=src/test/resources/certs/server.key
5+
6+
plugin.bundles=\
7+
../presto-oracle/pom.xml,\
8+
../presto-postgresql/pom.xml,\
9+
../presto-mysql/pom.xml,\
10+
../presto-singlestore/pom.xml
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#
2+
# WARNING
3+
# ^^^^^^^
4+
# This configuration file is for development only and should NOT be used be
5+
# used in production. For example configuration, see the Presto documentation.
6+
#
7+
com.=INFO
8+
com.facebook.presto.execution=DEBUG
9+
com.facebook.presto=INFO
10+
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory=WARN
11+
com.ning.http.client=WARN
12+
com.facebook.presto.server.PluginManager=DEBUG
13+
com.facebook.presto.flightshim=DEBUG

0 commit comments

Comments
 (0)