Skip to content

Commit beabbc3

Browse files
BryanCutlerpdabre12
andcommitted
Add FlightShim module for connector federation. This includes an Apache
Arrow Flight server that can load Presto Java connectors, and provide a record batch stream from the given connector split. The FlightShim server is designed to work with a native arrow connector, that will use a Flight client to forward the connector split to the server and process the record batch stream. See related PR at #26404 RFC: prestodb/rfcs#46 Co-authored-by: Pratik Joseph Dabre <pdabre12@gmail.com>
1 parent dae492a commit beabbc3

28 files changed

+4230
-0
lines changed

.github/workflows/arrow-flight-tests.yml

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,45 @@ jobs:
6666
- name: Maven Tests
6767
run: ./mvnw test ${MAVEN_TEST} -pl ${{ matrix.modules }} -Dtest="*,!TestArrowFlightNativeQueries*"
6868

69+
arrowflight-shim-tests:
70+
runs-on: ubuntu-latest
71+
strategy:
72+
fail-fast: false
73+
matrix:
74+
java: [17.0.15]
75+
modules:
76+
- :presto-flight-shim # Only run tests for the `presto-flight-shim` module
77+
78+
timeout-minutes: 80
79+
concurrency:
80+
group: ${{ github.workflow }}-test-${{ matrix.modules }}-${{ github.event.pull_request.number }}-${{ matrix.java }}
81+
cancel-in-progress: true
82+
83+
steps:
84+
# Checkout the code only if there are changes in the relevant files
85+
- uses: actions/checkout@v4
86+
with:
87+
show-progress: false
88+
persist-credentials: false
89+
90+
# Set up Java and dependencies for the build environment
91+
- uses: actions/setup-java@v4
92+
with:
93+
distribution: temurin
94+
java-version: ${{ matrix.java }}
95+
cache: maven
96+
- name: Download nodejs to maven cache
97+
run: .github/bin/download_nodejs
98+
99+
# Install dependencies for the target module
100+
- name: Maven Install
101+
run: |
102+
export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
103+
./mvnw install ${MAVEN_FAST_INSTALL} -e -am -pl ${{ matrix.modules }}
104+
# Run Maven tests for the target module, excluding native tests
105+
- name: Maven Tests
106+
run: ./mvnw test ${MAVEN_TEST} -pl ${{ matrix.modules }} -Dtest="*,!TestArrowFederationNativeQueries*"
107+
69108
prestocpp-linux-build-for-test:
70109
runs-on: ubuntu-22.04
71110
container:

pom.xml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -236,6 +236,7 @@
236236
<module>presto-plan-checker-router-plugin</module>
237237
<module>presto-sql-helpers/presto-sql-invoked-functions-plugin</module>
238238
<module>presto-sql-helpers/presto-native-sql-invoked-functions-plugin</module>
239+
<module>presto-flight-shim</module>
239240
</modules>
240241

241242
<dependencyManagement>
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
flight-shim.server=localhost
2+
flight-shim.server.port=9999
3+
flight-shim.server-ssl-certificate-file=src/test/resources/certs/server.crt
4+
flight-shim.server-ssl-key-file=src/test/resources/certs/server.key
5+
6+
plugin.bundles=\
7+
../presto-oracle/pom.xml,\
8+
../presto-postgresql/pom.xml,\
9+
../presto-mysql/pom.xml,\
10+
../presto-singlestore/pom.xml
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#
2+
# WARNING
3+
# ^^^^^^^
4+
# This configuration file is for development only and should NOT be used be
5+
# used in production. For example configuration, see the Presto documentation.
6+
#
7+
com.=INFO
8+
com.facebook.presto.execution=DEBUG
9+
com.facebook.presto=INFO
10+
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory=WARN
11+
com.ning.http.client=WARN
12+
com.facebook.presto.server.PluginManager=DEBUG
13+
com.facebook.presto.flightshim=DEBUG

0 commit comments

Comments
 (0)