Skip to content

Commit e56074a

Browse files
NihalJainNihalJain
committed
Improve getting-started section's docker compose
- Add support to optionally start kafka - Align with manual docker steps like networks - Improve script and add health checks - Make image tags configurable. create sub-sections as needed
1 parent 8d7d429 commit e56074a

File tree

1 file changed

+123
-26
lines changed

1 file changed

+123
-26
lines changed

‎basics/getting-started/running-pinot-in-docker.md

Lines changed: 123 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,16 @@ Create an isolated bridge network in docker
8484
docker network create -d bridge pinot-demo
8585
```
8686

87+
#### Export Docker Image tags
88+
89+
Export the necessary docker image tags for Pinot, Zookeeper, and Kafka.
90+
91+
```
92+
export PINOT_IMAGE=apachepinot/pinot:1.2.0
93+
export ZK_IMAGE=zookeeper:3.9.2
94+
export KAFKA_IMAGE= bitnami/kafka:3.6
95+
```
96+
8797
#### Start Zookeeper
8898

8999
Start Zookeeper in daemon mode. This is a single node zookeeper setup. Zookeeper is the central metadata store for Pinot and should be set up with replication for production use. For more information, see [Running Replicated Zookeeper](https://zookeeper.apache.org/doc/r3.6.0/zookeeperStarted.html#sc\_RunningReplicatedZooKeeper).
@@ -94,7 +104,7 @@ docker run \
94104
--name pinot-zookeeper \
95105
--restart always \
96106
-p 2181:2181 \
97-
-d zookeeper:3.9.2
107+
-d ${ZK_IMAGE}
98108
```
99109

100110
#### Start Pinot Controller
@@ -162,7 +172,7 @@ docker run --rm -ti \
162172
-e KAFKA_BROKER_ID=0 \
163173
-e KAFKA_ADVERTISED_HOST_NAME=kafka \
164174
-p 9092:9092 \
165-
-d bitnami/kafka:latest
175+
-d ${KAFKA_IMAGE}
166176
```
167177

168178
Now all Pinot related components are started as an empty cluster.
@@ -176,86 +186,173 @@ docker container ls -a
176186
**Sample Console Output**
177187

178188
```
179-
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
180-
9ec20e4463fa bitnami/kafka:latest "start-kafka.sh" 43 minutes ago Up 43 minutes kafka
181-
0775f5d8d6bf apachepinot/pinot:latest "./bin/pinot-admin.s…" 44 minutes ago Up 44 minutes 8096-8099/tcp, 9000/tcp pinot-server
182-
64c6392b2e04 apachepinot/pinot:latest "./bin/pinot-admin.s…" 44 minutes ago Up 44 minutes 8096-8099/tcp, 9000/tcp pinot-broker
183-
b6d0f2bd26a3 apachepinot/pinot:latest "./bin/pinot-admin.s…" 45 minutes ago Up 45 minutes 8096-8099/tcp, 0.0.0.0:9000->9000/tcp pinot-controller
184-
570416fc530e zookeeper:3.9.2 "/docker-entrypoint.…" 45 minutes ago Up 45 minutes 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, 8080/tcp pinot-zookeeper
189+
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
190+
accc70bc7f07 bitnami/kafka:3.6 "/opt/bitnami/script…" About a minute ago Up About a minute 0.0.0.0:9092->9092/tcp kafka
191+
1b8b80395959 apachepinot/pinot:1.2.0 "./bin/pinot-admin.s…" About a minute ago Up About a minute 8096-8097/tcp, 8099/tcp, 9000/tcp, 0.0.0.0:8098->8098/tcp pinot-server
192+
134a67eec957 apachepinot/pinot:1.2.0 "./bin/pinot-admin.s…" About a minute ago Up About a minute 8096-8098/tcp, 9000/tcp, 0.0.0.0:8099->8099/tcp pinot-broker
193+
4fcc72cb7302 apachepinot/pinot:1.2.0 "./bin/pinot-admin.s…" About a minute ago Up About a minute 8096-8099/tcp, 0.0.0.0:9000->9000/tcp pinot-controller
194+
144304524f6c zookeeper:3.9.2 "/docker-entrypoint.…" About a minute ago Up About a minute 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, 8080/tcp pinot-zookeeper
185195
```
186196

187197
### Docker Compose
188198

199+
#### Export Docker Image tags
200+
201+
Optionally, export the necessary docker image tags for Pinot, Zookeeper, and Kafka.
202+
203+
```
204+
export PINOT_IMAGE=apachepinot/pinot:1.2.0
205+
export ZK_IMAGE=zookeeper:3.9.2
206+
export KAFKA_IMAGE=bitnami/kafka:3.6
207+
```
208+
209+
#### Create _docker-compose.yml_ file
189210
Create a file called _docker-compose.yml_ that contains the following:
190211

191212
{% code title="docker-compose.yml" %}
192213
```yaml
193214
version: '3.7'
215+
194216
services:
195217
pinot-zookeeper:
196-
image: zookeeper:3.9.2
197-
container_name: pinot-zookeeper
218+
image: ${ZK_IMAGE:-zookeeper:3.9.2}
219+
container_name: "pinot-zookeeper"
220+
restart: unless-stopped
198221
ports:
199222
- "2181:2181"
200223
environment:
201224
ZOOKEEPER_CLIENT_PORT: 2181
202225
ZOOKEEPER_TICK_TIME: 2000
226+
networks:
227+
- pinot-demo
228+
healthcheck:
229+
test: ["CMD", "zkServer.sh", "status"]
230+
interval: 30s
231+
timeout: 10s
232+
retries: 5
233+
start_period: 10s
234+
235+
pinot-kafka:
236+
image: ${KAFKA_IMAGE:-bitnami/kafka:3.6}
237+
container_name: "kafka"
238+
restart: unless-stopped
239+
ports:
240+
- "9092:9092"
241+
environment:
242+
KAFKA_ZOOKEEPER_CONNECT: pinot-zookeeper:2181/kafka
243+
KAFKA_BROKER_ID: 0
244+
KAFKA_ADVERTISED_HOST_NAME: kafka
245+
depends_on:
246+
pinot-zookeeper:
247+
condition: service_healthy
248+
networks:
249+
- pinot-demo
250+
healthcheck:
251+
test: [ "CMD-SHELL", "kafka-broker-api-versions.sh -bootstrap-server kafka:9092" ]
252+
interval: 1s
253+
timeout: 10s
254+
retries: 5
255+
start_period: 10s
256+
deploy:
257+
replicas: ${KAFKA_REPLICAS:-0} # Default to 0, meaning Kafka won't start unless KAFKA_REPLICAS is set
258+
203259
pinot-controller:
204-
image: apachepinot/pinot:1.2.0
260+
image: ${PINOT_IMAGE:-apachepinot/pinot:1.2.0}
205261
command: "StartController -zkAddress pinot-zookeeper:2181"
206-
container_name: pinot-controller
262+
container_name: "pinot-controller"
207263
restart: unless-stopped
208264
ports:
209265
- "9000:9000"
210266
environment:
211267
JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log"
212268
depends_on:
213-
- pinot-zookeeper
269+
pinot-zookeeper:
270+
condition: service_healthy
271+
networks:
272+
- pinot-demo
273+
healthcheck:
274+
test: ["CMD-SHELL", "curl -f http://localhost:9000/health || exit 1"]
275+
interval: 30s
276+
timeout: 10s
277+
retries: 5
278+
start_period: 10s
279+
214280
pinot-broker:
215-
image: apachepinot/pinot:1.2.0
281+
image: ${PINOT_IMAGE:-apachepinot/pinot:1.2.0}
216282
command: "StartBroker -zkAddress pinot-zookeeper:2181"
217-
restart: unless-stopped
218283
container_name: "pinot-broker"
284+
restart: unless-stopped
219285
ports:
220286
- "8099:8099"
221287
environment:
222288
JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-broker.log"
223289
depends_on:
224-
- pinot-controller
290+
pinot-controller:
291+
condition: service_healthy
292+
networks:
293+
- pinot-demo
294+
healthcheck:
295+
test: ["CMD-SHELL", "curl -f http://localhost:8099/health || exit 1"]
296+
interval: 30s
297+
timeout: 10s
298+
retries: 5
299+
start_period: 10s
300+
225301
pinot-server:
226-
image: apachepinot/pinot:1.2.0
302+
image: ${PINOT_IMAGE:-apachepinot/pinot:1.2.0}
227303
command: "StartServer -zkAddress pinot-zookeeper:2181"
228-
restart: unless-stopped
229304
container_name: "pinot-server"
305+
restart: unless-stopped
230306
ports:
231307
- "8098:8098"
232308
environment:
233309
JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log"
234310
depends_on:
235-
- pinot-broker
311+
pinot-broker:
312+
condition: service_healthy
313+
networks:
314+
- pinot-demo
315+
healthcheck:
316+
test: ["CMD-SHELL", "curl -f http://localhost:8097/health/readiness || exit 1"]
317+
interval: 30s
318+
timeout: 10s
319+
retries: 5
320+
start_period: 10s
321+
322+
networks:
323+
pinot-demo:
324+
name: pinot-demo
325+
driver: bridge
236326
```
237327
{% endcode %}
238328
239-
Run the following command to launch all the components:
329+
#### Launch the components
330+
Run the following command to launch all the required components:
240331
241332
```
242333
docker compose --project-name pinot-demo up
243334
```
244335

336+
OR, optionally, run the following command to launch all the components, including kafka:
337+
```
338+
export KAFKA_REPLICAS=1
339+
docker compose --project-name pinot-demo up
340+
```
245341
Run the below command to check the container status:
246342

247343
```
248-
docker container ls
344+
docker container ls -a
249345
```
250346

251347
**Sample Console Output**
252348

253349
```
254-
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
255-
ba5cb0868350 apachepinot/pinot:1.2.0 "./bin/pinot-admin.s…" About a minute ago Up About a minute 8096-8099/tcp, 9000/tcp pinot-server
256-
698f160852f9 apachepinot/pinot:1.2.0 "./bin/pinot-admin.s…" About a minute ago Up About a minute 8096-8098/tcp, 9000/tcp, 0.0.0.0:8099->8099/tcp, :::8099->8099/tcp pinot-broker
257-
b1ba8cf60d69 apachepinot/pinot:1.2.0 "./bin/pinot-admin.s…" About a minute ago Up About a minute 8096-8099/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp pinot-controller
258-
54e7e114cd53 zookeeper:3.9.2 "/docker-entrypoint.…" About a minute ago Up About a minute 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp pinot-zookeeper
350+
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
351+
f34a046ac69f bitnami/kafka:3.6 "/opt/bitnami/script…" 9 minutes ago Up About a minute (healthy) 0.0.0.0:9092->9092/tcp kafka
352+
f28021bd5b1d apachepinot/pinot:1.2.0 "./bin/pinot-admin.s…" 18 minutes ago Up About a minute (healthy) 8096-8097/tcp, 8099/tcp, 9000/tcp, 0.0.0.0:8098->8098/tcp pinot-server
353+
e938453054b0 apachepinot/pinot:1.2.0 "./bin/pinot-admin.s…" 18 minutes ago Up About a minute (healthy) 8096-8098/tcp, 9000/tcp, 0.0.0.0:8099->8099/tcp pinot-broker
354+
e0d0c71303a8 apachepinot/pinot:1.2.0 "./bin/pinot-admin.s…" 18 minutes ago Up About a minute (healthy) 8096-8099/tcp, 0.0.0.0:9000->9000/tcp pinot-controller
355+
4be5f168f252 zookeeper:3.9.2 "/docker-entrypoint.…" 18 minutes ago Up About a minute (healthy) 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, 8080/tcp pinot-zookeeper
259356
```
260357

261358
Once your cluster is up and running, see [Exploring Pinot](../components/exploring-pinot.md) to learn how to run queries against the data.

0 commit comments

Comments
 (0)