Collector cannot push to elasticsearch behind TLS proxy

[tanner-bruce]

We are running a remote collector that should push to an Elasticsearch cluster behind a TLS reverse proxy.

Upon startup, the docker container will die after printing the message
{"level":"fatal","ts":1508514781.8733246,"caller":"collector/main.go:86","msg":"Unable to set up builder","error":"health check timeout: no Elasticsearch node available","errorVerbose":"no Elasticsearch node available\ngithub.com/uber/jaeger/vendor/github.com/olivere/elastic.init\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/olivere/elastic/client.go:84\ngithub.com/uber/jaeger/pkg/es/config.init\n\t/home/travis/gopath/src/github.com/uber/jaeger/pkg/es/config/config.go:102\ngithub.com/uber/jaeger/cmd/builder.init\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/builder/doc.go:20\nmain.init\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/main.go:161\nruntime.main\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/proc.go:172\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:2086\nhealth check timeout\ngithub.com/uber/jaeger/vendor/github.com/olivere/elastic.(*Client).startupHealthcheck\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/olivere/elastic/client.go:1067\ngithub.com/uber/jaeger/vendor/github.com/olivere/elastic.NewClient\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/olivere/elastic/client.go:240\ngithub.com/uber/jaeger/pkg/es/config.(*Configuration).NewClient\n\t/home/travis/gopath/src/github.com/uber/jaeger/pkg/es/config/config.go:50\ngithub.com/uber/jaeger/cmd/collector/app/builder.(*SpanHandlerBuilder).initElasticStore\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/builder/span_handler_builder.go:102\ngithub.com/uber/jaeger/cmd/collector/app/builder.NewSpanHandlerBuilder\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/app/builder/span_handler_builder.go:75\nmain.main.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/main.go:84\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:636\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:722\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:681\nmain.main\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/main.go:139\nruntime.main\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/proc.go:183\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/asm_amd64.s:2086","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:191\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:301\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Fatal\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:235\nmain.main.func1\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/main.go:86\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:636\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:722\ngithub.com/uber/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/spf13/cobra/command.go:681\nmain.main\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/collector/main.go:139"}

I've filed olivere/elastic#625 for the bad error message, but would like to open the discussion up here as to whether the jaeger-collector docker image should include ca-certificates.

[yurishkuro]

shouldn't the certificates be provided on the host and just mapped into container? I don't know how it's done typically, but putting them in the docker image sounds odd.

[tanner-bruce]

Keeping these two definitions from the docker site in mind:

A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings.

Docker containers isolate applications from one another and from the underlying infrastructure. Docker provides the strongest default isolation to limit app issues to a single container instead of the entire machine.

I would argue that the collector should include the ca-certificates - we don't want to rely on the underlying infrastructure to have those certificates or provide them for the container. Maybe I haven't even installed them on my server, it's not strictly necessary. Since without them, the collector cannot access an HTTPS server, I would say the current container is not meeting the goal of providing everything needed to run it.

FWIW I'm very OK with maintaining my own image, I just wanted to bring this up for discussion, as I think providing the certificates makes for a nicer out-of-the-box experience for users.

[jpkrohling]

I don't know how this would apply for "pure" Docker, but on platforms like OpenShift and Kubernetes, the CA that signs the certs for the "services" (like Elasticsearch here) is located at a well-known place (/var/run/secrets/kubernetes.io/serviceaccount/ca.crt). The CA cert itself isn't part of the image, but part of the orchestration service and is mounted as a volume at runtime.

[yurishkuro]

actually going back to the original issue, why do we need ca certificates in the first place? The collectors do not serve HTTPS traffic, and you don't need certificates being the client in the HTTPS connection, only being the server. Unless you have a setup where you use certificates to authenticate both sides of the connection, in this case it seems obvious that we're talking about deployment concerns for a specific installation, which cannot be part of the image.

[tanner-bruce]

The issue is when the you run jaeger collector with elasticsearch behind https. Running this with the official container will fail.

docker run -i --name collector -p 14267:14267 -p 14268:14268 -p 9411:9411 \
    jaeger-collector /go/bin/collector-linux \
    --span-storage.type=elasticsearch  \
    --es.server-urls=https://elastic.com \
    --es.username=elastic \
    --es.password=pass \
    --es.num-shards=5 \
    --es.num-replicas=1 \
    --es.sniffer=false \
    --log-level=debug

[freelinuxer]

Hi, @tanner-bruce @yurishkuro @jpkrohling do we have a solution/workaround for this issue?
I think that I am going throug same issue now.
My test k8s pod connects to a production ES (https) just fine as @yurishkuro mentioned. (Client does not need a cert).
But, Jager-collector and jaeger-query fail to connect the prod ElasticSearch(https).

[tanner-bruce]

@freelinuxer I built my own images using bitnami/minideb as the base and installed the ca certificates necessary.

If there is interest I can contribute a second docker build (i.e alongside the scratch base) using minideb or alpine as the base with ca-certs installed. We tend to prefer minideb as it is lightweight but still glibc based and is gives a familiar debian environment.

[freelinuxer]

@tanner-bruce That will help a lot. !
Could you provide a blog or instruction on this as I am not familiar how I'd put cert and create custom images for collector and query. Thanks !!

[yurishkuro]

@tanner-bruce would be useful if you could share your Dockerfiles.

[tanner-bruce]

@freelinuxer in a pinch you can use this:

If you compile jaeger:

FROM bitnami/minideb

EXPOSE 9411
EXPOSE 14267
EXPOSE 14268
EXPOSE 14269

RUN install_packages ca-certificates

COPY collector-linux /go/bin/
ENTRYPOINT ["/go/bin/collector-linux"]

If you don't want to compile, you can use a multi-stage build to get the binary from the official container image. This is for the query service, so note that it also requires copying the /go/jaeger-ui folder. The collector does not require this.

FROM jaegertracing/jaeger-query:0.9.0 AS jaeger
FROM bitnami/minideb

EXPOSE 16686

RUN install_packages ca-certificates

COPY --from=jaeger /go/bin/query-linux /go/bin/query-linux
COPY --from=jaeger /go/jaeger-ui/ /go/jaeger-ui/
ENTRYPOINT ["/go/bin/query-linux"]

You can replace this with alpine by changing the RUN install_packages ca-certificates to RUN apk add --update --no-cache ca-certificates and also updating the base image.

[freelinuxer]

@tanner-bruce I guess Jaeger-query image also needs similar changes as you described above since it also needs to talk to ElasticSearch. correct ?

[tanner-bruce]

@freelinuxer yes, it does. In the snippets I posted above, the first one is for collector, the second one, which shows how to build it without compiling, is for query.

[sagikazarmark]

actually going back to the original issue, why do we need ca certificates in the first place?

I suppose host verification is turned on by default in the underlying elasticsearch/http client.

I'm trying to make it work by using a self-signed cert and mounting it to /etc/ssl/certs/ca-certificates.crt file (based on this), but no luck so far. :(

[freelinuxer]

I tried with @tanner-bruce 's approach and it worked for me.
#485 (comment)
I built my own binaries and container images.

[sagikazarmark]

I managed to make it work in the meantime (turned out I was using the wrong CA)