failed to flush the buffer (use https, but elasticsearch received http traffic)

[chaseSpace]

(check apply)

• read the contribution guideline
• (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Problem

This is fluentd plugin log:

│ fluentd 2023-12-21 01:02:41 +0800 [error]: #0 [Faraday::ConnectionFailed] Connection refused - connect(2) for 20.1.47.16:9200 (Errno::ECONNREFUSED) {:host=>"elasticsearch-master.efk.svc.cluster.local", :port=>9200, :scheme=>"https", :user=>"elastic", :password=><REDACTED>, :protocol=>"https"}   │
│ fluentd The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the server is running an unsupported product.                                                                                                                                      │
│ fluentd 2023-12-21 01:02:41 +0800 [warn]: #0 failed to flush the buffer. retry_times=8 next_retry_time=2023-12-21 01:06:45 +0800 chunk="60cf3e4711529ea417a7cd0484ffa58b" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluste │
│ fluentd   2023-12-21 01:02:41 +0800 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.4.2/lib/fluent/plugin/out_elasticsearch.rb:1171:in `rescue in send_bulk'                                                                                                                            │
│ fluentd   2023-12-21 01:02:41 +0800 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.4.2/lib/fluent/plugin/out_elasticsearch.rb:1133:in `send_bulk'                                                                                                                                      │
│ fluentd   2023-12-21 01:02:41 +0800 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.4.2/lib/fluent/plugin/out_elasticsearch.rb:909:in `block in write'                                                                                                                                  │
│ fluentd   2023-12-21 01:02:41 +0800 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.4.2/lib/fluent/plugin/out_elasticsearch.rb:908:in `each'                                                                                                                                            │
│ fluentd   2023-12-21 01:02:41 +0800 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.4.2/lib/fluent/plugin/out_elasticsearch.rb:908:in `write'                                                                                                                                           │
│ fluentd   2023-12-21 01:02:41 +0800 [warn]: #0 /usr/local/bundle/gems/fluentd-1.16.2/lib/fluent/plugin/output.rb:1225:in `try_flush'                                                                                                                                                                    │
│ fluentd   2023-12-21 01:02:41 +0800 [warn]: #0 /usr/local/bundle/gems/fluentd-1.16.2/lib/fluent/plugin/output.rb:1538:in `flush_thread_run'                                                                                                                                                             │
│ fluentd   2023-12-21 01:02:41 +0800 [warn]: #0 /usr/local/bundle/gems/fluentd-1.16.2/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'                                                                                                                                                     │
│ fluentd   2023-12-21 01:02:41 +0800 [warn]: #0 /usr/local/bundle/gems/fluentd-1.16.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 

and elastic log:

│ elasticsearch {"@timestamp":"2023-12-20T17:09:50.754Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:58184}", "ecs.version": "1.2.0","service.name":"ES_ECS","event │
│ elasticsearch {"@timestamp":"2023-12-20T17:09:50.754Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:58184}", "ecs.version": "1.2.0","service.name":"ES_ECS","event │
repeated...

Steps to replicate

Paste my fluentd conf:

    <source>
        @type tail
        path /var/log/biz/container.log
        pos_file /var/log/biz/container.log.pos
        <parse>
          @type json
        </parse>
        tag json-svc
    </source>

    <match **>
        @type elasticsearch
        host elasticsearch-master.efk.svc.cluster.local
        port 9200
        logstash_format true

        user elastic
        password 123
        ca_file /elastic-ca-file/ca.crt
        ssl_verify false 
        scheme https 
        with_transporter_log true

        time_key_format "%Y-%m-%d %H:%M:%S.%N%z"
        time_key node_time
        utc_index true
    
        retry_forever true 
        log_es_400_reason true
        reconnect_on_error true
        reload_on_failure true
        reload_connections false
        request_timeout 30s
    
        <buffer tag,time>
          @type file
          path /var/log/fluentd-buffer
          timekey 1h  
          timekey_wait 10s
          flush_mode interval
          flush_interval 5s
          flush_thread_count 4
          overflow_action block 
        </buffer>
        verify_es_version_at_startup false
        default_elasticsearch_version 8 
    </match>

Using Fluentd and ES plugin versions

environment:

• fluentd: v1.16.2
• fluentd-plugin-elasticsearch: v5.4.2
• elastic v8.5.1 in helm
• k8s v1.27

Dockerfile for install plugin

ARG FLUENTD_TAG

FROM fluent/fluentd:$FLUENTD_TAG

USER root

ARG ES_PLUGIN_VERSION

RUN gem install fluent-plugin-elasticsearch --version $ES_PLUGIN_VERSION

USER fluent

docker build:

docker build --build-arg FLUENTD_TAG=v1.16.2-debian-1.1 --build-arg ES_PLUGIN_VERSION=5.4.2 . -t leigg/fluentd-with-es-plugin:v1.16.2

[chaseSpace]

It's worked fine when I disabled es ssl and turned scheme http in fluentd.conf. I was stucked here for two days, and I don't know where the problem is...

[gael-alfa]

Same problem for us, Upgrade fluent-plugin-elasticsearch to 6.0.0, but no changes.
We added a daily restart, no reduce issue occurences, but on heavy load the plugins switch to http, a restart fix it... totally weird