Fluentd old buffer file cannot flush to elasticsearch

[luckyzzr]

Problem

I run fluentd for a long time，input plugin config forward receive event, output pugin config elasticsearch

Recently I found there aure many residual old buffer file and cannot flush to elasticseach even if I restart fluentd

Steps to replicate

run fluentd for a long time, and buffer output rate is not large than elasticsearch cluster write rate

OR

Provide fluentd config and some start message and some buffer file

  <label @FLUENT_LOG>
    <match fluent.*>
      @type null
    </match>
  </label>
  <source>
    @type forward
    port 24224
    skip_invalid_event true
  </source>
  <match **>
    @type elasticsearch_dynamic
    @id elasticsearch
    host "elasticsearch"
    ssl_verify false
    validate_client_version true
    reconnect_on_error true
    index_name "${tag_parts[-1]}"
    reload_connections false
    reload_on_failure true
    time_key "timestamp"
    time_key_exclude_timestamp true
    utc_index false
    slow_flush_log_threshold 120.0
    request_timeout 120s
    bulk_message_request_threshold -1
    suppress_type_name true
    default_elasticsearch_version 7
    <buffer>
      @type "file_single"
      path "/log/buffer/elasticsearch"
      chunk_format text
      total_limit_size 9G
      chunk_limit_size 15M
      retry_type periodic
      retry_wait 60s
      flush_mode interval
      flush_interval 15s
      flush_thread_count 4
      retry_forever true
      overflow_action block
    </buffer>
  </match>
  <source>
    @type prometheus
    port 24231
    metrics_path "/prometheus"
    aggregated_metrics_path "/metrics"
  </source>
  <source>
    @type prometheus_output_monitor
  </source>
  <system>
    root_dir "/tmp/fluentd-buffers/"
    rpc_endpoint "0.0.0.0:24444"
    suppress_repeated_stacktrace true
    ignore_same_log_interval 60s
    ignore_repeated_log_interval 60s
    emit_error_log_interval 60s
  </system>

2025-05-12 09:26:19 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c71e00ab834f19da9d5db50bd634b.buf
2025-05-12 09:26:19 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c7ea812c3695eb6cc161bea831269.buf
2025-05-12 09:26:19 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c7eabb2f376bbd62ff6ccfef64390.buf
2025-05-12 09:26:19 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c7ef249400acd82b6d598e606feb2.buf
2025-05-12 09:26:19 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c7ef91a6bad2fe4a11961bcd606c3.buf
2025-05-12 09:26:19 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c7efecd58a8b9a0bf75c55a257448.buf
2025-05-12 09:26:19 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c80a0ef5503ff376690732b2d5ad4.buf
2025-05-12 09:26:19 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c80a51a8796a8df795a173129407f.buf
2025-05-12 09:26:19 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c80a9184ca99991197671ec212dcd.buf
2025-05-12 09:26:19 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c84ec8fbed06daa461ba22a545b60.buf
2025-05-12 09:26:20 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c84f087dd2ab57d0208527f64555a.buf
2025-05-12 09:26:20 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c84f593d35209eabdb9e6bed40514.buf
2025-05-12 09:26:20 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c8567e96570f0671e9e54aae2b0fe.buf
2025-05-12 09:26:20 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c8572a762f5b167529536fec2b847.buf
2025-05-12 09:26:20 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c85805a39aca7d469618b57d2427a.buf
2025-05-12 09:26:20 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c85e586f14c778b36452245446937.buf
2025-05-12 09:26:20 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c85e9e2cd0faca0688dfabc35a55e.buf
2025-05-12 09:26:21 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c85f052485f8ce9c1cc8fd19ce395.buf
2025-05-12 09:26:21 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c85f302bfe8668e18663093e96f9b.buf
2025-05-12 09:26:21 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c86384b198fe7cb900a53f766ca27.buf
2025-05-12 09:26:21 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c863c535025520f5d165fedbde326.buf
2025-05-12 09:26:21 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c863f8d8e30abe7506a652f2c7791.buf
2025-05-12 09:26:21 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c87e9d8833fcaf9278a14a5ed8f92.buf
2025-05-12 09:26:21 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c87eac0bdb01332586cf6eed10e5f.buf
2025-05-12 09:26:21 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c87eafb484e72f728622dc5c04ce3.buf
2025-05-12 09:26:21 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c8807801edb18ca4855922029530f.buf
2025-05-12 09:26:21 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c880b6279b4ab31274efa7ce5b163.buf
2025-05-12 09:26:21 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c8811ca7f52996ad570bb2b3846d2.buf
2025-05-12 09:26:22 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c881c5ff3eff403c83c5c96fe8771.buf
2025-05-12 09:26:22 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c89a80af066cc9cb9f2210140543d.buf
2025-05-12 09:26:22 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c89aaa881e45ef11b2b84647678cf.buf
2025-05-12 09:26:22 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.pod.openstack.b622c89ada4fe632312b3a241ef5eb634.buf
2025-05-12 09:26:22 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.syslog.system.b634ece54e21704344a7bc57047a62b55.buf
2025-05-12 09:26:22 +0000 [debug]: #2 [elasticsearch] restoring buffer file: path = /log/buffer/elasticsearch/worker2/fsb.volume.audit.b634ece57b786e69735c67b06a6a0a6c6.buf
2025-05-12 09:26:22 +0000 [debug]: #2 [elasticsearch] buffer started instance=2520 stage_size=46423238 queue_size=0
2025-05-12 09:26:22 +0000 [debug]: #2 fluent/log.rb:341:debug: listening prometheus http server on http:://0.0.0.0:24233//prometheus for worker2
2025-05-12 09:26:22 +0000 [debug]: #2 [elasticsearch] flush_thread actually running
2025-05-12 09:26:22 +0000 [debug]: #2 [elasticsearch] flush_thread actually running
2025-05-12 09:26:22 +0000 [debug]: #2 [elasticsearch] flush_thread actually running
2025-05-12 09:26:22 +0000 [debug]: #2 [elasticsearch] flush_thread actually running
2025-05-12 09:26:22 +0000 [debug]: #2 fluent/log.rb:341:debug: Start async HTTP server listening http://0.0.0.0:24233
2025-05-12 09:26:22 +0000 [debug]: #2 [elasticsearch] enqueue_thread actually running
2025-05-12 09:26:22 +0000 [debug]: #2 fluent/log.rb:341:debug:   0.0s: Async::IO::Socket
      | Binding to #<Addrinfo: 0.0.0.0:24233 TCP>
2025-05-12 09:26:22 +0000 [info]: #2 fluent/log.rb:362:info: listening port port=24224 bind="0.0.0.0"
2025-05-12 09:26:22 +0000 [info]: #2 fluent/log.rb:362:info: fluentd worker is now running worker=2

-rw-r--r-- 1 root root 15140776 Sep 23  2024 fsb.pod.openstack.b622c89a80af066cc9cb9f2210140543d.buf
-rw-r--r-- 1 root root 15503561 Sep 23  2024 fsb.pod.openstack.b622c89aaa881e45ef11b2b84647678cf.buf
-rw-r--r-- 1 root root 15679860 Sep 23  2024 fsb.pod.log.b622c8a645e8e4e8279b5f6c96b6fd8d2.buf
-rw-r--r-- 1 root root 15097368 Sep 23  2024 fsb.pod.log.b622c8a6afe0c4a68f1421cbd725d23da.buf
-rw-r--r-- 1 root root 15140143 Sep 23  2024 fsb.pod.log.b622c8a75e18ff5a710060ac859220fc7.buf
-rw-r--r-- 1 root root 15043892 Sep 23  2024 fsb.pod.log.b622c8c27709c06be4b61c188b51203ff.buf
-rw-r--r-- 1 root root 15142960 Sep 23  2024 fsb.pod.log.b622c8c2aba3e3b0156fe869a3ca1b34f.buf
-rw-r--r-- 1 root root 15001516 Sep 23  2024 fsb.pod.log.b622c8c2e83a15ae83de4e8265b20d8ea.buf
-rw-r--r-- 1 root root 14945271 Sep 23  2024 fsb.pod.log.b622c8c7827dd8b1aad3dd9d13034334e.buf
-rw-r--r-- 1 root root 14993627 Sep 23  2024 fsb.pod.log.b622c8c7924e717059032fe5f604328d5.buf
-rw-r--r-- 1 root root 14961563 Sep 23  2024 fsb.pod.log.b622c8c7b1c9e14ebc5b1b4e781ee9e48.buf
-rw-r--r-- 1 root root 15029793 Sep 23  2024 fsb.pod.log.b622c8c7c4a55ef03876af1a7011b8eae.buf
-rw-r--r-- 1 root root 15027225 Sep 23  2024 fsb.pod.log.b622c8c7e52a7a0fb03731cc72a94d2cc.buf
-rw-r--r-- 1 root root 15400896 Sep 23  2024 fsb.pod.log.b622c8d01b999f82a2fc6b4f1415b4821.buf
-rw-r--r-- 1 root root 15427766 Sep 23  2024 fsb.pod.log.b622c8d07d331a17c8b7866b4440aeeb2.buf
-rw-r--r-- 1 root root 15319755 Sep 23  2024 fsb.pod.log.b622c8d0b05891029d86404a103e503f6.buf
-rw-r--r-- 1 root root 15384163 Sep 23  2024 fsb.pod.log.b622c8d0e1e937b7eddc4a3609069dbeb.buf
-rw-r--r-- 1 root root 15377167 Sep 23  2024 fsb.pod.log.b622c8d1205ad018018c335779963dbec.buf
-rw-r--r-- 1 root root     3552 May 12 09:34 fsb.syslog.system.b634ed06629e693e14cdb8272eddde615.buf
-rw-r--r-- 1 root root   525598 May 12 09:34 fsb.pod.log.q634ed05d8a6727806be186b60af1e2fe.buf
-rw-r--r-- 1 root root     2062 May 12 09:34 fsb.kubelet.kubernetes.b634ed0641914ab0d55a354cda45eef16.buf

Expected Behavior or What you need to ask

old buffer file that create at 2024 year both flush to elasticsearch normally

Using Fluentd and ES plugin versions

• OS version
  Debian GNU/Linux 12 (bookworm)
• Bare Metal or within Docker or Kubernetes or others?
  docker
• Fluentd v0.12 or v0.14/v1.0
  fluentd 1.16.8
• ES plugin 3.x.y/2.x.y or 1.x.y
  elasticsearch (7.17.11)
  elasticsearch-api (7.17.11)
  elasticsearch-transport (7.17.11)
  fluent-plugin-elasticsearch (5.4.4)
• ES version (optional)
  7.17.23
• ES template(s) (optional)

[like-inspur]

I also met this problem with fluentd 1.15.3 and fluent-plugin-elasticsearch 4.2.2 , after fluentd restart and upgrade version, the problem still exist