Data Node: gl-datanode-metrics not re-created if it disappears

[damianharouff]

In the event that gl-datanode-metrics disappears from Opensearch, Data Node will repeatedly emit into logs that it can't post metrics:

2025-12-18T08:30:40.441-07:00 ERROR [MetricsCollector] Error indexing metrics
org.graylog.shaded.opensearch2.org.opensearch.OpenSearchStatusException: OpenSearch exception [type=index_not_found_exception, reason=no such index [gl-datanode-metrics]]
	at org.graylog.shaded.opensearch2.org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:210)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2235)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2212)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:2123)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:851)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient$1.completed(RestClient.java:580)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient$1.completed(RestClient.java:560)
	at org.graylog.shaded.opensearch2.org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:182)
	at org.graylog.shaded.opensearch2.org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
	at org.graylog.shaded.opensearch2.org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: org.graylog.shaded.opensearch2.org.opensearch.client.ResponseException: method [POST], host [https://gl.cekkent.net:9200], URI [/gl-datanode-metrics/_doc?timeout=1m], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [gl-datanode-metrics]","index":"gl-datanode-metrics","resource.id":"gl-datanode-metrics","resource.type":"index_expression","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [gl-datanode-metrics]","index":"gl-datanode-metrics","resource.id":"gl-datanode-metrics","resource.type":"index_expression","index_uuid":"_na_"},"status":404}
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.convertResponse(RestClient.java:479)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.access$1800(RestClient.java:128)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient$1.completed(RestClient.java:564)
		... 16 more
2025-12-18T08:30:40.577-07:00 ERROR [MetricsCollector] Uncaught exception in Periodical
org.graylog.shaded.opensearch2.org.opensearch.OpenSearchStatusException: OpenSearch exception [type=index_not_found_exception, reason=no such index [gl-datanode-metrics]]
	at org.graylog.shaded.opensearch2.org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:210)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2235)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2212)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1931)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1884)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1852)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1095)
	at org.graylog.datanode.periodicals.MetricsCollector.getPreviousMetricsForCluster(MetricsCollector.java:172)
	at org.graylog.datanode.periodicals.MetricsCollector.lambda$doRun$0(MetricsCollector.java:123)
	at java.base/java.util.Optional.ifPresent(Unknown Source)
	at org.graylog.datanode.periodicals.MetricsCollector.doRun(MetricsCollector.java:109)
	at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99)
	at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedPeriodicRunnable.run(InstrumentedScheduledExecutorService.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: org.graylog.shaded.opensearch2.org.opensearch.client.ResponseException: method [POST], host [https://gl.cekkent.net:9200], URI [/gl-datanode-metrics/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [gl-datanode-metrics]","index":"gl-datanode-metrics","resource.id":"gl-datanode-metrics","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [gl-datanode-metrics]","index":"gl-datanode-metrics","resource.id":"gl-datanode-metrics","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.convertResponse(RestClient.java:479)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:371)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:346)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1918)
		... 15 more
2025-12-18T08:31:40.447-07:00 ERROR [MetricsCollector] Error indexing metrics
org.graylog.shaded.opensearch2.org.opensearch.OpenSearchStatusException: OpenSearch exception [type=index_not_found_exception, reason=no such index [gl-datanode-metrics]]
	at org.graylog.shaded.opensearch2.org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:210)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2235)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2212)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:2123)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:851)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient$1.completed(RestClient.java:580)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient$1.completed(RestClient.java:560)
	at org.graylog.shaded.opensearch2.org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:182)
	at org.graylog.shaded.opensearch2.org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
	at org.graylog.shaded.opensearch2.org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: org.graylog.shaded.opensearch2.org.opensearch.client.ResponseException: method [POST], host [https://gl.cekkent.net:9200], URI [/gl-datanode-metrics/_doc?timeout=1m], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [gl-datanode-metrics]","index":"gl-datanode-metrics","resource.id":"gl-datanode-metrics","resource.type":"index_expression","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [gl-datanode-metrics]","index":"gl-datanode-metrics","resource.id":"gl-datanode-metrics","resource.type":"index_expression","index_uuid":"_na_"},"status":404}
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.convertResponse(RestClient.java:479)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.access$1800(RestClient.java:128)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient$1.completed(RestClient.java:564)
		... 16 more
2025-12-18T08:31:40.574-07:00 ERROR [MetricsCollector] Uncaught exception in Periodical
org.graylog.shaded.opensearch2.org.opensearch.OpenSearchStatusException: OpenSearch exception [type=index_not_found_exception, reason=no such index [gl-datanode-metrics]]
	at org.graylog.shaded.opensearch2.org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:210)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2235)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2212)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1931)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1884)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1852)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1095)
	at org.graylog.datanode.periodicals.MetricsCollector.getPreviousMetricsForCluster(MetricsCollector.java:172)
	at org.graylog.datanode.periodicals.MetricsCollector.lambda$doRun$0(MetricsCollector.java:123)
	at java.base/java.util.Optional.ifPresent(Unknown Source)
	at org.graylog.datanode.periodicals.MetricsCollector.doRun(MetricsCollector.java:109)
	at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99)
	at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedPeriodicRunnable.run(InstrumentedScheduledExecutorService.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: org.graylog.shaded.opensearch2.org.opensearch.client.ResponseException: method [POST], host [https://gl.cekkent.net:9200], URI [/gl-datanode-metrics/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [gl-datanode-metrics]","index":"gl-datanode-metrics","resource.id":"gl-datanode-metrics","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [gl-datanode-metrics]","index":"gl-datanode-metrics","resource.id":"gl-datanode-metrics","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.convertResponse(RestClient.java:479)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:371)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:346)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1918)
		... 15 more
2025-12-18T08:32:40.439-07:00 ERROR [MetricsCollector] Error indexing metrics
org.graylog.shaded.opensearch2.org.opensearch.OpenSearchStatusException: OpenSearch exception [type=index_not_found_exception, reason=no such index [gl-datanode-metrics]]
	at org.graylog.shaded.opensearch2.org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:210)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2235)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2212)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:2123)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:851)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient$1.completed(RestClient.java:580)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient$1.completed(RestClient.java:560)
	at org.graylog.shaded.opensearch2.org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:182)
	at org.graylog.shaded.opensearch2.org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
	at org.graylog.shaded.opensearch2.org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.graylog.shaded.opensearch2.org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: org.graylog.shaded.opensearch2.org.opensearch.client.ResponseException: method [POST], host [https://gl.cekkent.net:9200], URI [/gl-datanode-metrics/_doc?timeout=1m], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [gl-datanode-metrics]","index":"gl-datanode-metrics","resource.id":"gl-datanode-metrics","resource.type":"index_expression","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [gl-datanode-metrics]","index":"gl-datanode-metrics","resource.id":"gl-datanode-metrics","resource.type":"index_expression","index_uuid":"_na_"},"status":404}
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.convertResponse(RestClient.java:479)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.access$1800(RestClient.java:128)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient$1.completed(RestClient.java:564)
		... 16 more
2025-12-18T08:32:40.578-07:00 ERROR [MetricsCollector] Uncaught exception in Periodical
org.graylog.shaded.opensearch2.org.opensearch.OpenSearchStatusException: OpenSearch exception [type=index_not_found_exception, reason=no such index [gl-datanode-metrics]]
	at org.graylog.shaded.opensearch2.org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:210)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2235)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2212)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1931)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1884)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1852)
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1095)
	at org.graylog.datanode.periodicals.MetricsCollector.getPreviousMetricsForCluster(MetricsCollector.java:172)
	at org.graylog.datanode.periodicals.MetricsCollector.lambda$doRun$0(MetricsCollector.java:123)
	at java.base/java.util.Optional.ifPresent(Unknown Source)
	at org.graylog.datanode.periodicals.MetricsCollector.doRun(MetricsCollector.java:109)
	at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99)
	at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedPeriodicRunnable.run(InstrumentedScheduledExecutorService.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: org.graylog.shaded.opensearch2.org.opensearch.client.ResponseException: method [POST], host [https://gl.cekkent.net:9200], URI [/gl-datanode-metrics/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [gl-datanode-metrics]","index":"gl-datanode-metrics","resource.id":"gl-datanode-metrics","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [gl-datanode-metrics]","index":"gl-datanode-metrics","resource.id":"gl-datanode-metrics","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.convertResponse(RestClient.java:479)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:371)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:346)
		at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1918)
		... 15 more

Restarting Data Node did not recreate the index or data stream.

In this case, the trauma to my indexes was self-inflicted, so I manually re-created the indexes and data stream:

# List all the backing indices
curl -k --cert datanode-cert.crt --key datanode-key.key \
  -X GET "https://localhost:9200/_cat/indices/.ds-gl-datanode-metrics-*?v"

# Delete all backing indices at once (wildcard)
curl -k --cert datanode-cert.crt --key datanode-key.key \
  -X DELETE "https://localhost:9200/.ds-gl-datanode-metrics-*"

# Now create the data stream
curl -k --cert datanode-cert.crt --key datanode-key.key \
  -X PUT "https://localhost:9200/_data_stream/gl-datanode-metrics"

# Verify
curl -k --cert datanode-cert.crt --key datanode-key.key \
  -X GET "https://localhost:9200/_data_stream/gl-datanode-metrics?pretty"

Would be good to check on Data Node start if the DS and indices exist, and re-create them if not.

Your Environment

• Graylog Version: 7.0.2
• Java Version: embedded
• OpenSearch Version: Data Node 7.0.2
• MongoDB Version: 8.0.4
• Operating System: Debian 12

[todvora]

I tried to reproduce this issue by manually triggering DELETE on _data_stream/gl-datanode-metrics while the datanode service is running. Then it starts logging the same errors as in the stacktrace above.

But after restarting datanode, the datastream is recreated and everything is running fine, without any exceptions anywhere. Is it possible that there is some other issue causing this? 🤔

[damianharouff]

Did you delete both the data stream and the backing indexes?

I was moving my lab from a 3x DN cluster back to a 1x AIO install. As part of that, I moved index sets that Graylog was tracking, but didn't move indexes that weren't visible in System -> Indicies, so things like the data node metrics were missed.

With Data Node setup that was previously running, try stopping Graylog but leaving Data Node running, deleting the data node metrics and data stream, starting Graylog, and see if you get the periodical triggering that it can't write data node metrics.

If you still can't replicate at that point; I'm willing to agree it's a "That's Weird" kind of situation stemming from me doing something we wouldn't support as a company anyway, and we can close this out as being unable to replicate.

[todvora]

I deleted just the data stream itself and got the same error message. The stream will be recreated during datanode restart, so maybe that's the difference between our tests?