`_search` calls to OpenSearch with `terminate_after` throw `null_pointer_exception`. Among others, this makes same visualizations that worked on OpenSearch 2.13 to fail on 2.14

[franfreire]

Visualization type: Controls

I had this visualization working in opensearch until I updated to 2.14. I tested it back in opensearch 2.13 and in this version it works properly

When I check opensearch logs I see the following error:

org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:757) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:394) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:796) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:542) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:315) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:104) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:75) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:766) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.transport.TransportService$9.handleException(TransportService.java:1729) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:410) [opensearch-security-2.14.0.0.jar:2.14.0.0]
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1515) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1629) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1603) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:81) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:75) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:70) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.ActionRunnable.onFailure(ActionRunnable.java:104) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:54) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.14.0.jar:2.14.0]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.OpenSearchException$3: Cannot invoke "org.opensearch.core.common.io.stream.NamedWriteable.getWriteableName()" because "namedWriteable" is null
	at org.opensearch.OpenSearchException.guessRootCauses(OpenSearchException.java:710) ~[opensearch-core-2.14.0.jar:2.14.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:392) [opensearch-2.14.0.jar:2.14.0]
	... 24 more
Caused by: java.lang.NullPointerException: Cannot invoke "org.opensearch.core.common.io.stream.NamedWriteable.getWriteableName()" because "namedWriteable" is null
	at org.opensearch.core.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1134) ~[opensearch-core-2.14.0.jar:2.14.0]
	at org.opensearch.core.common.io.stream.StreamOutput.writeNamedWriteableList(StreamOutput.java:1233) ~[opensearch-core-2.14.0.jar:2.14.0]
	at org.opensearch.search.aggregations.InternalAggregations.writeTo(InternalAggregations.java:154) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.io.stream.DelayableWriteable$Referencing.writeToBuffer(DelayableWriteable.java:147) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.io.stream.DelayableWriteable$Referencing.writeTo(DelayableWriteable.java:120) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.search.query.QuerySearchResult.writeToNoId(QuerySearchResult.java:445) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.cache.policy.CachedQueryResult.writeToNoId(CachedQueryResult.java:60) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.indices.IndicesService.lambda$loadIntoContext$24(IndicesService.java:1707) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.indices.IndicesService.lambda$cacheShardLevelResult$25(IndicesService.java:1764) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:345) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:328) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.cache.store.OpenSearchOnHeapCache.lambda$computeIfAbsent$0(OpenSearchOnHeapCache.java:98) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.cache.Cache.compute(Cache.java:470) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.cache.Cache.computeIfAbsent(Cache.java:427) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.cache.store.OpenSearchOnHeapCache.computeIfAbsent(OpenSearchOnHeapCache.java:98) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:287) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1770) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.indices.IndicesService.loadIntoContext(IndicesService.java:1701) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:557) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:623) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:592) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-2.14.0.jar:2.14.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.14.0.jar:2.14.0]
	... 8 more

OpenSearch Version
2.14

Dashboards Version
2.14

[mjakob-gh]

I am seeing the same error with 2.14, the visualization worked fine with 2.13

I see this notification:

in the details is this information

Request:

{
  "timeout": "1000ms",
  "terminate_after": 100000,
  "size": 0,
  "aggs": {
    "termsAgg": {
      "terms": {
        "order": {
          "_count": "desc"
        },
        "field": "fields.umgebung.keyword"
      }
    }
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    }
  ],
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [],
      "should": [],
      "must_not": []
    }
  }
}

Response (removed all but two mentioned indices, so the numbers dont match):

{
  "took": 38,
  "timed_out": false,
  "terminated_early": false,
  "_shards": {
    "total": 110,
    "successful": 53,
    "skipped": 0,
    "failed": 57,
    "failures": [
      {
        "shard": 0,
        "index": "filebeat-kmewo-prodo-7.12.1-2024.05.06",
        "node": "7VUNjvDMSoSWpgKVDlGTiA",
        "reason": {
          "type": "null_pointer_exception",
          "reason": "Cannot invoke \"org.opensearch.core.common.io.stream.NamedWriteable.getWriteableName()\" because \"namedWriteable\" is null"
        }
      },
      {
        "shard": 0,
        "index": "filebeat-kmewo-prodo-7.12.1-2024.05.07",
        "node": "7VUNjvDMSoSWpgKVDlGTiA",
        "reason": {
          "type": "null_pointer_exception",
          "reason": "Cannot invoke \"org.opensearch.core.common.io.stream.NamedWriteable.getWriteableName()\" because \"namedWriteable\" is null"
        }
      }
    ]
  },
  "hits": {
    "total": 1831452,
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "termsAgg": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "qso",
          "doc_count": 897627
        },
        {
          "key": "prodo",
          "doc_count": 504180
        },
        {
          "key": "sem",
          "doc_count": 238448
        },
        {
          "key": "prodw",
          "doc_count": 119040
        },
        {
          "key": "qsw",
          "doc_count": 72157
        }
      ]
    }
  }
}

[kavilla]

@AMoo-Miki are you able to check on this.

[kavilla]

@franfreire could you please provide information on your indices and any configurations within OpenSearch and OpenSearch Dashboards. Also your local setup?

Thank you.

[disaster37]

Hi, it's the same here.
I have remove the control visu, The dashboard work fine. If I add it, it work, but I gest the same shard failed pop describe above

[AMoo-Miki]

@disaster37, I am assuming you and @franfreire are together or the same person. If so and you have ascertained that the issue is caused by a call Controls viz is triggering, I would recommend looking at the Network tab of your browser's DevTools to see what specific query was run to cause this failure.

@mjakob-gh, it would be great if you could provide some info about what the underlying failure is by checking Dev Tools too.

On the surface, both of your errors are directly arriving from OpenSearch and I want to make sure we get more info while we transfer it there.

[AMoo-Miki]

@opensearch-project/admin please transfer to OpenSearch engine.

[disaster37]

I have the same issue than @franfreire but we are not the same person and we are not working on same organization. I will try to grab more useful trace on devtools like you suggest tomorrow.

[rasituum]

Hi, i have the same issue on 2.14 with Controls viz, with exactly the same stack traces than @franfreire on my opensearch client node. Works fine in 2.13.