java.lang.UnsupportedOperationException on X-pack enabled ABAC system - elasticsearch

Question

KB/ES version: 7.2
I am trying to build an ABAC security based Multi-Tenant system with X-Pack platinum license.
There is an Index which have 3 Client's data. The Admins who have access to all three Client's data, able to view the Kibana Dashboards properly.
However, Users and Admins with 2 out of 3 Clients' data access are unable to view the dashboard.
Whenever they access the Dashboard the following Error pops up on Kibana.
Error: Request to Elasticsearch failed:
{"error":{"root_cause":[{"type":"unsupported_operation_exception","reason":"unsupported_operation_exception:
null"}],"type":"search_phase_execution_exception","reason":"all shards
failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"unified_asset_processed","node":"MtkL9yGKQPyJZu3an3ICvw","reason":{"type":"unsupported_operation_exception","reason":"unsupported_operation_exception:
null"}}]},"status":500}
Detailed Logs in ES can be seen like below:
org.elasticsearch.transport.RemoteTransportException: [elkxx-1][ip-address:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:306) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:114) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:335) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:360) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.UnsupportedOperationException
at org.elasticsearch.search.internal.ContextIndexSearcher$1.scorer(ContextIndexSearcher.java:156) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.xpack.core.security.authz.accesscontrol.SecurityIndexSearcherWrapper$IndexSearcherWrapper.search(SecurityIndexSearcherWrapper.java:164) ~[?:?]
at org.apache.lucene.search.XIndexSearcher.search(XIndexSearcher.java:44) ~[elasticsearch-7.2.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:177) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:271) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:114) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:335) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:360) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) ~[?:?]
How to resolve this error?
Kibana request posted to ES:
Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[xxxx_xxxx_processed], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='1569247377788', requestCache=false, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=32, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":0,"timeout":"30000ms","query":{"bool":{"must":[{"range":{"CREATE_DATE":{"from":"2019-02-24T06:00:49.223Z","to":"2019-09-24T06:00:49.223Z","include_lower":true,"include_upper":true,"format":"strict_date_optional_time","boost":1.0}}}],"filter":[{"match_all":{"boost":1.0}},{"match_all":{"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"_source":{"includes":[],"excludes":[]},"stored_fields":"*","docvalue_fields":[{"field":"WOW_CREATE_DTTM","format":"date_time"},{"field":"CREATE_DATE","format":"date_time"},{"field":"INSTALL_DATE","format":"date_time"},{"field":"LASTMODIFIEDTIME","format":"date_time"},{"field":"LAST_REFRESH_DTTM","format":"date_time"},{"field":"PRODDATE","format":"date_time"}],"script_fields":{},"track_total_hits":2147483647,"aggregations":{"2":{"terms":{"field":"OSNAME2","missing":"__missing__","size":50,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}]

Answer 1

On what ES version was the index created on? might be that the index was created in ES 5 or less? that might be the case, as mapping types have changed and query syntax might not fit. see this issue for a similar example: github.com/elastic/kibana/issues/13950