Elasticsearch catch-all field slowness after upgrade - elasticsearch

Question

We upgraded a 2.4 cluster to 6.2 cluster using the reindex from remote approach. In 2.4, we were using the catch-all _all field to perform searches and were seeing response times under 500 ms for all our queries.
In 6.2, the _all field is no longer available for the new index, so we ended up creating a new text type field called all like "all": {"type": "text"} and set copy_to on all our other fields (about 2000 of them). But now, searches on this new catch-all field all are taking 2 to 10 times longer than the search on the 2.4 _all field. (We flushed the caches on both clusters before performing the queries.)
Both clusters are single data center, single node 8GB memory on the same AWS zone, hosted through elastic cloud. Both indices have the same number of documents (about 6M) and have about 150 Lucene segment files.
Any clues as to why?
UPDATE: Both indices return documents without the catch-all field i.e. they do not store the catch-all field.
Here is an example query and response:
$ curl --user "$user:$password" \
> -H 'Content-Type: application/json' \
> -XGET "$es/$index/$mapping/_search?pretty" -d'
> {
> "size": 1,
> "query" : {
> "match" : { "all": "sherlock" }
> }
> }
> '
{
"took" : 42,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 28133,
"max_score" : 2.290815,
"hits" : [ {
"_index" : "sherlock",
"_type" : "doc",
"_id" : "513763",
"_score" : 2.290815,
"_source" : {
"docid" : 513763,
"age" : 115,
"essay" : "Has Mr. Sherlock Holmes?",
"name" : {
"last" : "Pezzetti",
"first" : "Lilli"
},
"ssn" : 834632279
}
} ]
}
}
UPDATE 2: Another point I forgot to mention is that the 2.4 cluster is currently being used by a staging app, which sends a few queries to it every few minutes. Could this bring other factors like OS caching into play?

Answer 1

Did you store the _all field and returned it in your original setup? Do you return it now? If you didn't and now you do then that's a response overhead you are seeing and not a search overhead. Basically you should omit that field in your response (from you _source) if you don't need it (and any other field for that matter).
Check _source filtering for more