go-elasticsearch fetch all documents - go

Question

I am trying to use go-elasticsearch to fetch all data from ES.
"github.com/elastic/go-elasticsearch/v8"
"github.com/elastic/go-elasticsearch/v8/esapi"
So far I have written this code:
var r map[string]interface{}
cfg := elasticsearch.Config{
Addresses: []string{
fmt.Sprint(viper.Get("ELASTICSEARCH_URL")),
},
}
es, err := elasticsearch.NewClient(cfg)
if err != nil {
log.Fatalf("Error creating the client: %s", err)
}
res, err := es.Search(
es.Search.WithContext(context.Background()),
es.Search.WithIndex(index_name),
es.Search.WithTrackTotalHits(true),
es.Search.WithPretty(),
es.Search.WithFrom(0),
es.Search.WithSize(1000),
)
if err != nil {
log.Printf("Error getting response: %s", err)
}
defer res.Body.Close()
if res.IsError() {
var e map[string]interface{}
if err := json.NewDecoder(res.Body).Decode(&e); err != nil {
log.Printf("Error parsing the response body: %s", err)
} else {
// Print the response status and error information.
log.Printf("[%s] %s: %s",
res.Status(),
e["error"].(map[string]interface{})["type"],
e["error"].(map[string]interface{})["reason"],
)
}
}
if err := json.NewDecoder(res.Body).Decode(&r); err != nil {
log.Printf("Error parsing the response body: %s", err)
}
// Print the response status, number of results, and request duration.
log.Printf(
"[%s] %d hits; took: %dms",
res.Status(),
int(r["hits"].(map[string]interface{})["total"].(map[string]interface{})["value"].(float64)),
int(r["took"].(float64)),
)
fmt.Println("length", len(r["hits"].(map[string]interface{})["hits"].([]interface{})))
domain := fmt.Sprint(viper.Get("DOMAIN"))
for _, hit := range r["hits"].(map[string]interface{})["hits"].([]interface{}) {
doc := hit.(map[string]interface{})
source := doc["_source"]
}
I want to fetch all documents from this index, I won't know how many documents will be there in the index. How do I do that? The max limit, by default, to fetch data at a time, is set to 10000.

Answer 1

You can do that by Scrolling search results.
Prefer not to scroll unless needed, as scrolling operation is costly