I need to get in Go the latest date from a datetime field of the elasticsearch index's documents. Basically, I have this query that I do directly in the elasticsearch that returns me just what I need:
GET localhost:9200/index-name/_search
{
"aggs" : {
"max_date": {"max": {"field": "dateTime"}}
}
}
I need to do that same query in Go. I saw that there is a MaxAgregation in Olivere library, but I'm not sure about how to use it. Does someone knows how to do that?
Here's how to get started w/ the go library.
After you instantiate a client, you can do the following:
maxDateAgg := NewMaxAggregation().Field("dateTime")
builder := client.Search().Index("index-name").Pretty(true)
builder = builder.Aggregation("max_date", maxDateAgg)
After that, call .Do(ctx) on the builder.
I manage to make it work with one of the answers and some things that I already had:
maxDateAgg := elastic.NewMaxAggregation().Field("dateTime")
builder := esClient.Search().Index("index-name").Pretty(true)
builder = builder.Aggregation("max_datetime", maxDateAgg) builder = builder.Size(1).Sort("dateTime",false)
searchResult, err := builder.Do(ctx)
This returns all the fields from the documents with the max datetime, so I made a Struct, and json.unmarshall the search.hit.hit source into the struct, so i could take only the datetime field from the struct.
For those wondering how to get the date
ctx := context.Background()
termQuery := elastic.NewMatchAllQuery()
agg := elastic.NewMaxAggregation().Field("le_nested.last_updated")
sr, err := elastic.Client.Search("your_index").Query(termQuery).Aggregation("max_date", agg).Do(ctx)
if err != nil {
m := fmt.Sprint("Error getting getting last updated data", err)
fmt.Println(m)
}
max, found := sr.Aggregations.MaxBucket("max_date")
if !found {
m := fmt.Sprint("max_date aggregation not found", err)
fmt.Println(m)
}
fmt.Println("MAX DATE", max.ValueAsString)
tm := int64(*max.Value) / 1000
mu := time.Unix(tm, 0)
fmt.Println("DATE", mu)
ES returns the dates in millis, no need to unmarshall anything
Related
I'm new in Golang, what I am trying to do is to query Prometheus and save the query result in an object (such as a map) that has all timestamps and their values of the metric.
I started from this example code with only a few changes (https://github.com/prometheus/client_golang/blob/master/api/prometheus/v1/example_test.go)
func getFromPromRange(start time.Time, end time.Time, metric string) model.Value {
client, err := api.NewClient(api.Config{
Address: "http://localhost:9090",
})
if err != nil {
fmt.Printf("Error creating client: %v\n", err)
os.Exit(1)
}
v1api := v1.NewAPI(client)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
r := v1.Range{
Start: start,
End: end,
Step: time.Second,
}
result, warnings, err := v1api.QueryRange(ctx, metric, r)
if err != nil {
fmt.Printf("Error querying Prometheus: %v\n", err)
os.Exit(1)
}
if len(warnings) > 0 {
fmt.Printf("Warnings: %v\n", warnings)
}
fmt.Printf("Result:\n%v\n", result)
return result
}
The result that is printed is for example:
"TEST{instance="localhost:4321", job="realtime"} =>\n21 #[1597758502.337]\n22 #[1597758503.337]...
These are actually the correct values and timestamps that are on Prometheus. How can I insert these timestamps and values into a map object (or another type of object that I can then use in code)?
The result coming from QueryRange has the type model.Matrix.
This will then contain a pointer of type *SampleStream. As your example then contains only one SampleStream, we can access the first one directly.
The SampleStream then has a Metric and Values of type []SamplePair. What you are aiming for is the slice of sample pairs. Over this we then can iterate and build for instance a map.
mapData := make(map[model.Time]model.SampleValue)
for _, val := range result.(model.Matrix)[0].Values {
mapData[val.Timestamp] = val.Value
}
fmt.Println(mapData)
You have to know the type of result you're getting returned. For example, model.Value can be of type Scalar, Vector, Matrix or String. Each of these types have their own way of getting the data and timestamps. For example, a Vector has an array of Sample types which contain the data you're looking for. The godocs and the github repo for the prom/go client have really great documentation if you want to dive deeper.
maybe you can find your answer in this issue
https://github.com/prometheus/client_golang/issues/194
switch {
case val.Type() == model.ValScalar:
scalarVal := val.(*model.Scalar)
// handle scalar stuff
case val.Type() == model.ValVector:
vectorVal := val.(model.Vector)
for _, elem := range vectorVal {
// do something with each element in the vector
// etc
Sorry if this is a dumb question. I'm using a service which was built using Elasticsearch client for Go. I run the service and now seems like the elasticsearch server have the index of the data. However, when I tried to query those data with http://129.94.14.234:9200/chromosomes/chromosome/1, I got {"_index":"chromosomes","_type":"chromosome","_id":"1","_version":1,"found":true,"_source":{"id":"1","length":249250621}} I checked that the SQL query from the database have those data. Now the question is that, how do I check that my elasticsearch index have those data? Or If anyone can tell me what might be wrong on the code that'll be great as well.
Here's the code that I assume adding the documents to chromosomes index.
func (c ChromosomeIndexer) AddDocuments(db *sql.DB, client *elastic.Client, coordID int) {
sqlQuery := fmt.Sprintf("SELECT seq_region.name, seq_region.length FROM seq_region WHERE seq_region.`name` REGEXP '^[[:digit:]]{1,2}$|^[xXyY]$|(?i)^mt$' AND seq_region.`coord_system_id` = %d", coordID)
stmtOut, err := db.Prepare(sqlQuery)
check(err)
defer stmtOut.Close()
stmtOut.Query()
rows, err := stmtOut.Query()
defer rows.Close()
check(err)
chromoFn := func(rows *sql.Rows, bulkRequest *elastic.BulkService) {
var name string
var length int
err = rows.Scan(&name, &length)
check(err)
chromo := Chromosome{ID: name, Length: length}
fmt.Printf("chromoID: %s\n", chromo.ID)
req := elastic.NewBulkIndexRequest().
OpType("index").
Index("chromosomes").
Type("chromosome").
Id(chromo.ID).
Doc(chromo)
bulkRequest.Add(req)
}
elasticutil.IterateSQL(rows, client, chromoFn)
}
This service have other index which I can query the data with no problem, I only have problem when querying chromosomes data.
Please let me know if I need to put more code so that I can give a bit more context on the problem, I just started on Go and Elasticsearch, and I tried reading the documentation, but it just leads to more confusion.
My question is specific to the "gopkg.in/olivere/elastic.v2" package I am using.
I am trying to return all documents that match my query:
termQuery := elastic.NewTermQuery("item_id", item_id)
searchResult, err := es.client.Search().
Index(index).
Type(SegmentsType). // search segments type
Query(termQuery). // specify the query
Sort("time", true). // sort by "user" field, ascending
From(0).Size(9).
Pretty(true). // pretty print request and response JSON
Do() // execute
if err != nil {
// Handle error
return timeline, err
}
The problem is I get an internal server error if I increase the size to something large. If I eliminate the line that states:
From(0).Size(9).
then the default is used (10 documents). How may I return all documents?
Using a scroller to retrieve all results is just a bit different and in the interest of brevity I'm not including much error handling that you might need.
Basically you just need to slightly change your code from Search to Scroller and then loop with the Scroller calling Do and handling pages of results.
termQuery := elastic.NewTermQuery("item_id", item_id)
scroller := es.client.Scroller().
Index(index).
Type(SegmentsType).
Query(termQuery).
Sort("time", true).
Size(1)
docs := 0
for {
res, err := scroller.Do(context.TODO())
if err == io.EOF {
// No remaining documents matching the search so break out of the 'forever' loop
break
}
for _, hit := range res.Hits.Hits {
// JSON parse or do whatever with each document retrieved from your index
item := make(map[string]interface{})
err := json.Unmarshal(*hit.Source, &item)
docs++
}
}
Currently on what I've seen so far is that, converting database rows to JSON or to []map[string]interface{} is not simple. I have to create two slices and then loop through columns and create keys every time.
...Some code
tableData := make([]map[string]interface{}, 0)
values := make([]interface{}, count)
valuePtrs := make([]interface{}, count)
for rows.Next() {
for i := 0; i < count; i++ {
valuePtrs[i] = &values[i]
}
rows.Scan(valuePtrs...)
entry := make(map[string]interface{})
for i, col := range columns {
var v interface{}
val := values[i]
b, ok := val.([]byte)
if ok {
v = string(b)
} else {
v = val
}
entry[col] = v
}
tableData = append(tableData, entry)
}
Is there any package for this ? Or I am missing some basics here
I'm dealing with the same issue, as far as my investigation goes it looks that there is no other way.
All the packages that I have seen use basically the same method
Few things you should know, hopefully will save you time:
database/sql package converts all the data to the appropriate types
if you are using the mysql driver(go-sql-driver/mysql) you need to add
params to your db string for it to return type time instead of a string
(use ?parseTime=true, default is false)
You can use tools that were written by the community, to offload the overhead:
A minimalistic wrapper around database/sql, sqlx, uses similar way internally with reflection.
If you need more functionality, try using an "orm": gorp, gorm.
If you interested in diving deeper check out:
Using reflection in sqlx package, sqlx.go line 560
Data type conversion in database/sql package, convert.go line 86
One thing you could do is create a struct that models your data.
**Note: I am using MS SQLServer
So lets say you want to get a user
type User struct {
ID int `json:"id,omitempty"`
UserName string `json:"user_name,omitempty"`
...
}
then you can do this
func GetUser(w http.ResponseWriter, req *http.Request) {
var r Role
params := mux.Vars(req)
db, err := sql.Open("mssql", "server=ServerName")
if err != nil {
log.Fatal(err)
}
err1 := db.QueryRow("select Id, UserName from [Your Datavse].dbo.Users where Id = ?", params["id"]).Scan(&r.ID, &r.Name)
if err1 != nil {
log.Fatal(err1)
}
json.NewEncoder(w).Encode(&r)
if err != nil {
log.Fatal(err)
}
}
Here are the imports I used
import (
"database/sql"
"net/http"
"log"
"encoding/json"
_ "github.com/denisenkom/go-mssqldb"
"github.com/gorilla/mux"
)
This allowed me to get data from the database and get it into JSON.
This takes a while to code, but it works really well.
Not in the Go distribution itself, but there is the wonderful jmoiron/sqlx:
import "github.com/jmoiron/sqlx"
tableData := make([]map[string]interface{}, 0)
for rows.Next() {
entry := make(map[string]interface{})
err := rows.MapScan(entry)
if err != nil {
log.Fatal("SQL error: " + err.Error())
}
tableData = append(tableData, entry)
}
If you know the data type that you are reading, then you can read into the data type without using generic interface.
Otherwise, there is no solution regardless of the language used due to nature of JSON itself.
JSON does not have description of composite data structures. In other words, JSON is a generic key-value structure. When parser encounters what is supposed to be a specific structure there is no identification of that structure in JSON itself. For example, if you have a structure User the parser would not know how a set of key-value pairs maps to your structure User.
The problem of type recognition is usually addressed with document schema (a.k.a. XSD in XML world) or explicitly through passed expected data type.
One quick way to go about being able to get an arbirtrary and generic []map[string]interface{} from these query libraries is to populate an array of interface pointers with the same size of the amount of columns on the query, and then pass that as a parameter on the scan function:
// For example, for the go-mssqldb lib:
queryResponse, err := d.pool.Query(query)
if err != nil {
return nil, err
}
defer queryResponse.Close()
// Holds all the end-results
results := []map[string]interface{}{}
// Getting details about all the fields from the query
fieldNames, err := queryResponse.Columns()
if err != nil {
return nil, err
}
// Creating interface-type pointers within an array of the same
// size of the number of columns we have, so that we can properly
// pass this to the "Scan" function and get all the query parameters back :)
var scanResults []interface{}
for range fieldNames {
var v interface{}
scanResults = append(scanResults, &v)
}
// Parsing the query results into the result map
for queryResponse.Next() {
// This variable will hold the value for all the columns, named by the column name
rowValues := map[string]interface{}{}
// Cleaning up old values just in case
for _, column := range scanResults {
*(column.(*interface{})) = nil
}
// Scan into the array of pointers
err := queryResponse.Scan(scanResults...)
if err != nil {
return nil, err
}
// Map the pointers back to their value and the associated column name
for index, column := range scanResults {
rowValues[fieldNames[index]] = *(column.(*interface{}))
}
results = append(results, rowValues)
}
return results, nil
I think I did a silly mistake somewhere, but could not figure where for long time already :( The code is rough, I just testing things.
It deletes, but by some reasons not all documents, I have rewritten to delete it all one by one, and that went OK.
I use official package for Couchbase http://github.com/couchbase/gocb
Here is code:
var items []gocb.BulkOp
myQuery := gocb.NewN1qlQuery([Selecting ~ 283k documents from 1.5mln])
rows, err := myBucket.ExecuteN1qlQuery(myQuery, nil)
checkErr(err)
var idToDelete map[string]interface{}
for rows.Next(&idToDelete) {
items = append(items, &gocb.RemoveOp{Key: idToDelete["id"].(string)})
}
if err := rows.Close(); err != nil {
fmt.Println(err.Error())
}
if err := myBucket.Do(items);err != nil {
fmt.Println(err.Error())
}
This way it deleted ~70k documents, I run it again it got deleted 43k more..
Then I just let it delete one by one, and it worked fine:
//var items []gocb.BulkOp
myQuery := gocb.NewN1qlQuery([Selecting ~ 180k documents from ~1.3mln])
rows, err := myBucket.ExecuteN1qlQuery(myQuery, nil)
checkErr(err)
var idToDelete map[string]interface{}
for rows.Next(&idToDelete) {
//items = append(items, &gocb.RemoveOp{Key: idToDelete["id"].(string)})
_, err := myBucket.Remove(idToDelete["id"].(string), 0)
checkErr(err)
}
if err := rows.Close(); err != nil {
fmt.Println(err.Error())
}
//err = myBucket.Do(items)
By default, queries against N1QL use a consistency level called 'request plus'. Thus, your second time running the program to query will use whatever index update is valid at the time of the query, rather than considering all of your previous mutations by waiting until the index is up to date. You can read more about this in Couchbase's Developer Guide and it looks like the you'll want to add the RequestPlus parameter to your myquery through the consistency method on the query.
This kind of eventually consistent secondary indexing and the flexibility is pretty powerful because it gives you as a developer the ability to decide what level of consistency you want to pay for since index recalculations have a cost.