I have an array of objects that I would like to reformat using a jsonata expression
{
"items": [
{
"time": 1575417919282,
"message": {
"data": 21,
"type": "temperature"
}
},
{
"time": 1575417919282,
"message": {
"data": 45,
"type": "temperature"
}
}
]
}
Desired format
[
{
"data": 21,
"type": "temperature",
"time": 1575417919282
},
{
"data": 45,
"type": "temperature"
"time": 1575417919282
}
]
Is there an easy one liner for this? I started with merging time into the message object using $merge([$.items.message, {"time":$.items.time}]) but his gives me
{
"data": 45,
"type": "temperature",
"time": [
1575417919282,
1575417919282
]
}
I'm finding the documentation hard to follow. How do you start with just merging two objects iteratively?
This will do it:
items.{
"data": message.data,
"type": message.type,
"time": time
}
http://try.jsonata.org/SJZDsyHTr
Related
I want to sort objects in ascending order but the sort doesn't work.
Here is a sort query below.
"sort":[
{
"category.position": {
"order":"asc",
"mode":"min",
"nested": {
"path": "category",
"filter": {
"term": {"category_category_id":42} }
}
}
}]
And here are the objects below.
"name": "Yeti",
"category": [
{
"category_id": 42,
"name": "Raamiga",
"position": 3
},
],
"name": "Venus",
"category": [
{
"category_id": 42,
"name": "Raamiga",
"position": 4
}
],
Please, help! Many thanks in advance!
Solved. There was a typo… Must be "category.category_id" indtead of "category_category_id".
I found an interesting article that used several data models on Vega-Lite. Tabular data were combined by key like in relational databases.
{
"$schema": "https://vega.github.io/schema/vega-lite/v2.json",
"title": "Test",
"datasets": {
"stores": [
{"cmdb_id1": 1, "group": "type1"},
{"cmdb_id1": 2, "group": "type1"},
{"cmdb_id1": 3, "group": "type2"}
],
"labelsTimelines": [
{"cmdb_id2": 1, "value": 83},
{"cmdb_id2": 2, "value": 53},
{"cmdb_id2": 3, "value": 23}
]
},
"data": {"name": "stores"},
"transform": [
{
"lookup": "cmdb_id1",
"from": {
"data": {"name": "labelsTimelines"},
"key": "cmdb_id2",
"fields": ["value"]
}
}
],
"mark": "bar",
"encoding": {
"y": {"aggregate": "sum", "field": "value", "type": "quantitative"},
"x": {"field": "group", "type": "ordinal"}
}
}
Vega Editor
The question arose as to whether it was possible to obtain the same result using the construction:
"data": {"url": "...."}
Changed the source for Elasticsearch query:
{
"$schema": "https://vega.github.io/schema/vega-lite/v3.json",
"datasets": {
"stores": [{
"url": {
"%context%": "true"
"index": "test_cmdb"
"body": {
"size": 1000,
"_source": ["cmdb_id", "street","group"]
}
}
format: {property: "hits.hits"}
}]}
"data": {
"name": "stores"
},
"encoding": {
"x": {"field": "url.body.size", "type": "ordinal", "title": "X"},
"y": {"field": "url.body.size", "type": "ordinal", "title": "Y"}
},
"layer": [
{
"mark": "rect",
"encoding": {
"tooltip": [
{"field": "url"}]
}
}
]
}
I understand that there is a syntactical error, the data did not come from Elasticsearch.
Thanks in advance!
example.png
No, it is not currently possible to specify URL data within top-level "datasets". The relevant open feature request in Vega-Lite is here: https://github.com/vega/vega-lite/issues/4004.
Your much better off using Vega rather than Vega-lite for this. In Vega you can specify as many datasets as you like with a URL. For example...
...
data: [
{
name: dataset_1
url: {
...
}
}
{
name: dataset_2
url: {
...
}
}
]
...
This can actually get very interesting since it means you can combine data from multiple indices into one visualisation.
I know this is late, but figured this might help people who are looking around.
I have the index I attached below.
Each doc in the index holds the name and height of Alice or Bob and the age at which the height was measured. Measurements taken at the age of 10 are flagged as "baseline_height_at_age_10": true
I need to do the following:
Find the height of Alice and Bob at age 10.
List item Return for Alice and Bob, the records where the height is lower than their height at age 10.
So my question is: Can Painless do such type of search?
I'd appriciate if you could point me at a good example for that.
Also: Is ElasticSearch Painless even a good approach for this problem? Can you sugges
The Index Mappings
PUT /shlomi_test/
{
"mappings": {
"_doc": {
"properties": {
"first_name": {
"type": "keyword",
"fields": {
"raw": {
"type": "text"
}
}
},
"surname": {
"type": "keyword",
"fields": {
"raw": {
"type": "text"
}
}
},
"baseline_height_at_age_10": {
"type": "boolean"
},
"age": {
"type": "integer"
},
"height": {
"type": "integer"
}
}
}
}
}
The Index Data
POST /test/_doc/alice_green_8_110
{
"first_name": "Alice",
"surname": "Green",
"age": 8,
"height": 110,
"baseline_height_at_age_10": false
}
POST /test/_doc/alice_green_10_120
{
"first_name": "Alice",
"surname": "Green",
"age": 10,
"height": 120,
"baseline_height_at_age_10": true
}
POST /test/_doc/alice_green_13_140
{
"first_name": "Alice",
"surname": "Green",
"age": 13,
"height": 140,
"baseline_height_at_age_10": false
}
POST /test/_doc/alice_green_23_170
{
"first_name": "Alice",
"surname": "Green",
"age": 23,
"height": 170,
"baseline_height_at_age_10": false
}
POST /test/_doc/bob_green_8_120
{
"first_name": "Alice",
"surname": "Green",
"age": 8,
"height": 120,
"baseline_height_at_age_10": false
}
POST /test/_doc/bob_green_10_130
{
"first_name": "Alice",
"surname": "Green",
"age": 10,
"height": 130,
"baseline_height_at_age_10": true
}
POST /test/_doc/bob_green_15_160
{
"first_name": "Alice",
"surname": "Green",
"age": 15,
"height": 160,
"baseline_height_at_age_10": false
}
POST /test/_doc/bob_green_21_180
{
"first_name": "Alice",
"surname": "Green",
"age": 21,
"height": 180,
"baseline_height_at_age_10": false
}
You should be able to do it just using aggregations. Assuming people only ever get taller, and the measurements are accurate, you could restrict the query to only those documents aged 10 or under, find the max height of those, then filter the results of those to exclude the baseline result
POST test/_search
{
"size": 0,
"query": {
"range": {
"age": {
"lte": 10
}
}
},
"aggs": {
"names": {
"terms": {
"field": "first_name",
"size": 10
},
"aggs": {
"max_height": {
"max": {
"field": "height"
}
},
"non-baseline": {
"filter": {
"match": {
"baseline_height_at_age_10": false
}
},
"aggs": {
"top_hits": {
"top_hits": {
"size": 10
}
}
}
}
}
}
}
}
I've posted the same question, with emphasis on Painless scripting, ElasticSearch Support Forum How to find records matching the result of a previous search using ElasticSearch Painless scripting
and the answer was:
"I don't think the Painless approach will work here. You cannot use
the results of one query to execute a second query with Painless.
The two-step approach that you outline at the end of your post is the
way to go."
The bottom line is that you cannot use a result from one query as an input to another query. You can filter and aggregate and more, but not this.
So the approcah is pretty much as follows:
according to my understanding, suggests to do the 1st search, process
the data and do an additional search. This basically translates to:
Search the record where first_name=Alice and baseline_height_at_age_10=True.
Process externally, to extract the value of height for Alice at age 10.
Search for Alice's records where her height is lower than the value calculated externally.
I have documents in ElasticSearch with the following structure :
"_source": {
"last_updated": "2017-10-25T18:33:51.434706",
"country": "Italia",
"price": [
"€ 139",
"€ 125",
"€ 120",
"€ 108"
],
"max_occupancy": [
2,
2,
1,
1
],
"type": [
"Type 1",
"Type 1 - (Tag)",
"Type 2",
"Type 2 (Tag)",
],
"availability": [
10,
10,
10,
10
],
"size": [
"26 m²",
"35 m²",
"47 m²",
"31 m²"
]
}
}
Basically, the details records are split in 5 arrays, and fields of the same record have the same index position in the 5 arrays. As can be seen in the example data there are 5 array(price, max_occupancy, type, availability, size) that are containing values related to the same element. I want to extract the element that has max_occupancy field greater or equal than 2 (if there is no record with 2 grab a 3 if there is no 3 grab a four, ...), with the lower price, in this case the record and place the result into a new JSON object like the following :
{
"last_updated": "2017-10-25T18:33:51.434706",
"country": "Italia",
"price: ": "€ 125",
"max_occupancy": "2",
"type": "Type 1 - (Tag)",
"availability": 10,
"size": "35 m²"
}
Basically the result structure should show the extracted record(that in this case is the second index of all array), and add the general information to it(fields : "last_updated", "country").
Is it possible to extract such a result from elastic search? What kind of query do I need to perform?
Could someone suggest the best approach?
My best approach: go nested with Nested Datatype
Except for easier querying, it easier to read and understand the connections between those objects that are, currently, scattered in different arrays.
Yes, if you'll decide this approach you will have to edit your mapping and re-index your entire data.
How would the mapping is going to look like? something like this:
{
"mappings": {
"properties": {
"last_updated": {
"type": "date"
},
"country": {
"type": "string"
},
"records": {
"type": "nested",
"properties": {
"price": {
"type": "string"
},
"max_occupancy": {
"type": "long"
},
"type": {
"type": "string"
},
"availability": {
"type": "long"
},
"size": {
"type": "string"
}
}
}
}
}
}
EDIT: New document structure (containing nested documents) -
{
"last_updated": "2017-10-25T18:33:51.434706",
"country": "Italia",
"records": [
{
"price": "€ 139",
"max_occupancy": 2,
"type": "Type 1",
"availability": 10,
"size": "26 m²"
},
{
"price": "€ 125",
"max_occupancy": 2,
"type": "Type 1 - (Tag)",
"availability": 10,
"size": "35 m²"
},
{
"price": "€ 120",
"max_occupancy": 1,
"type": "Type 2",
"availability": 10,
"size": "47 m²"
},
{
"price": "€ 108",
"max_occupancy": 1,
"type": "Type 2 (Tag)",
"availability": 10,
"size": "31 m²"
}
]
}
Now, its more easy to query for any specific condition with Nested Query and Inner Hits. for example:
{
"_source": [
"last_updated",
"country"
],
"query": {
"bool": {
"must": [
{
"term": {
"country": "Italia"
}
},
{
"nested": {
"path": "records",
"query": {
"bool": {
"must": [
{
"range": {
"records.max_occupancy": {
"gte": 2
}
}
}
]
}
},
"inner_hits": {
"sort": {
"records.price": "asc"
},
"size": 1
}
}
}
]
}
}
}
Conditions are: Italia AND max_occupancy > 2.
Inner hits: sort by price ascending order and get the first result.
Hope you'll find it useful
So assuming that i have a mapping structure like the following
{
"mappings": {
"users": {
"properties": {
"user": {
"type": "nested"
}
}
}
}
}
and I have indexed the following
users/52
{
"user": [
{
"id": 52,
"first": "John",
"last": "Smith",
"age": 21,
"school": {
"name": "STC",
"location": "Mt LV",
"District": "Western"
}
}
]
}
users/57
{
"user": [
{
"id": 57,
"first": "Alice",
"last": "White",
"age": 25,
"school": {
"name": "HFC",
"location": "DEH WLA",
"District": "Western"
}
}
]
}
What if I want to get certain fields using the id and without destroying the relationship link of each other.
For an example
If id == 57
then the return structure should consists only "first","age","school.name","school.District"
{
"user": [
{
"first": "Alice",
"age": 25,
"school": {
"name": "HFC",
"District": "Western"
}
}
]
}
How should you write a query for this sort of response in Elasticsearch?
Use response filtering in Elasticsearch. According to your scenario, a GET request would look like GET /_search?user=57&filter_path=first,age,school.name,school.District