here are my logs:
index: purchase
{details: { name: john, corID: 12345678 , UUID : 555gotr}}
{details: { name: bill, corID: 96657545 , UUID : ggg777}}
other indexes
{town: NY, ID: 12345678 }
{a:b , v: g}
{a: hi, b: 12345678}
{g:f , k:ggg777 }
I would like to create a dashboard where a the user can enter a name, process will search in purchase index for details.name=<name> , extract details.UUID and search in all other indexes for logs which contain the details.UUID
example for above details:
user enters: john
kibana dashboard result:
(all logs which have 12345678)
{details: { name: john, corID: 12345678 , UUID : 555gotr}}
{town: NY, ID: 12345678 }
{a: hi, b: 12345678}
You need to create multiple visualisations, some on purchase index and some of other index, then search in the navbar text input ,12345678 and each visualisation will show the documents on specific index.
Related
I'm trying to add a map to an array of maps in ytt to modify a YAML doc.
I tried the below but it errors out and says it expects a map but getting an array.
https://gist.github.com/amalagaura/c8b5c7c92402120ed76dec95dfafb276
---
id: 1
type: book
awards:
books:
- id: 1
title: International Botev
reviewers:
- id: 2
name: PersonB
- id: 2
title: Dayton Literary Peace Prize
reviewers:
- id: 3
name: PersonC
#! How do I add a map to an array of maps?
## load("#ytt:overlay", "overlay")
##overlay/match by=overlay.all
---
awards:
books:
##overlay/match by=overlay.all, expects="1+"
##overlay/match missing_ok=True
reviewers:
##overlay/append
- id: 1
name: PersonA
## load("#ytt:overlay", "overlay")
#! Add a map to an array of maps:
##overlay/match by=overlay.all
---
awards:
books:
##overlay/match by=overlay.all, expects="1+"
- reviewers:
##overlay/append
- id: 1
name: Person A
You were really close in your solution, all you really needed was to make reviewers an array item. If you want to be able to add reviewers to a book that does not have that key, then you will have to add a matcher on the array item and the map item; a gist is included below to see this behavior overlay in action.
If you have more than one ##overlay/match annotation on the same item, the last one wins. There are plans to improve this behavior: https://github.com/k14s/ytt/issues/114.
https://get-ytt.io/#gist:https://gist.github.com/gcheadle-vmware/a6243ee73fa5cc139dba870690eb15c5
here are my logs:
index: purchase
{details: { name: john, corID: 12345678 }}
{details: { name: bill, corID: 96657545}}
{town: NY, ID: 12345678 }
{a:b , v: g}
{a: hi, b: 12345678}
{g:f , k:ggg777 }
I would like to create a query which for a given name, the query will search in purchase index for details.name=<name> , extract details.corID and search in the indexe for logs which contain the details.corID
example for above details:
name = "john"
query result:
(all logs which have 12345678)
{details: { name: john, corID: 12345678 }}
{town: NY, ID: 12345678 }
{a: hi, b: 12345678}
EDIT
this is how I would do it in SQL:
SELECT * FROM purchase
where corID=
(SELECT details.corID
FROM [purchase]
where details.name = "john")
I have a large data set of rental listings that I want to generate the average price for each city based on the number of bedrooms. I have the following types of rows:
{( city: 'New York', num_bedrooms: 1, price: 1000.00 ),
( city: 'New York', num_bedrooms: 2, price: 2000.00 ),
( city: 'New York', num_bedrooms: 1, price: 2000.00 ),
( city: 'Chicago', num_bedrooms: 1, price: 4000.00 ),
( city: 'Chicago', num_bedrooms: 1, price: 1500.00 )}
Using Pig, I want to get results in the following format:
{( city: 'New York', 1: 1500.00, 2: 2000.00),
( city: 'Chicago', 1: 2750.00 )}
Alternatively, I could deal with this too:
{( city: 'New York', num_bedrooms: 1, price: 1500.00),
( city: 'New York', num_bedrooms: 2, price: 2000.00),
( city: 'Chicago', num_bedrooms: 1, price: 2750.00 )}
My plan is to create bar charts using this data with the number of bedrooms along the X axis, and the price on the Y axis for a given city. I have been able to group by city and number of bedrooms and then average that, but I don't know how to put the data in the format I want. So far this is what I have:
D = GROUP blah BY (city, num_bedrooms);
C = FOREACH D GENERATE blah.city, blah.num_bedrooms, AVG(blah.price);
However this causes the city and num_bedrooms to be repeated for each time they appear!
Input :
New York,1,1000.00
New York,2,2000.00
New York,1,2000.00
Chicago,1,4000.00
Chicago,1,1500.00
Approach 1 :
Pig Script :
rental_data = LOAD 'rental_data.csv' USING PigStorage(',') AS (city:chararray, num_bedrooms: long, price:double);
rental_data_grp_city = GROUP rental_data BY (city);
rental_kpi = FOREACH rental_data_grp_city {
one_bed_room = FILTER rental_data BY num_bedrooms==1;
two_bed_room = FILTER rental_data BY num_bedrooms==2;
GENERATE group AS city, AVG(one_bed_room.price) AS one_bed_price, AVG(two_bed_room.price) AS tow_bed_price;
};
Output : DUMP rental_kpi :
(Chicago,2750.0,)
(New York,1500.0,2000.0)
Approach 2 :
Pig Script :
rental_data = LOAD 'rental_data.csv' USING PigStorage(',') AS (city:chararray, num_bedrooms: long, price:double);
rental_data_grp_city = GROUP rental_data BY (city,num_bedrooms);
rental_kpi = FOREACH rental_data_grp_city {
prices_bag = rental_data.price;
GENERATE group.city AS city, group.num_bedrooms AS num_bedrooms, AVG(prices_bag) AS price;
}
Output : DUMP rental_kpi :
(Chicago,1,2750.0)
(New York,2,2000.0)
(New York,1,1500.0)
I'm trying to turn the following schema:
{
id: chararray,
v: chararray,
paid: chararray,
ts: {(ts: int)}
}
into the following JSON output:
{
"id": "abcdef123456",
v: "some identifier",
paid: "another identifier",
ts: [ 1,2,3,4,5,6 ]
}
I know how to generate the JSON output, but I can't figure out how to turn the ts attribute in my Pig Schema to just the array of numeric values.
The number of items in the ts bag is known, but they all have the same schema (ts: int).
Pig doesn't support array kind of datatype, one option could be you can try something like this.
input
1 1 100 {(1),(2),(3)}
2 2 200 {(4),(5)}
3 3 300 {(1),(2),(3),(4),(5),(6)}
PigScript:
A = LOAD 'input' USING PigStorage() AS (id: chararray, v: chararray,paid: chararray,ts: {(ts: int)});
B = FOREACH A GENERATE id,v,paid,CONCAT('[',BagToString(ts,','),']') AS ts;
STORE B INTO 'output' USING JsonStorage();
Output:
{"id":"1","v":"1","paid":"100","ts":"[1,2,3]"}
{"id":"2","v":"2","paid":"200","ts":"[4,5]"}
{"id":"3","v":"3","paid":"300","ts":"[1,2,3,4,5,6]"}
I have 2 documents:
{ path: "/monster/green", name: "Green monster", age: 105, timeline: 1 }
{ path: "/monster/green" name: "Really Green Monster", timeline: 2 }
The question:
How can I create a CouchDB view, where the documents are sorted by 'timeline' and the result is a combination of them:
{ path: "/monster/green", name: "Really Green Monster", age: 105 }
You can easily create a view which will give you the following output for key="/monster/green" :
{[
{"key":"/monster/green", "value":{"name": "Green monster", "age": 105},
{"key":"/monster/green", "value":{"name": "Really Green monster"}
]}
Except the format, this is really similar to the data you wanted.
If you really need to change the format, you can define a list on top of your view.