elasticsearch: creating inner queries - elasticsearch

here are my logs:
index: purchase
{details: { name: john, corID: 12345678 }}
{details: { name: bill, corID: 96657545}}
{town: NY, ID: 12345678 }
{a:b , v: g}
{a: hi, b: 12345678}
{g:f , k:ggg777 }
I would like to create a query which for a given name, the query will search in purchase index for details.name=<name> , extract details.corID and search in the indexe for logs which contain the details.corID
example for above details:
name = "john"
query result:
(all logs which have 12345678)
{details: { name: john, corID: 12345678 }}
{town: NY, ID: 12345678 }
{a: hi, b: 12345678}
EDIT
this is how I would do it in SQL:
SELECT * FROM purchase
where corID=
(SELECT details.corID
FROM [purchase]
where details.name = "john")

Related

Unique filter in Nunjuck

systems:
- name: Fred
country: DE
- name: Wilma
country: US
- name: Pebbles
country: DE
- name: Dino
country: US
---
# Systems
Countries: {{ page.systems | join(",", "country") }}
I am trying to create a GitBook page with a list of items containing no duplicates. I.e I would want to apply a 'unique' filter or 'distinct' filter in my Nunjucks template for the page. The template needs to process the page variables (YAML). The above template generates the output:
Countries: DE,US,DE,US
I would like it to produce the output
Countries: DE,US
How could I achive that? (Given that 'unique' filter is not supported with Nunjucks.)
You can extend your Nunjucks through Custom filter
const nunjucks = require('nunjucks');
const env = new nunjucks.Environment(/* loaders etc... */);
env.addFilter('unique', arr => arr instanceof Array && arr.filter((e, i, arr) => arr.indexOf(e) == i) || arr);
let out = env.renderString(`{{[1, 2, 3, 2] | unique }}`);
console.log(out);

create complex kibana dashboard

here are my logs:
index: purchase
{details: { name: john, corID: 12345678 , UUID : 555gotr}}
{details: { name: bill, corID: 96657545 , UUID : ggg777}}
other indexes
{town: NY, ID: 12345678 }
{a:b , v: g}
{a: hi, b: 12345678}
{g:f , k:ggg777 }
I would like to create a dashboard where a the user can enter a name, process will search in purchase index for details.name=<name> , extract details.UUID and search in all other indexes for logs which contain the details.UUID
example for above details:
user enters: john
kibana dashboard result:
(all logs which have 12345678)
{details: { name: john, corID: 12345678 , UUID : 555gotr}}
{town: NY, ID: 12345678 }
{a: hi, b: 12345678}
You need to create multiple visualisations, some on purchase index and some of other index, then search in the navbar text input ,12345678 and each visualisation will show the documents on specific index.

Neo4j query: conditional match

I have this model:
Bob and Alice are Users
CVI is a clinic
Pluto is an animal
The users have a property called identityId (CONSTRAINT UNIQUE) to identify the user.
I would like to select the User with a given id only if it is the user itself (same identityId) or if it exists a relationship SHARED_WITH between the Alice and Bob.
In terms of performance, is the query below the best query for that?
MATCH (u:User)
WHERE id(u) = {id} AND ((u.identityId = {identityId})
OR ((:User { identityId: {identityId} }) - [:OWNS] -> (:Clinic) <- [:SHARED_WITH] - (u)))
RETURN u
Example
Alice { id: 6, identityId: "5678"}
Bob { id: 3, identityId: "1234"}
Mallory { id: 5, identityId: "2222"}
First case: The caller is Alice
MATCH (u:User)
WHERE id(u) = 6 AND ((u.identityId = "5678")
OR ((:User { identityId: "5678" }) - [:OWNS] -> (:Clinic) <- [:SHARED_WITH] - (u)))
RETURN u
The u is Alice
Second case: The caller is Bob
MATCH (u:User)
WHERE id(u) = 6 AND ((u.identityId = "1234")
OR ((:User { identityId: "1234" }) - [:OWNS] -> (:Clinic) <- [:SHARED_WITH] - (u)))
RETURN u
The u is Alice
Third case: The caller is Mallory
MATCH (u:User)
WHERE id(u) = 6 AND ((u.identityId = "2222")
OR ((:User { identityId: "2222" }) - [:OWNS] -> (:Clinic) <- [:SHARED_WITH] - (u)))
RETURN u
The u is NULL (mallory is neither the user nor the user with Alice has shared its user)
Taking into account your additional explanations:
// Get user by id
MATCH (I:User) WHERE id(I) = {id}
// Whom with given {identityId} shard with him
OPTIONAL MATCH (U:User {identityId: {identityId} })
-[:OWNS]->()<-[:SHARED_WITH]-
(I)
WITH I, COUNT(U) as UC
// Test user {identityId}
// or there are those who with {identityId} are with him shares
WHERE I.identityId = {identityId} OR UC > 0
RETURN I

How to generate a distinct average of lots of data in Pig Latin?

I have a large data set of rental listings that I want to generate the average price for each city based on the number of bedrooms. I have the following types of rows:
{( city: 'New York', num_bedrooms: 1, price: 1000.00 ),
( city: 'New York', num_bedrooms: 2, price: 2000.00 ),
( city: 'New York', num_bedrooms: 1, price: 2000.00 ),
( city: 'Chicago', num_bedrooms: 1, price: 4000.00 ),
( city: 'Chicago', num_bedrooms: 1, price: 1500.00 )}
Using Pig, I want to get results in the following format:
{( city: 'New York', 1: 1500.00, 2: 2000.00),
( city: 'Chicago', 1: 2750.00 )}
Alternatively, I could deal with this too:
{( city: 'New York', num_bedrooms: 1, price: 1500.00),
( city: 'New York', num_bedrooms: 2, price: 2000.00),
( city: 'Chicago', num_bedrooms: 1, price: 2750.00 )}
My plan is to create bar charts using this data with the number of bedrooms along the X axis, and the price on the Y axis for a given city. I have been able to group by city and number of bedrooms and then average that, but I don't know how to put the data in the format I want. So far this is what I have:
D = GROUP blah BY (city, num_bedrooms);
C = FOREACH D GENERATE blah.city, blah.num_bedrooms, AVG(blah.price);
However this causes the city and num_bedrooms to be repeated for each time they appear!
Input :
New York,1,1000.00
New York,2,2000.00
New York,1,2000.00
Chicago,1,4000.00
Chicago,1,1500.00
Approach 1 :
Pig Script :
rental_data = LOAD 'rental_data.csv' USING PigStorage(',') AS (city:chararray, num_bedrooms: long, price:double);
rental_data_grp_city = GROUP rental_data BY (city);
rental_kpi = FOREACH rental_data_grp_city {
one_bed_room = FILTER rental_data BY num_bedrooms==1;
two_bed_room = FILTER rental_data BY num_bedrooms==2;
GENERATE group AS city, AVG(one_bed_room.price) AS one_bed_price, AVG(two_bed_room.price) AS tow_bed_price;
};
Output : DUMP rental_kpi :
(Chicago,2750.0,)
(New York,1500.0,2000.0)
Approach 2 :
Pig Script :
rental_data = LOAD 'rental_data.csv' USING PigStorage(',') AS (city:chararray, num_bedrooms: long, price:double);
rental_data_grp_city = GROUP rental_data BY (city,num_bedrooms);
rental_kpi = FOREACH rental_data_grp_city {
prices_bag = rental_data.price;
GENERATE group.city AS city, group.num_bedrooms AS num_bedrooms, AVG(prices_bag) AS price;
}
Output : DUMP rental_kpi :
(Chicago,1,2750.0)
(New York,2,2000.0)
(New York,1,1500.0)

Get Unique contents from Ruby Hash

I have a Hash #estate:
[#<Estate id: 1, Name: "Thane ", Address: "Thane St.", created_at: "2013-06-21 16:40:50", updated_at: "2013-06-21 16:40:50", user_id: 2, asset_file_name: "DSC02358.JPG", asset_content_type: "image/jpeg", asset_file_size: 5520613, asset_updated_at: "2013-06-21 16:40:49", Mgmt: "abc">,
#<Estate id: 2, Name: "Mumbai", Address: "Mumbai St.", created_at: "2013-06-21 19:13:59", updated_at: "2013-06-21 19:14:28", user_id: 2, asset_file_name: "DSC02359.JPG", asset_content_type: "image/jpeg", asset_file_size: 5085580, asset_updated_at: "2013-06-21 19:13:57", Mgmt: "abc">]
Is it possible to make new Hash with unique values according to the user_id: 2, because currently 2 elements have the user_id same i.e 2, I just want it once in the hash, what should I do ?
It seems to be something like a has_many relation between User model and Estate model, right? If I understood you correctly, than you need in fact to group your Estate by user_id:
PostgreSQL:
Estate.select('DISTINCT ON (user_id) *').all
MySQL:
Estate.group(:user_id).all
P.S. I'd not recommend to select all records from a database and then process them with Ruby as databases handle operations with data in much more efficient way.
Here is an sample example to get you a good start:
h = [ { a: 2, b: 3}, { a: 2, c: 3 } ]
h.uniq { |i| i[:a] }
# => [{:a=>2, :b=>3}]

Resources