I have to group the data by model number, get the sum of the grouped items, and do a custom sort. I'm having problems returning the value of "type", which is the sort order.
I've tried a number of different things, this is where I'm at right now.
type = x.type is not correct.
Sample Request Data:
[
{
"type": "ANTENNA MAKE - MODEL",
"sector": "F",
"position": "7",
"qty": "3",
"model": "NNH4-65D-R6"
},
{
"type": "ANTENNA MAKE - MODEL",
"sector": "A",
"position": "1",
"qty": "3",
"model": "NNH4-65D-R6"
},
{
"type": "RRH - WCS band",
"sector": "B",
"position": "1",
"qty": "1",
"model": "B2/B66A 8843"
},
{
"type": "RRH - WCS band",
"sector": "B",
"position": "1",
"qty": "3",
"model": "B2/B66A 8843"
}
]
Query
var ModelList = current.Where(x => !string.IsNullOrWhiteSpace(x.model) && !string.IsNullOrWhiteSpace(x.type))
.OrderBy(x => x.type == "ANTENNA MAKE - MODEL" ? 1 :
x.type == "Antenna RET Motor" ? 2 :
x.type == "SURGE ARRESTOR" ? 3 :
x.type == "DIPLEXER" ? 4 :
x.type == "DUPLEXER" ? 5 :
x.type == "Antenna RET CONTROL UNIT" ? 6 :
x.type == "TMA/LNA" ? 7 :
x.type == "CURRENT INJECTORS FOR TMA" ? 8 :
x.type == "PDU FOR TMAS" ? 9 :
x.type == "FILTER" ? 10 :
x.type == "SQUID" ? 11 :
x.type == "RRH - 700 band" ? 12 :
x.type == "RRH - 850 band" ? 13 :
x.type == "RRH - 1900 band" ? 14 :
x.type == "RRH - AWS band" ? 15 :
x.type == "RRH - WCS band" ? 16 :
x.type == "Additional RRH #1 - any band" ? 17 :
x.type == "Additional RRH #2 - any band" ? 18 :
x.type == "Additional Component 1" ? 19 :
x.type == "Additional Component 2" ? 20 :
x.type == "Additional Component 3" ? 21 :
x.type == "DC TRUNK" ? 22 :
x.type == "DC BLOCK" ? 23 : 23)
.GroupBy(x => x.model)
.Select(x => new {
model = x.Key,
type = x.type,
qty = x.Sum(y => y.qty),
elev = x.Average(z => z.elev)
}).ToList();
In this example I expect to receive the following:
[
{
"model": "NNH4-65D-R6",
"type": "ANTENNA MAKE - MODEL",
"qty": 6,
"elev": null
}
{
"model": "B2/B66A 8843",
"type": "RRH - WCS band",
"qty": 4,
"elev": null
}
]
Adjust your select like this:
.Select(x => new {
model = x.Key,
type = x.Select(y => y.type).First(),
qty = x.Sum(y => y.qty),
elev = x.Average(z => z.elev)
}).ToList();
Related
PRODUCT 1
"attributes_list" : {
"src1" : [
{
"aname" : "Manufacturer Standard Lead Time",
"avalue" : "16 Weeks"
},
{
"aname" : "Color",
"avalue" : "red"
},
{
"aname" : "Detailed Description",
"avalue" : "Pre-Biased Bipolar
},
{
"aname" : "DC Current Gain (hFE) (Min) # Ic, Vce",
"avalue" : "100 # 10mA, 5V"
},
{
"aname" : "Transistor Type",
"avalue" : "2 NPN - Pre-Biased (Dual)"
},
{
"aname" : "Mounting Type",
"avalue" : "Surface Mount"
}
]
}
PRODUCT 2-------------
"attributes_list" : {
"src1" : [
{
"aname" : "Lead Time",
"avalue" : "16 Weeks"
},
{
"aname" : "Color",
"avalue" : "green"
},
{
"aname" : "Description",
"avalue" : "Pre-Biased Bipolar
},
{
"aname" : "DC Current Gain (hFE) ",
"avalue" : "100 # 10mA, 5V"
},
{
"aname" : "Transistor",
"avalue" : "2 NPN - Pre-Biased (Dual)"
},
{
"aname" : "Type",
"avalue" : "Surface Mount"
}
]
}
The question is
How can i get all the colors (avalues) by key.
like:
aggs
Colors
Red
doc_count 10
Green
doc_count 5
Simple is
We need to get all the colors by key.
We have tried different aggs queries on avalue but all are getting all the avalues. We need just to get all the avalue:colors [red,green etc ] by aname:color.
I have a query just like this:
{
"script": {
"script": {
"lang": "painless",
"params": {
"maxPrice": 1000000,
"minPrice": 50,
"now": "{{now}}"
},
"source": "(doc['selling_price'].value == 0 && doc['normal_price'].value >= params.minPrice && doc['normal_price'].value <= params.maxPrice) || (doc['selling_price'].value > 0 && doc['selling_price'].value >= params.minPrice && doc['selling_price'].value <= params.maxPrice && params.now <= doc['selling_price_end_time'].value.getMillis() && params.now >= doc['selling_price_start_time'].value.getMillis())"
}
}
}
When I trying to put minPrice and maxPrice into parameters in a template just like now like this :
"params": {
"maxPrice": "{{maxPrice}}",
"minPrice": "{{minPrice}}",
"now": "{{now}}"
},
Then when I run :
{
"id": "my_template",
"params": {
"query_all": "My goi",
"maxPrice": 100000000,
"minPrice": 50,
"now": 1421427600000,
I received this:
"failures": [
{
"shard": 4,
"index": "dev_index_sku_list",
"node": "jSpDqKHwSMeWsUxxoD9-5w",
"reason": {
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"(doc['selling_price'].value == 0 && doc['normal_price'].value >= params.minPrice && doc['normal_price'].value <= params.maxPrice) || (doc['selling_price'].value > 0 && doc['selling_price'].value >= params.minPrice && doc['selling_price'].value <= params.maxPrice &&
Ridiculously, when I set up now as a parameter it works, but not for the case for minPrice and maxPrice:
The logic behind the source is:
Our data have:
NormalPrice: 200 #this is the normal price
SellingPrice: 100 #this is actually the promotion price
selling_price_start_time: "2020-01-01" #duration of promotion price
selling_price_end_time: "2020-01-30"
The logic:
if selling_price != 0 and now in the promotion time then filter within minPrice and maxPrice
else then filter by normalPrice within minPrice and maxPrice
Currently working on version 7 of elasticSearch
I am using logstash with input jdbc, and would like to embed one object inside another with aggregate.
How can I use add recursive?
Ie add an object inside another object?
This would be an example:
{
"_index": "my-index",
"_type": "test",
"_id": "1",
"_version": 1,
"_score": 1,
"_source": {
"id": "1",
"properties": {
"nested_1": [
{
"A": 0,
"B": "true",
"C": "PEREZ, MATIAS ROGELIO Y/O",
"Nested_2": [
{
"Z1": "true",
"Z2": "99999"
}
},
{
"A": 0,
"B": "true",
"C": "SALVADOR MATIAS ROMERO",
"Nested_2": [
{
"Z1": "true",
"Z2": "99999"
}
}
]
}
}
}
I'm using something like that but it doesn't work
aggregate {
task_id => "%{id}"
code => "
map['id'] = event.get('id')
map['nested_1_list'] ||= []
map['nested_1'] ||= []
if (event.get('id') != nil)
if !( map['nested_1_list'].include?event.get('id') )
map['nested_1_list'] << event.get('id')
map['nested_1'] << {
'A' => event.get('a'),
'B' => event.get('b'),
'C' => event.get('c'),
map['nested_2_list'] ||= []
map['nested_2'] ||= []
if (event.get('id_2') != nil)
if !( map['nested_2_list'].include?event.get('id_2') )
map['nested_2_list'] << event.get('id_2')
map['nested_2'] << {
'Z1' => event.get('z1'),
'Z2' => event.get('z2')
}
end
end
}
end
end
event.cancel()
"
push_previous_map_as_event => true
timeout => 3
}
Any idea how to implement this?........................
..........
Finally what I did was, generate the JSON from the input, that is, from a stored procedure that is consumed from a view (vw) from the input statement of logstash.
Once consumed, I process it as json and I already have that json to work as one more variable.
# Convierto el string a json real (quita comillas y barras invertidas)
ruby {
code => "
require 'json'
json_value = JSON.parse(event.get('field_db').to_s)
event.set('field_convert_to_json',json_value)
"
}
Maybe you can try this. Note This will be applicable only when you want to have a single object and not an array of object.
Please do visit my blog for other formats.
https://xyzcoder.github.io/2020/07/29/indexing-documents-using-logstash-and-python.html
input {
jdbc {
jdbc_driver_library => "/usr/share/logstash/javalib/mssql-jdbc-8.2.2.jre11.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://host.docker.internal;database=StackOverflow2010;user=pavan;password=pavankumar#123"
jdbc_user => "pavan"
jdbc_password => "pavankumar#123"
statement => "select top 500 p.Id as PostId,p.AcceptedAnswerId,p.AnswerCount,p.Body,u.Id as userid,u.DisplayName,u.Location
from StackOverflow2010.dbo.Posts p inner join StackOverflow2010.dbo.Users u
on p.OwnerUserId=u.Id"
}
}
filter {
aggregate {
task_id => "%{postid}"
code => "
map['postid'] = event.get('postid')
map['accepted_answer_id'] = event.get('acceptedanswerid')
map['answer_count'] = event.get('answercount')
map['body'] = event.get('body')
map['user'] = {
'id' => event.get('userid'),
'displayname' => event.get('displayname'),
'location' => event.get('location')
}
map['user']['test'] = {
'test_body' => event.get('postid')
}
event.cancel()"
push_previous_map_as_event => true
timeout => 30
}
}
output {
elasticsearch {
hosts => ["http://elasticsearch:9200", "http://elasticsearch:9200"]
index => "stackoverflow_top"
}
stdout {
codec => rubydebug
}
}
and my output is
{
"_index" : "stackoverflow_top",
"_type" : "_doc",
"_id" : "S8WEmnMBrXsRTNbKO0JJ",
"_score" : 1.0,
"_source" : {
"#version" : "1",
"body" : """<p>How do I store binary data in MySQL?</p>
""",
"#timestamp" : "2020-07-29T12:20:22.649Z",
"answer_count" : 10,
"user" : {
"displayname" : "Geoff Dalgas",
"location" : "Corvallis, OR",
"test" : {
"test_body" : 17
},
"id" : 2
},
"postid" : 17,
"accepted_answer_id" : 26
}
Here test object is nested into the user object
I'm a newer and studying MongoDB with Laravel.
This is my problem.
I need to get records in 60 days and group them by type.
In each type group, I have to group them by range time:
60 - 30 days ago
30 days ago - now
I researched in 2 days and these are my codes so far.
$collection->aggregate([
[
'$match' => $match,
],
[
'$project' => [
'range' => [
'$concat' => [
[
'$cond' => [
[
'$and' => [
['$gte' => ['$created_at', $ago_60_days]],
['$lte' => ['$created_at', $ago_30_days]],
],
],
'before',
'',
],
],
[
'$cond' => [
[
'$and' => [
['$gt' => ['$created_at', $ago_30_days]],
['$lte' => ['$created_at', $now]],
],
],
'current',
'',
],
],
],
],
],
],
[
'$group' => [
'_id' => '$type,
'total' => ['$sum' => 1],
],
],
... anything after ???
]);
Please let me know anything, any clues could help me.
I can handle with mongo shell.
Thank you guys so much.
The query:
db.test.aggregate( [
{
$match: {
$expr: {
$gte: [ "$dt", { $subtract: [ ISODate(), 60*24*60*60*1000 ] } ]
}
}
},
{
$addFields: {
rangeTime: {
$cond: {
if: { $gte: [ "$dt", { $subtract: [ ISODate(), 30*24*60*60*1000 ] } ] },
then: "range_30_now",
else: "range_60_30"
}
}
}
},
{
$group: {
_id: {
t: "$type", r: "$rangeTime"
},
total: { $sum: 1 }
}
},
{
$project: {
type: "$_id.t",
rangeTime: "$_id.r",
total: 1,
_id: 0
}
},
{
$sort: {
type: 1,
rangeTime: -1
}
}
] )
The Output:
{ "total" : 2, "type" : "A", "rangeTime" : "range_60_30" }
{ "total" : 1, "type" : "B", "rangeTime" : "range_60_30" }
{ "total" : 2, "type" : "B", "rangeTime" : "range_30_now" }
The Input Documents:
{ "_id" : 1, "dt" : ISODate("2019-10-15T00:00:00Z"), "type" : "A" }
{ "_id" : 2, "dt" : ISODate("2019-10-20T00:00:00Z"), "type" : "B" }
{ "_id" : 3, "dt" : ISODate("2019-11-08T00:00:00Z"), "type" : "A" }
{ "_id" : 4, "dt" : ISODate("2019-11-29T00:00:00Z"), "type" : "B" }
{ "_id" : 5, "dt" : ISODate("2019-11-15T00:00:00Z"), "type" : "A" }
{ "_id" : 9, "dt" : ISODate("2019-12-15T00:00:00Z"), "type" : "B" }
I have the following data
[{"devcount" : 1 , "dayofweek" :0, "hour" : 1 },
{"devcount" : 2 , "dayofweek" :0, "hour" : 2 },
{"devcount" : 3 , "dayofweek" :1, "hour" : 2 },
{"devcount" : 4 , "dayofweek" :1, "hour" : 3 },
{"devcount" : 6 , "dayofweek" :1, "hour" : 4 },
{"devcount" : 5 , "dayofweek" :1, "hour" : 5 },
{"devcount" : 7 , "dayofweek" :2, "hour" : 5 },
{"devcount" : 8 , "dayofweek" :2, "hour" : 6 },
{"devcount" : 9 , "dayofweek" :2, "hour" : 7 },
{"devcount" : 10 , "dayofweek" :2, "hour" : 9 }]
It is required to compare the devcount with the group average of devcount for each dayofweek.
i.e. for the fist row, devcount=1 is to be compared with the the average device count for the dayofweek-0 (= 1.5) and "yes" to be returned if the devcount is lesser. Else "No" should be returned.
I have coded as below.
smry=d3.nest()
.key( function(d) { return d.dayofweek;})
.rollup(function(d) {return d3.mean(d, function(g) {return g.devcount; })})
.entries(result);
I am not sure how to compare the smry data and the original data.
The original data will be used in selectAll for creating rectangles and the output after comparison needs for determining the colour of the rectangle
You can do it as shown in the snippet below.
test = [{
"devcount": 1,
"dayofweek": 0,
"hour": 1
}, {
"devcount": 2,
"dayofweek": 0,
"hour": 2
},
{
"devcount": 3,
"dayofweek": 1,
"hour": 2
}, {
"devcount": 4,
"dayofweek": 1,
"hour": 3
}, {
"devcount": 6,
"dayofweek": 1,
"hour": 4
}, {
"devcount": 5,
"dayofweek": 1,
"hour": 5
},
{
"devcount": 7,
"dayofweek": 2,
"hour": 5
}, {
"devcount": 8,
"dayofweek": 2,
"hour": 6
}, {
"devcount": 9,
"dayofweek": 2,
"hour": 7
}, {
"devcount": 10,
"dayofweek": 2,
"hour": 9
}
];
//make the summary using nest
smry = d3.nest()
.key(function(d) {
return d.dayofweek;
})
.rollup(function(d) {
return d3.mean(d, function(g) {
return g.devcount;
})
})
.entries(test);
test.forEach(function(t) {
//find the value from summary for dayofweek
var k = smry.find(function(s) {
return (s.key == t.dayofweek)
});
//check the day of week with the mean, set the flag in the data
if(k.values<t.devcount){
t.flag = true;
} else {
t.flag = false;
}
});
console.log(test);//this now has the flag to determine the color
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.4.11/d3.min.js"></script>