Elastic search aggregation key value - elasticsearch

PRODUCT 1
"attributes_list" : {
"src1" : [
{
"aname" : "Manufacturer Standard Lead Time",
"avalue" : "16 Weeks"
},
{
"aname" : "Color",
"avalue" : "red"
},
{
"aname" : "Detailed Description",
"avalue" : "Pre-Biased Bipolar
},
{
"aname" : "DC Current Gain (hFE) (Min) # Ic, Vce",
"avalue" : "100 # 10mA, 5V"
},
{
"aname" : "Transistor Type",
"avalue" : "2 NPN - Pre-Biased (Dual)"
},
{
"aname" : "Mounting Type",
"avalue" : "Surface Mount"
}
]
}
PRODUCT 2-------------
"attributes_list" : {
"src1" : [
{
"aname" : "Lead Time",
"avalue" : "16 Weeks"
},
{
"aname" : "Color",
"avalue" : "green"
},
{
"aname" : "Description",
"avalue" : "Pre-Biased Bipolar
},
{
"aname" : "DC Current Gain (hFE) ",
"avalue" : "100 # 10mA, 5V"
},
{
"aname" : "Transistor",
"avalue" : "2 NPN - Pre-Biased (Dual)"
},
{
"aname" : "Type",
"avalue" : "Surface Mount"
}
]
}
The question is
How can i get all the colors (avalues) by key.
like:
aggs
Colors
Red
doc_count 10
Green
doc_count 5
Simple is
We need to get all the colors by key.
We have tried different aggs queries on avalue but all are getting all the avalues. We need just to get all the avalue:colors [red,green etc ] by aname:color.

Related

Square point of sale web app-"auto_return" does not work

I am setting the return to true, but when i complete the payment on square, the app does not redirect on its own.
{
"amount_money": {
"amount" : "100",
"currency_code" : "USD"
},
"auto_return":true,
"callback_url" : "https://floating-inlet-19449.herokuapp.com/redirect",
"client_id" : "sq0idp-U8x6mJyLFtHuhCfv9sqL5g",
"version": "1.3",
"notes": "notes for the transaction",
"options" : {
"supported_tender_types" : ["CREDIT_CARD"]
}
auto_return should be nested in the options object, like this:
{
"amount_money": { "amount" : 100, "currency_code" : "USD" },
"callback_url" : "https://floating-inlet-19449.herokuapp.com/redirect",
"client_id" : "sq0idp-U8x6mJyLFtHuhCfv9sqL5g",
"version": "1.3",
"notes": "notes for the transaction",
"options" : {
"supported_tender_types" : ["CREDIT_CARD"],
"auto_return": true
}

Google places API not returning address components

My program is trying to determine the City, state, and country based on some text, for example "New york yankee stadium" I want to get New york city, NY, USA. I am using Google places API to do this. According to the documentations, the API should return a list of address component https://developers.google.com/places/web-service/details. However, right now its only returning formatted address "1 E 161st St, Bronx, NY 10451, United States".
here is my web service url
https://maps.googleapis.com/maps/api/place/textsearch/json?key=MY_KEY&query=new%20york%20yankee%20stadium
Anyone familiar with google places API that can let me know if I am not writting the right query or parameter?
{
"html_attributions" : [],
"results" : [
{
"formatted_address" : "1 E 161st St, Bronx, NY 10451, United States",
"geometry" : {
"location" : {
"lat" : 40.82964260000001,
"lng" : -73.9261745
},
"viewport" : {
"northeast" : {
"lat" : 40.83279975,
"lng" : -73.92236575000001
},
"southwest" : {
"lat" : 40.82643674999999,
"lng" : -73.93052034999999
}
}
},
"icon" : "https://maps.gstatic.com/mapfiles/place_api/icons/generic_business-71.png",
"id" : "3d78036d61d35f48650bda737226432b57d82511",
"name" : "Yankee Stadium",
"opening_hours" : {
"open_now" : true,
"weekday_text" : []
},
"photos" : [
{
"height" : 540,
"html_attributions" : [
"\u003ca href=\"https://maps.google.com/maps/contrib/101696810905045719819/photos\"\u003eYankee Stadium\u003c/a\u003e"
],
"photo_reference" : "CoQBdwAAAIxmCLrNS_XZ2FcJqVvRVtBUlNYMBVTVKppOWBu7sICj2q70cqJARBoJlTcZpydbMTzURKWWMVJhYpVCqsnia5pjmDhjvjsTirrEnAc6gvmRYKuUwgewB9Re--FulXzXZ5DY3P9fkwIwuc4U9BJVbqHD5O-N6SbbHcqn4XHUj_OdEhCoNPZ3kiNJhxOCGdYG5O4DGhTqVfUjdq7JzasqYATvQxkL1-H3xg",
"width" : 1242
}
],
"place_id" : "ChIJcWnnWiz0wokRCB6aVdnDQEk",
"rating" : 4.4,
"reference" : "CmRRAAAA5dHiw1YmLxW60_jITBZjMiUs48L4aVUqlPnPDpN_ySa7rw8kPp04WWk0qf8mG-kkMFSNzh39lP0YwfynW54tLcY4s_EYbAPvNWTMe6wXHm_FJiVbI0Lfenyxz4yOTzunEhDgI64EWoXkQe9k45y6qP3-GhSVSdCMPPZA3joFbnYGV-bqo2e0lw",
"types" : [ "stadium", "point_of_interest", "establishment" ]
}
],
"status" : "OK"
}
Its a two step process, first search and get the place_id from google places search service,
use the returned place_id and pass it with another call to receive the individual address components,
https://maps.googleapis.com/maps/api/place/details/json?place_id=ChIJcWnnWiz0wokRCB6aVdnDQEk&key=
{"html_attributions": [],
"result": {
"address_components": [
{
"long_name": "1",
"short_name": "1",
"types": [
"street_number"
]
},
{
"long_name": "East 161st Street",
"short_name": "E 161st St",
"types": [
"route"
]
},
{
"long_name": "Concourse",
"short_name": "Concourse",
"types": [
"neighborhood",
"political"
]
},
{
"long_name": "Bronx",
"short_name": "Bronx",
"types": [
"sublocality_level_1",
"sublocality",
"political"
]
},
{
"long_name": "Bronx County",
"short_name": "Bronx County",
"types": [
"administrative_area_level_2",
"political"
]
},

AWS::EC2::Instance refuses to boot. Mounting the wrong drive as root?

I am attempting to provision a few EC2 instances that need multiple EBS drives. I am attempting to define the root volume and 4 other volumes through BlockDeviceMappings.
Problem:
As far as I can tell, the code below conforms to any example I have seen online. But when Windows boots up, it dies instantly. And looking at the EC2 console (screenshot), I can see that the instance has seven EBS volumes attached (instead of 5) and that /dev/xda is set to root instead of /dev/sda1.
"Mappings" : {
"AWSRegionToAMI" : {
"us-east-1" : { "Windows2012R2" : "ami-5d1b984a" },
"us-west-1" : { "Windows2012R2" : "ami-07713767" },
"us-west-2" : { "Windows2012R2" : "ami-241bd844" }
},
"VolumeSize" : {
"DataDrive" : { "Size" : "50" },
"LogDrive" : { "Size" : "50" },
"TempDrive" : { "Size" : "400" },
"BackupDrive" : { "Size" : "100" }
},
"stackmap" : {
"sqlha" : {
"Name": "MS SQL Server 2014 Always On",
"chefjson" : "https://s3.amazonaws.com/[redacted]",
"os" : "win",
"bootstrapurl" : "https://s3.amazonaws.com/[redacted]"
}
}
},
"WSFCNode1": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId" : { "Fn::FindInMap" : [ "AWSRegionToAMI", { "Ref" : "AWS::Region" }, "Windows2012R2" ] },
"InstanceType": { "Ref": "InstanceType" },
"EbsOptimized": "true",
"NetworkInterfaces": [
{
"DeleteOnTermination": "true",
"DeviceIndex": 0,
"SubnetId": { "Ref": "PrivateSubnet1ID" },
"SecondaryPrivateIpAddressCount": 2,
"GroupSet": [
{ "Ref": "WSFCSecurityGroup" },
{ "Ref": "WSFCClientSecurityGroup" }
]
}
],
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"Ebs" : {"VolumeSize": "60"}
},
{
"DeviceName": "/dev/xvdb",
"Ebs" : {"VolumeSize": { "Fn::FindInMap" : [ "VolumeSize", "DataDrive", "Size" ]} }
},
{
"DeviceName": "/dev/xvdc",
"Ebs" : {"VolumeSize": { "Fn::FindInMap" : [ "VolumeSize", "LogDrive", "Size" ]} }
},
{
"DeviceName": "/dev/xvdd",
"Ebs" : {"VolumeSize": { "Fn::FindInMap" : [ "VolumeSize", "TempDrive", "Size" ]} }
},
{
"DeviceName": "/dev/xvde",
"Ebs" : {"VolumeSize": { "Fn::FindInMap" : [ "VolumeSize", "BackupDrive", "Size" ]} }
}
],
"KeyName": { "Ref": "KeyPairName" },
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
"<powershell>\n",
"# Disable UAC and allow scripts to run\n",
"New-ItemProperty -Path HKLM:Software\\Microsoft\\Windows\\CurrentVersion\\policies\\system -Name EnableLUA -PropertyType DWord -Value 0 -Force\n",
"Set-ExecutionPolicy Unrestricted -force\n",
"c:\\windows\\System32\\WindowsPowershell\\v1.0\\powershell.exe -noninteractive -noprofile Set-ExecutionPolicy unrestricted -force\n",
"c:\\windows\\syswow64\\windowspowershell\\v1.0\\powershell.exe -noninteractive -noprofile Set-ExecutionPolicy unrestricted -force\n",
"#Change TimeZone\n",
"tzutil /s ", {"Ref" : "Timezone"}, "\n",
"#Run Bootstrap PS1\n",
"$newname = '", { "Fn::Join" : ["", [{"Ref" : "Environment"}, {"Ref" : "Location"}, {"Ref" : "Stack"}, {"Ref" : "Role"} ]]},"'\n",
"$region = '", {"Ref" : "VPCRegion"}, "'\n",
"$role = '", {"Ref" : "Role"}, "'\n",
"$chef_rb = '", { "Fn::FindInMap" : [ "stackmap", { "Ref" : "Role" }, "chefjson"]}, "'\n",
"mkdir 'c:\\temp' -force\n",
"(new-object System.Net.WebClient).DownloadFile( 'https://s3.amazonaws.com/[redacted]', 'c:\\temp\\bootstrap.ps1')\n",
"powershell c:\\temp\\bootstrap.ps1 -newname $newname -region $region -role $role -chef_rb $chef_rb -logfile c:\\temp\\bootstrap.log -verbose true\n",
"#Reboot if needed\n",
"Start-Sleep -s 10\n",
"Restart-Computer\n",
"mkdir 'c:\\temp\\cf_reboot_cmd_ran' -force\n",
"shutdown -r\n",
"mkdir 'c:\\temp\\cf_shut_cmd_ran' -force\n",
"Start-Sleep -s 10\n",
"mkdir 'c:\\temp\\cf_ran_again' -force\n",
"</powershell>"
] ] }
},
"Tags": [
{ "Key": "Name", "Value": "SQL Node 1" }
]
}
},
Confusingly, even when I drop all the extra drives and just do a block device mapping of one disk
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"Ebs" : {"VolumeSize": "60"}
}
],
...I still end up with three volumes, and the wrong one (/dev/xda) assigned as root. Screenshot.
Is this a Windows thing? What do my block device mappings need to look like to mount correctly as root (or C:, in this case)?
Nevermind. Root problem was in the AMI I chose. Once I selected a proper Windows AMI, everything worked perfectly.
For anybody else running into this problem, double check your AMI selection.

MongoDB scans entire index when using $all and $elemMatch

I have a collection of user documents, where each user can have an arbitrary set of properties. Each user is associated to an app document. Here is an example user:
{
"appId": "XXXXXXX",
"properties": [
{ "name": "age", "value": 30 },
{ "name": "gender", "value": "female" },
{ "name": "alive", "value": true }
]
}
I would like to be able to find/count users based on the values of their properties. For example, find me all users for app X that have property Y > 10 and Z equals true.
I have a compound, multikey index on this collection db.users.ensureIndex({ "appId": 1, "properties.name": 1, "properties.value": 1}). This index is working well for single condition queries, ex:
db.users.find({
appId: 'XXXXXX',
properties: {
$elemMatch: {
name: 'age',
value: {
$gt: 10
}
}
}
})
The above query completes in < 300ms with a collection of 1M users. However, when I try and add a second condition, the performance degrades considerably (7-8s), and the explain() output indicates that the whole index is being scanned to fulfill the query ("nscanned" : 2752228).
Query
db.users.find({
appId: 'XXXXXX',
properties: {
$all: [
{
$elemMatch: {
name: 'age',
value: {
$gt: 10
}
}
},
{
$elemMatch: {
name: 'alive',
value: true
}
}
]
}
})
Explain
{
"cursor" : "BtreeCursor appId_1_properties.name_1_properties.value_1",
"isMultiKey" : true,
"n" : 256,
"nscannedObjects" : 1000000,
"nscanned" : 2752228,
"nscannedObjectsAllPlans" : 1018802,
"nscannedAllPlans" : 2771030,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 21648,
"nChunkSkips" : 0,
"millis" : 7425,
"indexBounds" : {
"appId" : [
[
"XXXXX",
"XXXXX"
]
],
"properties.name" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"properties.value" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
]
},
"filterSet" : false
}
I assume this is because Mongo is unable to create suitable bounds since I am looking for both boolean and integer values.
My question is this: Is there a better way to structure my data, or modify my query to improve performance and take better advantage of my index? Is it possible to instruct mongo to treat each condition separately, generate appropriate bounds, and then perform the intersection of the results, instead of scanning all documents? Or is mongo just not suited for this type of use case?
I know this is an old question, but I think it would be much better to structure your data without the "name" and "value" tags:
{
"appId": "XXXXXXX",
"properties": [
{ "age": 30 },
{ "gender: "female" },
{ "alive": true }
]
}

Why is my mongo query not using index only?

Please, observe:
MongoDB shell version: 2.4.1
connecting to: test
> use dummy
switched to db dummy
> db.invoices.find({'items.nameTags': /^z/}, {_id: 1}).explain()
{
"cursor" : "BtreeCursor items.nameTags_1_created_1_special_1__id_1_items.qty_1_items.total_1 multi",
"isMultiKey" : true,
"n" : 55849,
"nscannedObjects" : 223568,
"nscanned" : 223568,
"nscannedObjectsAllPlans" : 223568,
"nscannedAllPlans" : 223568,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 86,
"nChunkSkips" : 0,
"millis" : 88864,
"indexBounds" : {
"items.nameTags" : [
[
"z",
"{"
],
[
/^z/,
/^z/
]
],
"created" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"special" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"_id" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"items.qty" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"items.total" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
]
},
"server" : "IL-Mark-LT:27017"
}
>
Here is the definition of the index:
> db.system.indexes.find({name : 'items.nameTags_1_created_1_special_1__id_1_items.qty_1_items.total_1'}).pretty()
{
"v" : 1,
"key" : {
"items.nameTags" : 1,
"created" : 1,
"special" : 1,
"_id" : 1,
"items.qty" : 1,
"items.total" : 1
},
"ns" : "dummy.invoices",
"name" : "items.nameTags_1_created_1_special_1__id_1_items.qty_1_items.total_1"
}
>
Finally, here is an example invoice document (with just 2 items):
> db.invoices.findOne({itemCount: 2})
{
"_id" : "85923",
"customer" : "Wgtd Fm 91",
"businessNo" : "314227928",
"billTo_name" : "Wgtd Fm 91",
"billTo_addressLine1" : "3839 Ross Street",
"billTo_addressLine2" : "Kingston, ON",
"billTo_postalCode" : "K7L 4V4",
"purchaseOrderNo" : "boi",
"terms" : "COD",
"shipDate" : "2013-07-10",
"shipVia" : "Moses Transportation Inc.",
"rep" : "Snowhite",
"items" : [
{
"qty" : 4,
"name" : "CA 7789",
"desc" : "3 pc. Coffee Table set (Silver)",
"price" : 222.3,
"total" : 889.2,
"nameTags" : [
"ca 7789",
"a 7789",
" 7789",
"7789",
"789",
"89",
"9"
],
"descTags" : [
"3",
"pc",
"c",
"coffee",
"offee",
"ffee",
"fee",
"ee",
"e",
"table",
"able",
"ble",
"le",
"e",
"set",
"et",
"t",
"silver",
"ilver",
"lver",
"ver",
"er",
"r"
]
},
{
"qty" : 4,
"name" : "QP 8681",
"desc" : "Ottoman Bed",
"price" : 1179.1,
"total" : 4716.4,
"nameTags" : [
"qp 8681",
"p 8681",
" 8681",
"8681",
"681",
"81",
"1"
],
"descTags" : [
"ottoman",
"ttoman",
"toman",
"oman",
"man",
"an",
"n",
"bed",
"ed",
"d"
]
}
],
"itemCount" : 2,
"discount" : "10%",
"delivery" : 250,
"hstPercents" : 13,
"subTotal" : 5605.6,
"totalBeforeHST" : 5295.04,
"total" : 5983.4,
"totalDiscount" : 560.56,
"hst" : 688.36,
"modified" : "2012-10-08",
"created" : "2014-06-25",
"version" : 0
}
>
My problem is that mongodb does not use index only according to the aforementioned explain() output. Why? After all I only request the _id field, which is part of the index.
In general, I feel that I am doing something very wrong. My invoices collection has 65,000 invoices with the total of 3,291,092 items. It took almost 89 seconds to explain() the query.
What am I doing wrong?
You are using arrays and subdocuments. Covered Indexes dont work with either of these.
From the mongo docs:
An index cannot cover a query if:
any of the indexed fields in any of the documents in the collection includes an array. If an indexed field is an array, the index becomes a multi-key index index and cannot support a covered query.
any of the indexed fields are fields in subdocuments. To index fields in subdocuments, use dot notation. For example, consider a collection users with documents of the following form:
http://docs.mongodb.org/manual/tutorial/create-indexes-to-support-queries/

Resources