Related
I have two index 1.ther 2.part.
"ther" index has 24 fields, ''part" index has 19 fields. I have to enrich "ther" index with "part" index fields.
Field "user_id" is common between two indexes. Using enrich process I tried creating 3 rd index "part_ther"
get ther/_search
{
"_index" : "ther",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"Ahi" : 42.6,
"Index" : 90,
"ipapPressureAverage" : 49,
"MinuteVentAverage" : 20,
"DeviceSerialNumber" : "<>",
"ClearAirwayApneaIndex" : 72.26135463,
"PeriodicBreathingTime" : 9311,
"cpapPressure90Pct" : 22,
"epapPressureAverage" : 73,
"ipapPressure90Pct" : 27,
"Usage" : 93904,
"epapPressure90Pct" : 10,
"AverageBreathRate" : 65,
"#timestamp" : "2021-08-29T00:00:00.000+05:30",
"user_id" : "39,476",
"TrigBreathsPct" : 93,
"cpapPressureAverage" : 29,
"AverageExhaledTidalVolume" : 20,
"DayStorageDate" : "29-08-2021",
"UnintendedLeakageAverage" : 67.58
}
get part/_search
{
"_index" : "part",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"email" : "<>",
"program_id" : 1849,
"program_name" : "<> ",
"scheduled_program_id" : 4765,
"scheduled_program_name" : "<>",
"scheduled_program_status" : 0,
"experience_id" : 10129,
"experience_name" : "<>",
"response_id" : 364482,
"user_id" : 39476,
"firing_time" : "2021-09-19T09:28:51.000-04:00",
"opened_at" : null,
"participation_percentage" : 0,
"status" : "<>",
"created_at" : "2021-09-18T12:26:02.455-04:00",
"updated_at" : "2021-09-19T09:30:04.228-04:00",
"last_frame_completed_id" : null,
"realm" : "affective-part-364482",
"realms_organization_ids" : [
"<>"
]
}
}
Using enrich process I tried creating 3 rd index "ther-part" so that all fields from "ther" index got enriched with "part" index.
Steps-
PUT /_enrich/policy/ther-policy
{
"match": {
"indices": "ther",
"match_field": "user_id",
"enrich_fields": ["Ahi","AverageBreatheRate","AverageExhaledTidalVolume","ClearAirwayApneaIndex","cpapPressureAverage","cpapPressure90Pct","DayStorageDate","epapPressureAverage","epapPressure90Pct","ipapPressureAverage","ipapPressure90Pct","MinuteVentAverage","PeriodicBreathingTime","TrigBreathsPct","UnintendedLeakageAverage","Usage","DeviceSerialNumber"]
}
}
*****
Output-
{
"acknowledged" : true
}
*************
POST /_enrich/policy/ther-policy/_execute
Output-
{
"status" : {
"phase" : "COMPLETE"
}
}
*************************
PUT /_ingest/pipeline/ther_lookup
{
"description" : "Enriching user details with tracks",
"processors" : [
{
"enrich" : {
"policy_name": "ther-policy",
"field" : "user_id",
"target_field": "tmp",
"max_matches": "1"
}
},
{
"script": {
"if": "ctx.tmp != null",
"source": "ctx.putAll(ctx.tmp); ctx.remove('tmp');"
}
}
]
}
Output-
{
"acknowledged" : true
}
**************************
POST _reindex
{
"source": {
"index": "part"
},
"dest": {
"index": "part_ther",
"pipeline": "ther_lookup"
}
}
Fields are getting enriched in part_ther index. Index "part" and "ther" has only 2 docs as I am testing the enrichment processor.
get part_ther/_search
{
"_index": "part_ther",
"_type": "_doc",
"_id": "1",
"_score": 1,
"_source": {
"Ahi": 42.6,
"response_id": 364482,
"program_id": 1849,
"program_name": "<> ",
"created_at": "2021-09-18T12:26:02.455-04:00",
"ipapPressureAverage": 49,
"MinuteVentAverage": 20,
"DeviceSerialNumber": "<>",
"scheduled_program_id": 4765,
"experience_id": 10129,
"ClearAirwayApneaIndex": 72.26135463,
"PeriodicBreathingTime": 9311,
"updated_at": "2021-09-19T09:30:04.228-04:00",
"scheduled_program_status": 0,
"cpapPressure90Pct": 22,
"email": "<>",
"epapPressureAverage": 73,
"ipapPressure90Pct": 27,
"last_frame_completed_id": null,
"Usage": 93904,
"participation_percentage": 0,
"epapPressure90Pct": 10,
"firing_time": "2021-09-19T09:28:51.000-04:00",
"opened_at": null,
"realms_organization_ids": [
"<>"
],
"user_id": "39476",
"scheduled_program_name": "<>",
"TrigBreathsPct": 93,
"cpapPressureAverage": 29,
"experience_name": "<>",
"AverageExhaledTidalVolume": 20,
"DayStorageDate": "29-08-2021",
"realm": "<>",
"UnintendedLeakageAverage": 67.58,
"status": "<>"
}
In Original Index "therapy" index has 9k docs and "participation index" has 80k docs.
When I am trying to create third index on original indices, I am getting errors.
Error 502, {"ok":false,"message":"backend closed connection"}
How can I avoid this error and enrich indices successfully.
POST _reindex
{
"source": {
"index": "-participation"
},
"dest": {
"index": "data-participation-therapy1",
"pipeline": "therapy_lookup"
}
}
output-
{"ok":false,"message":"backend closed connection"}
Thanks, #Val for clarifying this error.
backend closed connection simply means that the client (i.e. Kibana Dev Tools in your browser) timed out. But the reindex process is still ongoing in the background.
If your source index is big, odds are high that it will take longer than the timeout for the operation to terminate. So you should start your reindex to run asynchronously using
POST _reindex?wait_for_completion=false
The call will return immediately and give you a task ID which you can use to check the task status as it progresses using
GET _tasks/<task_id>
Ref link-Enrich fields in new index using enrich processor
MIT APP INVENTOR 2 I have this problem: I have a dictionary with many sub-classes. But I have no idea how I could access to specific data inside the dictionary. For example a have the next list:
{
"keystatus" : "valid",
"status" : 0,
"result" : [
{
"stores" : {
"0" : {
"category" : "",
"min_price" : "64.90",
"manufacturer" : "adidas",
"url" : "https:\/\/www.rakuten.de\/%20produkt%20\/adidas-leonero-herrenschuhe-sneaker-2040331525.html?portaldv=6",
"title" : "adidas Leonero Herrenschuhe Sneaker",
"advertiser" : "Rakuten Deutschland GmbH",
"currency" : "€",
"max_price" : ""
}
},
"details" : {
"barcode_formats" : "EAN 4058025901204",
"color" : "",
"publisher" : "",
"weight" : "",
"actor" : "",
"brand" : "adidas",
"product_name" : "adidas Leonero Herrenschuhe Sneaker",
"category" : "",
"prod_details" : "",
"manufacturer" : "adidas",
"length" : "",
"label" : "",
"clothing_size" : "",
"artist" : "",
"height" : "",
"model" : "",
"width" : "",
"director" : "",
"genre" : "",
"manufacturer_part_number" : "Adidas Leonero",
"long_description" : "Nach 70 Jahren als einer der weltbesten Hersteller von FuBballschuhen hat adidas jede Menge Erfahrung gesammelt, wenn es um cleanen Sportstyle geht. Wir haben uns von einigen der legendärsten Designs aus unseren Archiven inspirieren lassen und m",
"features" : "",
"author" : ""
},
"reviews" : {
},
"type" : "EAN",
"images" : {
"0" : "https:\/\/images.barcodelookup.com\/9152\/91526482-1.jpg"
},
"barcode" : "4058025901204"
}
]
}
I want to access to this part of the dictionary ["result"][0]["stores"]["0"]["title"] . Any thoughts?
All ideas are welcome. Thanks
You will have to learn how to work with lists and list of lists...
See these examples:
How to work with Lists by Saj
How to work with Lists and Lists of lists (pdf) by appinventor.org
JSON and list of lists: example1 and example2
Please, observe:
MongoDB shell version: 2.4.1
connecting to: test
> use dummy
switched to db dummy
> db.invoices.find({'items.nameTags': /^z/}, {_id: 1}).explain()
{
"cursor" : "BtreeCursor items.nameTags_1_created_1_special_1__id_1_items.qty_1_items.total_1 multi",
"isMultiKey" : true,
"n" : 55849,
"nscannedObjects" : 223568,
"nscanned" : 223568,
"nscannedObjectsAllPlans" : 223568,
"nscannedAllPlans" : 223568,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 86,
"nChunkSkips" : 0,
"millis" : 88864,
"indexBounds" : {
"items.nameTags" : [
[
"z",
"{"
],
[
/^z/,
/^z/
]
],
"created" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"special" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"_id" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"items.qty" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"items.total" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
]
},
"server" : "IL-Mark-LT:27017"
}
>
Here is the definition of the index:
> db.system.indexes.find({name : 'items.nameTags_1_created_1_special_1__id_1_items.qty_1_items.total_1'}).pretty()
{
"v" : 1,
"key" : {
"items.nameTags" : 1,
"created" : 1,
"special" : 1,
"_id" : 1,
"items.qty" : 1,
"items.total" : 1
},
"ns" : "dummy.invoices",
"name" : "items.nameTags_1_created_1_special_1__id_1_items.qty_1_items.total_1"
}
>
Finally, here is an example invoice document (with just 2 items):
> db.invoices.findOne({itemCount: 2})
{
"_id" : "85923",
"customer" : "Wgtd Fm 91",
"businessNo" : "314227928",
"billTo_name" : "Wgtd Fm 91",
"billTo_addressLine1" : "3839 Ross Street",
"billTo_addressLine2" : "Kingston, ON",
"billTo_postalCode" : "K7L 4V4",
"purchaseOrderNo" : "boi",
"terms" : "COD",
"shipDate" : "2013-07-10",
"shipVia" : "Moses Transportation Inc.",
"rep" : "Snowhite",
"items" : [
{
"qty" : 4,
"name" : "CA 7789",
"desc" : "3 pc. Coffee Table set (Silver)",
"price" : 222.3,
"total" : 889.2,
"nameTags" : [
"ca 7789",
"a 7789",
" 7789",
"7789",
"789",
"89",
"9"
],
"descTags" : [
"3",
"pc",
"c",
"coffee",
"offee",
"ffee",
"fee",
"ee",
"e",
"table",
"able",
"ble",
"le",
"e",
"set",
"et",
"t",
"silver",
"ilver",
"lver",
"ver",
"er",
"r"
]
},
{
"qty" : 4,
"name" : "QP 8681",
"desc" : "Ottoman Bed",
"price" : 1179.1,
"total" : 4716.4,
"nameTags" : [
"qp 8681",
"p 8681",
" 8681",
"8681",
"681",
"81",
"1"
],
"descTags" : [
"ottoman",
"ttoman",
"toman",
"oman",
"man",
"an",
"n",
"bed",
"ed",
"d"
]
}
],
"itemCount" : 2,
"discount" : "10%",
"delivery" : 250,
"hstPercents" : 13,
"subTotal" : 5605.6,
"totalBeforeHST" : 5295.04,
"total" : 5983.4,
"totalDiscount" : 560.56,
"hst" : 688.36,
"modified" : "2012-10-08",
"created" : "2014-06-25",
"version" : 0
}
>
My problem is that mongodb does not use index only according to the aforementioned explain() output. Why? After all I only request the _id field, which is part of the index.
In general, I feel that I am doing something very wrong. My invoices collection has 65,000 invoices with the total of 3,291,092 items. It took almost 89 seconds to explain() the query.
What am I doing wrong?
You are using arrays and subdocuments. Covered Indexes dont work with either of these.
From the mongo docs:
An index cannot cover a query if:
any of the indexed fields in any of the documents in the collection includes an array. If an indexed field is an array, the index becomes a multi-key index index and cannot support a covered query.
any of the indexed fields are fields in subdocuments. To index fields in subdocuments, use dot notation. For example, consider a collection users with documents of the following form:
http://docs.mongodb.org/manual/tutorial/create-indexes-to-support-queries/
I'm working on retrieving all the orders for a given shop, using code that looks like this:
orders = ShopifyAPI::Order.find(:all, :params => {:financial_status => 'paid'})
orders.each do |order|
order_json = order.to_json
post_json_to_server(order_json)
end
But for some reason, when I inspect the JSON created by order.to_json, the discount_codes and client_details attributes look like this:
"client_details":{"":{"accept_language":"en-US,en;q=0.8","browser_ip":"199.185.98.174","session_hash":"6b37d22ebcdf097f5ab4e1c9e596a504c4cdc4c41c4f2b29a3a7aae4ead559c3","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11"}}
"discount_codes":[{"":{"amount":"20.00","code":"CZV57KSE8VMV"}}]
Why is there a leading {"": in both of these lists? Is something misconfigured in my test store?
Here is a complete dump (captured with RequestBin) of the JSON I'm sending:
{ "billing_address" : { "address1" : "asdf",
"address2" : "",
"city" : "asdf",
"company" : "",
"country" : "United States",
"country_code" : "US",
"first_name" : "asdf",
"last_name" : "asdf",
"latitude" : "45.176384",
"longitude" : "-123.045601",
"name" : "asdf asdf",
"phone" : "",
"province" : "Alaska",
"province_code" : "AK",
"zip" : "asdf"
},
"browser_ip" : "199.185.98.174",
"buyer_accepts_marketing" : true,
"cancel_reason" : null,
"cancelled_at" : null,
"cart_token" : "bbf42c99f456f9ccda30554022fec659",
"client_details" : { "" : { "accept_language" : "en-US,en;q=0.8",
"browser_ip" : "199.185.98.174",
"session_hash" : "6b37d22ebcdf097f5ab4e1c9e596a504c4cdc4c41c4f2b29a3a7aae4ead559c3",
"user_agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11"
} },
"closed_at" : null,
"created_at" : "2012-07-10T17:23:53-04:00",
"currency" : "CAD",
"customer" : { "accepts_marketing" : true,
"created_at" : "2012-07-10T17:23:53-04:00",
"email" : "asdf#example.com",
"first_name" : "asdf",
"id" : 94366870,
"last_name" : "asdf",
"last_order_id" : null,
"last_order_name" : null,
"note" : null,
"orders_count" : 0,
"state" : "disabled",
"tags" : "",
"total_spent" : "0.00",
"updated_at" : "2012-07-10T17:35:23-04:00"
},
"discount_codes" : [ { "" : { "amount" : "20.00",
"code" : "CZV57KSE8VMV"
} } ],
"email" : "asdf#example.com",
"financial_status" : "authorized",
"fulfillment_status" : null,
"fulfillments" : [ ],
"gateway" : "bogus",
"id" : 134753494,
"landing_site" : "/",
"landing_site_ref" : null,
"line_items" : [ { "fulfillment_service" : "manual",
"fulfillment_status" : null,
"grams" : 0,
"id" : 219421970,
"name" : "Grass-roots methodical instruction set",
"price" : "19.00",
"product_id" : 95843140,
"quantity" : 2,
"requires_shipping" : true,
"sku" : "",
"title" : "Grass-roots methodical instruction set",
"variant_id" : 224399478,
"variant_inventory_management" : null,
"variant_title" : null,
"vendor" : "Shopify"
} ],
"name" : "#1004",
"note" : "",
"note_attributes" : [ ],
"number" : 4,
"order_number" : 1004,
"payment_details" : { "avs_result_code" : null,
"credit_card_bin" : "1",
"credit_card_company" : "Bogus",
"credit_card_number" : "XXXX-XXXX-XXXX-1",
"cvv_result_code" : null
},
"processing_method" : "direct",
"referring_site" : "",
"shipping_address" : { "address1" : "asdf",
"address2" : "",
"city" : "asdf",
"company" : "",
"country" : "United States",
"country_code" : "US",
"first_name" : "asdf",
"last_name" : "asdf",
"latitude" : "45.176384",
"longitude" : "-123.045601",
"name" : "asdf asdf",
"phone" : "",
"province" : "Alaska",
"province_code" : "AK",
"zip" : "asdf"
},
"shipping_lines" : [ { "code" : "International Shipping",
"price" : "20.00",
"source" : "shopify",
"title" : "International Shipping"
} ],
"subtotal_price" : "18.00",
"tax_lines" : [ ],
"taxes_included" : false,
"token" : "51116b93d2d774b6c537a3bcc8861506",
"total_discounts" : "20.00",
"total_line_items_price" : "38.00",
"total_price" : "38.00",
"total_price_usd" : "37.28",
"total_tax" : "0.00",
"total_weight" : 0,
"updated_at" : "2012-07-10T17:35:20-04:00"
}
In an attempt to get the Order formatted the same way as the JSON sent by a webhook, I ended up doing it myself - here is the code:
ActiveResource::Base.include_root_in_json = true
orders = ShopifyAPI::Order.find(:all, :params => {:financial_status => 'paid'})
orders.each do |order|
order_json = order.as_json
%w(billing_address customer line_items payment_details shipping_address shipping_lines).each do |attribute|
order_json[attribute] = order.send(attribute).as_json
end
if order_json['discount_codes'].length > 0
order_json['discount_codes'] = [order_json['discount_codes'].as_json[0][nil]]
end
order_json['client_details'] = order_json['client_details'].as_json[nil]
post_json_to_server(order_json)
end
By doing it this way, I was able to turn the JSON into something that mapped exactly to what Shopify's order paid webhook sends.
When you use to_json without telling ActiveRecord how to treat the root it can do that.
You can tell ActiveRecord to include the root when rendering JSON.
ActiveRecord::Base.include_root_in_json = true
You would then see
{"order":{...}} and not {"":{...}}
Often you can just use the syntax
order_json = order.to_json(:root => true)
in order to get the key (which is the root) you want. Using JSON is still kinda like walking inside a carnival jumpy ride for kids...
I have a MongoDB query that's taking an unreasonably long time to run, but it:
is only scanning 6 objects
hits an index
consistently takes ~1500ms (wasn't paging or otherwise occupied)
index miss% is 0 in mongostat
It showed up in the profiler (without the explain()), and I don't understand why it's so slow. Any ideas?
gimmebar:PRIMARY> db.assets.find({ owner: "123", avatar: false, private: false }).sort({date: -1}).explain()
{
"cursor" : "BtreeCursor owner_1_avatar_1_date_-1",
"nscanned" : 6,
"nscannedObjects" : 6,
"n" : 6,
"millis" : 1567,
"nYields" : 0,
"nChunkSkips" : 0,
"isMultiKey" : false,
"indexOnly" : false,
"indexBounds" : {
"owner" : [
[
"123",
"123"
]
],
"avatar" : [
[
false,
false
]
],
"date" : [
[
{
"$maxElement" : 1
},
{
"$minElement" : 1
}
]
]
}
}
Missing the index on the private key?
BtreeCursor owner_1_avatar_1_date_-1 vs .find({ owner: "123", avatar: false, private: false }).sort({date: -1})