I'm using the sharding support of spring-data-mongodb by adding the #Sharded annotation on a document:
#Sharded(immutableKey = true, shardKey = { "location.locationId", "ts" })
However, when spring-data is generating the request, the added filter contains:
filterWithShardKey = {Document#20158} size = 3
"_id" -> {ObjectId#20139} "60489730a9687c7c83a287df"
"location.locationId" -> null
"ts" -> {Date#20141} "Mon Feb 17 12:13:06 CET 2020"
I looked over the QueryOperations class and its applyShardKey method:
getMappedShardKeyFields(domainType).forEach(key -> filterWithShardKey.putIfAbsent(key, shardKeySource.get(key)));
shardKeySource is a bson document and presents the nested documents also as bson documents (not flat). Hence, when getting the "location.locationId" field, it is not found:
documentAsMap = {LinkedHashMap#20083} size = 9
"_id" -> {ObjectId#20099} "60489996721f557bb2c8e5d6"
"ts" -> {Date#20101} "Mon Feb 17 12:13:06 CET 2020"
"location" -> {Document#20103} size = 3
key = "location"
value = {Document#20103} size = 3
"source" -> {Document#20105} size = 7
"device" -> {Document#20107} size = 2
"timezoneRawOffset" -> {Double#20109} 1.0
Is there a way to declare a nested sharding key with spring-data-mongodb?
Thanks for your answers!
Related
Reproduce
With Xcode 14.1, following steps can reproduce the phenomenon:
add a pdf file to the assets, and in attributes inspector, select "Preserve Vector Data", set "Scales" as "Single Scale".
run Xcode->Product->Archive
follow the instruction of create the app size report from apple official site. In the archives list, select "Distribute App" -> Ad Hoc ->App Thinning: All compatible device variants -> Next -> Automatically menage signing -> Next -> Export.
Now we check the exported folder, in the sub folder named "Apps", we can get 3 ipas.
-rw-r--r--# 1 xxxx staff 386048 Nov 10 21:01 111002-582C7144-401C-4F99-8FA7-3B164292726E.ipa
-rw-r--r--# 1 xxxx staff 639517 Nov 10 21:01 111002-5A4FE73D-2DCB-4079-BB68-59FA54D31921.ipa
-rw-r--r-- 1 xxxx staff 1190004 Nov 10 21:01 111002.ipa
choose one specific ipa (NOT the universal one 111002.ipa), we can choose one base on file "App Thinning Size Report.txt", unzip that ipa, enter the app folder, now we can find the file which name is "Assets.car".
we run the script "xcrun --sdk iphoneos assetutil --info Assets.car", then we get a output message, part of it like this:
{
"AssetType" : "Image",
"BitsPerComponent" : 8,
"ColorModel" : "RGB",
"Colorspace" : "srgb",
"Compression" : "deepmap-lzfse",
"Encoding" : "ARGB",
"Idiom" : "universal",
"Name" : "pdf",
"NameIdentifier" : 19593,
"Opaque" : false,
"PixelHeight" : 842,
"PixelWidth" : 595,
"Preserved Vector Representation" : true,
"RenditionName" : "pdf.pdf",
"Scale" : 1,
"SHA1Digest" : "5C71A688035EDA13684D904E6F272BC9B0F7ABF8D5AFE1C44915FDC99AB5DC4E",
"SizeOnDisk" : 71134,
"State" : "Normal",
"Template Mode" : "automatic",
"Value" : "Off"
},
{
"AssetType" : "Image",
"BitsPerComponent" : 8,
"ColorModel" : "RGB",
"Colorspace" : "srgb",
"Compression" : "deepmap-lzfse",
"Encoding" : "ARGB",
"Idiom" : "universal",
"Name" : "pdf",
"NameIdentifier" : 19593,
"Opaque" : false,
"PixelHeight" : 2526,
"PixelWidth" : 1785,
"Preserved Vector Representation" : true,
"RenditionName" : "pdf.pdf",
"Scale" : 3,
"SHA1Digest" : "BB0E06CEB24DBF6A89D38BAFCBAE5D6322CED922FDD30E4A5E4A22C9079B7390",
"SizeOnDisk" : 369765,
"State" : "Normal",
"Template Mode" : "automatic",
"Value" : "Off"
},
{
"AssetType" : "Vector",
"Colorspace" : "generic",
"Height" : 842,
"Idiom" : "universal",
"Name" : "pdf",
"NameIdentifier" : 19593,
"RenditionName" : "pdf.pdf",
"Scale" : 1,
"SHA1Digest" : "03F9900F23A26D421A8A612372849CCFB2952E7C47D99DA514B770AE1908C283",
"SizeOnDisk" : 43409,
"State" : "Normal",
"Value" : "Off",
"Width" : 595
}
This means there are 3 files in the Assets.car. And we can use some tools to export them out. e.g QLCARFiles and we can get these 3 files.
-rw-r--r-- 1 xxxxx staff 43185 Nov 10 21:20 pdf.pdf
-rw-r--r-- 1 xxxxx staff 113023 Nov 10 21:20 pdf-UIAppearanceAny.png
-rw-r--r-- 1 xxxxx staff 343966 Nov 10 21:20 pdf-UIAppearanceAny#2x.png
My question is
as I already selected "Preserve Vector Data" in Xcode, why my device still get two unnecessary png files?
In my opinion, what my device need is only that pdf.pdf file, do not need these two png file.(pdf-UIAppearanceAny.png / pdf-UIAppearanceAny.png), these have no benefit except increased unnecessary ipa's size.
I also used svg file and can got same phenomenon, I googled a lot, no useful message found.
Could it be the minimum iOS version for your app is <= iOS 12?
The "vector data preservation" + SVG support is only available on macOS 10.15 or later, iOS 13 or later, and iPadOS 13 or later.
Else Xcode will fallback to generating png files when building an app archive.
The following blogs explain it a lot better than i can, have read.
https://sarunw.com/posts/svg-image-assets-supported-in-xcode12/
https://www.avanderlee.com/xcode/svg-image-assets/
https://www.willowtreeapps.com/craft/what-svg-support-means-for-your-ios-app-graphics
I have a time-based Kibana and the time-field is a type [number] (e.g. 1599026948 ) called timestamp. The time field name is not set and I need to set it to timestamp. Because of the bad format (number) of timestamp, I created a scripted field called datetime to get a better date-format.
The code for the scripted field datetime:
doc['timestamp'].value * 1000;
It works as it should:
Samples
Input Output
1599027119802 Sep 2, 2020 # 08:11:59.802
1577833200000 Jan 1, 2020 # 00:00:00.000
1609455599999 Dec 31, 2020 # 23:59:59.999
When I look at my "Discover"-tab, I get the warning:
"1 of 2 shards failed The data you are seeing might be incomplete or wrong."
Warning-detail:
script_exception at shard 0 index MY-INDEX-PATTERN node JgWo2yNxRBeR9mEuYLpK2R
Type script_exeption
Reason runtime error
Script stack org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)
org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)
doc['timestamp'].value * 1000;
^---- HERE
Script doc['timestamp'].value * 1000;
Lang painless
Caused by type illegal_argument_exeption
Caused by reason No field found for [timestamp] in mapping with types []
I ignored the warning and moved on.
I needed to set the time field name to the new field datetime.
I tried this by following code in the console:
POST .kibana/_update/MY-INDEX-PATTERN
{
"doc": {
"index-pattern": {
"timeFieldName" : "datetime"
}
}
}
I get the following result:
{
"_index" : ".kibana",
"_type" : "_doc",
"_id" : "MY-INDEX-PATTERN",
"_version" : 8,
"result" : "updated",
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"_seq_no" : 887,
"_primary_term" : 4
}
Looks like it worked, but when i try to create a time-based component inside my dashboard or just change the time-filter, it does not work and i get the same warning as above (1 of 2 shards failed..)
I think the time field name was not set to datetime.
I feel like I tried everything but nothing workes. I really want to avoid reindexing or is this the only solution? I am new to Kibana and hope someone can help me.
-Tobi
Solved: Kibana-administrator has to reindex the index, and set the type of timestamp to date. Now timestamp is visible when creating a pattern and can be choosen for time field name.
[ {
"id" : 57592,
"code" : "village1023",
"created_by_id" : null,
"created_date" : "Tue Mar 31 23:08:47 IST 2020",
"l_village_name" : "village1023",
"modified_by_id" : 70806,
"modified_date" : "Tue Mar 31 23:08:47 IST 2020",
"name" : "village1023",
"status" : "{0}",
"taluk_id" : 386
} ]
Any suggestion on how could I convert the value received from JSON as "status" : "{0}" as "TRUE" for the MySQL value "1" in bit data type.
If you want to replace the JSON value content "{0}" in to true, we can update the content of the flow file using ReplaceText processor, so you can simply use the ReplaceText processor to achieve the reported scenario but still it have its own limitations. Please check and proceed further.
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ReplaceText/
I got the expected result, I made mistake in the expression. It should be ${field.value:equals("{0}"):ifElse("TRUE","FALSE")} in "UpdateRecord" processor.
Doumentation - Example 5 - Use Expression Language to Modify Value
How can I format digit in logstash?
I am using the '' % format expression in ruby code in filter plugin but I get nil as format result. I tried sprintf and format function but same result.
Below is my code snippet.
ruby {
code => "
event.set( 'positioning', event.get('branch_lat') + ',' + event.get('branch_lon') )
event.set( 'report_datetime', event.get('report_date') + '%04d' % event.get('report_time') )
"
}
As a format result, I get below error in the log.
[2016-10-28T12:31:43,217][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `+' for nil:NilClass
My platform information is below.
[root#elk-analytic logstash]# rpm -qi logstash
Name : logstash
Epoch : 1
Version : 5.0.0
Release : 1
Architecture: noarch
Install Date: Thu 27 Oct 2016 01:26:03 PM JST
Group : default
Size : 198320729
License : ASL 2.0
Signature : RSA/SHA512, Wed 26 Oct 2016 01:57:59 PM JST, Key ID d27d666cd88e42b4
Source RPM : logstash-5.0.0-1.src.rpm
Build Date : Wed 26 Oct 2016 01:10:26 PM JST
Build Host : packer-virtualbox-iso-1474648640
Relocations : /
Packager : <vagrant#packer-virtualbox-iso-1474648640>
Vendor : Elasticsearch
URL : http://www.elasticsearch.org/overview/logstash/
Summary : An extensible logging pipeline
Description :
An extensible logging pipeline
Added on 2016.10.28 14:32
My Goal is to parse below csv columns into timestamp field in elasticsearch.
Please notice that hour of time has mixed patterns of 1 and 2 digits.
date,time
20160204,1000
20160204,935
I tried using date function in filter plugin but it did not work properly by logging error.
[2016-10-28T11:00:10,233][WARN ][logstash.filters.date ] Failed parsing date from field {:field=>"report_datetime",
:value=>"20160204 935", :exception=>"Cannot parse \"20160204 935\": Value 93 for hourOfDay must be in the range [0,23]", :config_parsers=>"YYYYMMdd Hmm", :config_locale=>"default=en_US"}
Below is the code snippet when above error appeared.
ruby {
code => "
event.set( 'positioning', event.get('branch_lat') + ',' + event.get('branch_lon') )
event.set( 'report_datetime', event.get('report_date') + ' ' + event.get('report_time') )
"
}
# Set the #timestamp according to report_date and time
date {
"match" => ["report_datetime", "YYYYMMdd Hmm"]
}
I did some modification and ended up with the code I first posted.
I suggest to do it like this without any ruby filter:
filter {
# your other filters...
# if 3-digit hours, pad the time with one zero
if [time] =~ /^\d{3}$/ {
mutate {
add_field => { "report_datetime" => "%{date} 0%{time}" }
}
# otherwise just concat the fields
} else {
mutate {
add_field => { "report_datetime" => "%{date} %{time}" }
}
}
# match date and time
date {
"match" => ["report_datetime", "yyyyMMdd HHmm"]
"target" => "report_datetime"
}
}
I have a dataset that looks like this:
Date
Category
Rate
Quantity
There will be 0 or 1 row for each Category for any given Date.
What is a good way to get this data into a summary type of view?
For example:
Date
Category1_Rate
Category2_Rate
Category3_Rate
Category4_Rate
I have a fixed number of Categories.
I'm using linq.
Here is an example. If I have this data:
Date Category Rate Quantity
1/1/12 toys 15 12
1/1/12 games 20 20
1/1/12 dvds 18 30
1/2/12 toys 19 13
1/2/12 dvds 20 17
I want to produce a summary that looks like this:
Date toys_rate games_rate dvds_rate
1/1/12 15 20 18
1/2/12 19 null 20
Possibly something like this
var summarydata =
from r in table
group r by r.Date into g
select new
{
Date = g.Key,
ToysRate = g.Where(e=> e.Category == "toys").Count() > 0 ?
(int?)g.Where(e=> e.Category == "toys").First().Rate : null,
GamesRate = g.Where(e=> e.Category == "games").Count() > 0 ?
(int?)g.Where(e=> e.Category == "games").First().Rate : null,
DvdsRate = g.Where(e=> e.Category == "dvds").Count() > 0 ?
(int?)g.Where(e=> e.Category == "dvds").First().Rate : null
};
Note I haven't tested this as I don't current have access to a C# environment.
EDIT - Added nullable int casts to properly set the type of the various rate fields in the resulting anonymous type.