Using Dynamic variables in aggregate query from Vega (in Kibana) to ElasticSearch - elasticsearch

We have a report that fetches data from elasticsearch and displays them as buckets. We want to show a band, customers and profit amount for each band. The idea is that we should give a band size parameter and then Vega should create 5 buckets for that when querying the data. If I mention the band size to be 50, then vega will pass this on in the query and create buckets 0-50, 51 to 100, 101 to 150, 151 to 200 and 200+. I am really struggling with this a lot and can not make it work. I am pasting the vega script here that works with hard coded values but when it comes to doing it through signals, I have not reached any clue at all. I would like to use the value from signal to define to band parameter. Please help.
{
"$schema":"https://vega.github.io/schema/vega/v4.json",
"data":[
{
"name":"SalesData",
"url":{
"index":"orderaggregates",
"body":{
"size":0,
"aggs":{
"bands":{
"range":{
"field":"total_sales",
"ranges":[
{
"from":0,
"to":50
},
{
"from":50.01,
"to":100
},
{
"from":100.01,
"to":150
},
{
"from":150.01,
"to":200
},
{
"from":200.01
}
]
},
"aggs":{
"total_sales":{
"sum":{
"field":"total_sales"
}
},
"customers":{
"cardinality":{
"field":"user_id"
}
},
"confirmed_orders":{
"sum":{
"field":"confirmedorders"
}
},
"cost_total":{
"sum":{
"field":"cost_total"
}
},
"shipping_collected":{
"sum":{
"field":"freightcollected"
}
},
"shipping_paid":{
"sum":{
"field":"freight_paid"
}
}
}
}
}
}
},
"format":{
"property":"aggregations.bands.buckets"
},
"transform":[
{
"type":"project",
"fields":[
"key",
"doc_count",
"customers.value",
"cost_total.value",
"total_sales.value"
],
"as":[
"range",
"orders",
"customers",
"costs",
"sales"
]
},
{
"type":"identifier",
"as":"id"
},
{
"type":"formula",
"as":"y",
"expr":"datum.id * 30"
},
{
"type":"formula",
"as":"orders",
"expr":"format(datum.orders,',.0f')"
},
{
"type":"formula",
"as":"customers",
"expr":"format(datum.customers,',.0f')"
},
{
"type":"formula",
"as":"costs",
"expr":"format(datum.costs,'$,.0f')"
},
{
"type":"formula",
"as":"sales",
"expr":"format(datum.sales,'$,.0f')"
}
]
}
],
"signals":[
{
"name":"Group",
"value":40,
"bind":{
"input":"range",
"min":10,
"max":100,
"step":10
}
},
{
"name":"GroupSize",
"update":"Group/2"
}
],
"marks":[
{
"from":{
"data":"SalesData"
},
"type":"text",
"encode":{
"enter":{
"fill":{
"value":"#000"
},
"text":{
"field":"range"
},
"x":{
"value":50
},
"y":{
"field":"y"
}
},
"update":{
"opacity":{
"value":1
},
"fontSize":{
"signal":"GroupSize"
}
},
"hover":{
"opacity":{
"value":0.5
}
}
}
},
{
"from":{
"data":"SalesData"
},
"type":"text",
"encode":{
"enter":{
"fill":{
"value":"#000"
},
"text":{
"field":"customers"
},
"x":{
"value":250
},
"y":{
"field":"y"
}
},
"update":{
"opacity":{
"value":1
},
"fontSize":{
"signal":"GroupSize"
}
},
"hover":{
"opacity":{
"value":0.5
}
}
}
},
{
"from":{
"data":"SalesData"
},
"type":"text",
"encode":{
"enter":{
"fill":{
"value":"#000"
},
"text":{
"field":"orders"
},
"x":{
"value":350
},
"y":{
"field":"y"
}
},
"update":{
"opacity":{
"value":1
},
"fontSize":{
"signal":"GroupSize"
}
},
"hover":{
"opacity":{
"value":0.5
}
}
}
},
{
"from":{
"data":"SalesData"
},
"type":"text",
"encode":{
"enter":{
"fill":{
"value":"#000"
},
"text":{
"field":"sales"
},
"x":{
"value":450
},
"y":{
"field":"y"
}
},
"update":{
"opacity":{
"value":1
},
"fontSize":{
"signal":"GroupSize"
}
},
"hover":{
"opacity":{
"value":0.5
}
}
}
}
]
}

You have to change the structure. Move the task which is done in your query to data's transform section where you can use the signals:
Query all relevant data
Transform your data with usage of your signal "Group" to the bins (e.g. calculate "extend" and "bin" transforms similar to this: http://blog.wafrat.com/building-a-log-scaled-histogram-of-aaves-loans-in-vega/)

Related

INDEXER Lake Framework - outcome.status.failure not showing execution error

The following transaction
https://explorer.near.org/transactions/CkZYiXB3BEoXNuLDqcn5GjErkXghKGCB47yPyyze4kTf#BPYU8T3o5mdsU4ewG3JdCLRe8ivpsGTehKxkifFAsboD
failed whith the following error:
{
"type": "action",
"error": {
"type": "functionCallError",
"error": {
"type": "executionError",
"error": "Smart contract panicked: The contract is not initialized"
}
}
}
But what I got on the indexer is the following:
{
"outcome":{
"executionOutcome":{
"blockHash":"8NP4Z2i7CBQzRUxa8CfuummnYZvwfZaCFmif4d6mcgyB",
"id":"CkZYiXB3BEoXNuLDqcn5GjErkXghKGCB47yPyyze4kTf",
"outcome":{
"executorId":"test1.csandi.near",
"gasBurnt":2428211671420,
"logs":[
],
"metadata":{
"gasProfile":null,
"version":1
},
"receiptIds":[
"GrXdFKA9byvCmxGiQkGc9iaFn88FfePk4tfJhvcTGMwc"
],
"status":{
"SuccessReceiptId":"GrXdFKA9byvCmxGiQkGc9iaFn88FfePk4tfJhvcTGMwc"
},
"tokensBurnt":"242821167142000000000"
},
"proof":[
{
"direction":"Left",
"hash":"BWhmGBmvzbX3WBP1RZQ59DB749sHSV2xS5pNyF6xAbbh"
},
{
"direction":"Right",
"hash":"H4YMDcd9Fh68KTwyPTwYuFb89KhfdGUoxRbBYzDcNAze"
},
{
"direction":"Right",
"hash":"9jv152euKvurdVkDYS5FKiiT4ej6xL1gRYTHoNbjwEZi"
},
{
"direction":"Right",
"hash":"BqQFHzSYzqDE8tMQkJT8PHFGxLmQrkFBcuTqFP4nTjx7"
}
]
},
"receipt":null
},
"transaction":{
"actions":[
{
"FunctionCall":{
"args":"eyJpbml0aWFsX3ByaWNlIjoxLjksInNwYWNpbmciOjIsImFscGhhX2lkIjoidXNuIiwiYmV0YV9pZCI6IndyYXAubmVhciIsInN3YXBfZmVlX3JhdGUiOjAuMDAzLCJwcm90b2NvbF90b2xsIjowLjAxfQ==",
"deposit":"0",
"gas":200000000000000,
"methodName":"create_new_pool"
}
}
],
"hash":"CkZYiXB3BEoXNuLDqcn5GjErkXghKGCB47yPyyze4kTf",
"nonce":75627675000007,
"publicKey":"ed25519:85yAx5mY8DinTWQfMn47cZTHMhZtLRAWZLpazYmetwno",
"receiverId":"dev-cliquidity-perk-jorge.near",
"signature":"ed25519:1CieS8AUSc7Y3S8By8adJaEXXydXcjvSA1uVXPgrdhkmGPpf5ykA7XWqJGZnDhMtymkp6eWDBEQzsBXkhybAbUV",
"signerId":"test1.csandi.near"
}
}
So I would like to know if there is anyway to get the failure message from the indexer?
Look in the next blocks for the corresponding receipt from the SuccessReceiptId (id GrXdFKA9byvCmxGiQkGc9iaFn88FfePk4tfJhvcTGMwc), and in its execution outcome you will find that status.
I cannot recommend more to watch or read the Data Flow in NEAR
FYI, just to follow up and clarify for anyone with this question, after following the above response by "Vlad Frolov", in order to get the Receipt Status, I had to check
shard.receiptExecutionOutcomes
which will provide the results of the receipts, so I started to track the receipts that I am interested in, in this case for
SuccessReceiptId (GrXdFKA9byvCmxGiQkGc9iaFn88FfePk4tfJhvcTGMwc)
"executionOutcome":{
"blockHash":"59F5pES9Myc2VT4JkppE95epCwtGAXYux9okbzpsh95K",
"id":"**GrXdFKA9byvCmxGiQkGc9iaFn88FfePk4tfJhvcTGMwc**",
"outcome":{
"executorId":"dev-cliquidity-perk-jorge.near",
"gasBurnt":2824274578054,
"logs":[
],
"metadata":{
"gasProfile":[
{
"cost":"BASE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"1323840555"
},
{
"cost":"CONTRACT_LOADING_BASE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"35445963"
},
{
"cost":"CONTRACT_LOADING_BYTES",
"costCategory":"WASM_HOST_COST",
"gasUsed":"106584645000"
},
{
"cost":"READ_MEMORY_BASE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"5219726400"
},
{
"cost":"READ_MEMORY_BYTE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"136847988"
},
{
"cost":"READ_REGISTER_BASE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"2517165186"
},
{
"cost":"READ_REGISTER_BYTE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"11334630"
},
{
"cost":"STORAGE_READ_BASE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"56356845750"
},
{
"cost":"STORAGE_READ_KEY_BYTE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"154762665"
},
{
"cost":"TOUCHING_TRIE_NODE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"193223471112"
},
{
"cost":"UTF8_DECODING_BASE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"3111779061"
},
{
"cost":"UTF8_DECODING_BYTE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"9038994849"
},
{
"cost":"WASM_INSTRUCTION",
"costCategory":"WASM_HOST_COST",
"gasUsed":"11928316488"
},
{
"cost":"WRITE_MEMORY_BASE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"2803794861"
},
{
"cost":"WRITE_MEMORY_BYTE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"313233780"
},
{
"cost":"WRITE_REGISTER_BASE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"2865522486"
},
{
"cost":"WRITE_REGISTER_BYTE",
"costCategory":"WASM_HOST_COST",
"gasUsed":"437179860"
}
],
"version":1
},
"receiptIds":[
"BPYU8T3o5mdsU4ewG3JdCLRe8ivpsGTehKxkifFAsboD"
],
**"status":{
"Failure":{
"ActionError":{
"index":0,
"kind":{
"FunctionCallError":{
"ExecutionError":"Smart contract panicked: The contract is not initialized"
}
}
}
}**
},
"tokensBurnt":"282427457805400000000"
},
"proof":[
{
"direction":"Left",
"hash":"5j1RhLnf5jGChUPxU9CPNbAGwSn6VnzBw7taJe5vSnYD"
},
{
"direction":"Right",
"hash":"AJ2kARUcYAkzsL3xmw4jAETZ61jtQuswf7RvhSxscKCR"
},
{
"direction":"Left",
"hash":"9vXQjFdTaQw5XjUKsRK6N3WFCg5zMVca5Ze2Qqznn3AM"
}
] }, "receipt":{
"predecessorId":"test1.csandi.near",
"receipt":{
"Action":{
"actions":[
{
"FunctionCall":{
"args":"eyJpbml0aWFsX3ByaWNlIjoxLjksInNwYWNpbmciOjIsImFscGhhX2lkIjoidXNuIiwiYmV0YV9pZCI6IndyYXAubmVhciIsInN3YXBfZmVlX3JhdGUiOjAuMDAzLCJwcm90b2NvbF90b2xsIjowLjAxfQ==",
"deposit":"0",
"gas":200000000000000,
"methodName":"create_new_pool"
}
}
],
"gasPrice":"346069590",
"inputDataIds":[
],
"outputDataReceivers":[
],
"signerId":"test1.csandi.near",
"signerPublicKey":"ed25519:85yAx5mY8DinTWQfMn47cZTHMhZtLRAWZLpazYmetwno"
}
},
"receiptId":"GrXdFKA9byvCmxGiQkGc9iaFn88FfePk4tfJhvcTGMwc",
"receiverId":"dev-cliquidity-perk-jorge.near" } }
And as you can see in the json response,the following status:
"status":{
"Failure":{
"ActionError":{
"index":0,
"kind":{
"FunctionCallError":{
"ExecutionError":"Smart contract panicked: The contract is not initialized"
}
}
}
}
Anyway to fully understand how transactions/receipts work this document helped a lot.

Vega/ Vega-lite in Kibana - how to make the chart responsive to resizing

I created a dashboard in Kibana and included also some Vega visualizations. My problem is that I cannot make the Vega bar chart responsive in sense of dynamically adjusting the size when the window size changes:
The bar chart on the dashboard
Settings:
autosize: {
type: "fit",
contains: "padding",
resize: true,
}
Bar chart on the dashboard
The bar chart after resizing - data does not adjust to the window
Bar char after resizing
I also tried to follow the instructions here https://vega.github.io/vega-lite/docs/size.html and used setting width or height to "container", but nothing worked, or I am defining them incorrectly.
The complete Vega specification is available below:
{
$schema: https://vega.github.io/schema/vega-lite/v5.json
title: Test
autosize: {
type: "fit",
contains: "padding",
resize: true,
},
// Define the data source
data: {
url: {
%context%: true
%timefield%: time
index: test
body: {
"aggs":{
"date_hist":{
"date_histogram": {
"field": "time",
"calendar_interval": "day",
"format": "strict_date"
},
"aggs":{
"state":{
"terms": {
"field": "f.keyword",
"include":["active","revoked"],
"min_doc_count": 0,
"size": 10000
}
},
"increased":{
"bucket_script": {
"buckets_path": {
"in": "state['active']>_count",
"out": "state['revoked']>_count"
},
"script": "params.in - params.out"
}
},
"active_patients":{
"cumulative_sum": {
"buckets_path": "increased"
}
}
}
}
},
"size":0
}
}
format: {property: "aggregations.date_hist.buckets"}
}
mark: bar
encoding: {
x: {
field: key_as_string
type: ordinal
axis: {title: "Date", "labelAngle": 45}
}
y: {
field: active_patients.value
type: quantitative
axis: {title: " "}
}
"color": {"value": "#00A6CE "}
}
}
I would appreciate any assistance!
You need to create a signal that will listen window resize event.
This signal will read the container width.
"signals": [
{
"name": "width",
"init": "containerSize()[0]",
"on": [{ "events": "window:resize", "update": "containerSize()[0]" }]
}
],
After that, you need to define a range on xscale to read the signal previously created.
"scales": [
{
"name": "xscale",
"type": "band",
"domain": {"data": "table", "field": "your_value"},
"range": [0, { "signal": "width" }],
"padding": 0.05
},
]

Why is my graphql query return all product?

I'm testing graphql query on Shopify GraphiQL Explorer for filtering products on Storefront API.
My query is like this:
query ProductType {
collectionByHandle(handle: "pants") {
handle
products(first: 10, filters: {
productType: "pants"
}) {
edges {
node {
handle
productType
}
}
}
}
}
And got the result like this:
{
"data": {
"collectionByHandle": {
"handle": "pants",
"products": {
"edges": [
{
"node": {
"handle": "my-test-product-pants",
"productType": "pants"
}
},
{
"node": {
"handle": "pleated-straight-pants-mens-fashion-elastic-waist-casual-pants-men-streetwear-loose-ice-silk-trousers-mens-wide-leg-pants-s-2xl",
"productType": ""
}
},
...
Basically, the result has all the products that I have for that collection. Can anybody help me with this? This is literally the code I got from shopify website
As we can see in the tutorial filters is an array
{
"product_filters": [
{
"productType": "shoes"
},
{
"productVendor": "bestshop"
},
{
"variantOption": {
"name": "color",
"value": "blue"
}
}
]
}
So, try this instead
query ProductType {
collectionByHandle(handle: "pants") {
handle
products(first:10, filters: [{ productType: "pants" ]}) {
...
}
}
}

Does index size impact document indexing speed, tips?

I currently want to index 132 Million documents over at my ES services hosted in aws ec2, I was able to do 98 Million, during a week.
I noticed that indexing speed progressively decreased as the index grew in size, its currently sitting at 44GB.
I tried pausing the process, and resuming it from certain points, but the speed was definitely not consistent.
Is there a relation between index size and document indexing speed?
Would appreciate a tip on how to improve index speed in these case, if possible guys, thanks in advance.
Cluster Settings
{
"persistent":{
},
"transient":{
},
"defaults":{
"cluster":{
"routing":{
"use_adaptive_replica_selection":"false",
"rebalance":{
"enable":"all"
},
"allocation":{
"node_concurrent_incoming_recoveries":"2",
"node_initial_primaries_recoveries":"4",
"same_shard":{
"host":"false"
},
"total_shards_per_node":"-1",
"type":"balanced",
"disk":{
"threshold_enabled":"true",
"watermark":{
"low":"85%",
"flood_stage":"95%",
"high":"90%"
},
"include_relocations":"true",
"reroute_interval":"60s"
},
"awareness":{
"attributes":[
]
},
"balance":{
"index":"0.55",
"threshold":"1.0",
"shard":"0.45"
},
"enable":"all",
"node_concurrent_outgoing_recoveries":"2",
"allow_rebalance":"indices_all_active",
"cluster_concurrent_rebalance":"2",
"node_concurrent_recoveries":"2"
}
},
"indices":{
"tombstones":{
"size":"500"
},
"close":{
"enable":"true"
}
},
"nodes":{
"reconnect_interval":"10s"
},
"persistent_tasks":{
"allocation":{
"enable":"all"
}
},
"blocks":{
"read_only_allow_delete":"false",
"read_only":"false"
},
"service":{
"slow_task_logging_threshold":"30s"
},
"name":"roseland",
"info":{
"update":{
"interval":"30s",
"timeout":"15s"
}
}
},
"no":{
"model":{
"state":{
"persist":"false"
}
}
},
"logger":{
"level":"INFO"
},
"bootstrap":{
"memory_lock":"false",
"system_call_filter":"false",
"ctrlhandler":"true"
},
"processors":"2",
"ingest":{
"grok":{
"watchdog":{
"max_execution_time":"1s",
"interval":"1s"
}
}
},
"network":{
"host":[
"_local:ipv4_",
"_eth0:ipv4_"
],
"tcp":{
"reuse_address":"true",
"keep_alive":"true",
"connect_timeout":"30s",
"receive_buffer_size":"-1b",
"no_delay":"true",
"send_buffer_size":"-1b"
},
"bind_host":[
"_local:ipv4_",
"_eth0:ipv4_"
],
"server":"true",
"breaker":{
"inflight_requests":{
"limit":"100%",
"overhead":"1.0"
}
},
"publish_host":[
"_local:ipv4_",
"_eth0:ipv4_"
]
},
"pidfile":"/var/run/elasticsearch/elasticsearch.pid",
"path":{
"data":[
"/data/elasticsearch"
],
"logs":"/var/log/elasticsearch",
"shared_data":"",
"home":"/usr/share/elasticsearch",
"repo":[
]
},
"search":{
"default_search_timeout":"-1",
"highlight":{
"term_vector_multi_value":"true"
},
"default_allow_partial_results":"true",
"max_buckets":"-1",
"low_level_cancellation":"false",
"keep_alive_interval":"1m",
"remote":{
"node":{
"attr":""
},
"initial_connect_timeout":"30s",
"connect":"true",
"connections_per_cluster":"3"
},
"default_keep_alive":"5m",
"max_keep_alive":"24h"
},
"security":{
"manager":{
"filter_bad_defaults":"true"
}
},
"repositories":{
"fs":{
"compress":"false",
"chunk_size":"9223372036854775807b",
"location":""
},
"url":{
"supported_protocols":[
"http",
"https",
"ftp",
"file",
"jar"
],
"allowed_urls":[
],
"url":"http:"
}
},
"action":{
"auto_create_index":"true",
"search":{
"shard_count":{
"limit":"9223372036854775807"
}
},
"destructive_requires_name":"false",
"master":{
"force_local":"false"
}
},
"client":{
"type":"node",
"transport":{
"ignore_cluster_name":"false",
"nodes_sampler_interval":"5s",
"sniff":"false",
"ping_timeout":"5s"
}
},
"xpack":{
"watcher":{
"execution":{
"scroll":{
"size":"0",
"timeout":""
},
"default_throttle_period":"5s"
},
"internal":{
"ops":{
"bulk":{
"default_timeout":""
},
"index":{
"default_timeout":""
},
"search":{
"default_timeout":""
}
}
},
"thread_pool":{
"queue_size":"1000",
"size":"10"
},
"index":{
"rest":{
"direct_access":""
}
},
"history":{
"cleaner_service":{
"enabled":"true"
}
},
"trigger":{
"schedule":{
"ticker":{
"tick_interval":"500ms"
}
}
},
"enabled":"true",
"input":{
"search":{
"default_timeout":""
}
},
"encrypt_sensitive_data":"false",
"transform":{
"search":{
"default_timeout":""
}
},
"stop":{
"timeout":"30s"
},
"watch":{
"scroll":{
"size":"0"
}
},
"require_manual_start":"false",
"actions":{
"bulk":{
"default_timeout":""
},
"index":{
"default_timeout":""
}
}
},
"license":{
"self_generated":{
"type":"basic"
}
},
"logstash":{
"enabled":"true"
},
"notification":{
"hipchat":{
"host":"",
"port":"443",
"default_account":""
},
"pagerduty":{
"default_account":""
},
"email":{
"default_account":"",
"html":{
"sanitization":{
"allow":[
"body",
"head",
"_tables",
"_links",
"_blocks",
"_formatting",
"img:embedded"
],
"disallow":[
],
"enabled":"true"
}
}
},
"reporting":{
"retries":"40",
"interval":"15s"
},
"jira":{
"default_account":""
},
"slack":{
"default_account":""
}
},
"security":{
"dls_fls":{
"enabled":"true"
},
"transport":{
"filter":{
"allow":[
],
"deny":[
],
"enabled":"true"
},
"ssl":{
"enabled":"false"
}
},
"enabled":"true",
"filter":{
"always_allow_bound_address":"true"
},
"encryption":{
"algorithm":"AES/CTR/NoPadding"
},
"audit":{
"outputs":[
"logfile"
],
"index":{
"bulk_size":"1000",
"rollover":"DAILY",
"flush_interval":"1s",
"events":{
"emit_request_body":"false",
"include":[
"ACCESS_DENIED",
"ACCESS_GRANTED",
"ANONYMOUS_ACCESS_DENIED",
"AUTHENTICATION_FAILED",
"REALM_AUTHENTICATION_FAILED",
"CONNECTION_DENIED",
"CONNECTION_GRANTED",
"TAMPERED_REQUEST",
"RUN_AS_DENIED",
"RUN_AS_GRANTED",
"AUTHENTICATION_SUCCESS"
],
"exclude":[
]
},
"queue_max_size":"10000"
},
"enabled":"false",
"logfile":{
"events":{
"emit_request_body":"false",
"include":[
"ACCESS_DENIED",
"ACCESS_GRANTED",
"ANONYMOUS_ACCESS_DENIED",
"AUTHENTICATION_FAILED",
"CONNECTION_DENIED",
"TAMPERED_REQUEST",
"RUN_AS_DENIED",
"RUN_AS_GRANTED"
],
"exclude":[
]
},
"prefix":{
"emit_node_host_name":"false",
"emit_node_name":"true",
"emit_node_host_address":"false"
}
}
},
"authc":{
"anonymous":{
"authz_exception":"true",
"roles":[
],
"username":"_anonymous"
},
"password_hashing":{
"algorithm":"bcrypt"
},
"run_as":{
"enabled":"true"
},
"reserved_realm":{
"enabled":"true"
},
"token":{
"delete":{
"interval":"30m",
"timeout":"-1"
},
"enabled":"false",
"thread_pool":{
"queue_size":"1000",
"size":"1"
},
"timeout":"20m"
}
},
"fips_mode":{
"enabled":"false"
},
"encryption_key":{
"length":"128",
"algorithm":"AES"
},
"http":{
"filter":{
"allow":[
],
"deny":[
],
"enabled":"true"
},
"ssl":{
"enabled":"false"
}
},
"automata":{
"max_determinized_states":"100000"
},
"user":null,
"authz":{
"store":{
"roles":{
"field_permissions":{
"cache":{
"max_size_in_bytes":"104857600"
}
},
"index":{
"cache":{
"ttl":"20m",
"max_size":"10000"
}
},
"cache":{
"max_size":"10000"
}
}
}
}
},
"http":{
"default_connection_timeout":"10s",
"proxy":{
"host":"",
"scheme":"",
"port":"0"
},
"default_read_timeout":"10s",
"max_response_size":"10mb"
},
"monitoring":{
"enabled":"true",
"collection":{
"cluster":{
"stats":{
"timeout":"10s"
}
},
"node":{
"stats":{
"timeout":"10s"
}
},
"indices":[
],
"index":{
"stats":{
"timeout":"10s"
},
"recovery":{
"active_only":"false",
"timeout":"10s"
}
},
"interval":"10s",
"enabled":"false",
"ml":{
"job":{
"stats":{
"timeout":"10s"
}
}
}
},
"history":{
"duration":"168h"
}
},
"graph":{
"enabled":"true"
},
"ml":{
"utility_thread_pool":{
"queue_size":"500",
"size":"80"
},
"max_machine_memory_percent":"30",
"max_open_jobs":"20",
"min_disk_space_off_heap":"5gb",
"autodetect_process":"true",
"datafeed_thread_pool":{
"queue_size":"200",
"size":"20"
},
"node_concurrent_job_allocations":"2",
"max_model_memory_limit":"0b",
"enabled":"true",
"autodetect_thread_pool":{
"queue_size":"80",
"size":"80"
}
},
"rollup":{
"enabled":"true",
"task_thread_pool":{
"queue_size":"4",
"size":"4"
}
},
"sql":{
"enabled":"true"
}
},
"rest":{
"action":{
"multi":{
"allow_explicit_index":"true"
}
}
},
"cache":{
"recycler":{
"page":{
"limit":{
"heap":"10%"
},
"type":"CONCURRENT",
"weight":{
"longs":"1.0",
"ints":"1.0",
"bytes":"1.0",
"objects":"0.1"
}
}
}
},
"reindex":{
"remote":{
"whitelist":[
]
}
},
"max":{
"percent":{
"date":{
"errors":"25"
},
"outoforder":{
"errors":"25"
}
},
"anomaly":{
"records":"500"
}
},
"resource":{
"reload":{
"enabled":"true",
"interval":{
"low":"60s",
"high":"5s",
"medium":"30s"
}
}
},
"thread_pool":{
"force_merge":{
"queue_size":"-1",
"size":"1"
},
"fetch_shard_started":{
"core":"1",
"max":"4",
"keep_alive":"5m"
},
"listener":{
"queue_size":"-1",
"size":"1"
},
"index":{
"queue_size":"200",
"size":"2"
},
"refresh":{
"core":"1",
"max":"1",
"keep_alive":"5m"
},
"generic":{
"core":"4",
"max":"128",
"keep_alive":"30s"
},
"warmer":{
"core":"1",
"max":"1",
"keep_alive":"5m"
},
"search":{
"max_queue_size":"1000",
"queue_size":"1000",
"size":"4",
"auto_queue_frame_size":"2000",
"target_response_time":"1s",
"min_queue_size":"1000"
},
"fetch_shard_store":{
"core":"1",
"max":"4",
"keep_alive":"5m"
},
"flush":{
"core":"1",
"max":"1",
"keep_alive":"5m"
},
"management":{
"core":"1",
"max":"5",
"keep_alive":"5m"
},
"analyze":{
"queue_size":"16",
"size":"1"
},
"get":{
"queue_size":"1000",
"size":"2"
},
"bulk":{
"queue_size":"200",
"size":"2"
},
"estimated_time_interval":"200ms",
"write":{
"queue_size":"200",
"size":"2"
},
"snapshot":{
"core":"1",
"max":"1",
"keep_alive":"5m"
}
},
"index":{
"codec":"default",
"store":{
"type":"",
"fs":{
"fs_lock":"native"
},
"preload":[
]
}
},
"monitor":{
"jvm":{
"gc":{
"enabled":"true",
"overhead":{
"warn":"50",
"debug":"10",
"info":"25"
},
"refresh_interval":"1s"
},
"refresh_interval":"1s"
},
"process":{
"refresh_interval":"1s"
},
"os":{
"refresh_interval":"1s"
},
"fs":{
"refresh_interval":"1s"
}
},
"transport":{
"tcp":{
"reuse_address":"true",
"connect_timeout":"30s",
"compress":"false",
"port":"9300-9400",
"keep_alive":"true",
"receive_buffer_size":"-1b",
"send_buffer_size":"-1b"
},
"bind_host":[
],
"ping_schedule":"-1",
"connections_per_node":{
"recovery":"2",
"state":"1",
"bulk":"3",
"reg":"6",
"ping":"1"
},
"tracer":{
"include":[
],
"exclude":[
"internal:discovery/zen/fd*",
"cluster:monitor/nodes/liveness"
]
},
"type":"security4",
"type.default":"netty4",
"features":{
"x-pack":"true"
},
"host":[
],
"publish_port":"-1",
"tcp_no_delay":"true",
"publish_host":[
],
"netty":{
"receive_predictor_size":"64kb",
"receive_predictor_max":"64kb",
"worker_count":"4",
"receive_predictor_min":"64kb",
"boss_count":"1"
}
},
"script":{
"allowed_contexts":[
],
"max_compilations_rate":"75/5m",
"cache":{
"max_size":"100",
"expire":"0ms"
},
"painless":{
"regex":{
"enabled":"false"
}
},
"max_size_in_bytes":"65535",
"allowed_types":[
]
},
"node":{
"data":"true",
"enable_lucene_segment_infos_trace":"false",
"local_storage":"true",
"max_local_storage_nodes":"1",
"name":"node01",
"id":{
"seed":"0"
},
"attr":{
"xpack":{
"installed":"true"
},
"ml":{
"machine_memory":"4075884544",
"max_open_jobs":"20",
"enabled":"true"
}
},
"portsfile":"false",
"ingest":"true",
"master":"true",
"ml":"true"
},
"indices":{
"cache":{
"cleanup_interval":"1m"
},
"mapping":{
"dynamic_timeout":"30s"
},
"memory":{
"interval":"5s",
"max_index_buffer_size":"-1",
"shard_inactive_time":"5m",
"index_buffer_size":"10%",
"min_index_buffer_size":"48mb"
},
"breaker":{
"request":{
"limit":"60%",
"type":"memory",
"overhead":"1.0"
},
"total":{
"limit":"70%"
},
"accounting":{
"limit":"100%",
"overhead":"1.0"
},
"fielddata":{
"limit":"60%",
"type":"memory",
"overhead":"1.03"
},
"type":"hierarchy"
},
"fielddata":{
"cache":{
"size":"-1b"
}
},
"query":{
"bool":{
"max_clause_count":"1024"
},
"query_string":{
"analyze_wildcard":"false",
"allowLeadingWildcard":"true"
}
},
"admin":{
"filtered_fields":"true"
},
"recovery":{
"recovery_activity_timeout":"1800000ms",
"retry_delay_network":"5s",
"internal_action_timeout":"15m",
"retry_delay_state_sync":"500ms",
"internal_action_long_timeout":"1800000ms",
"max_bytes_per_sec":"40mb"
},
"requests":{
"cache":{
"size":"1%",
"expire":"0ms"
}
},
"store":{
"delete":{
"shard":{
"timeout":"30s"
}
}
},
"analysis":{
"hunspell":{
"dictionary":{
"ignore_case":"false",
"lazy":"false"
}
}
},
"queries":{
"cache":{
"count":"10000",
"size":"10%",
"all_segments":"false"
}
}
},
"plugin":{
"mandatory":[
]
},
"max_running_jobs":"20",
"discovery":{
"type":"zen",
"zen":{
"commit_timeout":"30s",
"no_master_block":"write",
"join_retry_delay":"100ms",
"join_retry_attempts":"3",
"ping":{
"unicast":{
"concurrent_connects":"10",
"hosts":[
],
"hosts.resolve_timeout":"5s"
}
},
"master_election":{
"ignore_non_master_pings":"false",
"wait_for_joins_timeout":"30000ms"
},
"send_leave_request":"true",
"ping_timeout":"3s",
"join_timeout":"60000ms",
"publish_diff":{
"enable":"true"
},
"publish":{
"max_pending_cluster_states":"25"
},
"minimum_master_nodes":"-1",
"hosts_provider":[
],
"publish_timeout":"30s",
"fd":{
"connect_on_network_disconnect":"false",
"ping_interval":"1s",
"ping_retries":"3",
"register_connection_listener":"true",
"ping_timeout":"30s"
},
"max_pings_from_another_master":"3"
},
"initial_state_timeout":"30s"
},
"tribe":{
"name":"",
"on_conflict":"any",
"blocks":{
"metadata":"false",
"read":{
"indices":[
]
},
"write.indices":[
],
"write":"false",
"metadata.indices":[
]
}
},
"http":{
"cors":{
"max-age":"1728000",
"allow-origin":"",
"allow-headers":"X-Requested-With,Content-Type,Content-Length",
"allow-credentials":"false",
"allow-methods":"OPTIONS,HEAD,GET,POST,PUT,DELETE",
"enabled":"false"
},
"max_chunk_size":"8kb",
"compression_level":"3",
"max_initial_line_length":"4kb",
"type":"security4",
"pipelining":"true",
"enabled":"true",
"type.default":"netty4",
"content_type":{
"required":"true"
},
"host":[
],
"publish_port":"-1",
"read_timeout":"0ms",
"max_content_length":"100mb",
"netty":{
"receive_predictor_size":"64kb",
"max_composite_buffer_components":"69905",
"receive_predictor_max":"64kb",
"worker_count":"4",
"receive_predictor_min":"64kb"
},
"tcp":{
"reuse_address":"true",
"keep_alive":"true",
"receive_buffer_size":"-1b",
"send_buffer_size":"-1b"
},
"bind_host":[
],
"reset_cookies":"false",
"max_warning_header_count":"-1",
"max_warning_header_size":"-1b",
"detailed_errors":{
"enabled":"true"
},
"port":"9200-9300",
"max_header_size":"8kb",
"pipelining.max_events":"10000",
"tcp_no_delay":"true",
"compression":"true",
"publish_host":[
]
},
"gateway":{
"recover_after_master_nodes":"0",
"expected_nodes":"-1",
"recover_after_data_nodes":"-1",
"expected_data_nodes":"-1",
"recover_after_time":"0ms",
"expected_master_nodes":"-1",
"recover_after_nodes":"-1"
}
}
}
As #leandrojmp mentioned in the comment, you need to provide much more information for us to provide a specific recommendation, but for the general tip on improving the reindex(one-time) https://opster.com/blogs/improve-elasticsearch-reindex-performance/ and for ongoing indexing performance follow https://opster.com/blogs/improve-elasticsearch-indexing-rate/ .

Spring Mongo - An aggregation to order by objects in an array

I have the following data:
{
"_id": ObjectID("5e2fa881c3a1a70006c5743c"),
"name": "Some name",
"policies": [
{
"cId": "dasefa-2738-4cf0-90e0d568",
"weight": 12
},
{
"cId": "c640ad67dasd0-92f981583568",
"weight": 50
}
]
}
I'm able to query this with Spring Mongo fine, however I want to be able to order the policies by weight
At the moment I get my results fine with:
return mongoTemplate.find(query, CArea::class.java)
However say I make the following aggregations:
val unwind = Aggregation.unwind("policies")
val sort = Aggregation.sort(Sort.Direction.DESC,"policies.weight")
How can I go and actually apply those to the returned results above? I was hoping that the dot annotation would do the job in my query however didnt do anything e.g. Query().with(Sort.by(options.sortDirection, "policies.weight"))
Any help appreciated.
Thanks.
I am not familier with Spring Mongo, but I guess you can convert the following aggregation to spring code.
db.collection.aggregate([
{
$unwind: "$policies"
},
{
$sort: {
"policies.weight": -1
}
},
{
$group: {
_id: "$_id",
"policies": {
"$push": "$policies"
},
parentFields: {
$first: "$$ROOT"
}
}
},
{
$replaceRoot: {
newRoot: {
$mergeObjects: [
"$parentFields",
{
policies: "$policies"
}
]
}
}
}
])
This will result:
[
{
"_id": "5e2fa881c3a1a70006c5743c",
"name": "Some name",
"policies": [
{
"cId": "c640ad67dasd0-92f981583568",
"weight": 50
},
{
"cId": "dasefa-2738-4cf0-90e0d568",
"weight": 12
}
]
}
]
Playground

Resources