Ok, here is my problem:
At backend I have java spring boot, with mongodb and elasticsearch.
I have documents structured like this in my elasticsearch:
{ "id": 1,
"prefixes": [
{
"prefName": "RS",
"value": "c"
},
{
"prefName": "99c",
"value": "c"
},
{
"prefName": "RT",
"value": "a"
},
{
"prefName": "001b",
"value": "a"
},
{
"prefName": "DT",
"value": "m"
},
{
"prefName": "001c",
"value": "m"
},
{
"prefName": "001d",
"value": "b"
},
{
"prefName": "RN",
"value": "16"
},
{
"prefName": "001e",
"value": "16"
},
{
"prefName": "BN",
"value": "2-228-32590-2"
},
{
"prefName": "010a",
"value": "2-228-32590-2"
},
{
"prefName": "SS",
"value": "d"
},
{
"prefName": "100b",
"value": "d"
},
{
"prefName": "PY",
"value": "1975"
},
{
"prefName": "100c",
"value": "1975"
},
{
"prefName": "LA",
"value": "fre"
},
{
"prefName": "101a",
"value": "fre"
},
{
"prefName": "CO",
"value": "fra"
},
{
"prefName": "102a",
"value": "fra"
},
{
"prefName": "KW",
"value": "Ekole et psychologie individuelle comparee"
}] }
From client app I'm passing object with this structure:
{
"pref1": "(prefixName - value)",
"pref2": "(prefixName - value)",
"pref3": "(prefixName - value)",
"pref4": "(prefixName - value)",
"pref5": "(prefixName - value)",
"text1": "(prefix value - value)",
"text2": "(prefix value - value)",
"text3": "(prefix value - value)",
"text4": "(prefix value - value)",
"text5": "(prefix value - value)",
"operator1": "(logical OR,AND,NOT)",
"operator2": "(logical OR,AND,NOT)",
"operator3": "(logical OR,AND,NOT)",
"operator4": "(logical OR,AND,NOT)"
}
So, I have 5 optional prefixes to chose( pref1, pref2... for example RS, RT..),
text fields are their values and should be matched by phrase prefix. The problem is that I have 4 logical operators that links those queries. Can someone direct me how to structure adequate elasticsearch query to fulfill these needs.
Search form img
Here is the search form and the result should be documet which:
Has prefix AU - Sheakspeare* AND prefix 041a - 99c OR prefix RN - 23 AND prefix LA NOT 23.
Related
I have a json data which is in the form of key and all values in a array but I need to transform it into a array of key value pairs, here is the data
Source data
"2022-08-30T06:58:56.573730Z": [
{ "tag": "AC 3 Phase/7957", "value": 161.37313113545272 },
{ "tag": "AC 3 Phase/7956", "value": 285.46869739695853 }
]
}
Transformation looking for
[
{ "tag": "AC 3 Phase/7957",
"ts": 2022-08-30T06:58:56.573730Z,
"value": 161.37313113545272
},
{ "tag": "AC 3 Phase/7956",
"ts": 2022-08-30T06:58:56.573730Z,
"value": 285.46869739695853
}
]
I would do it like this:
$each($$, function($entries, $ts) {
$entries.{
"tag": tag,
"ts": $ts,
"value": value
}
}) ~> $reduce($append, [])
Feel free to play with this example on the playground: https://stedi.link/g6qJGcP
I'm new to Clickhouse. I'm trying to read Jaeger logs from Kafka into Clickhouse db. I have following Kafka messages format:
{
"traceId": "omFv9AGFHOAfWQ+tJcxDZQ==",
"spanId": "Lai3jc8v6Pg=",
"operationName": "GET",
"startTime": "2021-02-24T09:26:12.771Z",
"duration": "0.006s",
"tags": [
{
"key": "component",
"vStr": "kong"
},
{
"key": "http.method",
"vStr": "GET"
},
{
"key": "http.path",
"vStr": "/static/bootstrap/js/bootstrap.min.js"
},
{
"key": "http.status_code",
"vStr": "304"
},
{
"key": "kong.node.id",
"vStr": "0b2d3a89-67c8-43b2-a56f-450c45689b7d"
},
{
"key": "peer.service"
},
{
"key": "peer.ipv4",
"vType": "INT64",
"vInt64": "2130706433"
},
{
"key": "peer.port",
"vType": "INT64",
"vInt64": "44076"
},
{
"key": "span.kind",
"vStr": "server"
},
{
"key": "internal.span.format",
"vStr": "zipkin"
}
],
"logs": [
{
"timestamp": "2021-02-24T09:26:12.771Z",
"fields": [
{
"key": "event",
"vStr": "krf"
}
]
}
],
"process": {
"serviceName": "kong"
}
}
I was able to input traceID, spanID and Operation into Clickhouse using the following table:
CREATE TABLE IF NOT EXISTS db1.jaeger
(
traceId String,
spanId String,
operationName String
) ENGINE = Kafka()
SETTINGS
kafka_broker_list = 'broker.net:9092',
kafka_topic_list = 'jaeger',
kafka_group_name = 'sample_group',
kafka_format = 'JSONEachRow';
But I failed to input tags. Any idea which Clickhouse data type I should use for it?
I try to implement NLog into my .NET Core Api web service.
I want to log to an Oracle database. All works well through an nlog.config XML file.
But the goal is to implement NLog config into appsettings.json and here problem occurs.
I get the error set in title:
LayoutRenderer cannot be found: 'aspnet-user-identity
My config file is like this :
"NLog": {
"autoReload": true,
"throwConfigExceptions": true,
"internalLogLevel": "info",
"internalLogFile": "c:/app/log/dev/internal-appsetting-nlog.txt",
"extensions": {
"NLog.Extensions.Logging": {
"assembly": [
"NLog.Extensions.Logging",
"NLog.Web.AspNetCore"
]
}
},
"variables": {
"var_logdir": "c:/app/log/dev"
},
"default-wrapper": {
"type": "AsyncWrapper",
"overflowAction": "Block"
},
"targets": {
"all-file": {
"type": "File",
"fileName": "${var_logdir}/nlog-all-${shortdate}.log",
"layout": {
"type": "JsonLayout",
"Attributes": [
{
"name": "timestamp",
"layout": "${date:format=o}"
},
{
"name": "level",
"layout": "${level}"
},
{
"name": "logger",
"layout": "${logger}"
},
{
"name": "message",
"layout": "${message:raw=true}"
},
{
"name": "properties",
"encode": false,
"layout": {
"type": "JsonLayout",
"includeallproperties": "true"
}
}
]
}
},
"db": {
"type": "Database",
"commandText": "INSERT INTO logtable (LOGLEVEL,LOGGER,MESSAGE,MACHINENAME,USERNAME,CALLSITE, THREADID,EXCEPTIONMESSAGE,STACKTRACE,SESSIONID) VALUES (:pLEVEL,:pLOGGER,:pMESSAGE,:pMACHINENAME, :pCALLSITE,:pTHREADID,:pEXCEPTIONMESSAGE,:pSTACKTRACE)",
"parameters": [
{
"name": "#pLEVEL",
"layout": "${level}"
},
{
"name": "#pLOGGER",
"layout": "${logger}"
},
{
"name": "#pMESSAGE",
"layout": "${message}"
},
{
"name": "#pMACHINENAME",
"layout": "${machinename}"
},
{
"name": "#pUSERNAME",
"layout": "${aspnet-user-identity}"
},
{
"name": "#pCALLSITE",
"layout": "${callsite:filename=true}"
},
{
"name": "#pTHREADID",
"layout": "${threadid}"
},
{
"name": "#pEXCEPTIONMESSAGE",
"layout": "${exception}"
},
{
"name": "#pSTACKTRACE",
"layout": "${stacktrace}"
},
{
"name": "#pSESSIONID",
"layout": "${aspnet-sessionid}"
}
],
"dbProvider": "Oracle.ManagedDataAccess.Client.OracleConnection, Oracle.ManagedDataAccess",
"connectionString": "xxxxxxxxxxxx"
}
},
"rules": [
{
"logger": "*",
"minLevel": "Trace",
"writeTo": "all-file"
},
{
"logger": "*",
"minLevel": "Trace",
"writeTo": "db"
},
{
"logger": "Microsoft.*",
"maxLevel": "Info",
"final": true
}
]
},
The internal debugger reports:
2019-10-09 16:48:48.6665 Info Adding target AsyncTargetWrapper(Name=all-file)
2019-10-09 16:48:48.7859 Warn Error when setting property 'Layout' on 'NLog.Targets.DatabaseParameterInfo' Exception: System.ArgumentException: LayoutRenderer cannot be found: 'aspnet-user-identity'. Is NLog.Web not included?
at NLog.Config.Factory`2.CreateInstance(String itemName)
at NLog.Layouts.LayoutParser.GetLayoutRenderer(ConfigurationItemFactory configurationItemFactory, String name)
at NLog.Layouts.LayoutParser.ParseLayoutRenderer(ConfigurationItemFactory configurationItemFactory, SimpleStringReader stringReader)
at NLog.Layouts.LayoutParser.CompileLayout(ConfigurationItemFactory configurationItemFactory, SimpleStringReader sr, Boolean isNested, String& text)
at NLog.Layouts.SimpleLayout.set_Text(String value)
at NLog.Internal.PropertyHelper.TryNLogSpecificConversion(Type propertyType, String value, Object& newValue, ConfigurationItemFactory configurationItemFactory)
at NLog.Internal.PropertyHelper.SetPropertyFromString(Object obj, String propertyName, String value, ConfigurationItemFactory configurationItemFactory)
Error occurs on ${aspnet-sessionid}. If I comment out both layout, everything works well.
I found different things on GitHub issue report but all I tried was a fail.
Could someone help?
The unknown aspnet-user-identity is probably an issue with your extensions:
"extensions": [
{ "assembly": "NLog.Extensions.Logging" },
{ "assembly": "NLog.Web.AspNetCore" }
],
Could you try the above suggestion?
P.S. Updated the wiki to include example of multiple "extensions"
Given data model structure like this,
{
Id: 123,
"string_facet": [
{
"name": "make",
"value": "Audi"
},
{
"name": "carListType",
"value": "PERSON EU"
},
{
"name": "modelType",
"value": ""
},
{
"name": "engineBrand",
"value": "APT"
},
{
"name": "typeDescription",
"value": "8D2"
}
],
"number_facet": [
{
"name": "typeNumber",
"value": 4614
},
{
"name": "serialNumber",
"value": 2
},
{
"name": "engineSize",
"value": 18
},
{
"name": "horsePower",
"value": 125
},
{
"name": "kw",
"value": 92
},
{
"name": "engineVolume",
"value": 1781
},
{
"name": "listType",
"value": 0
}
],
"dateTime_facet": [
{
"name": "fromDate",
"value": "1999-04-01T00:00:00"
},
{
"name": "toDate",
"value": "2000-10-01T00:00:00"
}
]
}
I want to get aggregates facet names, and values per name. However, I'm only interested in facets that have specific names, such as: make and engineBrand. Note that facets are of type nested.
I have tried the following .NEST expression, but it still returns all of the facet names.
.Global("global", g => g
.Aggregations(ag => ag
.Filter("global_makes", f => f
.Filter(ff => ff
.Nested(n => n
.Path("string_facet")
.Filter(pf => pf.Term("string_facet.name", "make")))
)
.Aggregations(agg => agg
.Nested("nested_string_facet", nested => nested
.Path("string_facet")
.Aggregations(stringFacet => stringFacet
.Terms("name", nameAgg => nameAgg.Field("string_facet.name").Size(0)
.Aggregations(nameAggNext => nameAggNext
.Terms("value", valueAgg => valueAgg.Field("string_facet.value").Size(0))
)
)
)
)
)
)
)
)
);
I have a filter within global (to lose scope of a passed in query), and then filter only on string_facet.name which match "make", but results still include all other names as well. How do I filter out aggregation to include only buckets where name is "make"?
This helped. https://github.com/elastic/elasticsearch/issues/4449
Essentially had to move filter part deeper into aggregation.
We are trying to update a single answer in our sub sub array.
However our query is causing the following error:
{
"deleted": 0 ,
"errors": 1 ,
"first_error": "Inserted value must be an OBJECT (got ARRAY):
[
{
"answers": [
{
"answer": "wassup",
"owner": 12201836
}
],
"question": "Vraag 1?",
"questionId": 0,
"time": "10"
},
{
"answers": [],
"question": "Vraag 2?",
"questionId": 1,
"time": "15"
},
{
"answers": [],
"question": "Vraga 3?",
"questionId": 2,
"time": "20"
}
]" ,
"inserted": 0 ,
"replaced": 0 ,
"skipped": 0 ,
"unchanged": 0
}
Our table structure looks like the following:
Youtube
- Id
- Course
- Unit
- Session
- Number
- Group
- Questions (array)
- Question Id
- Time
- Answers (array)
- Id
- Answer
- Owner
Our query:
r.db('GitSmurf')
.table('youtube')
.update(function (row) {
return row('questions').merge(function (q) {
return r.branch(q('questionId').eq(0), { "answers": q('answers').merge(function(answer) {
return r.branch(answer('owner').eq(12201836), {"answer": "wassup"}, {})} )},{})
})
})
Test content:
{
"completed": [ ],
"course": "swd" ,
"group": "dwa-group-b" ,
"id": "44443377-ed15-4358-a005-f561e7b6a42d" ,
"number": 1 ,
"session": 1 ,
"unit": 1,
"questions": [
{
"answers": [
{
"answer": "hallo" ,
"owner": 12201836
}
] ,
"question": "Vraag 1?" ,
"questionId": 0 ,
"time": "10"
} ,
{
"answers": [ ],
"question": "Vraag 2?" ,
"questionId": 1 ,
"time": "15"
} ,
{
"answers": [ ],
"question": "Vraga 3?" ,
"questionId": 2 ,
"time": "20"
}
] ,
}
Any help is greatly appreciated!
We forgot to return a new object in the update query.
When we added that it worked.
r.db('GitSmurf')
.table('youtube')
.update(function (row) {
return { questions: row('questions').merge(function (q) {
return r.branch(q('questionId'), { "answers": q('answers').merge(function(answer) {
return r.branch(answer('owner').eq(12201836), {"answer": "tom"}, {})
})},{})
})}
})