UPDATE: I can actually change the indexPrefix using the below code, but the actual _index which is used to filter in Kibana gets its name from the original indexPrefix. It seems changing the indexPrefix in the transformer method is too late, because the _index has already been created with the original prefix.
I'm using winston and winston-elasticsearch in a nodejs/express setup and want to use the same logger to log to different indexes (different indexPrefix)
const logger = winston.createLogger({
transports
});
The transports is an array of different transports. One of them is an ElasticsearchTransport that takes in some parameters like indexPrefix and level among others. The level can be changed based on the type of log by passing in a transformer method as parameter.
new winstonElastic.ElasticsearchTransport({
level: logLevel,
indexPrefix: getIndex(),
messageType: 'log',
ensureMappingTemplate: true,
mappingTemplate: indexTemplateMapping,
transformer: (logData: any) => {
const transformed: any = {};
transformed['#timestamp'] = logData.timestamp ? logData.timestamp : new Date().toISOString();
transformed.message = logData.message;
transformed.level = parseWinstonLevel(logData.level);
transformed.fields = _.extend({}, staticMeta, logData.meta);
transformed.indexPrefix = getIndex(logData.indexPrefix);
return transformed;
},
The transformer method is called whenever the logger writes a new entry and i can verify that it works by setting properties like message. It also overwrites the level to whatever the current log level is. For some reason it doesn't work on the property indexPrefix - even when it changes, nothing overwrites the initial indexPrefix. I even tried to remove the initial value but then the logging fails, having never set the indexPrefix.
Does anyone know why? Does it have to do with the mappingTemplate listed below?:
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"index": {
"refresh_interval": "5s"
}
},
"mappings": {
"properties": {
"#timestamp": { "type": "date" },
"#version": { "type": "keyword" },
"message": { "type": "text" },
"severity": { "type": "keyword" },
"fields": {
"dynamic": true,
"properties": {}
}
}
}
}
Ok, if anyone is interrested. I ended up making a loggerFactory instead. I create a logger seeded with the correct indexPrefix through the factory - that way I end up with one logger instance per indexPrefix i want...
For those who are having the same problem i solved this in a different way.
1 - I created a variable inside ElasticsearchTransport scope;
2 - I changed the value of the variable inside the transformer method;
3 - I used the variable inside the indexPrefix method to define which prefix to use:
indexPrefix: () => return variable ? 'test1' : 'test2'
Related
I have a requirement to throw different error in case of different scenarios like below, and there are many such fields not just 1.
e.g.
{
"id": 1,
"name": "nameWithSpecialChar$"
}
Here it should throw error for special character.
{
"id": 1,
"name": null
}
Here throw field null error.
{
"id": 1
}
Here throw field missing error.
Handling, 1st and 2nd scenario is easy, but for 3rd one, is there any way we can have a List of name of fields that were passed in input json at the time of serialization itself with Jackson?
One way, I am able to do it is via mapping request to JsonNode and then check if nodes are present for required fields and after that deserialize that JsonNode manually and then validate rest of the members as below.
public ResponseEntity myGetRequest(#RequestBody JsonNode requestJsonNode) {
if(!requestJsonNode.has("name")){
throw some error;
}
MyRequest request = ObjectMapper.convertValue(requestJsonNode, MyRequest .class);
validateIfFieldsAreInvalid(request);
But I do not like this approach, is there any other way of doing it?
You can define a JSON schema and validate your object against it. In your example, your schema may look like this:
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"id": {
"description": "The identifier",
"type": "integer"
},
"name": {
"description": "The item name",
"type": "string",
"pattern": "^[a-zA-Z]*$"
}
},
"required": [ "id", "name" ]
}
To validate your object, you could use the json-schema-validator library. This library is built on Jackson. Since you're using Spring Boot anyway, you already have Jackson imported.
The example code looks more or less like this:
String schema = "<define your schema here>";
String data = "<put your data here>";
JsonSchemaFactory factory = JsonSchemaFactory.byDefault();
ObjectMapper m = new ObjectMapper();
JsonSchema jsonSchema = factory.getJsonSchema(m.readTree(schema));
JsonNode json = m.readTree(data);
ProcessingReport report = jsonSchema.validate(json);
System.out.println(report);
The report includes detailed errors for different input cases. For example, with this input
{
"id": 1,
"name": "nameWithSpecialChar$"
}
this output is printed out
--- BEGIN MESSAGES ---
error: ECMA 262 regex "^[a-zA-Z]*$" does not match input string "nameWithSpecialChar$"
level: "error"
schema: {"loadingURI":"#","pointer":"/properties/name"}
instance: {"pointer":"/name"}
domain: "validation"
keyword: "pattern"
regex: "^[a-zA-Z]*$"
string: "nameWithSpecialChar$"
--- END MESSAGES ---
Or instead of just printing out the report, you can loop through all errors and have your specific logic
for (ProcessingMessage message : report) {
// Add your logic here
}
You could check the example code to gain more information about how to use the library.
I want to create a custom angular schematic that can accept a collection of action names. I will then generate 3 ngrx actions for every action name provided from the user.
for example I want to create a schematic that can be invoked like this:
ng g my-collection:my-schematic --actions=GetById,GetByFirstName
Then I'll generate code for GetById, GetByIdSuccess, GetByIdError, GetByFirstName, GetByFirstNameSuccess, GetByFirstNameError.
The issue is I've only seen angular schematics that will accept a single value as an input parameter. Anyone know how to handle collections in a custom angular schematic?
you can follow this blog, it will teach you how to create your own schematics project:
https://blog.angular.io/schematics-an-introduction-dc1dfbc2a2b2
after you generate your schematics project in file collection.json you can extend the #ngrx/schematics:
{
...
"extends": ["#ngrx/schematics"],
}
and then use the ngrx schematics to generate 3 actions like this:
externalSchematic('#ngrx/schematics', 'action')
I haven't found a good example of how to an array of string into a schematic parameter, but I found a workaround. What I did was have a simple string input parameter that is consistently delimited (I used , to delimit the values). Here is my schema:
export interface Schema {
name: string;
path?: string;
project?: string;
spec?: boolean;
actions: string;
__actions: string[];
store?: string;
}
I parse the actions param that is provided and generate the string array __actions and use that property in my templates. Here is a snippet from my index.ts:
export default function(options: ActionOptions): Rule {
return (host: Tree, context: SchematicContext) => {
options.path = getProjectPath(host, options);
const parsedPath = parseName(options.path, options.name);
options.name = parsedPath.name;
options.path = parsedPath.path;
options.__actions = options.actions.split(',');
options.__actions = options.__actions.map(_a => classify(_a));
If you know of a better way to process these please share. Thanks!
You need to pass the actions multiple times.
ng g my-collection:my-schematic --actions=GetById --actions=GetByFirstName
Define the parameter as an array within your schema.json file.
...
"actions": {
"type": "array",
"items": {
"type": "string"
},
"description": "The name of the actions."
},
...
Also in your schema.ts.
export interface Schema {
actions: string[]
}
If you want to pull them right off of the command args, you can do the following:
{
"$schema": "http://json-schema.org/draft-07/schema",
"$id": "Sample",
"title": "Sample Schematic",
"type": "object",
"description": "Does stuff.",
"additionalProperties": false,
"properties": {
"things": {
"type": "array",
"items": {
"type": "string"
},
"$default": {
"$source": "argv"
},
"description": "Things from the command-line args."
}
}
}
Then when you run your schematic you can do:
schematics schematic-lib:sample stuff other-stuff more-stuff
In this case, the things property will be ['stuff', 'other-stuff', 'more-stuff'].
Edit: Note that the required value in the schema won't cause the schematic to fail if you don't provide any args. You'd need to do validation on that property in your schematic.
I have the problem with merging splitted FlowFiles. let me explain the problem step by step.
This is my sequence of processors.
In Elasticsearch I have this index and mapping:
PUT /myindex
{
"mappings": {
"myentries": {
"_all": {
"enabled": false
},
"properties": {
"yid": {"type": "keyword"},
"days": {
"properties": {
"Type1": { "type": "date" },
"Type2": { "type": "date" }
}
},
"directions": {
"properties": {
"name": {"type": "keyword"},
"recorder": { "type": "keyword" },
"direction": { "type": "integer" }
}
}
}
}
}
}
I get directions from Elasticsearch using QueryElasticsearchHTTP and then I split directions into using SplitJson in order to get 10 FlowFiles. Each FlowFile has this content: {"name": "X","recorder": "X", "direction": "X"}
After this, for each of 10 FlowFiles I generate the attribute filename using UpdateAttribute and ${UUID()}. Then, I enrich each FlowFile with some constant data from ElasticSearch. In fact, the data that I merge to each FlowFile is the same. Therefore, ideally, I would like to run Get constants from Elastic only once instead of running it 10 times.
But anyway the key problem is different. FlowFiles that come from Gets constants from Elastic have other values of filename and they cannot be merged with the files that come from Set the attribute "filename". I also tried to use EvaluateJsonPath, but had the same problem. Any idea of how to solve this problem?
UPDATE:
The Groovy code used in Merge inputs.... I am not sure if it works when in the input queues come batches of 10 and 10 files that should be merged:
import org.apache.nifi.processor.FlowFileFilter;
import groovy.json.JsonSlurper
import groovy.json.JsonBuilder
//get first flow file
def ff0 = session.get()
if(!ff0)return
def filename = ff0.getAttribute('filename')
//try to find files with same attribute in the incoming queue
def ffList = session.get(new FlowFileFilter(){
public FlowFileFilterResult filter(FlowFile ff) {
if( filename == ff.getAttribute('filename') )return FlowFileFilterResult.ACCEPT_AND_CONTINUE
return FlowFileFilterResult.REJECT_AND_CONTINUE
}
})
//let's assume you require two additional files in queue with the same attribute
if( !ffList || ffList.size()<1 ){
session.rollback(false)
return
}
//let's put all in one list to simplify later iterations
ffList.add(ff0)
//create empty map (aka json object)
def json = [:]
//iterate through files parse and merge attributes
ffList.each{ff->
session.read(ff).withStream{rawIn->
def fjson = new JsonSlurper().parse(rawIn)
json.putAll(fjson)
}
}
//create new flow file and write merged json as a content
def ffOut = session.create()
ffOut = session.write(ffOut,{rawOut->
rawOut.withWriter("UTF-8"){writer->
new JsonBuilder(json).writeTo(writer)
}
} as OutputStreamCallback )
//set mime-type
ffOut = session.putAttribute(ffOut, "mime.type", "application/json")
session.remove(ffList)
session.transfer(ffOut, REL_SUCCESS)
I have model like this:
#Getter
#Setter
#Document(indexName = "indexName", type = "typeName")
#Setting(settingPath = "/elastic/elastic-setting.json")
public class Model extends BaseModel {
#Field(type = FieldType.String, index = FieldIndex.analyzed, analyzer = "customAnalyzer")
private String name;
}
And i have elastic-setting.json inside ../resources/elastic/elastic-setting.json:
{
"index": {
"number_of_shards": "1",
"number_of_replicas": "0",
"analysis": {
"analyzer": {
"customAnalyzer": {
"type": "custom",
"tokenizer": "uax_url_email"
}
}
}
}
}
I clean my elastic DB and when i start my application i have exception:
MapperParsingException[analyzer [customAnalyzer] not found for field [name]]
What's wrong with my code?
Help me, please!
EDIT
Val, I thought #Setting is like an addition for #Document, but looks like they are interchangeably.
In my case i also have another model, with:
#Document(indexName = "indexName", type = "anotherTypeName")
So, first i create index with name "indexName" for anotherModel, next when elastic preparing Model, it see, that index with name "indexName" already created, and he does not use #Setting.
Now i have another quesion.
How to add custom analyzer to already created index in java code, for example in InitializingBean. Something like - is my analyzer created? no - create. yes - do not create.
Modify your elastic-setting.json file like this:
{
"index": {
"number_of_shards": "1",
"number_of_replicas": "0"
},
"analysis": {
"analyzer": {
"customAnalyzer": {
"type": "custom",
"tokenizer": "uax_url_email"
}
}
}
}
}
Note that you need to delete your index first and recreate it.
UPDATE
You can certainly add a custom analyzer via Java code, however, you won't be able to change your existing mapping in order to use that analyzer, so you're really better off wiping your index and recreating it from scratch with a proper elastic-setting.json JSON file.
For Val:
Yeah, i use something like this.
Previously, i had added #Setting in one of my entity class, but when i started app, index with same name was already created, before Spring Data had analysed entity with #Setting, and index was not modified, because index with same name was already created.
Now I add annotation #Setting(path = "elastic-setting.json") on abstract baseModel, and high level hierarchy class was scanned firstly and analyzer was created as well.
Use case :
The index/indices will be built dynamically from a template, so that
it will pick some custom settings (number of shards/replicas/ etc).
Generate dynamically the types and their mappings, enabling for them all the timestamp
& ttl fields. Also define the same parent type for all types, except
the parent type itself (I know its name).
{
"template": "*",
"settings": {
...................
},
"mappings": {
"_default_": {
"_parent": {
"type": "ParentType"
},
"_timestamp": {
"enabled": true,
"store": true
},
"_ttl": {
"enabled": true
}
}
}
}
How could I disable the _parent field for the ParentType type itself ?
After a little big of digging on the docs, I found out that you can override __default__ mappings using this work around.
{
"mappings": {
"_default_": {
"_parent": {
"type": "my_parent_type" # 1
}
},
"my_parent_type": {
"_parent": {
"type": "_dummy" # 2
}
}
}
}
1) This mapping configuration will be inherited by all your types including my_parent_typetype. Step 2 explains how to override your parent type _parent meta-field.
2) I found here https://www.elastic.co/guide/en/elasticsearch/reference/current/default-mapping.html that you can override any of the mappings from __default__ when you provide a specific mapping for the type in question.
In this case we are assigning the _parent mapping of my_parent_type to a _dummy parent type that does not have to exists in your index.
Note: ElasticSearch 2.x documentation states:
The _parent.type setting can only point to a type that doesn’t exist yet. This means that a type cannot become a parent type after it is has been created.
For this reason, you have to make sure you index first, a document that is not my_parent_type type, otherwise you'll get an at indexing time.
I hope this helps.