I'm creating a winston-elasticsearch logger usin Kibana, where I need to save all what is happen in the app (request, apis, ...).I used this( https://github.com/vanthome/winston-elasticsearch) I defined a json mapping template. Also a transformer.js file where I define a Transformer function to transform log data as provided by winston into a message structure which is more appropriate for indexing in ES.
then I use the winston-elasticsearch to create new Es instance:
exports.mappingTemplate = require('../../../index-template-mapping.json');
const TransportersElastic = new Elasticsearch({
index: 'soapserver',
level: 'info',
indexPrefix: 'logs',
transformer: transformer,
ensureMappingTemplate: true,
mappingTemplate: mappingTemplate,
flushInterval: 2000,
waitForActiveShards: 1,
handleExceptions: false,...
But I keep getting Elasticsearch index error of type: 'mapper_parsing_exception' reasons like : failed to parse field [fields.result.status] of type [text]'
illegal_argument_exception' reason : 'failed to parse field [fields.rows.tmcode] of type [long]'
Here transformer.js:
exports.mappingTemplate = require('../../../index-template-mapping.json');
exports.transformer = (logData) => {
const transformed = {};
transformed['#timestamp'] = new Date().toISOString();
transformed.source_host = os.hostname();
transformed.message = logData.message;
if (typeof transformed.message === 'object') {
transformed.message = JSON.stringify(transformed.message);
I need help, suggestion how to resolve those error and succeed the Mapping.
Here the index-mapping-template.json
{
"mapping": {
"_doc": {
"properties": {
"#timestamp": {
"type": "date"
},
"fields": {
"ignore_malformed": true,
"dynamic": "true",
"properties": {}
},
"message": {
"type": "text"
},
"severity": {
"type": "keyword"
}
}
}
}
}
Related
I'm trying to validate JSON response messages according to schema.
But as soon as an attribute has the type "array" or "object", AJV seems to ignore them and does not see the nested attribute within that object or array.
I started trying to break up the schema into multiple smaller ones so all objects/arrays are covered, but this is a tedious task and I'm hoping I'm just using AJV incorrectly.
Example schema (simplified):
{
"type": "object",
"required": [
"MainThing"
],
"properties": {
"MainThing": {
"type": "string",
"description": "Some kind of string.",
"example": "This is some kind of string."
},
"NestedThing": {
"type": "object",
"required": [
"NestedAttribute"
],
"properties": {
"NestedAttribute": {
"type": "String",
"description": "A nested string.",
"example": "This is a nested string"
}
}
}
}
}
If I change the NestedAttribute to an Integer instead of a String, the schema validation in Postman will tell me the response is valid against the schema. While it's not, because the response will contain a NestedAttribute which is a String value.
The following response would be deemed valid, while it's not:
{
"MainThing": "The main thing",
"NestedThing": {
"NestedAttribute": 12345
}
}
My Postman testcode:
var Ajv = require('ajv'),
ajv = new Ajv({logger: console}),
schema = JSON.parse(pm.variables.get("THIS_IS_MY_AWESOME_SCHEMA"));
pm.test('My schema is valid', function() {
var data = pm.response.json();
pm.expect(ajv.validate(schema, data)).to.be.true;
});
If anyone can point me in the right direction, that would be greatly appreciated!
In default elastic search only returns 10k results. But I need to go to the last page which exceeds 10k results.
I did some reach and found a solution by setting "max_result_window" : 100000
And I execute it in Kibana and even more thanx 5000pages works fine after this setting.
PUT jm-stage-products/_settings
{
"max_result_window" : 100000
}
Now I need to include this setting when I'm creating an index in my source code.But I coundn't find a way to do it.
This is my index create function. How should I set "max_result_window" : 100000?
public string InitIndexing()
{
var indexName = string.Format(_config.ElasticIndexName, _config.HostingEnvironment);
//-----------------------------------------------------------
if (!_client.Indices.Exists(indexName).Exists)
{
//----------------------------------------------
var indexSettings = new IndexSettings
{
NumberOfReplicas = 0, // If this is set to 1 or more, then the index becomes yellow.
NumberOfShards = 5,
};
var indexConfig = new IndexState
{
Settings = indexSettings
};
var createIndexResponses = _client.Indices.Create(indexName, c => c
.InitializeUsing(indexConfig)
.Map<ElasticIndexGroupProduct>(m => m.AutoMap())
);
return createIndexResponses.DebugInformation;
}
else
{
return $"{_config.ElasticIndexName} already exists";
}
}
You can create an index with max_result_window setting with following code snippet:
var createIndexResponse = await elasticClient.Indices.CreateAsync("index_name", c => c
.Settings(s => s
.Setting(UpdatableIndexSettings.MaxResultWindow, 100000)))
Already existing index can be updated with this fluent syntax:
await elasticClient.Indices.UpdateSettingsAsync("index_name", s => s
.IndexSettings(i => i.Setting(UpdatableIndexSettings.MaxResultWindow, 100000)));
I don't know the nest but it can be easily done while creating the index and
below is the REST API to show that.
PUT HTTP://localhost:9200/your-index-name/
{
"settings": {
"max_result_window" : 1000000 // note this
},
"mappings": {
"properties": {
"first_name": {
"type": "text"
},
"last_name": {
"type": "text"
},
"country": {
"type": "text"
},
"state": {
"type": "text"
},
"city": {
"type": "text"
}
}
}
}
Check if its created successfully in index-settings
GET HTTP://localhost:9200/your-index-name/_settings
{
"so_1": {
"settings": {
"index": {
"number_of_shards": "1",
"provided_name": "so_1",
"max_result_window": "1000000", // note this
"creation_date": "1601273239277",
"number_of_replicas": "1",
"uuid": "eHBxaGf2TBG9GdmG5bvwkQ",
"version": {
"created": "7080099"
}
}
}
}
}
In kibana it looks like this
PUT index_name/_settings
{
"max_result_window": 10000
}
I'm not using BulkIndex request in my code.
Below is the Jest code used for indexing the document.
Builder builder = new Index.Builder(source).index(indexName).type(typeName)
.id(optionalDocId == null ? UUID.randomUUID().toString() : optionalDocId);
datatype of source is Map
Caused by: java.lang.Exception: ES Error Message while writing to raw: {"root_cause":[{"type":"remote_transport_exception","reason":"[datanodes-364022667-13-482069107.azure-ebf.opsds.cp.prod-az-westus-1.prod.us.walmart.net][10.12.10.171:9300][indices:data/write/bulk[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of processing of [557850852][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[sceventstorev1][15]] containing [index {[SCEventStore][scevents][93adca0b-7404-4405-8f72-9fa5e32a167c], source[n/a, actual length: [2.1kb], max length: 2kb]}], target allocation id: iTbBHe7vT_ihTHdJwqVRhA, primary term: 7 on EsThreadPoolExecutor[name = datanodes-364022667-13-482069107.azure-ebf.opsds.cp.prod-az-westus-1.prod.us.walmart.net/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor#4d2bc57e[Running, pool size = 8, active threads = 8, queued tasks = 200, completed tasks = 148755948]]"}
Given is the index mapping.
{
"sceventstore_v1": {
"mappings": {
"scevents": {
"properties": {
"eventID": {
"type": "keyword"
},
"eventId": {
"type": "keyword"
},
"eventName": {
"type": "keyword"
},
"message": {
"type": "text"
},
"producerName": {
"type": "keyword"
},
"receivedTimestamp": {
"type": "date",
"format": "epoch_millis"
},
"timestamp": {
"type": "date",
"format": "epoch_millis"
}
}
}
}
}
}```
The section of error message clearly explains the cause of it.
source[n/a, actual length: [2.1kb], max length: 2kb]}], it means max length allowed is 2kb somewhere configured in your application and you are sending more than which is 2.1 kb.
Try to inspect your document and see which field value is crossing this limit.
More resource can be found https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster and https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html
I have a C# object like so:
public class Foo {
public decimal Number {get;set;}
}
And I create a json schema like so:
var schema = await JsonSchema4.FromTypeAsync(typeof(Foo));
var jsonSchema = schema.ToJson();
return jsonSchema;
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Foo",
"type": "object",
"additionalProperties": false,
"properties": {
"SomeNumber": {
"type": "number",
"format": "decimal"
}
}
}
How do I prevent the property "format": "decimal" from outputting in the resulting schema?
edit to add: Without modifying the class Foo -- I don't have access to it.
I'm working in a project with Elasticsearch and Spring Data Elasticsearch.
I need to get the mapping of an object type of my index. My #document class looks like:
#Document(indexName = "esbsdocuments", type = ESBSDocumentEls.MAPPING_TYPE)
public class ESBSDocumentEls extends ESBSDomainModel {
...
#Field(type =FieldType.Object, store = false)
private Object metadata;
...
}
If I try to get it via http://xxx:9200/_mapping I can get the mapping for "metadata" field correctly:
...
"metadata": {
"properties": {
"APP": {
"type": "string"
},
"APPDST": {
"type": "string"
},
"APPSUB": {
"type": "string"
},
"COUNTSUB": {
"type": "string"
},
"DOMINIO": {
"type": "string"
},
"DUPLICATE": {
"type": "string"
},
"EXCLUDEFIELDNAMES": {
"type": "string"
},
"FECHA": {
"type": "string"
},
"ID": {
"type": "string"
},
"IDF": {
"type": "string"
},
"IDSUB": {
"type": "string"
},
"LOCALEDATE": {
"type": "string"
},
"MENSAJE": {
"type": "string"
},
"TAMANYO": {
"type": "string"
},
"TIPO": {
"type": "string"
},
"VERSION": {
"type": "string"
}
}
},
...
But when I try it in code with
Map mapping = elasticsearchTemplate.getMapping(ESBSDocumentEls.class);
I can only get:
... (definition of metadata dynamic templates)
metadata={type=object}
...
How can I get the detailed mapping definition using ElasticSearchTemplate or another Spring Data Elasticsearch class??
Thank you very much!
Finally I got a solution.
Instead of using elasticSearchTemplate, I have used the els java api:
GetMappingsResponse res = elasticSearchClient.admin().indices().getMappings(new GetMappingsRequest().indices(indexName)).get();
ImmutableOpenMap<String, MappingMetaData> indexMappings = res.mappings().get(indexName);
MappingMetaData mappingMetaData = indexMappings.get(mappingType);
Map<String, Object> map = mappingMetaData.getSourceAsMap();
Map<String, Object> metadataProperties = (Map<String, Object>) ((Map<String, Object>) map.get("properties")).get("metadata");
Map<String, Object> metadataList = (Map<String, Object>) metadataProperties.get("properties");
This way I can get a Map with all my "metadata" fields. You can't get the api client from the elasticSearchTemplate (https://jira.spring.io/browse/DATAES-124) so you have to inject it :(
I hope this help someone!
You can now use the public <T> Map getMapping(Class<T> clazz) method which is already available in the template. It returns a map of fields and types.
elasticSearchTemplate.getMapping(ESBSDocumentEls.class);