I'm working in a project with Elasticsearch and Spring Data Elasticsearch.
I need to get the mapping of an object type of my index. My #document class looks like:
#Document(indexName = "esbsdocuments", type = ESBSDocumentEls.MAPPING_TYPE)
public class ESBSDocumentEls extends ESBSDomainModel {
...
#Field(type =FieldType.Object, store = false)
private Object metadata;
...
}
If I try to get it via http://xxx:9200/_mapping I can get the mapping for "metadata" field correctly:
...
"metadata": {
"properties": {
"APP": {
"type": "string"
},
"APPDST": {
"type": "string"
},
"APPSUB": {
"type": "string"
},
"COUNTSUB": {
"type": "string"
},
"DOMINIO": {
"type": "string"
},
"DUPLICATE": {
"type": "string"
},
"EXCLUDEFIELDNAMES": {
"type": "string"
},
"FECHA": {
"type": "string"
},
"ID": {
"type": "string"
},
"IDF": {
"type": "string"
},
"IDSUB": {
"type": "string"
},
"LOCALEDATE": {
"type": "string"
},
"MENSAJE": {
"type": "string"
},
"TAMANYO": {
"type": "string"
},
"TIPO": {
"type": "string"
},
"VERSION": {
"type": "string"
}
}
},
...
But when I try it in code with
Map mapping = elasticsearchTemplate.getMapping(ESBSDocumentEls.class);
I can only get:
... (definition of metadata dynamic templates)
metadata={type=object}
...
How can I get the detailed mapping definition using ElasticSearchTemplate or another Spring Data Elasticsearch class??
Thank you very much!
Finally I got a solution.
Instead of using elasticSearchTemplate, I have used the els java api:
GetMappingsResponse res = elasticSearchClient.admin().indices().getMappings(new GetMappingsRequest().indices(indexName)).get();
ImmutableOpenMap<String, MappingMetaData> indexMappings = res.mappings().get(indexName);
MappingMetaData mappingMetaData = indexMappings.get(mappingType);
Map<String, Object> map = mappingMetaData.getSourceAsMap();
Map<String, Object> metadataProperties = (Map<String, Object>) ((Map<String, Object>) map.get("properties")).get("metadata");
Map<String, Object> metadataList = (Map<String, Object>) metadataProperties.get("properties");
This way I can get a Map with all my "metadata" fields. You can't get the api client from the elasticSearchTemplate (https://jira.spring.io/browse/DATAES-124) so you have to inject it :(
I hope this help someone!
You can now use the public <T> Map getMapping(Class<T> clazz) method which is already available in the template. It returns a map of fields and types.
elasticSearchTemplate.getMapping(ESBSDocumentEls.class);
Related
I posted this question on the elastic forums, but I thought I should try it here as well. The problem is as follows:
Hi,
We have Elasticsearch with logstash (version 8.2). It is inserting data in elastic index from a jdbc source. In logstash we use an aggregate filter. The config looks like this:
jdbc {
jdbc_connection_string => "jdbc:oracle:thin:#wntstdb03.izaaksuite.nl:1521:wntstf2"
jdbc_user => "webnext_zaken"
jdbc_password => "webnext_zaken"
jdbc_driver_library => ""
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
statement_filepath =>"/appl/sw/webnext/logstash/config_documenten/queries/documenten.sql"
last_run_metadata_path => "/appl/sw/webnext/logstash/config_documenten/parameters/.jdbc_last_run_doc"
}
}
# The filter part of this file is commented out to indicate that it is
# optional.
filter {
aggregate {
task_id => "%{zaakdoc_id}"
code => "
map['zaak_id'] ||= event.get('zaak_id')
map['result_type'] ||= event.get('result_type')
map['mutatiedatum'] ||= event.get('mutatiedatum')
map['oge_id'] ||= event.get('oge_id')
map['zaakidentificatie'] ||= event.get('zaakidentificatie')
map['zaakomschrijving'] ||= event.get('zaakomschrijving')
map['titel'] ||= event.get('titel')
map['beschrijving'] ||= event.get('beschrijving')
map['zaakdoc_id'] ||= event.get('zaakdoc_id')
map['groepsrollenlijst'] ||= []
map['groepsrollenlijst'] << {'groepsrol' => event.get('rol')}
event.cancel()
"
push_previous_map_as_event => true
timeout => 5
}
}
output {
# stdout { codec => rubydebug }
# file {
# path => ["/appl/sw/webnext/logstash/config_documenten/output/documenten.txt"]
# }
elasticsearch {
hosts => ["localhost:9200"]
index => "documenten"
document_id => "%{zaakdoc_id}"
}
}
The index config looks like this:
{
"documenten": {
"aliases": {
"izaaksuite": {}
},
"mappings": {
"properties": {
"#timestamp": {
"type": "date"
},
"#version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"beschrijving": {
"type": "text"
},
"groepsrollenlijst": {
"properties": {
"groepsrol": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"mutatiedatum": {
"type": "date"
},
"oge_id": {
"type": "text"
},
"result_type": {
"type": "text"
},
"rol": {
"type": "text"
},
"tags": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"titel": {
"type": "text"
},
"zaak_id": {
"type": "text"
},
"zaakdoc_id": {
"type": "long"
},
"zaakidentificatie": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"zaakomschrijving": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"settings": {
"index": {
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
},
"number_of_shards": "1",
"provided_name": "documenten",
"creation_date": "1654158264412",
"number_of_replicas": "1",
"uuid": "bf4xj4TwQ-mP5K4Orc5HEA",
"version": {
"created": "8010399"
}
}
}
}
}
One document in the index that is eventually build, looks like this:
"_index": "documenten",
"_id": "25066386",
"_version": 1,
"_seq_no": 33039,
"_primary_term": 6,
"found": true,
"_source": {
"groepsrollenlijst": [
{
"groepsrol": "7710_AFH1"
},
{
"groepsrol": "7710_AFH2"
},
{
"groepsrol": "MR_GRP1"
}
],
"zaak_id": 44973087,
"oge_id": 98,
"#version": "1",
"#timestamp": "2022-07-11T08:24:07.717572Z",
"zaakdoc_id": 25066386,
"zaakomschrijving": "testOSiZaakAOS",
"result_type": "doc",
"titel": "Test4",
"zaakidentificatie": "077215353",
"mutatiedatum": "2022-06-27T09:51:52.078119Z",
"beschrijving": "Test4"
}
}
As you can see, the "groepsrollenlijst" is present. Now our problem: when searching we need to match one of the values in groepsrollenlijst (which is dutch for grouprole which is basically an autorisation within the application where the data is coming from) with the grouprole present on the user doing the search. This to prevent users to be able to have data in their search results they don't have acces to.
Our java code looks like this (sorry for the dutch sentences):
List<SearchResult> searchResults = new ArrayList<>();
SearchRequest searchRequest = new SearchRequest(index);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
BoolQueryBuilder fieldsQuery = new BoolQueryBuilder();
/*
* Haal per index alle velden op waarop gezocht kan en mag worden. We kunnen
* niet over alle velden zoeken omdat er dan ook hits voorkomen op de
* groepsrollenlijst (als je op bv op rutten zoekt worden er ook hits gevonden
* op groepsrol "RUTTENGROEP" wat je niet wilt) Ook bij documenten en
* betrokkenen wil je bv niet dat er hits gevonden worden op de
* zaakomschrijving.
*/
String indexFields = index + "Fields";
indexFields = indexFields.substring(0, 1).toUpperCase() + indexFields.substring(1);
List<String> fields = getFieldsFor(indexFields);
// Voeg per veld een query toe voor de ingegeven zoektekst
HighlightBuilder highlightBuilder = new HighlightBuilder();
QueryStringQueryBuilder queryStringQueryBuilder = new QueryStringQueryBuilder(autoCompleteText);
for (String field : fields) {
queryStringQueryBuilder.field(field);
highlightBuilder.field(field);
}
fieldsQuery.should(queryStringQueryBuilder);
// Manipuleer de roles tbv test
roles.clear();
roles.add("7710_AFH1");
roles.add("7710_AFH2");
BoolQueryBuilder rolesQuery = QueryBuilders.boolQuery();
for (String role : roles) {
rolesQuery.should(QueryBuilders.wildcardQuery("groepsrol", "*" + role + "*"));
}
LOG.info("Rollen medewerker: " + roles);
BoolQueryBuilder mainQuery = new BoolQueryBuilder();
mainQuery.must(new TermsQueryBuilder("oge_id", String.valueOf(ogeId)));
mainQuery.must(fieldsQuery);
mainQuery.must(rolesQuery);
searchSourceBuilder.query(mainQuery);
searchSourceBuilder.highlighter(highlightBuilder);
searchRequest.source(searchSourceBuilder);
searchRequest.validate();
// Execute search
LOG.info("Search query: {}", searchRequest.source().toString());
SearchResponse searchResponse = null;
try {
searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
} catch (IOException | ElasticsearchStatusException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return;
}
if (searchResponse == null) {
return;
}
SearchHits hits = searchResponse.getHits();
For the test we hardcoded the user's grouproles into the code.
The issue is that when we search for "testOSiZaakAOS" (one of the values in the document previously shown) which should be a hit, we don't get a result. If we comment out the "mainQuery.must(rolesQuery);" part, we do get a result. But then the roles are not taking into account.
How do we go about fixing this? So user has role x, some documents in the index have key-value pairs for role x, y and z. And some do have only y and z.
Search should only show those where role x is present.
Basically at least one of the roles of the user should match one of the roles present in the document in the index.
Your help is greatly appreciated! Let me know if you need more info.
I've a requirement to upload any file content using Swagger-3 along with some metadata information as a JSON within a single request. Therefore I configured following into my swagger:
"requestBody": {
"content": {
"multipart/form-data": {
"schema": {
"type": "object",
"properties": {
"metadata": {
"$ref": "#/components/schemas/Attachment"
},
"file": {
"type": "string",
"format":"binary",
"description": "Actual File Attachment"
}
}
}
}
},
"description": "The Attachment record / entry be created",
"required": true
}
It translates to following when I build the controller object:
#ApiOperation(value = "Upload attachments to a existing Ticket", nickname = "uploadAttachment", notes = "", response = Attachment.class, responseContainer = "List", tags={ "changeRequest", })
#RequestMapping(value = "/changeRequest/attachment/{id}",
produces = { "application/json" },
consumes = { "multipart/form-data" },
method = RequestMethod.POST)
public ResponseEntity<List<Attachment>> uploadAttachment(#ApiParam(value = "Identifier of the Change Request",required=true) #PathVariable("id") String id,#ApiParam(value = "Application ID invoking the call" ,required=true) #RequestHeader(value="X-App-Id", required=true) String xAppId,#NotNull #ApiParam(value = "To track unique transaction across multiple systems for audit trail", required = true) #Valid #RequestParam(value = "X-Transaction-Id", required = true) String xTransactionId,#ApiParam(value = "Authorization header" ) #RequestHeader(value="authorization", required=false) String authorization,#ApiParam(value = "", defaultValue="null") #RequestParam(value="metadata", required=false) Attachment metadata, #ApiParam(value = "file detail") #Valid #RequestPart("file") MultipartFile file) {
ResponseEntity<List<Attachment>> responseEntity = new ResponseEntity<>(HttpStatus.OK);
responseEntity.getBody().add(metadata);
return responseEntity;
}
Following is the Attachment schema definition:
"Attachment": {
"type": "object",
"description": "Attachment Metadata definition",
"properties": {
"description": {
"type": "string",
"description": "A narrative text describing the content of the attachment"
},
"href": {
"type": "string",
"description": "Reference of the attachment"
},
"id": {
"type": "string",
"description": "Unique identifier of the attachment"
},
"mimeType": {
"type": "string",
"description": "The mime type of the document as defined in RFC 2045 and RFC 2046 specifications."
},
"name": {
"type": "string",
"description": "The name of the file"
},
"path": {
"type": "string",
"description": "The path of the attached file"
},
"size": {
"type": "integer",
"description": "The size of the file (sizeUnit if present indicates the unit, otherwise kilobytes is the default)."
},
"sizeUnit": {
"type": "integer",
"description": "The unit size for expressing the size of the file (MB,kB...)"
},
"url": {
"type": "string",
"description": "Uniform Resource Locator, is a web page address (a subset of URI)"
},
"validFor": {
"$ref": "#/components/schemas/TimePeriod",
"description": "Period of validity of the attachment"
},
"#type": {
"type": "string",
"description": "The class type of the actual resource (for type extension)."
},
"#schemaLocation": {
"type": "string",
"description": "A link to the schema describing a resource (for type extension)."
},
"#baseType": {
"type": "string",
"description": "The base type for use in polymorphic collections"
}
}
}
In the example above, Attachment metadata is what I am trying to pass as part of the SOAP API test. However I keep getting following error:
Mon Oct 12 17:06:28 IST 2020:DEBUG:<< "{"timestamp":"2020-10-12T11:36:28.371Z","status":500,"error":"Internal Server Error","message":"Failed to convert value of type 'java.lang.String' to required type 'com.bell.na.nt.change.swagger.model.Attachment'; nested exception is java.lang.IllegalStateException: Cannot convert value of type 'java.lang.String' to required type 'com.bell.na.nt.change.swagger.model.Attachment': no matching editors or conversion strategy found","path":"/changeManagement/api/v1/changeRequest/attachment/1234"}"
Why is the string not being converted and mapped to the JSON object. Not sure I am missing anything. Following is what my json looks like.
{"#baseType": "string", "#schemaLocation": "string", "#type": "string", "description": "string", "href": "string", "id": "string", "mimeType": "string", "name": "string", "path": "string", "size": 0, "sizeUnit": 0, "url": "string", "validFor": { "endDateTime": "2020-10-11T19:06:40.586Z", "startDateTime": "2020-10-11T19:06:40.586Z"}}
Postman Request
Turns out I had to add the a converter for converting the string representation of the JSON to desired Swagger generated Model object something like:
#Component
public class StringToAttachmentObjectConverter implements Converter<String, Attachment> {
Logger logger = LoggerFactory.getLogger(StringToAttachmentObjectConverter.class);
#Autowired
private ObjectMapper objectMapper;
DocumentContext docContext = null;
#Override
public Attachment convert(String source) {
try {
String sourceString = JsonPath.using(NetsUtilityJSONDatumUtils.jsonPathConfig).parse(source).jsonString();
return objectMapper.readValue(sourceString, Attachment.class);
} catch (JsonParseException e) {
logger.error("Error While converting the String: \n" + source, e);
} catch (JsonMappingException e) {
logger.error("Error While converting the String: \n" + source, e);
} catch (IOException e) {
logger.error("Error While converting the String: \n" + source, e);
}
return null;
}
}
Not sure if there is any better way or if I am defying any best practice(s) but this did the trick for me.
I'm creating a winston-elasticsearch logger usin Kibana, where I need to save all what is happen in the app (request, apis, ...).I used this( https://github.com/vanthome/winston-elasticsearch) I defined a json mapping template. Also a transformer.js file where I define a Transformer function to transform log data as provided by winston into a message structure which is more appropriate for indexing in ES.
then I use the winston-elasticsearch to create new Es instance:
exports.mappingTemplate = require('../../../index-template-mapping.json');
const TransportersElastic = new Elasticsearch({
index: 'soapserver',
level: 'info',
indexPrefix: 'logs',
transformer: transformer,
ensureMappingTemplate: true,
mappingTemplate: mappingTemplate,
flushInterval: 2000,
waitForActiveShards: 1,
handleExceptions: false,...
But I keep getting Elasticsearch index error of type: 'mapper_parsing_exception' reasons like : failed to parse field [fields.result.status] of type [text]'
illegal_argument_exception' reason : 'failed to parse field [fields.rows.tmcode] of type [long]'
Here transformer.js:
exports.mappingTemplate = require('../../../index-template-mapping.json');
exports.transformer = (logData) => {
const transformed = {};
transformed['#timestamp'] = new Date().toISOString();
transformed.source_host = os.hostname();
transformed.message = logData.message;
if (typeof transformed.message === 'object') {
transformed.message = JSON.stringify(transformed.message);
I need help, suggestion how to resolve those error and succeed the Mapping.
Here the index-mapping-template.json
{
"mapping": {
"_doc": {
"properties": {
"#timestamp": {
"type": "date"
},
"fields": {
"ignore_malformed": true,
"dynamic": "true",
"properties": {}
},
"message": {
"type": "text"
},
"severity": {
"type": "keyword"
}
}
}
}
}
I have a C# object like so:
public class Foo {
public decimal Number {get;set;}
}
And I create a json schema like so:
var schema = await JsonSchema4.FromTypeAsync(typeof(Foo));
var jsonSchema = schema.ToJson();
return jsonSchema;
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Foo",
"type": "object",
"additionalProperties": false,
"properties": {
"SomeNumber": {
"type": "number",
"format": "decimal"
}
}
}
How do I prevent the property "format": "decimal" from outputting in the resulting schema?
edit to add: Without modifying the class Foo -- I don't have access to it.
I've been developing a Spring Boot REST API. I've done so many things so far except my problem. I'm using springfox swagger UI for documentation, and I separated models and dtos for better structure.
I have a base dto model:
public class BaseDto {
private int code;
private boolean success;
public BaseDto() {
this.code = HttpStatus.OK.value();
this.success = HttpStatus.OK.is2xxSuccessful();
}
}
And of course I'm using this class by extending it like:
#ApiModel("User")
public class UserDto extends BaseDto {
private String email;
private String username;
// stuffs
}
If I do user requests when I use this structure, I get:
{
code: 200,
success: true,
email: "",
username: ""
}
and so on... That's fine, but in other dtos, like post model, I have List of UserDto and it's showed in that form. In every object, "code" and "success" fields are written; however, this is not I want to.
The goal that I want to achieve is only once "code" and "success" are written in the response JSON body not all returning list objects.
To clarify more this is Post Dto Model and returns like this:
{
"code": 0,
"createdAt": "2016-05-17T21:59:37.512Z",
"id": "string",
"likes": [
{
"code": 0,
"createdAt": "2016-05-17T21:59:37.512Z",
"deviceType": "string",
"email": "string",
"fbAccessToken": "string",
"fbId": "string",
"followers": [
{}
],
"followings": [
{}
],
"id": "string",
"profileImage": "string",
"success": true,
"token": "string",
"udid": "string",
"updatedAt": "2016-05-17T21:59:37.512Z",
"username": "string",
"version": 0
}
],
"pictures": [
"string"
],
"postedBy": {
"code": 0,
"createdAt": "2016-05-17T21:59:37.512Z",
"deviceType": "string",
"email": "string",
"fbAccessToken": "string",
"fbId": "string",
"followers": [
{}
],
"followings": [
{}
],
"id": "string",
"profileImage": "string",
"success": true,
"token": "string",
"udid": "string",
"updatedAt": "2016-05-17T21:59:37.512Z",
"username": "string",
"version": 0
},
"success": true,
"text": "string",
"updatedAt": "2016-05-17T21:59:37.512Z",
"userId": "string",
"userIds": [
"string"
],
"version": 0
}
You can see at Post Dto model where User Dto is used, code and success fields are added redundant.
I don't know most probably I got wrong approach. Perhaps, I should use adding global HTTP status response to all returning DTOs.
Can anyone help?
You can have an AppUtil class were you can create your response and set HttpStatus. In your AppUtil class write a method like so :
public static ResponseEntity<ResponseEnvelope> successResponse(Object data,
int messageCode, String message) {
ResponseEnvelope envelope = new ResponseEnvelope(data, true, message,
messageCode);
ResponseEntity<ResponseEnvelope> responseEntity = new ResponseEntity<>(
envelope, HttpStatus.OK);
return responseEntity;
}
In the successResponse method you can set your data object in the ResponseEnvelope which along with the HttpStatus will be sent in the ResponseEntity that you want to return.
Check out my previous answer here