I'am just started with ES 5.2.2
Trying ad analyzer with support russian morhology.
Run ES using docker, i create image with installed elasticsearch-analysis-morphology.
then i:
Create index,
then put settings
after that get settings, and all sems right
curl http://localhost:9200/news/_settings?pretty
{
"news" : {
"settings" : {
"index" : {
"number_of_shards" : "5",
"provided_name" : "news",
"creation_date" : "1489343955314",
"analysis" : {
"analyzer" : {
"russian_analyzer" : {
"filter" : [
"stop",
"custom_stop",
"russian_stop",
"custom_word_delimiter",
"lowercase",
"russian_morphology",
"english_morphology"
],
"char_filter" : [
"html_strip",
"ru"
],
"type" : "custom",
"tokenizer" : "standard"
}
},
"char_filter" : {
"ru" : {
"type" : "mapping",
"mappings" : [
"Ё=>Е",
"ё=>е"
]
}
},
"filter:" : {
"custom_stop" : {
"type" : "stop",
"stopwords" : [
"n",
"r"
]
},
"russian_stop" : {
"ignore_case" : "true",
"type" : "stop",
"stopwords" : [
"а",
"без",
]
},
"custom_word_delimiter" : {
"split_on_numerics" : "false",
"generate_word_parts" : "false",
"preserve_original" : "true",
"catenate_words" : "true",
"generate_number_parts" : "true",
"catenate_all" : "true",
"split_on_case_change" : "false",
"type" : "word_delimiter",
"catenate_numbers" : "false"
}
}
},
"number_of_replicas" : "1",
"uuid" : "IUkHHwWrStqDMG6fYOqyqQ",
"version" : {
"created" : "5020299"
}
}
}
}
}
then i try open index but ES give me this:
{
"error" : {
"root_cause" : [
{
"type" : "exception",
"reason" : "Failed to verify index [news/IUkHHwWrStqDMG6fYOqyqQ]"
}
],
"type" : "exception",
"reason" : "Failed to verify index [news/IUkHHwWrStqDMG6fYOqyqQ]",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Custom Analyzer [russian_analyzer] failed to find filter under name [custom_stop]"
}
},
"status" : 500
}
Can't understand where i'm wrong.
Can anyone see what the problem is?
There was mistake in "filter" section
was:
look here this This colon was a mistake
|
v
"filter:" : {
"custom_stop" : {
"type" : "stop",
"stopwords" : [
"n",
"r"
]
}...
Thanks #asettou and #andrey-morozov
Related
I am doing a DIY Tweet Sentiment analyser, I have an index of tweets like these
"_source" : {
"id" : 26930655,
"status" : 1,
"title" : "Here’s 5 underrated #BTC and realistic crypto accounts that everyone should follow: #Quinnvestments , #JacobOracle , #jevauniedaye , #ginsbergonomics , #InspoCrypto",
"hashtags" : null,
"created_at" : 1622390229,
"category" : null,
"language" : 50
},
{
"id" : 22521897,
"status" : 1,
"title" : "#bulls gonna overtake the #bears soon #ATH coming #ALTSEASON #BSCGem #eth #btc #memecoin #100xgems #satyasanatan 🙏🚩🚩🇮🇳""",
"hashtags" : null,
"created_at" : 1620045296,
"category" : null,
"language" : 50
}
There Mappings are settings are like
"sentiment-en" : {
"mappings" : {
"properties" : {
"category" : {
"type" : "text"
},
"created_at" : {
"type" : "integer"
},
"hashtags" : {
"type" : "text"
},
"id" : {
"type" : "long"
},
"language" : {
"type" : "integer"
},
"status" : {
"type" : "integer"
},
"title" : {
"type" : "text",
"fields" : {
"raw" : {
"type" : "keyword"
},
"raw_text" : {
"type" : "text"
},
"stop" : {
"type" : "text",
"index_options" : "docs",
"analyzer" : "stop_words_filter"
},
"syn" : {
"type" : "text",
"index_options" : "docs",
"analyzer" : "synonyms_filter"
}
},
"index_options" : "docs",
"analyzer" : "all_ok_filter"
}
}
}
}
}
"settings" : {
"index" : {
"number_of_shards" : "10",
"provided_name" : "sentiment-en",
"creation_date" : "1627975717560",
"analysis" : {
"filter" : {
"stop_words" : {
"type" : "stop",
"stopwords" : [ ]
},
"synonyms" : {
"type" : "synonym",
"synonyms" : [ ]
}
},
"analyzer" : {
"stop_words_filter" : {
"filter" : [ "stop_words" ],
"tokenizer" : "standard"
},
"synonyms_filter" : {
"filter" : [ "synonyms" ],
"tokenizer" : "standard"
},
"all_ok_filter" : {
"filter" : [ "stop_words", "synonyms" ],
"tokenizer" : "standard"
}
}
},
"number_of_replicas" : "0",
"uuid" : "Q5yDYEXHSM-5kvyLGgsYYg",
"version" : {
"created" : "7090199"
}
}
Now the problem is i want to extract all the Hashtags and mentions in a seprate field.
What i want as O/P
"id" : 26930655,
"status" : 1,
"title" : "Here’s 5 underrated #BTC and realistic crypto accounts that everyone should follow: #Quinnvestments , #JacobOracle , #jevauniedaye , #ginsbergonomics , #InspoCrypto",
"hashtags" : BTC,
"created_at" : 1622390229,
"category" : null,
"language" : 50
},
{
"id" : 22521897,
"status" : 1,
"title" : "#bulls gonna overtake the #bears soon #ATH coming #ALTSEASON #BSCGem #eth #btc #memecoin #100xgems #satyasanatan 🙏🚩🚩🇮🇳""",
"hashtags" : bulls,bears,ATH, ALTSEASON, BSCGem, eth , btc, memecoin, 100xGem, satyasanatan
"created_at" : 1620045296,
"category" : null,
"language" : 50
}
What i have tried so far
Create a pattern based tokenizer to just read Hashtags and mentions and no other token for field hashtag and mentions did not had much success there.
Tried to write an n-gram tokenizer without any analysers did not achive much success there as well.
Any help would be appreciated, I am open to reindex my data. Thanks in advance !!!
You can use Logstash Twitter input plugin for indexing data and configured below ruby script in filter plugin as mentioned in blog.
if [message] {
ruby {
code => "event.set('hashtags', event.get('message').scan(/\#[a-z]*/i))"
}
}
You can use Logtstash Elasticsearch Input plugin for source index and configured about ruby code in Filter plugin and Logtstash elasticsearch output plugin with destination index.
input {
elasticsearch {
hosts => "localhost:9200"
index => "current_twitter"
query => '{ "query": { "query_string": { "query": "*" } } }'
size => 500
scroll => "5m"
}
}
filter{
if [message] {
ruby {
code => "event.set('hashtags', event.get('message').scan(/\#[a-z]*/i))"
}
}
}
output {
elasticsearch {
index => "new_twitter"
}
}
Another option is to use reingest API with ingest pipeline but ingest pipeline not support ruby code. So you need to convert above ruby code to the painless script.
I am tired of reindexing every 2 3 weeks i have to do reindex.
{
"winlogbeat_sysmon" : {
"order" : 0,
"index_patterns" : [
"log-wlb-sysmon-*"
],
"settings" : {
"index" : {
"lifecycle" : {
"name" : "winlogbeat_sysmon_policy",
"rollover_alias" : "log-wlb-sysmon"
},
"refresh_interval" : "1s",
"number_of_shards" : "1",
"number_of_replicas" : "1"
}
},
"mappings" : {
"properties" : {
"thread_id" : {
"type" : "long"
},
"z_elastic_ecs.event.code" : {
"type" : "long"
},
"geoip" : {
"type" : "object",
"properties" : {
"ip" : {
"type" : "ip"
},
"latitude" : {
"type" : "half_float"
},
"location" : {
"type" : "geo_point"
},
"longitude" : {
"type" : "half_float"
}
}
},
"dst_ip_addr" : {
"type" : "ip"
}
}
},
"aliases" : { }
}
}
this is the template i set earlier from then i didn't change anything
in current and previous indices of log-wlb-sysmon has dst_ip_addr has ip field and older indices of log-wlb-sysmon has text field in logstash i didn't see any warnning for this issue
I am working on Elasticsearch. I want to use search suggestor for multiple indices at a time. I have two indices, tags and pool_tags which has name field in each index. How to use suggestor on this two indices having a similarly named field name.
I tried naming the suggestor (pool_tag_suggest in pool_tags) differently and I tried. Here are the mappings
tags:
{
"tags" : {
"mappings" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword"
},
"suggest" : {
"type" : "completion",
"analyzer" : "simple",
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
}
}
}
}
}
}
}
pool_tags:
{
"pool_tags" : {
"mappings" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword"
},
"pool_tag_suggest" : {
"type" : "completion",
"analyzer" : "simple",
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
}
}
}
}
}
}
}
WHAT I TRIED
POST pool_tags,tags/_search
{
"suggest": {
"tags_suggestor": {
"text": "ww",
"term": {
"field": "name.suggest"
}
},
"pooltags_suggestor": {
"text": "ww",
"term": {
"field": "name.pool_tag_suggest"
}
}
}
}
ERROR
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "no mapping found for field [name.suggest]"
},
{
"type" : "illegal_argument_exception",
"reason" : "no mapping found for field [name.pool_tag_suggest]"
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "pool_tags",
"node" : "g2rCnS4PQMWyldWABVJawQ",
"reason" : {
"type" : "illegal_argument_exception",
"reason" : "no mapping found for field [name.suggest]"
}
},
{
"shard" : 0,
"index" : "tags",
"node" : "g2rCnS4PQMWyldWABVJawQ",
"reason" : {
"type" : "illegal_argument_exception",
"reason" : "no mapping found for field [name.pool_tag_suggest]"
}
}
],
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "no mapping found for field [name.suggest]",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "no mapping found for field [name.suggest]"
}
}
},
"status" : 400
}
I have been trying to make one of the fields being indexed to be dynamic and also changed the elasticsearch.yml for the same by adding
index.mapper.dynamic: false
in the end and also restarted elasticsearch and kibana sense. Also tried with different fields and index names but I am still getting the same error :
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [albumdetailid] has unsupported parameters: [dynamic : true]"
}
],
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping [package]: Mapping definition for [albumdetailid] has unsupported parameters: [dynamic : true]",
"caused_by": {
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [albumdetailid] has unsupported parameters: [dynamic : true]"
}
},
"status": 400
}
the code for adding index is below :
PUT /worldtest221
{
"settings" : {
"index" : {
"creation_date" : "1474989052008",
"uuid" : "Ae-7aFrLT466ZJL4U9QGLQ",
"number_of_replicas" : "0",
"analysis" : {
"char_filter" : {
"quotes" : {
"type" : "mapping",
"mappings" : [ "=>'", "=>'", "‘=>'", "’=>'", "‛=>'" ]
}
},
"filter" : {
"nGram_filter" : {
"max_gram" : "400",
"type" : "nGram",
"min_gram" : "1",
"token_chars" : [ "letter", "digit", "punctuation", "symbol" ]
}
},
"analyzer" : {
"quotes_analyzer" : {
"char_filter" : [ "quotes" ],
"tokenizer" : "standard"
},
"nGram_analyzer" : {
"type" : "custom",
"filter" : [ "lowercase", "asciifolding", "nGram_filter" ],
"tokenizer" : "whitespace"
},
"whitespace_analyzer" : {
"type" : "custom",
"filter" : [ "lowercase", "asciifolding" ],
"tokenizer" : "whitespace"
}
}
},
"cache" : {
"query_result" : {
"enable" : "true"
}
},
"number_of_shards" : "1",
"version" : {
"created" : "2030099"
}
}
},
"mappings" : {
"package" : {
"properties" : {
"autosuggestionpackagedetail" : {
"type" : "string",
"index" : "not_analyzed"
},
"availability" : {
"type" : "nested",
"include_in_all" : false,
"properties" : {
"displaysequence" : {
"type" : "long",
"include_in_all" : false
},
"isquantityavailable" : {
"type" : "boolean"
},
.....
"metatags" : {
"type" : "string",
"include_in_all" : false
},
"minadultage" : {
"type" : "long",
"include_in_all" : false
},
"newmemberrewardpoints" : {
"type" : "long",
"include_in_all" : false
},
"packagealbum" : {
"include_in_all" : false,
"properties" : {
"albumdetailid" : {
"type" : "string",
"include_in_all" : false,
"dynamic": true
},
....
Look at the second last line where I mention "dynamic" : true
This happens because you are trying to set "dynamic: true" on a field ("albumdetailid") of type string! This make no sense! no new fields can be created under a "string" field. The "dynamic" parameter should be set under an "object" field. so Either define "albumdetailid" as "object" or put the "dynamic: true" one level higher - under "packagealbum" like this:
"packagealbum" : {
"include_in_all" : false,
"dynamic": true,
"properties" : {
"albumdetailid" : {
"type" : "string",
"include_in_all" : false
},
....
I've this mapping for fuas type:
curl -XGET 'http://localhost:9201/living_team/_mapping/fuas?pretty'
{
"living_v1" : {
"mappings" : {
"fuas" : {
"properties" : {
"backlogStatus" : {
"type" : "long"
},
"comment" : {
"type" : "string"
},
"dueTimestamp" : {
"type" : "date",
"format" : "strict_date_optional_time||epoch_millis"
},
"matter" : {
"type" : "string"
},
"metainfos" : {
"properties" : {
"category 1" : {
"type" : "string"
},
"key" : {
"type" : "string"
},
"null" : {
"type" : "string"
},
"processos" : {
"type" : "string"
}
}
},
"resources" : {
"properties" : {
"noteId" : {
"type" : "string"
},
"resourceId" : {
"type" : "string"
}
}
},
"status" : {
"type" : "long"
},
"timestamp" : {
"type" : "date",
"format" : "strict_date_optional_time||epoch_millis"
},
"user" : {
"type" : "string",
"index" : "not_analyzed"
}
}
}
}
}
}
I'm trying to perform this aggregation:
curl -XGET 'http://ESNode01:9201/living_team/fuas/_search?pretty' -d '
{
"aggs" : {
"demo" : {
"nested" : {
"path" : "metainfos"
},
"aggs" : {
"key" : { "terms" : { "field" : "metainfos.key" } }
}
}
}
}
'
ES realizes me:
"error" : {
"root_cause" : [ {
"type" : "aggregation_execution_exception",
"reason" : "[nested] nested path [metainfos] is not nested"
} ],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query_fetch",
"grouped" : true,
"failed_shards" : [ {
"shard" : 3,
"index" : "living_v1",
"node" : "HfaFBiZ0QceW1dpqAnv-SA",
"reason" : {
"type" : "aggregation_execution_exception",
"reason" : "[nested] nested path [metainfos] is not nested"
}
} ]
},
"status" : 500
}
Any ideas?
You're missing "type":"nested" from your metainfos mapping.
Should have been:
"metainfos" : {
"type":"nested",
"properties" : {
"category 1" : {
"type" : "string"
},
"key" : {
"type" : "string"
},
"null" : {
"type" : "string"
},
"processos" : {
"type" : "string"
}
}
}