I created an index which has type completion in elasticsearch, however, when I tried to use logstash to automate fetching data from the database, I faced the following error:
failed to parse [x]: expected text or object, but got VALUE_NUMBER
only on x which its type is completion.
The only solution that I'm thinking of is to change type of x to number rather than string when fetching it from a database.
I tried to search on every support site for elastic search and I did not get any answer.
It seems the only way is to create a new column with the same number but now AS STRING.
I hope someone will benefit for it in the future.
Related
i have an issue which i suspect is quite basic but i have been stuck on this for too long and i fear i am missing something so basic that i can't see it by now.
we are using the ELK stack today for log analysis of our application logs.
logs are created by the JAVA application into JSON format, shipped using filebeat into logstash which in turn processes the input and queues it into ES.
some of the messages contain unstructured data in the message field which i currently cannot parse into separate fields so i need to catch them in the message field. problem is this:
the string i need to catch is: "57=1" this is an indication of something which i need to filter documents upon. i need to get documents which contain this exact string.
no matter what i try i can't get kibana to match this. it seems to always ignore the equal char and match either 57 or 1.
please advise.
thanks
You may check the Elasticsearch mapping on the field type of the referring field. If it is analyzed, the '=' may not have been indexed due to the default-analyzer. (source 1, source 2)
So im working on a system that logs bad data sent to an api and what the full request was. Would love to be able to see this in Kibana.
Issue is the datatypes could be random, so when I send them to the bad_data field it fails if it dosen't match the original mapping.
Anyone have a suggestion for the right way to handle this?
(2.X Es is required due to a sub dependancy)
You could use ignore_malformed flag in your field mappings. In that case wrong format values will not be indexed and your document will be saved.
See elastic documentation for more information.
If you want to be able to query such fields as original text you could use fields in your mapping for multi-type indexing, to get fast queries on raw text values.
I've been given a set up where I have a program creating live data and posting them into Elasticsearch.
I am trying to visualise this data in Kibana, but I'm coming across many problems such as numbers for a field being of type string instead of integers or there being certain missing fields.
But mainly for now certain fields being integer instead of string would be useful. How do I go about this? Is it possible?
I have no access to source code of the system creating the live events data.
Thanks in advance.
Update: I should also mention additionally that for now I am restricted to Elasticsearch version 2.4
If your data is coming straight into Elasticsearch, your options are limited.
The best option is to have the program that is creating the data send valid, properly formatted data.
If that's not an option, you can set your Elasticsearch mapping to force the field to be numeric. This will have the side-effect of dropping all documents where this field is not numeric.
There is also the elasticsearch injest node, which allows for some (logstash-like) transformations of the data. Converting the type is one such allowed "processor".
Basic usecase that we are trying to solve is for users to be able to search from the contents of the log file .
Lets say a simple situation where user searches for a keyword and this is present in a log file which i want to render it back to the user.
We plan to use ElasticSearch for handling this. The idea that i have in mind is to use elastic search as a mechanism to store the indexed log files.
Having this concept in mind, i went through https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
Couple of questions i have,
1) I understand the input provided to elastic search is a JSON doc. It is going to scan this JSON provided and create/update indexes. So i need a mechanism to convert my input log files to JSON??
2) Elastic search would scan this input document and create/update inverted indexes. These inverted indexes actually point to the exact document. So does that mean, ES would store these documents somewhere?? Would it store them as JSON docs? Is it purely in memory or on file sytem/database?
3) No when user searches for a keyword , ES returns back the document which contains the searched keyword. Now do i need to have the ability to convert back this JSON doc to the original log document that user expects??
Clearly im missing something.. Sorry for asking questions this silly , but im trying to improve my skills and its WIP.
Also , i understand that there is ELK stack out there. For some reasons we just want to use ES and not the LogStash and Kibana part of the stack..
Thanks
Logs needs to be parsed to JSON before they can be inserted into Elasticsearch
All documents are stored on the filesystem and some data is kept in memory but all data is persistent.
When you search Elasticsearch you get back matching JSON documents. If you want to display the original error message, you can store that original message in one of the JSON fields and display just that.
So if you just want to store log messages and not break them into fields or anything, you can simply take each row and send it to Elasticsearch like so:
{ "message": "This is my log message" }
To parse logs, break them into fields and add some logic, you will need to use some sort of app, like Logstash for example.
I'm having some issues getting elasticsearch to interpret an epoch millis timestamp field. I have some old bro logs I want to ingest and have them be in the proper orders and spacing. Thanks to Logstash filter to convert "$epoch.$microsec" to "$epoch_millis"
I've been able to convert the field holding the bro timestamp to the proper length of digits. I've also inserted a mapping into elasticsearch for that field, and it says that the type is "Date" with the format being the default. However, when I go and look at the entries it still has a little "t" next to it instead of a little clock. And hence I can't use it for my filter view reference in kibana.
Anyone have any thoughts or have dealt with this before? Unfortunately it's a stand alone system so I would have to manually enter any of the configs I'm using.
I did try and convert my field "ts" back to an integer after using the method described in the link above. So It should be a logstash integer before hitting the elasticsearch mapping.
So I ended up just deleting all my mappings in Kibana, and elasticsearch. I then resubmitted and this time it worked. Must have been some old junk in there that was messing me up. But now it's working great!