I'm trying to test our Loki log data source. From the Queries I've been executing nothing is returned.
It's possible that the logs are in a different format to what I'm expecting, or that no Logs are ingested by Loki, and my pipeline is broken somewhere.
Is there a Loki query that returns all the logs?
I've looked through documentation, and so far, I haven't found any such Loki query. Any other queries to help debug would be appreciated!
You can use a match-all regex together with a stream you have for all your logs.
For example if you collect a stream named host for all your incoming logs you'd query for:
{host=~ ".*"}
You should note that at present a stream selector is always required for querying logs.
{host=~ ".*"} doesn't work for me. Use {host=~ ".+"} That should work always.
Related
I have elastic configured with Grafana and it has logs. I tried to query logs for the elasticsearch in grafana but did not have much succes. I went online to try to learn how to do so, but when I do it talks about Loki. Are you able to use Loki with Elasticsearch? Do not see a definite answer for this online.
Using Loki with ES defeats the purpose of using Loki itself.
Loki prides itself on indexing only the metadata/labels of the logs and storing the actual log data separately in a compressed manner.
This reduces storage costs and leads to faster retrieval of data as there is less data to index as compared to the an ES index which indexes everything in a log line and worse still ,if the data is missing ,stores the index attribute as empty. (Almost similar to the diff between SQL vs NoSQL)
As of now, Loki does not support ES as the index store.
It uses two types of indices:- Labels and log chunks and stores them separately to be queried as and when required.
Label/metadata/index :- uses Cassandra,GCS,File System,S3
Data chunks:- Cassandra,BigTable,DynamoDB,BoltDB
For more info see Loki storage.
I have a Elasticsearch cluster running on AWS Elasticsearch instance. It is up running for a few months. I'd like to know the most used query requests over the last few months. Does Elasticsearch save all queries somewhere I can search? Or do I have to programmatically save the requests for analysis?
As far as I'm aware, Elasticsearch doesn't by default save a record or frequency histogram of all queries. However, there's a way you could have it log all queries, and then ship the logs somewhere to be aggregated/searched for the top results (incidentally this is something you could use Elasticsearch for :D). Sadly, you'll only be able to track queries after you configure this, I doubt that you'll be able to find any record of your historical queries the last few months.
To do this, you'd take advantage of Elasticsearch's slow query log. The default thresholds are designed to only log slow queries, but if you set those defaults to 0s then Elasticsearch would log any query as a slow query, giving you a record of all queries. See that link above for detailed instructions how, you could set this for a whole cluster in your yaml configuration file like
index.search.slowlog.threshold.fetch.debug: 0s
or set it dynamically per-index with
PUT /<my-index-name>/_settings
{
"index.search.slowlog.threshold.query.debug": "0s"
}
To be clear the log level you choose doesn't strictly matter, but utilizing debug for this would allow you to keep logging actually slow queries at the more dangerous levels like info and warn, which you might find useful.
I'm not familiar with how to configure an AWS elasticsearch cluster, but as the above are core Elasticsearch settings in all the versions I'm aware of there should be a way to do it.
Happy searching!
I have configured Fluent-bit on my ECS cluster . I can see the logs in Kibana. But all the log data are sent to a single field "log". How can I extract each field into a separate field. There is a solution for fluentd already in this question.
But how can I achieve the same with fluent-bit?
There is a solution in Kuberntetes with fluent-bit: https://docs.fluentbit.io/manual/filter/kubernetes
How do I achieve the same thing in ECS?
Generally fluent-bit send exactly docker log file that taking from /var/lib/docker/containers/*/*.log
You can browse this path on your machine and see that it contains JSON strings with exactly two fields you mentioned.
From here you have number ways, I'll discover two that I know well:
Use logstash:
You should know well the log structure. This helps you to create the right filters pipeline for the parse log field. Usually, people use filter plugins for this. If you add log examples I will be able to make an example of a filter like this
Use the elasticsearch ingest node.
You should know well the log structure. For be able easy to create processors pipeline for parse log field. More one time, specific log examples help's us to help you.
The most used filter/processor is grok filter/processor. This tool have a lot of options for parse structured text from any log.
Consider the following use case:
I want the information from one particular log line to be indexed into Elasticsearch, as a document X.
I want the information from some log line further down the log file to be indexed into the same document X (not overriding the original, just adding more data).
The first part, I can obviously achieve with filebeat.
For the second, does anyone have any idea about how to approach it? Could I still use filebeat + some pipeline on an ingest node for example?
Clearly, I can use the ES API to update the said document, but I was looking for some solution that doesn't require changes to my application - rather, it is all possible to achieve using the log files.
Thanks in advance!
No, this is not something that Beats were intended to accomplish. Enrichment like you describe is one of the things that Logstash can help with.
Logstash has an Elasticsearch input that would allow you to retrieve data from ES and use it in the pipeline for enrichment. And the Elasticsearch output supports upsert operations (update if exists, insert new if not). Using both those features you can enrich and update documents as new data comes in.
You might want to consider ingesting the log lines as is to Elasticearch. Then using Logstash, build a separate index that is entity specific and driven based on data from the logs.
I'm trying to build some kind of monitor for my ELK stack. I want to know when/if my ELK is down. This will be just a simple solution. I was tasked with integrating a on/off signal within a bigger, global monitoring tool.
So I want to query my ELKs elasticsearch for the latest entry that matches one particular field value. My ELK data contains a field for each access.log row that states which server was the origin. So there is always say server_node.raw=Tomcat1 oder Tomcat2 or ...
I do get a result from my index but this seems like metadata to me. http://10.170.121.148:9100/logstash-2015.11.10/?pretty
Is there a way to query ES for the latest entry that matches server_node.raw=Tomcat1 using a simple HTTP request?
Using server_node.raw in Kibana works perfectly fine.
Anyone with an idea? I'd appreciate it.
Thanks in advance and regards. Sebastian
Yes, you are on the right path, you can simply query your logstash index with a URI search and &q=server_node.raw:... like this
curl -XGET 'http://10.170.121.148:9100/logstash-2015.11.10/_search?q=server_node.raw:Tomcat1&pretty'