I have elastic search cluster.
Currently designing a python service for client for read and write query to my elastic search. The python service will not be maintained by me. Only internally python service will call our elastic search for fetching and writing
Is there any way to configure the elastic search so that we get to know that the requests are coming from python service, Or any way we can pass some extra fields while querying based on that fields we will get the logs
There is no online feature in elasticsearch to resolve your request. (you want to check the source and add fields to query).
but there is a solution for audit logs.
https://www.elastic.co/guide/en/elasticsearch/reference/current/enable-audit-logging.html
What you can do is placing a proxy in front of it and do the logging there, we have an Apache in front of our Elastic clusters to enable SSL-offloading there and add logging and ACL possibilities.
Related
I'm using Elastic search to analyze my logs in WSO2 API Manager. I'm using basic authentication mode. After setting up Elastic and Kibana and configuring its setting, these errors appear when I want to see Kibana dashboards. How can I solve these problems?
In you Elasticsearch looks like there is no index which starts with apim_event_faulty or apim_event*, you can check all the indices in your Elasticsearch cluster by hitting _cat/indices?v API of Elasticsearch.
Check whether there is /repository/logs/apim_metrics.log inside your WSO2 API Manager home directory.
If you don't have the apim_metrics.log file, most like there is an issue in configurations you have done in API Manager. Refer this documentation https://apim.docs.wso2.com/en/latest/api-analytics/on-prem/elk-installation-guide/
If you have the apim_metrics.log file, check the content. If it does not have any logs, most likely API Manger haven't gone through any event to trigger apim_event_faulty, apim_event_response logs. Try invoking an API and observe the logs.
I'm using Elasticsearch to drive a "search website" feature. I'd like to collect statistics about what people search for (and which search queries are popular).
Elasticsearch is currently running behind Nginx, so I could extract this information from the Nginx access logs - but maybe Elasticsearch can be made to track this iinformation itself?
I found the Index stats API but that seems to be more abstract. It can be used to determne the average time needed to answer a query and such things, but it does not keep track of individual queries.
I am using a similar configuration (ES behind nginx), and I up to now I always just checked nginx' logfiles directly. However, thinking about your question, it makes much sense to route the nginx log files through the Elastic stack to Elastic Search using logstash, this seems to be the cleanest way.
Apparently in deprecated version there were some security auditing options using a plugin termed Shield or Security, but as I said, configuring logstash to ingest nginx logfiles directly seems most endurable way for your purposes.
Further reading and detailed instructions
discuss.elastic.co: How to get elaticsearch access logs
https://sysadmins.co.za/how-to-ingest-nginx-access-logs-to-elasticsearch-using-filebeat-and-logstash/
Elasticsearch Access Log
how to enable ElasticSearch http access log
can kibana's console (in Dev Tools) be used for writing and implementing elasticsearch ? I am new to elasticsearch and very confused when it comes to doing hands-on it. thank you in advance.
kibana Dev tools makes calling elastic search API's easier so you can develop what ever you want in kibana Dev tools to make aggregation call or make query string to call the API's.
on the other hand you should use it with an SDK in your application like Elasticsearch JS for javascript so you can use the developed queries and aggregations in kibana to be used in your application and more you can monitor your shards health or put mapping for your indexes and more of functionality which can be found in Documentation, Although, you can find JS API's Documentation here
You can use Kibana Dev Tools to invoke REST API commands to perform cluster level actions such as taking snapshots, restore etc and also index simple documents. But, if you are looking to writing data to Elastic on a regular basis like ingesting server/ app logs or server metrics (CPU, memory, Disk usage etc) you should look at installing filebeats or metricbeats.
We are building log analytics applicaton in which we are using Graylog & Elasticsearch. Since I have installed Elasticsearch but somehow I want to take the data from elasticsearch and create relational graphs with the data on my own instead of using Xpack-Graph.
i could have used xpack graph api and do http calls to get data but its not free ware and i'm not sure that we will be able to buy one licence
is there any other alternative for xpack graph api which is free ??
or can i query directly to elastic using aggregation if so how feasible it is?? can yo share me some resource on this
Kindly share your thoughts on this.
I have configured a Kibana using AWS infrastructure. I'm using AWS CloudWatch Logs and AWS ElasticSearch which contains Kibana software. As you can imagine, I'm uploading all my logs to Kibana.
I'm trying to obtain a list of the most traffic URL's omitting query strings... but I don't know if this is possible.
Can you help me? I've search it on Google and on Elastic documentation, but I didn't found anything.
Here is an example:
Suppose that I have the following URL's:
abc.com/helloWorld.html?param=1
abc.com/helloWorld.html?param=2
abc.com/helloWorld.html?param=3
abc.com/bye.html?anotherParam=1
I want to see the following URL's in order to compute the sum of requests per file. Is it possible?
abc.com/helloWorld.html
abc.com/helloWorld.html
abc.com/helloWorld.html
abc.com/bye.html
Thanks,