I am using the Elasticsearch, Logstash, and Kibana collect log a website
Elasticsearch, Logstash, and Kibana How does this ?
The API uses of the website?
Javascript code is?
Or the other way?
You can log all website content access to nginx access log
Parse the nginx access log with logstash and send to elasticsearch. See more here
Assuming you can identify resources on your server by URL / context path, you can create separate dashboards in Kibana for APIs/JS/other resources based on path.
Related
I'm using Elastic search to analyze my logs in WSO2 API Manager. I'm using basic authentication mode. After setting up Elastic and Kibana and configuring its setting, these errors appear when I want to see Kibana dashboards. How can I solve these problems?
In you Elasticsearch looks like there is no index which starts with apim_event_faulty or apim_event*, you can check all the indices in your Elasticsearch cluster by hitting _cat/indices?v API of Elasticsearch.
Check whether there is /repository/logs/apim_metrics.log inside your WSO2 API Manager home directory.
If you don't have the apim_metrics.log file, most like there is an issue in configurations you have done in API Manager. Refer this documentation https://apim.docs.wso2.com/en/latest/api-analytics/on-prem/elk-installation-guide/
If you have the apim_metrics.log file, check the content. If it does not have any logs, most likely API Manger haven't gone through any event to trigger apim_event_faulty, apim_event_response logs. Try invoking an API and observe the logs.
I would like to implement elastic search and Kibana instance as shared between multiple users, where multiple users can have their applications(services) and write their logs in the same of different files. But when they logged from Kibana they can only able to see their application(s) logs. I am using Fluentd as a log collector
I've installed filebeat in a server, collecting all the logs from all the containers i have. With filebeat i indicate to which elasticsearch and kibana hosts he must send them (both, elasticsearch and kibana are running as a service in another server). So now all the logs appear in kibana. My question is, all those logs that appear there, are stored somewhere? In elasticsearch or in kibana?
Thank you in advance
All the data is stored inside Elasticsearch.
Kibana is a visualization engine on top of Elasticsearch. Kibana itself also stores its configuration data inside an internal Elasticsearch index called .kibana.
Whatever you can see from Kibana always comes from Elasticsearch.
You can learn more about Elasticsearch here and Kibana here.
I'm using Elasticsearch to drive a "search website" feature. I'd like to collect statistics about what people search for (and which search queries are popular).
Elasticsearch is currently running behind Nginx, so I could extract this information from the Nginx access logs - but maybe Elasticsearch can be made to track this iinformation itself?
I found the Index stats API but that seems to be more abstract. It can be used to determne the average time needed to answer a query and such things, but it does not keep track of individual queries.
I am using a similar configuration (ES behind nginx), and I up to now I always just checked nginx' logfiles directly. However, thinking about your question, it makes much sense to route the nginx log files through the Elastic stack to Elastic Search using logstash, this seems to be the cleanest way.
Apparently in deprecated version there were some security auditing options using a plugin termed Shield or Security, but as I said, configuring logstash to ingest nginx logfiles directly seems most endurable way for your purposes.
Further reading and detailed instructions
discuss.elastic.co: How to get elaticsearch access logs
https://sysadmins.co.za/how-to-ingest-nginx-access-logs-to-elasticsearch-using-filebeat-and-logstash/
Elasticsearch Access Log
how to enable ElasticSearch http access log
I don't want to use filebeat or logstash. I have installed only Elasticsearch and I want to store log records in it and then index them. Is it possible.
Yes, it's possible, you can use any client library of your choosing and then hit the Index API endpoint or the Bulk API endpoint to index your data.