When a new agent joins the fleet, I want to make a few notes on my application. Is it possible to get a notification whenever a new agent joins to fleet?
I checked elasticsearch watcher but haven't found solution so far.
Tldr;
There are no built in mechanism to perform such a task.
Nonetheless it is possible to create it, but it will be a bit of a hack.
Use the .fleet-agents index which won't be accessible in the 8.x releases
Use the Kibana agent api which is in experimental state.
Related
I am using Elasticsearch, and I am building a client (using the Java Client API) to export logs indexed via Logstash.
I would like to be able to be notified (by adding a listener somewhere) when a new document is index (= a new log line have been added) instead of querying the last X documents.
Is it possible ?
This is what you're looking for: https://github.com/ForgeRock/es-change-feed-plugin
Using this plugin, you can register to a websocket channel to receive indexation/deletion events as they happen. It has some limitations, though.
Back in the days, it was possible to install river plugins to stream documents to ES. The river feature has been removed, but this plugin above is like a "reverse river", where outside clients are notified by ES as documents get indexed.
Very useful and seemingly up-to-date with ES 6.x
UPDATE (April 14th, 2019):
According to what was said at Elastic{ON} Zurich 2019, at some point in the 7.x series, there will be a Changes API that will provide index changes notifications (document creation, update, deletion and more).
UPDATE (July 22nd, 2022):
ES 8.x is out and the Changes API is still nowhere in sight ... Good to know, though, that's it's still open at least.
I have elasticsearch 5.6 and using log4j2 to config it now,
I save data in elasticsearch, And now i want to push data to sentry 8.22.
If elasticsearch reviceve a data then push the data to sentry automatically.
Can someone tell me how to do this?
PS:
I found some links like this Using sentry logging with elasticsearch
But the solution there is too old.
IMO that's not what Sentry is for: You want to find errors in your application, but it isn't a general log collector. You're also not trying to get your operating system, webserver, database,... hooked into Sentry, right?
If anything in Elasticsearch is going wrong, Sentry should collect the error in your application and you can dig deeper from there. No need to connect Elasticsearch directly.
PS: Adding logging libraries is definitely untested and you might run into various issues (at the very least every upgrade will be more complicated) — I'd be pretty careful with this.
I'm new to elasticsearch and am still trying to set it up. I have installed elasticsearch 5.5.1 using default values I have also installed Kibana 5.5.1 using the default values. I've also installed the ingest-attachment plugin with the latest x-pack plugin. I have elasticsearch running as a service and I have Kibana open in my browser. On the Kibana dashboardI have an error stating that it is unable to fetch mappings. I guess this is because I havn't set up any indices or pipelines yet. This is where I need some steer, all the documentation I've found so far on-line isn't particularly clear. I have a directory with a mixture of document types such as pdf and doc files. My ultimate goal is to be able to search these documents with values that a user will enter via an app. I'm guessing I need to use the Dev Tools/console window in Kibana using the 'PUT' command to create a pipeline next, but I'm unsure of how I should do this so that it points to my directory with the documents. Can anybody provide me an example of this for this version please.
If I understand you correctly, let's first set some basic understanding about elasticsearch:
Elasticsearch in it's simple definition is a "Search engine". so you need to store some data, and then elastic will help you to search using a search criteria, and it will retrieve relevant data back
You need a "Container" to save your data to, and elastic has this thing like any database engine to store your data, but the terms are somehow different. for example a "Database" in sql-like systems is called "Index", and what you know as "table" is called "Type" in elastic.
from my understanding, you will need to create your index (with or without mappings) to have a starting point, and I recommend you to start without mappings just to "start" and get things working, but later on it's highly recommend to work with "mappings" if applicable, because elastic is smart, but it cannot know more about your data than you do
Because Kibana has failed to find a proper index to start with, it has complained and asked you to either provide a syntax for index names, or a specific index name so it can infer the inline mappings and give you the nice features of querying, displaying charts, etc of your data, so once you create your index, you will provide that to the starting page of Kibana, and you will be ready to go.
Let me know if you need something more specific to your needs :)
I'm currently working on a POC with Couchbase, using Spring Data to put & get documents on/off a bucket on a cluster.
As I'm working in a big company, I'm lucky they gave me a bucket, but still I don't have the admin rights on the cluster, so I only have access to the bucket.
But as I'm digging into the Spring Data documentation, I'm not able to find a way to retrieve documents without creating views on the server. (I'm getting errors like "Unknown query param" ). Nevertheless with couchbase java sdk i'm able to, through n1ql queries, but the use of the Spring data layer is mandatory.
The answers I found always point me to the server-side function direction
ex : https://stackoverflow.com/a/30928169/3744307
What I would like to find, is a way to add a repository method like
List findReceiptByAccount(String Account)
without having to specificly declare the function server-side.
Is this possible, or have I to send a request to the administrators to create functions for me everytime I have to add a findByX method?
Thanks for your time,
What version of CB is it ?
I think that prior to 4.5, a n1ql access (which you seems to have) is enough to build your index yourself !
With Spring Data Couchbase 2.x that would use a N1QL index in the background, and it would work with a single primary index (although having 1 index per repository entity class would be best for performance). Maybe you can ask your admin to create that index once?
I am going to use logstash+ES+kibana for my project. I want to know how to use this framework for multi tenants. Can any one explain me how after the authentication Kibana query the elastic search index and load in Kibana's dashboard? Can I restrict kibana to look for a specifix index of Elastic search for a particular user or some-id? Anybody has tried this?
Thnx
You could, but depending on your use case it is probably not a good idea. There are a few gotchas, particularly regarding security and separating the users. First Kibana is just javascript running in the browser. So whatever Kibana is allowed to do so is your user. You can however have a separate index pattern for each "user", but elastic search does not provide you any ways of authenticating a users or authorizing a user access to a specific index. You would have to use some sort of proxy for this.
I recommend http://www.found.no/foundation/elasticsearch-in-production/ and http://www.found.no/foundation/elasticsearch-security/ for a more in depth explanation.
Create an index for each tenant.
In this way you can use a proxy (like the app the hosts kibana) to intercept the request and return a settings that includes the index to use.
The value that specifies the index to use can be the logged in user or you can get that value somewhere else.
To separate even more the data, you can use a prefix in each index name, and then when you specify an index you can use a pattern to take all the index related to only certain kind of data/entities.
Hope this help.
Elasticsearch announced today a plugin they are working on that should provide security features to ES product. Probably, this will contain ways of restricting access based on roles and users setup at cluster and indices level. If this happens I see no way for them not to extend this security layer to Kibana, as well. Also, it seems this plugin will have a commercial version only.