I'm running ElasticSearch on Ubuntu 14.04 and I can't seem to get ES to find a template for a specific index. The documentation is confusing in that it says you should put a templates directory under /etc/elasticsearch/config/, but then later in the documentation it says that configuration should be under /etc/elasticsearch, like the yaml file is at /etc/elasticsearch.
The reason I know its not finding it is I can do a:
curl -XGET 'http://localhost:9200/_template/my_template?pretty'
and get an empty JSON object back.
According to the config notes for Templates:
Index templates can also be placed within the config location (path.conf) under the templates directory (note, make sure to place them on all master eligible nodes).
In your case, if your main configuration directory is /etc/elasticsearch then you may place templates inside a folder called /etc/elasticsearch/templates. You'll need to place that file onto all of the servers which are running master-eligible nodes. (E.g., for a small cluster, onto all nodes.)
In my experience, it's a little more common to simply POST templates using the HTTP API. That way you can add and remove templates without having to worry about managing and deploying configurations on your servers.
Index Templates
Related
I'd like to edit a few ElasticSearch 8 settings (for example: network.host), but I don't want to edit /etc/elasticsearch/elasticsearch.yml directly. What I'd like to do is to add my settings to my own file and put it in an hypothetical /etc/elasticsearch/conf.d folder, do be included automatically.
This approach is widespread with multiple other software I use (Nginx, Apache, MySQL, PHP just to name a few).
ElasticSearch already does what i need with /etc/elasticsearch/jvm.options.d, but that folder is just for the the JVM options, not for ES own settings.
What am I missing here?
You can override the config path via env variables:
ES_PATH_CONF=/path/to/my/config ./bin/elasticsearch
https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html
There is an old discussion about doing what you are asking about but didn't make it this way:
https://github.com/elastic/elasticsearch/issues/11362
I want to add 3 extension to nifi (nifi-encryptMD5-nar-1.0.nar-unpacked,nifi-getOperator-nar-1.0-SNAPSHOT.nar-unpacked,nifi-splitAttributeValue-nar-1.0.nar-unpacked)
I added the extensions folder in the directory /opt/nifi/nifi-1.9.2/work/nar/extensions/
then when I restart the nifi service, nifi turns off and does not turn on, when I force the start with the user nifi, nifi turns on but the extentions have been deleted from the directory /opt/nifi/nifi-1.9.2/work/nar/extensions/
you have to put *.nar packages into nifi/lib directory.
nifi will extract it automatically on startup into nifi/work folder.
As daggett says, you need to use the .nar files, not any unpacked directories.
In your nifi.properties there will be two or more properties that provide locations for NiFi libraries:
nifi.nar.library.directory=./lib
nifi.nar.library.autoload.directory=./extensions
nifi.nar.library.directory.<something>=./<yourdir>
The first is the default and contains all the basic NiFi files. It is only checked on startup and any valid nars found are unpacked in the work directory and loaded. Generally you don't want to add anything here except in test environments as it complicates upgrades.
The second is empty by default but it is scanned every 30 seconds for new .nars. These will be unpacked and loaded if possible, but only for new libraries. Already loaded libraries will not be reloaded.
This is a good location to add your validated custom libraries without having to restart NiFi.
The third and further need to be added manually to the properties file. These are loaded on startup only and useful if you have a lot of custom processors and want to keep them organized.
In your situation I'd put the .nars in the extensions folder and check the logs to see if they were loaded successfully. You'll then need a full refresh of the browser window (Shift+F5 I think) before they show up in the list of processors.
In a cluster setup, add the .nars on all nodes and verify their availability before trying to add them to the canvas or things might get messy.
As of a few weeks ago we added filebeat, metricbeat and apm to our dotnet core application ran on our kubernetes cluster.
It works all nice and recently we discovered filebeat and metricbeat are able to write a different index upon several rules.
We wanted to do the same for APM, however searching the documentation we can't find any option to set the name of the index to write to.
Is this even possible, and if yes how is it configured?
I also tried finding the current name apm-* within the codebase but couldn't find any matches upon configuring it.
The problem which we'd like to fix is that every space in kibana gets to see the apm metrics of every application. Certain applications shouldn't be within this space so therefore i thought a new apm-application-* index would do the trick...
Edit
Since it shouldn't be configured on the agent but instead in the cloud service console. I'm having troubles to 'user-override' the settings to my likings.
The rules i want to have:
When an application does not live inside the kubernetes namespace default OR kube-system write to an index called apm-7.8.0-application-type-2020-07
All other applications in other namespaces should remain in the default indices
I see you can add output.elasticsearch.indices to make this happen: Array of index selector rules supporting conditionals and formatted string.
I tried this by copying the same i had for metricbeat and updated it to use the apm syntax and came to the following 'user-override'
output.elasticsearch.indices:
- index: 'apm-%{[observer.version]}-%{[kubernetes.labels.app]}-%{[processor.event]}-%{+yyyy.MM}'
when:
not:
or:
- equals:
kubernetes.namespace: default
- equals:
kubernetes.namespace: kube-system
but when i use this setup it tells me:
Your changes cannot be applied
'output.elasticsearch.indices.when': is not allowed
Set output.elasticsearch.indices.0.index to apm-%{[observer.version]}-%{[kubernetes.labels.app]}-%{[processor.event]}-%{+yyyy.MM}
Set output.elasticsearch.indices.0.when.not.or.0.equals.kubernetes.namespace to default
Set output.elasticsearch.indices.0.when.not.or.1.equals.kubernetes.namespace to kube-system
Then i updated the example but came to the same conclusion as it was not valid either..
In your ES Cloud console, you need to Edit the cluster configuration, scroll to the APM section and then click "User override settings". In there you can override the target index by adding the following property:
output.elasticsearch.index: "apm-application-%{[observer.version]}-{type}-%{+yyyy.MM.dd}"
Note that if you change this setting, you also need to modify the corresponding index template to match the new index name.
I'm following the steps from the Adobe instructions on How to Build AEM Projects using Maven and I'm not seeing how to populate or configure the meta data for the contents.
I can edit and configure the actual files, but when I push the zip file to the CQ instance, the installed component has a jcr:primaryType of nt:folder and the item I'm trying to duplicate has a jcr:primaryType of cq:Component (as well as many other properties). So is there a way to populate that data without needing to manual interact with CQ?
I'm very new to AEM, so it's entirely possible I've overlooked something very simple.
Yes, this is possible to configure JCR node types without manually changing with CQ.
Make sure you have .content.xml file in component folder and it contains correct jcr:primaryType ( e.g. jcr:primaryType="cq:Component").
This file contains metadata for mapping JCR node on File System.
For beginners it may be useful take a look VLT, that used for import/export JCR on File System. Actually component's files in your project should be similar to VLT component export result from JCR.
I'm trying to experiment with using scripts in the config/scripts directory. The Elasticsearch docs here say this:
Save the contents of the script as a file called config/scripts/my_script.groovy on every data node in the cluster:
This seems like it's probably really easy, but I'm afraid I don't understand how exactly to put a groovy file "on every data node in the cluster". Would this normally be done through the command line somehow, or can it be done by manually moving the groovy file (in Finder on OSX for example)? I have a test index, but when I look at the file structure on the nodes I'm confused where to put the groovy file. Help, pretty please.
You just need to copy the file to each server running elasticsearch. If you're just running elasticsearch on your computer then go to the folder you've installed elasticsearch into and add copy the file into config/scripts in there (you may have to create the folder first). Doesn't matter how the file gets there.
You should see an entry in the logs (or the console if you are running in the foreground) along the lines of
compiling script file [/path/to/elasticsearch/config/scripts/my_script.groovy
This won't show up straightaway - by default elasticsearch checks for new/updated scripts every 60 seconds (you can change this with the watcher.interval setting)
Since file scripts are deprecated (elastic/elasticsearch#24552 & elastic/elasticsearch#24555) this aproach is not going to work anymore.
API it's the only way.