I have a HDP cluster & have java client that needs to download the configuration files (hdfs-site.xml, core-site.xml, hbase-site) at runtime? how do i achieve this? Cloudera Manager provides provide URL way downloading config files, do we have something similar with ambari?
You may use Ambari API or command line utility to download properties and their values set for any configuration file. (But you can download in json format not config file as it is in xml format.)
API:
Step1: Find latest config version
curl -u admin:admin -H "X-Requested-By: ambari" -X GET http://<AMBARI_SERVER_HOST>:8080/api/v1/clusters/<CLUSTER_NAME>?fields=Clusters/desired_configs
Sample OUTPUT
{
"href" : "http://AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME?fields=Clusters/desired_configs",
"Clusters" : {
"cluster_name" : "CLUSTER_NAME",
"version" : "HDP-2.0.6",
"desired_configs" : {
...
"mapred-site" : {
"user" : "admin",
"tag" : "version1384716039631"
}
...
}
}
}
Step2: Download configuration
curl -u admin:admin -H "X-Requested-By: ambari" -X GET "http://<AMBARI_SERVER_HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/configurations?type=mapred-site&tag=version1384716039631"
Sample OUTPUT
{
"href" : "http://AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME/configurations?type=mapred-site&tag=version1384716039631",
"items" : [
{
"href" : "http://AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME/configurations?type=mapred-site&tag=version1384716039631",
"tag" : "version1384716039631",
"type" : "mapred-site",
"Config" : {
"cluster_name" : "CLUSTER_NAME"
},
"properties" : {
... THESE ARE THE PROPERTY KEY-VALUE PAIRS ...
}
}]
}
USING COMMAND LINE UTILITY
/var/lib/ambari-server/resources/scripts/configs.sh get <ambari host> <cluster name> <config file type>
Example: /var/lib/ambari-server/resources/scripts/configs.sh get localhost Sandbox mapred-site
More details are available at following link
https://cwiki.apache.org/confluence/display/AMBARI/Modify+configurations
Related
I just tried working with elasticsearch and now trying to create first watcher
There are some information I have read in elasticsearch documentation : https://www.elastic.co/guide/en/x-pack/current/watcher-getting-started.html
And now I trty to create one :
https://es.origin-test.cloud.rccf.ru/apiconnect508/_xpack/watcher/watch/audit_watch
PUT method + auth headers
I put in :
{ "trigger" : {
"schedule": {
"interval": "1h"
}
}, "actions" : { "send_email" : {
"email" : {
"to" : "ext_avolkova#rencredit.ru",
"subject" : "Watcher Notification",
"body" : "{{ctx.payload.hits.total}} logs found"
} } } }
But now I see mistake :
No handler found for uri
[/apiconnect508/_xpack/watcher/watch/log_audit] and method [PUT]
Please, help me to create one simple watcher
Based on the support matrix, elasticsearch 2.x is not compatible with x-pack.
You might want to install Watcher as a separate plugin using this document.
I want to create index alias in template file, I have specified index name as "test_2017_12_02" in logstash conf file, my template is as below
"aliases" : {"test_2017_12_02" : "test"}
but not working, the index getting created without alias
Try to use Dev Tools in kibana of elasticsearch.
POST /_aliases
{
"actions" : [
{ "add" : { "index" : "test_2017_12_02", "alias" : "test" } }
]
}
ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html
I'm trying to view my geojson on any sort of map in Kibana.
My original data is a geo polygon with an array of coordinates.
From my understanding, ElasticSearch/Kibana can't visualize geo shapes, so I'm trying to make a coordinate a geopoint so that I can view it on a tilemap.
Is this possible? I've tried to create a couple different mappings. My most current one won't let me index data to it. Any better approaches? (In an ideal world I could plot a polygon...though I don't think Kibana supports this). I am using version 5.3
Original Data (replacing actual values with long and lat):
{
"geometry": {
"type": "Polygon",
"coordinates": [
[
[long,lat],
[long,lat],
[long,lat],
[long,lat],
[long,lat]
]
]
},
This is the mapping elasticsearch defaults to when I index my json:
{
"indexname" : {
"mappings" : {
"my_type" : {
"properties" : {
"geometry" : {
"properties" : {
"coordinates" : {
"type" : "float"
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
This is the attempt I just tried to fix the mapping (however, this approach does not take to any indexed data:
curl -XPUT "http://localhost:9200/indexname" -d "{\"mappings\" : {\"my_type\" : {\"properties\" : {\"geometry\" : {\"type\":\"geo_point\"}}}}}"
If I try this command it shows up in kibana, but when I try to run a tilemap, the map just disappears, so I'm assuming it is not getting the correct data:
curl -XPUT "http://localhost:9200/indexname" -d "{\"mappings\" : {\"my_type\" : {\"properties\" : {\"coordinates\" : {\"type\":\"geo_point\"}}}}}"
EDIT
No success. I tried:
kibana-plugin install file:///kibana-5.3.0-windows-x86/kibana-5.3.0-windows-x86/plugins/enhanced-tilemap-v2017-03-17-5.2.2/kibana/enhanced_tilemap
Attempting to transfer from file:///kibana-5.3.0-windows-x86/kibana-5.3.0-windows-x86/plugins/enhanced-tilemap-v2017-03-17-5.2.2/kibana/enhanced_tilemap
Transferring unknown number of bytes
Error: EISDIR: illegal operation on a directory, read
Plugin installation was unsuccessful due to error "EISDIR: illegal operation on a directory, read"
kibana-plugin install http://artifacts.elastic.co/downloads/kibana-plugins/enhanced_tilemap/enhanced_tilemap-5.2.2.zip
Attempting to transfer from http://artifacts.elastic.co/downloads/kibana-plugins/enhanced_tilemap/enhanced_tilemap-5.2.2.zip
Attempting to transfer from https://artifacts.elastic.co/downloads/kibana-plugins/http://artifacts.elastic.co/downloads/kibana-plugins/enhanced_tilemap/enhanced_tilemap-5.2.2.zip/http://artifacts.elastic.co/downloads/kibana-plugins/enhanced_tilemap/enhanced_tilemap-5.2.2.zip-5.3.0.zip
Plugin installation was unsuccessful due to error "No valid url specified."
The next kibana plugin allow you visualize polygons.
If you using kibana 5.3, the plugin installation not support install operation above 5.2
You can do one of to option:
1) open issue in plugin github
2) clone the plugin, extract the zip file to plugin directory vin kibana home,
cd to the plugin that you download and type bower install, restart kibana
I'm following this tutorial.
I start with installing attachment-mapper (replaced their link with latest version).
bin/plugin -install elasticsearch/elasticsearch-mapper-attachments/2.4.1
Start new, delete "test" index and then create new one:
curl -X DELETE "localhost:9200/test"
Create index, I presume:
curl -X PUT "localhost:9200/test" -d '{
"settings" : { "index" : { "number_of_shards" : 1, "number_of_replicas" : 0 }}
}'
Then I try to create mapping:
curl -X PUT "localhost:9200/test/attachment/_mapping" -d '{
"attachment" : {
"properties" : {
"file" : {
"type" : "attachment",
"fields" : {
"title" : { "store" : "yes" },
"file" : { "term_vector":"with_positions_offsets", "store":"yes" }
}
}
}
}
}'
Then I get this error:
{
"error" : "NoSuchMethodError[org.elasticsearch.index.mapper.core.TypeParsers.parseMultiField(Lorg/elasticsearch/index/mapper/core/AbstractFieldMapper$Builder;Ljava/lang/String;Lorg/elasticsearch/index/mapper/Mapper$TypeParser$ParserContext;Ljava/lang/String;Ljava/lang/Object;)V]",
"status" : 500
}
Any idea what's going on?
Could it be a problem with the attachment-mapper plugin installation?
attachment-mapper uses Tika. I've installed Tika, maybe that's installed wrong? How do I check?
Any insight would be helpful.
I had the wrong version of ElasticSearch installed.
For the attachment-mapper plugin I had installed, I needed elasticsearch version 1.4.
Removed old version, installed new version, installed attachment-mapper plugin, started service, and ran through tutorial again and it worked.
I start to configure an ActiveMQ river, I'm already installed the (ActiveMQ plugin) but I feel confused about how to make it working, the documentation was so brief, Actually, I follow exactly the steps of creating a new river but I don't know what are the following steps to follow?
Note:
I have the an ActiveMQ server up and running and I tested it using a
simple JMS app to push a message into a queue.
I created a new river using:
curl -XPUT 'localhost:9200/_river/myindex_river/_meta' -d '{
"type" : "activemq",
"activemq" : {
"user" : "guest",
"pass" : "guest",
"brokerUrl" : "failover://tcp://localhost:61616",
"sourceType" : "queue",
"sourceName" : "elasticsearch",
"consumerName" : "activemq_elasticsearch_river_myindex_river",
"durable" : false,
"filter" : ""
},
"index" : {
"bulk_size" : 100,
"bulk_timeout" : "10ms"
}
}'
After creating the previous river, I could get it's status using
curl -XGET 'localhost:9200/my_index/_status', it give me the index
status, not the created river.
Please, any help to get me the right road with ActiveMQ river configuration with the elasticsearch.
I told you on the mailing list. Define index.index value or set the name of your river to be your index name (easier):
curl -XPUT 'localhost:9200/_river/my_index/_meta' -d '
{
"type":"activemq",
"activemq":{
"user":"guest",
"pass":"guest",
"brokerUrl":"failover://tcp://localhost:61616",
"sourceType":"queue",
"sourceName":"elasticsearch",
"consumerName":"activemq_elasticsearch_river_myindex_river",
"durable":false,
"filter":""
},
"index":{
"bulk_size":100,
"bulk_timeout":"10ms"
}
}'
or
curl -XPUT 'localhost:9200/_river/myindex_river/_meta' -d '
{
"type":"activemq",
"activemq":{
"user":"guest",
"pass":"guest",
"brokerUrl":"failover://tcp://localhost:61616",
"sourceType":"queue",
"sourceName":"elasticsearch",
"consumerName":"activemq_elasticsearch_river_myindex_river",
"durable":false,
"filter":""
},
"index":{
"index":"my_index",
"bulk_size":100,
"bulk_timeout":"10ms"
}
}'
It should help.
If not, update your question with what you can see in logs.