New to world of elastic search.I am trying to connect elastic search to mysql db using jdbc river plugin.
Query Used:
curl -XPUT 'localhost/_river/my_jdbc_river/_meta' -d '{
"type" : "jdbc",
"jdbc" : {
"url" : "jdbc:mysql://local/test",
"user" : "",
"password" : "",
"sql" : "select * from orders ",
}
}'
Exception Observed :
[ERROR][river.jdbc.BulkNodeClient] cluster state is RED and not YELLOW, cowardly refusing to continue with operations
java.io.IOException: cluster state is RED and not YELLOW, cowardly refusing to continue with operations
at org.xbib.elasticsearch.plugin.jdbc.client.ClientHelper.waitForCluster(ClientHelper.java:85)
at org.xbib.elasticsearch.plugin.jdbc.client.node.BulkNodeClient.waitForCluster(BulkNodeClient.java:411)
at org.xbib.elasticsearch.plugin.jdbc.client.node.BulkNodeClient.newClient(BulkNodeClient.java:205)
at org.xbib.elasticsearch.plugin.jdbc.river.JDBCRiver$1.create(JDBCRiver.java:237)
at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverMouth.setIngestFactory(SimpleRiverMouth.java:88)
at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverMouth.setIngestFactory(SimpleRiverMouth.java:45)
at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.createRiverMouth(SimpleRiverFlow.java:304)
at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.beforeFetch(SimpleRiverFlow.java:184)
at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.execute(SimpleRiverFlow.java:148)
at org.xbib.elasticsearch.plugin.jdbc.RiverPipeline.request(RiverPipeline.java:88)
at org.xbib.elasticsearch.plugin.jdbc.RiverPipeline.call(RiverPipeline.java:66)
at org.xbib.elasticsearch.plugin.jdbc.RiverPipeline.call(RiverPipeline.java:30)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2015-02-19 10:39:20,175][INFO ][river.jdbc.SimpleRiverMouth] creating index jdbc with settings {} and mappings null
[2015-02-19 10:39:20,175][ERROR][river.jdbc.SimpleRiverFlow] client is closed
org.elasticsearch.ElasticsearchIllegalStateException: client is closed
at org.xbib.elasticsearch.plugin.jdbc.client.node.BulkNodeClient.newIndex(BulkNodeClient.java:473)
at org.xbib.elasticsearch.plugin.jdbc.client.node.BulkNodeClient.newIndex(BulkNodeClient.java:53)
at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverMouth.beforeFetch(SimpleRiverMouth.java:106)
at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.beforeFetch(SimpleRiverFlow.java:207)
at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.execute(SimpleRiverFlow.java:148)
at org.xbib.elasticsearch.plugin.jdbc.RiverPipeline.request(RiverPipeline.java:88)
at org.xbib.elasticsearch.plugin.jdbc.RiverPipeline.call(RiverPipeline.java:66)
at org.xbib.elasticsearch.plugin.jdbc.RiverPipeline.call(RiverPipeline.java:30)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Can someone help fix this issue?
Additional info :
Elastic search version :-1.4.2
Plugin version:-1.4.0.9
you could install a plugin named "head".
cd $ES_HOME/bin
./plugin -install mobz/elasticsearch-head
then you can view it in a web browser by visiting http://localhost:9200/_plugin/head . It's easy to see the status of shard.
Related
I'm trying to setup an HdfsSinkConnector. This is my worker.properties config:
bootstrap.servers=kafkacluster01.corp:9092
group.id=nycd-og-kafkacluster
config.storage.topic=hive_conn_conf
offset.storage.topic=hive_conn_offs
status.storage.topic=hive_conn_stat
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://my-schemaregistry.co:8081
schema.registry.url=http://my-schemaregistry.co:8081
hive.integration=true
hive.metastore.uris=dev-hive-metastore
schema.compatibility=BACKWARD
value.converter.schemas.enable=true
logs.dir = /logs
topics.dir = /topics
plugin.path=/usr/share/java
and this is the post requst that i'm calling to setup the connector
curl -X POST localhost:9092/connectors -H "Content-Type: application/json" -d '{
"name":"hdfs-hive_sink_con_dom16",
"config":{
"connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector",
"topics": "dom_topic",
"hdfs.url": "hdfs://hadoop-sql-dev:10000",
"flush.size": "3",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url":"http://my-schemaregistry.co:8081"
}
}'
The topic dom_topic already exists (is Avro) but I get the following error from my worker:
INFO Couldn't start HdfsSinkConnector: (io.confluent.connect.hdfs.HdfsSinkTask:72)
org.apache.kafka.connect.errors.ConnectException: java.io.IOException:
Failed on local exception: com.google.protobuf.InvalidProtocolBufferException:
Protocol message end-group tag did not match expected tag.;
Host Details : local host is: "319dc5d70884/172.17.0.2"; destination host is: "hadoop-sql-dev":10000;
at io.confluent.connect.hdfs.DataWriter.<init>(DataWriter.java:202)
at io.confluent.connect.hdfs.HdfsSinkTask.start(HdfsSinkTask.java:64)
at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:207)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:139)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
the hdfs.url I've gotten from hive: jdbc:hive2://hadoop-sql-dev:10000
If I change the port to, say, 9092 I get
INFO Retrying connect to server: hadoop-sql-dev/xxx.xx.x.xx:9092. Already tried 0 time(s); maxRetries=45 (org.apache.hadoop.ipc.Client:837)
I'm running this all on Docker, and my Dockerfile is very simple
#FROM coinsmith/cp-kafka-connect-hdfs
FROM confluentinc/cp-kafka-connect:5.3.1
COPY confluentinc-kafka-connect-hdfs-5.3.1 /usr/share/java/kafka-connect-hdfs
COPY worker.properties worker.properties
# start
ENTRYPOINT ["connect-distributed", "worker.properties"]
Any help would be appreciated.
In a nutshell, I have a standalone ES master instance and a client node that is created within my Java application. The client node discovers the standalone ES instance correctly if the standalone ES instance is started before the client node, which is expected.
The problem I'm facing is this - if for some reason, the client node starts before the standalone ES instance, I see a "MasterNotDiscoveredException", which is again expected. However, I continue seeing the same exception even after I start the standalone ES instance. Is there some configuration I should be changing to fix this?
I'm using ES 1.7.1 with Unicast discovery.
EDIT
Cluster information: The standalone ES instance and the client node together make up a cluster.
Client node stack trace:
11:29:35,634 INFO http [496648366, id=7BCBFQLCTWOO2, ide=tcp://172.17.78.80:61616] [Squidboy] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/172.17.78.80:9200]}
11:29:35,635 INFO node [496648366, id=7BCBFQLCTWOO2, ide=tcp://172.17.78.80:61616] [Squidboy] started
11:30:10,279 ERROR ApplicationLifeCycle [299961584] System startup not complete after 120 seconds ...
11:30:14,706 WARN ElasticSearchStatus [278792216] An Exception occurred during cluster health status update - java.util.concurrent.ExecutionException: org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:292)
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:279)
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:117)
at com.harry.elastic.node.ElasticSearchStatus.updateClusterHealth(ElasticSearchStatus.java:90)
at com.harry.elastic.node.ElasticSearchStatus.access$000(ElasticSearchStatus.java:37)
at com.harry.elastic.node.ElasticSearchStatus$1.run(ElasticSearchStatus.java:62)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
at org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$4.onTimeout(TransportMasterNodeOperationAction.java:164)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:231)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:560)
... 3 more
Client creation code:
private Node createEmbeddedClientNode() {
ImmutableSettings.Builder settingsBuilder = ImmutableSettings.settingsBuilder()
.put("discovery.zen.ping.multicast.enabled", false)
.put("discovery.zen.ping.unicast.hosts", "localhost[9300-9400]");
return nodeBuilder().settings(settingsBuilder).clusterName("harryService")
.client(true).data(false).node();
}
Master discovery configuration
"discovery": {
"zen": {
"ping": {
"multicast": {
"enabled": false
}
}
}
By default your client node will retry pinging your master node each 30s for 3 times and then give up. So if you started your master node after that time has passed, your client node will not discover it.
Try increasing the retries and/or the timeout, that should help.
.put("discovery.zen.fd.ping_timeout", "1m")
.put("discovery.zen.fd.ping_retries", 5)
With those settings, your client node will keep trying during 5 minutes instead of only 1.5 minutes. However, your master node should really be up already when you start your application.
Another settings that might help is the following, as it is true by default and your master will ignore client pings during master election, but since there's a single master node it might not make any difference, still worth a try:
.put("discovery.zen.master_election.filter_client", false)
I solved the problem by adding unicast configuration in the master node explicitly.
"discovery": {
"zen": {
"ping": {
"multicast": {
"enabled": false
},
"unicast": {
"hosts": "localhost[9300-9400]"
}
}
}
}
In MAC elasticsearch to build repository
PUT http://localhost:9400/_snapshot/my_backup
{
"type": "fs",
"settings": {
"location": "/Users/Edison/Elasticsearch/Repository"
}
}
My computer is MacOS
I don't understand , how setting my location path..
This is my error message:
{
"error": "RepositoryException[[my_backup] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, org.elasticsearch.repositories.RepositoryException: [my_backup] location [/Users/Edison/Elasticsearch/Repository] doesn't match any of the locations specified by path.repo because this setting is empty\n at org.elasticsearch.repositories.fs.FsRepository.<init>(Unknown Source)\n while locating org.elasticsearch.repositories.fs.FsRepository\n while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: RepositoryException[[my_backup] location [/Users/Edison/Elasticsearch/Repository] doesn't match any of the locations specified by path.repo because this setting is empty]; ",
"status": 500
}
Elasticsearch nodes require a shared drive for each node to save to, this shared directory is what the location property is referring to.
The first task is to set up this shared storage, for example you could choose a straightforward NFS mount: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-ubuntu-12-04 Once you have a mounted directory on each node, you can then register your backup location.
Alternatively you can use a Samba share, for which this seems to be a guide for: http://vichargrave.com/creating-elasticsearch-snapshots/
Error which I had faced,
"repository_exception","reason":"[my_backup] location [/tmp/my_backup] doesn't match any of the locations specified by path.repo because this setting is empty"}}}
Operating system Centos
[ec2-user#ip-10-33-207-201 config]$ curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{
> "type": "fs",
> "settings": {
> "location": "/tmp/my_backup",
> "compress": true
> }
> }'
{"acknowledged":true}[ec2-user#ip-10-33-207-201 config]$
Solution
You need's to add the repository path to elasticsearch.yml file
path.repo: ["/tmp/my_backup"]
Reference
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html
I've been following the instructions here
and I've completed step 1, the installation of the plugin. I'm now trying to add a snapshot by executing the following command in sense:
PUT /_snapshot/ElasticSearch
{
"type": "s3",
"settings": {
"bucket": "SomeBucket",
"base_path" : "ElasticSearch",
"secret_key": "xxx",
"acess_key" : "xxx"
}
}
From this I get the following error:
{
"error": "RepositoryException[[ElasticSearch] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, java.lang.NoClassDefFoundError: org/elasticsearch/common/blobstore/ImmutableBlobContainer\n at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n while locating org.elasticsearch.repositories.s3.S3Repository\n while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: NoClassDefFoundError[org/elasticsearch/common/blobstore/ImmutableBlobContainer]; nested: ClassNotFoundException[org.elasticsearch.common.blobstore.ImmutableBlobContainer]; ",
"status": 500
}
Any ideas on how to start troubleshooting this? All the information I get in the logs is:
[2015-07-13 16:20:40,519][WARN ][repositories ] [app1050-ela-dacq] failed to create repository [s3][ElasticSearch -d]
org.elasticsearch.common.inject.CreationException: Guice creation errors:
1) Error injecting constructor, java.lang.NoClassDefFoundError: org/elasticsearch/common/blobstore/ImmutableBlobContainer
at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)
while locating org.elasticsearch.repositories.s3.S3Repository
while locating org.elasticsearch.repositories.Repository
1 error
at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:344)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:131)
at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:69)
at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:409)
at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:373)
at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:57)
at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:112)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:374)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:188)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:158)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/common/blobstore/ImmutableBlobContainer
at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:124)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
Note that the version of elasticsearch I'm using is 1.6.0 so the plugin version is 2.6.0. The output of the command 'java -version' is the following:
openjdk version "1.8.0_45"
OpenJDK Runtime Environment (build 1.8.0_45-b13)
OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
It turns out that the version of the plugin that had been installed was actually 2.2.0 instead of 2.6.0.
The answer tlrx gives in here is the clue.
It's also good to remember to restart elasticsearch after you've updated the plugin
I have Elasticsearch properly configured on my server. I can do everything from the command line using cURL. I can even connect to it using cURL from a PHP script outside Yii. However, I can't seem to get it to work from within Yii 2.0.
In my config, I have:
'elasticsearch' => [
'class' => 'yii\elasticsearch\Connection',
'nodes' => [
['http_address' => 'localhost:9200'],
// configure more hosts if you have a cluster
],
],
But when I try to do a simple query in Yii, I get this error. Note how it's using my server ip address rather than 'localhost' or '172.0.0.1'. Note: I've hashed out my ip address for sercurity.
Elasticsearch Database Exception – yii\elasticsearch\Exception
Elasticsearch request failed: 7 - Failed to connect to ##.##.##.### port 9200: Connection refused
Error Info: Array
(
[requestMethod] => GET
[requestUrl] => http://##.##.##.###:9200/profiles/profile/_search
[requestBody] => {"size":100,"query":{"match_all":{}}}
[responseHeaders] => Array
(
)
[responseBody] =>
)
I was able to fix this error by updating the version of Elasticsearch to something > 1.3.0 as this is the minimum requirement for YIISOFT/YII2-ELASTICSEARCH
run curl -X GET 'http://127.0.0.1:9200' to check what version you are running.
First follow this steps to download elastic search.
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.5.2.tar.gz
mkdir es
tar -xf elasticsearch-1.5.2.tar.gz -C es
cd es
./bin/elasticsearch
Then you must be able to access to localhost:9200 and get something like this below :
{
"name" : "Sigyn",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.4.0",
"build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55",
"build_timestamp" : "2016-08-29T09:14:17Z",
"build_snapshot" : false,
"lucene_version" : "5.5.2"
},
"tagline" : "You Know, for Search"
}
Then secondly,follow instruction in https://github.com/yiisoft/yii2-elasticsearch. Then you are done