I was following this tutorial on setting up es locally with springboot: https://piotrminkowski.com/2019/03/29/elasticsearch-with-spring-boot/.
In the tutorial, a docker container is created which spring can point to.
Following the steps taken, I launch 2 containers on ports 5000 and 9200/9300.
Accessing the link http://localhost:5000 I get an error that it cannot find other clusters. If I try to connect to http://localhost:9200 from the browser I see the configuration json.
{
"name": "ACIxTxG",
"cluster_name": "docker-cluster",
"cluster_uuid": "Tea0VEBrQhm8ys9WRQyivw",
"version": {
"number": "6.6.2",
"build_flavor": "default",
"build_type": "tar",
"build_hash": "3bd3e59",
"build_date": "2019-03-06T15: 16: 26.864148Z",
"build_snapshot": false,
"lucene_version": "7.6.0",
"minimum_wire_compatibility_version": "5.6.0",
"minimum_index_compatibility_version": "5.0.0"
},
"tagline": "You Know, for Search"
}
but the connection is refused. Anyone have directions on how to fix?
Related
I have a cluster of 3 nodes of consul servers. I have registered one service(FooService) with one of the server(Server1). When i check the registered services using http (/v1/agent/services) from the server(Server1) it is showing correctly. But when i try the same with any of other server(ie, Server1 /Server2) its not listing this registered service. This issue is not happening for KV Store. Can someone suggest a fix for this?
consul version : 1.2.1
I have pasted my configuration below
{
"bootstrap_expect": 3,
"client_addr": "0.0.0.0",
"datacenter": "DC1",
"data_dir": "/var/consul",
"domain": "consul",
"enable_script_checks": true,
"dns_config": {
"enable_truncate": true,
"only_passing": true
},
"enable_syslog": true,
"encrypt": "3scwcXQpgNVo1CZuqlSouA==",
"leave_on_terminate": true,
"log_level": "INFO",
"rejoin_after_leave": true,
"server": true,
"start_join": [
"10.0.0.242",
"10.0.0.243",
"10.0.0.244"
],
"ui": true
}
What i understood is , spring boot app should always connect to local consul client. Then this issue will not occur.
Newbie to Microservices here.
I have been looking into develop a microservice with spring actuator while having Consul for service discovery and fail recovery.
I have configured a cluster as explained in Consul documentation.
Now what I'm trying to do is configure a Consul Watch to trigger when any of my service is down and execute a shell script to restart my service. Following is my configuration file.
{
"bind_addr": "127.0.0.1",
"datacenter": "dc1",
"encrypt": "EXz7LsrhpQ4idwqffiFoQ==",
"data_dir": "/data",
"log_level": "INFO",
"enable_syslog": true,
"enable_debug": true,
"enable_script_checks": true,
"ui":true,
"node_name": "SpringConsulClient",
"server": false,
"service": { "name": "Apache", "tags": ["HTTP"], "port": 8080,
"check": {"script": "curl localhost >/dev/null 2>&1", "interval": "10s"}},
"rejoin_after_leave": true,
"watches": [
{
"type": "service",
"handler": "/Consul-Script.sh"
}
]
}
Any help/tip would be greatly appreciate.
Regards,
Chrishan
Take a closer look at the description of the service watch type in the official documentation. It has an example, how you can specify it:
{
"type": "service",
"service": "redis",
"args": ["/usr/bin/my-service-handler.sh", "-redis"]
}
Note that it has no property handler and but takes a path to the script as an argument. And one more:
It requires the "service" parameter
It seems, in you case you need to specify it as follows:
"watches": [
{
"type": "service",
"service": "Apache",
"args": ["/fully/qualified/path/to/Consul-Script.sh"]
}
]
I am trying to connect to my Elasticsearch server using the Java Api and shield. I can execute index, get, delete and search operations on the existing cluster using sense plugin (e.g) and via curl on 9200. I've seen other threads about this but none of them worked and none of them were trying to connect to a Elasticsearch webserver with shield.
I used the same API to connect with my localhost of elasticsearch and it worked fine however when I try to connect with my web server I always get the same error:
Error
1342 [main] DEBUG org.elasticsearch.shield.transport.netty - [Benjamin Jacob Grimm] connected to node [{#transport#-1}{HOST_IP}{HOST/HOST_IP:9300}]
1431 [elasticsearch[Benjamin Jacob Grimm][generic][T#1]] DEBUG org.elasticsearch.shield.transport.netty - [Benjamin Jacob Grimm] disconnecting from [{#transport#-1}{HOST_IP}{HOST/HOST_IP:9300}], channel closed event
1463 [main] INFO org.elasticsearch.client.transport - [Benjamin Jacob Grimm] failed to get node info for {#transport#-1}{HOST_IP}{HOST/HOST_IP:9300}, disconnecting...
NodeDisconnectedException[[][HOST/HOST_IP:9300][cluster:monitor/nodes/liveness] disconnected]
...9200/_nodes
"cluster_name": "elasticsearch",
"nodes": {
"UYdZbCQKQZavtFYOoUpawg": {
"name": "Desmond Pitt",
"transport_address": "HOST_IP:9300",
"host": "HOST_IP",
"ip": "HOST_IP",
"version": "2.3.3",
"build": "218bdf1",
"http_address": "HOST_IP:9200",
"settings": {
"pidfile": "/var/run/elasticsearch/elasticsearch.pid",
"cluster": {
"name": "elasticsearch"
},
"path": {
"conf": "/etc/elasticsearch",
"data": "/var/lib/elasticsearch",
"logs": "/var/log/elasticsearch",
"home": "/usr/share/elasticsearch"
},
"shield": {
"http": {
"ssl": "true"
},
"https": {
"ssl": "true"
},
"transport": {
"ssl": "true"
}
},
"name": "Desmond Pitt",
"client": {
"type": "node"
},
"http": {
"cors": {
"allow-origin": "*",
"allow-headers": "Authorization, Origin, X-Requested-With, Content-Type, Accept",
"allow-credentials": "true",
"allow-methods": "OPTIONS, HEAD, GET, POST, PUT, DELETE",
"enabled": "true"
}
},
"index": {
"queries": {
"cache": {
"type": "opt_out_cache"
}
}
},
"foreground": "false",
"config": {
"ignore_system_properties": "true"
},
"network": {
"host": "HOST_IP",
"bind_host": "0.0.0.0",
"publish_host": "HOST_IP"
}
}
Java code:
TransportClient client = TransportClient.builder()
.addPlugin(ShieldPlugin.class)
.settings(Settings.builder()
.put("cluster.name", ClusterName)
.put("shield.user", "USER:PASSWORD")
.build())
.build()
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(HOST), 9300));
I've tried as stated on Can't connect to ElasticSearch server using Java API to sync my Java API java version and my server and currently i'm using:
Java API:
C:\Program Files\Java\jdk1.8.0_92
Server:
"version": "1.8.0_91",
"vm_name": "OpenJDK 64-Bit Server VM",
I don't know if it has any problem using ...0_91 and 0_92 but doesn't seem to make any difference because the java API works weel on my localhost server.
If you need more information feel free to ask.
Thanks in advance!
UPDATE:
Changes I did in elasticsearch.yml
shield.ssl.keystore.path: /usr/share/elasticsearch/bin/shield/elastic.jks
shield.ssl.keystore.password: password
shield.ssl.keystore.key_password: password
shield.transport.ssl: true
shield.http.ssl: true
shield.https.ssl: true
network.host: HOST_IP
network.publish_host: HOST_IP
shield.ssl.hostname_verification.resolve_name: false
Result of https://HOST:9200/_cluster/health?pretty=true
{
"cluster_name": "elasticsearch",
"status": "yellow",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 5,
"active_shards": 5,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 5,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 50
}
UPDATE2:
I've tried activate SSL according to official documentation and I got the following errors:
2082 [elasticsearch[Steel Serpent][transport_client_worker][T#1]{New I/O worker #1}] DEBUG org.elasticsearch.shield.transport.netty - [Steel Serpent] SSL/TLS handshake failed, closing channel: null
java.nio.channels.ClosedChannelException
at org.jboss.netty.handler.ssl.SslHandler.channelDisconnected(SslHandler.java:575)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Temporary Solution
After that try I did as Vladislav Kysliy suggested and disabled SSL and it worked but I'm looking for a real solution and not a temporary one.
As i can see you enabled SSL encryption. But your java code didn't activate SSL. According official documentation you should use something like this:
TransportClient client = TransportClient.builder()
.addPlugin(ShieldPlugin.class)
.settings(Settings.builder()
.put("cluster.name", "myClusterName")
.put("shield.user", "transport_client_user:changeme")
.put("shield.ssl.keystore.path", "/path/to/client.jks") (1)
.put("shield.ssl.keystore.password", "password")
.put("shield.transport.ssl", "true")
...
.build())
Moreover i would test my code without any encryption and add some new features(e.g SSL) to config and code step by step.
UPD: To be honest remotely fixing ssl issues will be tricky. This errors often appeared when client sends an invalid SSL certificate. Probably you need to disable client auth
Because of you use SSL + Shield the main idea is check your functionality step-by-step: disable SSL - check in Java -API client, enable SSL - check again.
I made an application Dockerized in a Docker container. I intended to make the application able to access files from our HDFS. The Docker image is to be deployed on the same cluster where we have HDFS installed via Marathon-Mesos.
Below is the json to be POST to Marathon. It seems that my app is able to read and write files in the HDFS. Can someone comment on the safety of this? Would files changed by my app correctly changed in the HDFS as well? I Googled around and didn't find any similar approaches...
{
"id": "/ipython-test",
"cmd": null,
"cpus": 1,
"mem": 1024,
"disk": 0,
"instances": 1,
"container": {
"type": "DOCKER",
"volumes": [
{
"containerPath": "/home",
"hostPath": "/hadoop/hdfs-mount",
"mode": "RW"
}
],
"docker": {
"image": "my/image",
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 8888,
"hostPort": 0,
"servicePort": 10061,
"protocol": "tcp",
}
],
"privileged": false,
"parameters": [],
"forcePullImage": true
}
},
"portDefinitions": [
{
"port": 10061,
"protocol": "tcp",
"labels": {}
}
]
}
You might have a look at the Docker volume docs.
Basically, the volumes definition in the app.json would trigger the start of the Docker image with the flag -v /hadoop/hdfs-mount:/home:RW, meaning that the host path gets mapped to the Docker container as /home in read-write mode.
You should be able to verify this if you SSH into the node which is running the app and do a docker inspect <containerId>.
See also
https://mesosphere.github.io/marathon/docs/native-docker.html
I have two vagrant hosts running local on my machine and I am trying to hit a container within one host from a second container on the other host.
When I curl from within the container:
curl search.mydomain.localhost:9090/ping
I receive the curl: (7) Failed to connect to search.mydomain.localhost port 9090: Connection refused
However when I curl without specifying the port:
curl search.mydomain.localhost/ping
OK
I'm certain the port is properly exposed as if I try the same from within the host instead of within the container I get:
curl search.mydomain.localhost:9090/ping
OK
Which shows that the service at port 9090 is exposed, however there is some networking issue with the container trying to reach it.
A fellow dev running the same versions of vbox/vagrant/docker/docker-compose and using an identical commit of the repos has no trouble hitting the service from within the container. I'm really stumped as to what to try from here...
I'm using the default bridge network:
sudo brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.02427c9cea3c no veth5dc6655
vethd1867df
docker network inspect bridge
[
{
"Name": "bridge",
"Id": "e4b8df614d4b8c451cd4a26c5dda09d22d77de934a4be457e1e93d82e5321a8b",
"Scope": "local",
"Driver": "bridge",
"IPAM": {
"Driver": "default",
"Config": [
{
"Subnet": "172.17.0.1/16",
"Gateway": "172.17.0.1"
}
]
},
"Containers": {
"1d67a1567ff698694b5f10ece8a62a7c2cdcfcc7fac6bc58599d5992def8df5a": {
"EndpointID": "4ac99ce582bfad1200d59977e6127998d940a688f4aaf4f3f1c6683d61e94f48",
"MacAddress": "02:42:ac:11:00:03",
"IPv4Address": "172.17.0.3/16",
"IPv6Address": ""
},
"3e8b6cbd89507814d66a026fd9fad26d649ecf211f1ebd72ed4689b21e781e2c": {
"EndpointID": "2776560da3848e661d919fcc24ad3ab80e00a0bf96673e9c1e0f2c1711d6c609",
"MacAddress": "02:42:ac:11:00:02",
"IPv4Address": "172.17.0.2/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
}
}
]
I'm on Docker version 1.9.0, build 76d6bc9 and docker-compose version: 1.5.0.
Any help would be appreciated.
I resolved my issue which seems like it might be a bug. Essentially the container was inheriting the /etc/hosts from my local macbook, bypassing the /etc/hosts on the actual vagrant host running the container, causing my entry of "127.0.0.1 search.mydomain.localhost" to make all connection attempts within the container redirect to itself.