we are trying to setup a multiregion cassandra cluster on ec2. Our configuration looks like
5 nodes each on us-east-1a,us-east-1b,us-east-1c,us-west-1a. For this we have modified the cassandra-rackdc.properties file.
using GossipingPropertyFileSnitch and modified cassandra.yaml file accordingly
we are using all 20 public ips for the seeds configuration in cassandra.yaml file
We have commented out the listen_address and rpc_address property so that cassandra defaults to using InetAddress.getLocalHost()
We have uncommented the broadcast address to use public ip
we have modifed the agents address.yaml file to use public ip address for the properties stomp_interface and local_interface
We are starting the nodes one by one with a 3 min pause in between.
Issue:
When using the opscenter. It shows only one node in the cluster
the 'nodetool status' command also shows only one node
When using cql statement it does show all of its peers
What is the mistake we are doing?
I am doing something similar as a proof-of-concept. I have a working 2-region cluster. Here are the things that I did differently, from reading your question:
I used the Ec2MultiRegionSnitch, which is designed to handle the public and private IPs in EC2.In AWS, the Elastic IP is not bound to the interface by the instance and this causes problems with the cluster communications.
in cassandra.yaml, I left listen_address as the private IP.
also, set rpc_address to 0.0.0.0
uncomment broadcast_address and set it to the public IP (like you did).
i set up dc_suffix in the cassandra-rackdc.properties file and uncommented prefer_local=true (inside the region, Cassandra will prefer to use private IPs).
I opened the security groups for Cassandra so that tcp ports 7000 and 7001 could talk between the nodes in the 2 different regions. Opscenter uses ports 61620 and 61621.
All nodes have the same cluster name.
seed IPs are set to the public IPs. I didn't use all the nodes as seeds, that's not recommended.
Start the seeds first, followed by the other nodes.
This provided a working cluster. Now I am working on ssl node-to-node communication between regions.
Related
We have 6 redis servers running in ports (8001, 8002, 8003, 8004, 8005, 8006).
On the redis.conf of every Redis server we bind the ip in different ways like:
bind 0.0.0.0
bind PRIVATE PUBLIC
bind PUBLIC
If we access like it works fine:
redis-cli -h PUBLIC_IP -p 8001
But when we wanna create the clusters we run:
./src/redis-cli --cluster create PUBLIC_IP:8001 PUBLIC_IP:8002 PUBLIC_IP:8003 PUBLIC_IP:8004 PUBLIC_IP:8005 PUBLIC_IP:8006 --cluster-replicas 1
The console always shows and keeps in Waiting for the cluster forever:
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica PUBLIC_IP:8005 to PUBLIC_IP:8001
Adding replica PUBLIC_IP:8006 to PUBLIC_IP:8002
Adding replica PUBLIC_IP:8004 to PUBLIC_IP:8003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 7ab009459f7f5cf6cef5f46b691748dc236e4c26 PUBLIC_IP:8001
slots:[0-5460] (5461 slots) master
M: 0048ca2cd65c1315b8f0a7c952b69bfb494d5ace PUBLIC_IP:8002
slots:[5461-10922] (5462 slots) master
M: c6ee023719f200b0d175f428fa15e5ab767d0e04 PUBLIC_IP:8003
slots:[10923-16383] (5461 slots) master
S: cf636a1a46b1e947daec3e797cac524c613f08ca PUBLIC_IP:8004
replicates 7ab009459f7f5cf6cef5f46b691748dc236e4c26
S: 5d4bd1041457114353b0b30dbefd86ab8e4ae020 PUBLIC_IP:8005
replicates 0048ca2cd65c1315b8f0a7c952b69bfb494d5ace
S: 62f01289dc3f72cac4a1745fc77b7bd91ec5d107 PUBLIC_IP:8006
replicates c6ee023719f200b0d175f428fa15e5ab767d0e04
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
A lot of people says that we need to bind the private ip, but we wanna do it on public becase when we connect with the external machines the clustering redirect to the master that contains the key, if we bind the private ip the redirect will shows "redirect to PRIVATE_IP" and that will not work as expected.
Are we missing something to let the cluster join by public IP?
From redis security guide:
Redis is designed to be accessed by trusted clients inside trusted environments.
See also: How to connect to redis from remote guide
When a server binds on its public ip, it can get requests from everyone, so unless you built some security around it anyone can access and manipulate your data.
In redis cluster the rules are the same and the replicas which binds on public ips are exposed.
The default use case for a redis cluster is that one machine (or multiple machines) access it from within it's private network, and you shouldn't divert from that unless you know what you are doing security wise.
If it makes sense for your use case, you should make the machine which access the redis cluster a part of the cluster private network.
What I would be doing if I were at your place is:
Bind all the servers with private ip and loopback ip i.e bind {{ private_ip }} 127.0.0.1
Enable ufw (or other any firewalling tool) on each server and do (for ufw) allow from {{ private_ip }} to any port {{ redis_port }} or similar.
My internal DNS will have entry for all the servers with their respective private ip.
Voila! create and access redis cluster securely without any security breach.
NOTE: if you still want to access them over public network then you can do some workaround with SNAT
WARNING: binding redis server to 0.0.0.0 or public ip might cause serious vulnerability issues like:
https://www.exploit-db.com/exploits/47195
https://medium.com/#knownsec404team/rce-exploits-of-redis-based-on-master-slave-replication-ef7a664ce1d0
PS: You can also follow this medium tutorial.
For example; I have 3 nifi nodes in nifi cluster. Example hostnames of these nodes;
192.168.12.50:8080(primary)
192.168.54.60:8080
192.168.95.70:8080
I know that I can access to nifi-rest api from all nifi nodes. I have GetHTTP processor for get cluster summary from rest-api, and this processor runs on only pimary node. I did set "URL" property of this processor to 192.168.12.50:8080/nifi-api/controller/cluster.
But, if primary node is down, new primary node will be elected. Thus, I will not be able to access 192.168.12.50:8080 address from new primary node. Because this node was down. So, I will not be able to get cluster summary result from rest-api.
In this case, Can I use "localhost:8080/nifi-api/controller/cluster" instead of "192.168.12.50:8080/nifi-api/controller/cluster" for each node in nifi cluster?
It depends on a few things... if you are running securely then you have certificates that are generated for each node specific to the hostname, so the host in the web requests needs to match the host in the certificates, so you can't use localhost in that case.
It also depends how NiFi's web server is configured. If nifi.web.http.host or nifi.web.https.host has a specific hostname specified, then the web server is only bound to that hostname and may not accept connections with a different hostname. In a default unsecure setup, if you leave nifi.web.http.host blank then it binds to all interfaces.
You may be able to use the expression language function to obtain the hostname of the current node. So you could make the url something like "http://${hostname()}/nifi-api/controller/cluster".
my cluster it's not in the same network, and there is a tunnel between my pc and the server.
i have got this error :
org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available
If you are using elasticsearch default multicast mechanism to discover nodes in the cluster you must have all cluster nodes located on the same subnet (this is true till version 2.0).
In order to have your node discovering the other nodes in the cluster you may configure [elasticsearch home]/config/elasticsearch.yaml field name: discovery.zen.ping.unicast.hosts as described [here]
(https://www.elastic.co/guide/en/elasticsearch/reference/2.x/modules-network.html):
discovery.zen.ping.unicast.hosts
In order to join a cluster, a node needs to know the hostname or IP address of at least some of the other nodes in the cluster. This >settting provides the initial list of other nodes that this node will try to contact. Accepts IP addresses or hostnames.
Defaults to ["127.0.0.1", "[::1]"].
Hope it helps.
I have tried to recreate your configuration on my environment and managed to work with Elasticsearch (created an index). Here is how it goes:
Configure Putty tunneling for Elasticsearch 9300 and 9200 ports
After configuring you'll need to open the SSH connection and make sure it is connected
You may look at the SSH event log here is a link on how to do it
The code looks like this
public class App
{
public static void main( String[] args ) throws Exception
{
Settings settings = ImmutableSettings.settingsBuilder().
put("cluster.name", "my-cluster").build();
TransportClient client = new TransportClient(settings)
.addTransportAddress(
new InetSocketTransportAddress(
"localhost", 9093));
CreateIndexResponse rs = client.admin().indices().create(new CreateIndexRequest("tunnelingindex")).actionGet();
System.out.println(rs.isAcknowledged());
client.close();
}
}
The code creates an index named tunnelingindex
If it still do not work for you, I think that you may have an issue which is not related to tunneling or Elasticsearch.
Hope I have managed to help.
you must set:
transport.publish_host: localhost
details here:
Elasticsearch basic client connection
I've been trying to use the lovely ansible-elasticsearch project to set up a nine-node Elasticsearch cluster.
Each node is up and running... but they are not communcating with each other. The master nodes think there are zero data nodes. The data nodes are not connecting to the master nodes.
They all have the same cluster.name. I have tried with multicast enabled (discovery.zen.ping.multicast.enabled: true) and disabled (previous setting to false, and discovery.zen.ping.unicast.hosts:["host1","host2",..."host9"]) but in either case the nodes are not communicating.
They have network connectivity to one another - verified via telnet over port 9300.
Sample output:
$ curl host1:9200/_cluster/health
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":"waited for [30s]"}],"type":"master_not_discovered_exception","reason":"waited for [30s]"},"status":503}
I cannot think of any more reasons why they wouldn't connect - looking for any more ideas of what to try.
Edit: I finally resolved this issue. The settings that worked were publish_host to "_non_loopback:ipv4_" and unicast with discovery.zen.ping.unicast.hosts set to ["host1:9300","host2:9300","host3:9300"] - listing only the dedicated master nodes. I have a minimum master node count of 2.
The only reasons I can think that can cause that behavior are:
Connectivity issues - Ping is not a good tool to check that nodes can connect to each other. Use telnet and try connecting from host1 to host2 on port 9300.
Your elasticsearch.yml is set to bind 127.0.0.1 or the wrong host (if you're not sure, bind 0.0.0.0 to see if that solves your connectivity issues and then it's important to change it to bind only internal hosts to avoid exposure of elasticsearch directly to the internet).
Your publish_host is incorrect - This usually happens when you run ES inside a docker container for example, you need to make sure that the publish_host is set to an address that can be accessed via other hosts.
What is the correct configuration for a mulit-region setup in EC2 instances?
What should the listen_address, broadcast_address, rpc_address and seed ip/addresses be to work?
When do you use public IP address and when do you use private IP addresses?
According to the docs:
broadcast_address: (Default: listen_address) If your Cassandra cluster is deployed across multiple Amazon EC2 regions and you use the EC2MultiRegionSnitch, set the broadcast_address to public IP address of the node and the listen_address to the private IP.
listen_address: (Default: localhost) The IP address or hostname that other Cassandra nodes use to connect to this node. If left unset, the hostname must resolve to the IP address of this node using/etc/hostname, /etc/hosts, or DNS. Do not specify 0.0.0.0.
rpc_address: (Default: localhost) The listen address for client connections (Thrift remote procedure calls).
seed_provider: (Default: org.apache.cassandra.locator.SimpleSeedProvider) A list of comma-delimited hosts (IP addresses) to use as contact points when a node joins a cluster. Cassandra also uses this list to learn the topology of the ring. When running multiple nodes, you must change the - seeds list from the default value (127.0.0.1). In multiple data-center clusters, the - seeds list should include at least one node from each data center (replication group)
Trying to summarize:
the rpc_address is used for client connections and has nothing to do with multi-region EC2
the listen_address and broadcast_address are the 2 important options for multi-region EC2 configuration
in general when configuring any of these answer 2 questions:
who is connecting? (another nodes? clients?)
what IPs are accessible? (is this network interface accessible to who is connecting?)