Redis Cluster Create Replicas Bind Public IP - amazon-ec2

We have 6 redis servers running in ports (8001, 8002, 8003, 8004, 8005, 8006).
On the redis.conf of every Redis server we bind the ip in different ways like:
bind 0.0.0.0
bind PRIVATE PUBLIC
bind PUBLIC
If we access like it works fine:
redis-cli -h PUBLIC_IP -p 8001
But when we wanna create the clusters we run:
./src/redis-cli --cluster create PUBLIC_IP:8001 PUBLIC_IP:8002 PUBLIC_IP:8003 PUBLIC_IP:8004 PUBLIC_IP:8005 PUBLIC_IP:8006 --cluster-replicas 1
The console always shows and keeps in Waiting for the cluster forever:
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica PUBLIC_IP:8005 to PUBLIC_IP:8001
Adding replica PUBLIC_IP:8006 to PUBLIC_IP:8002
Adding replica PUBLIC_IP:8004 to PUBLIC_IP:8003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 7ab009459f7f5cf6cef5f46b691748dc236e4c26 PUBLIC_IP:8001
slots:[0-5460] (5461 slots) master
M: 0048ca2cd65c1315b8f0a7c952b69bfb494d5ace PUBLIC_IP:8002
slots:[5461-10922] (5462 slots) master
M: c6ee023719f200b0d175f428fa15e5ab767d0e04 PUBLIC_IP:8003
slots:[10923-16383] (5461 slots) master
S: cf636a1a46b1e947daec3e797cac524c613f08ca PUBLIC_IP:8004
replicates 7ab009459f7f5cf6cef5f46b691748dc236e4c26
S: 5d4bd1041457114353b0b30dbefd86ab8e4ae020 PUBLIC_IP:8005
replicates 0048ca2cd65c1315b8f0a7c952b69bfb494d5ace
S: 62f01289dc3f72cac4a1745fc77b7bd91ec5d107 PUBLIC_IP:8006
replicates c6ee023719f200b0d175f428fa15e5ab767d0e04
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
A lot of people says that we need to bind the private ip, but we wanna do it on public becase when we connect with the external machines the clustering redirect to the master that contains the key, if we bind the private ip the redirect will shows "redirect to PRIVATE_IP" and that will not work as expected.
Are we missing something to let the cluster join by public IP?

From redis security guide:
Redis is designed to be accessed by trusted clients inside trusted environments.
See also: How to connect to redis from remote guide
When a server binds on its public ip, it can get requests from everyone, so unless you built some security around it anyone can access and manipulate your data.
In redis cluster the rules are the same and the replicas which binds on public ips are exposed.
The default use case for a redis cluster is that one machine (or multiple machines) access it from within it's private network, and you shouldn't divert from that unless you know what you are doing security wise.
If it makes sense for your use case, you should make the machine which access the redis cluster a part of the cluster private network.

What I would be doing if I were at your place is:
Bind all the servers with private ip and loopback ip i.e bind {{ private_ip }} 127.0.0.1
Enable ufw (or other any firewalling tool) on each server and do (for ufw) allow from {{ private_ip }} to any port {{ redis_port }} or similar.
My internal DNS will have entry for all the servers with their respective private ip.
Voila! create and access redis cluster securely without any security breach.
NOTE: if you still want to access them over public network then you can do some workaround with SNAT
WARNING: binding redis server to 0.0.0.0 or public ip might cause serious vulnerability issues like:
https://www.exploit-db.com/exploits/47195
https://medium.com/#knownsec404team/rce-exploits-of-redis-based-on-master-slave-replication-ef7a664ce1d0
PS: You can also follow this medium tutorial.

Related

Can I access to Nifi Rest-API using localhost instead of actual node-ip address in Nifi cluster?

For example; I have 3 nifi nodes in nifi cluster. Example hostnames of these nodes;
192.168.12.50:8080(primary)
192.168.54.60:8080
192.168.95.70:8080
I know that I can access to nifi-rest api from all nifi nodes. I have GetHTTP processor for get cluster summary from rest-api, and this processor runs on only pimary node. I did set "URL" property of this processor to 192.168.12.50:8080/nifi-api/controller/cluster.
But, if primary node is down, new primary node will be elected. Thus, I will not be able to access 192.168.12.50:8080 address from new primary node. Because this node was down. So, I will not be able to get cluster summary result from rest-api.
In this case, Can I use "localhost:8080/nifi-api/controller/cluster" instead of "192.168.12.50:8080/nifi-api/controller/cluster" for each node in nifi cluster?
It depends on a few things... if you are running securely then you have certificates that are generated for each node specific to the hostname, so the host in the web requests needs to match the host in the certificates, so you can't use localhost in that case.
It also depends how NiFi's web server is configured. If nifi.web.http.host or nifi.web.https.host has a specific hostname specified, then the web server is only bound to that hostname and may not accept connections with a different hostname. In a default unsecure setup, if you leave nifi.web.http.host blank then it binds to all interfaces.
You may be able to use the expression language function to obtain the hostname of the current node. So you could make the url something like "http://${hostname()}/nifi-api/controller/cluster".

Overriding `tcp.publish_port` breaks clustering when elasticsearch is in a container

I'm trying to run an elasticsearch cluster with each es-node running in its own container. These containers are deployed using ECS across several machines that may be running other unrelated containers. To avoid port conflicts each port a container exposes is assigned a random value. These random ports are consistent across all running containers of the same type. In other words, all running es-node containers map port 9300 to the same random number.
Here's the config I'm using:
network:
host: 0.0.0.0
plugin:
mandatory: cloud-aws
cluster:
name: ${ES_CLUSTER_NAME}
discovery:
type: ec2
ec2:
groups: ${ES_SECURITY_GROUP}
any_group: false
zen.ping.multicast.enabled: false
transport:
tcp.port: 9300
publish_port: ${_INSTANCE_PORT_TRANSPORT}
cloud.aws:
access_key: ${AWS_ACCESS_KEY}
secret_key: ${AWS_SECRET_KEY}
region: ${AWS_REGION}
In this case _INSTANCE_PORT_TRANSPORT is the port that 9300 is bound to on the host machine. I've confirmed that all the environment variables used above are set correctly. I'm also setting network.publish_host to the host machine's local IP via a command line arg.
When I forced _INSTANCE_PORT_TRANSPORT (and in turn transport.publish_port) to be 9300, everything worked great, but as soon as it's given a random value, nodes can no longer connect to each other. I see errors like this using logger.discovery=TRACE:
ConnectTransportException[[][10.0.xxx.xxx:9300] connect_timeout[30s]]; nested: ConnectException[Connection refused: /10.0.xxx.xxx:9300];
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:952)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
It seems like the port a node binds to is the same as the port it pings while trying to connect to other nodes. Is there any way to make them different? If not, what's the point of transport.publish_port?
The way the discovery-ec2 plugin works is that it's collecting a list of IP addresses using AWS EC2 API and use this list as unicast list of nodes.
But it does not collect any information from the running cluster. Obviously the node is not yet connected!
So it does not know anything about the publish_port of other nodes.
It just adds an IP address. And that's all. Elasticsearch then is using the default port which is 9300.
So there is nothing you can do IMO to fix that in the short time.
But we can imagine adding a new feature which is close to what has been implemented for Google Compute Engine. We are using a specific metadata to get this port from the GCE APIs.
We could do the same for Azure and EC2. Do you want to open an issue so we can track the effort?

Cassandra 2.0 multi region nodes

we are trying to setup a multiregion cassandra cluster on ec2. Our configuration looks like
5 nodes each on us-east-1a,us-east-1b,us-east-1c,us-west-1a. For this we have modified the cassandra-rackdc.properties file.
using GossipingPropertyFileSnitch and modified cassandra.yaml file accordingly
we are using all 20 public ips for the seeds configuration in cassandra.yaml file
We have commented out the listen_address and rpc_address property so that cassandra defaults to using InetAddress.getLocalHost()
We have uncommented the broadcast address to use public ip
we have modifed the agents address.yaml file to use public ip address for the properties stomp_interface and local_interface
We are starting the nodes one by one with a 3 min pause in between.
Issue:
When using the opscenter. It shows only one node in the cluster
the 'nodetool status' command also shows only one node
When using cql statement it does show all of its peers
What is the mistake we are doing?
I am doing something similar as a proof-of-concept. I have a working 2-region cluster. Here are the things that I did differently, from reading your question:
I used the Ec2MultiRegionSnitch, which is designed to handle the public and private IPs in EC2.In AWS, the Elastic IP is not bound to the interface by the instance and this causes problems with the cluster communications.
in cassandra.yaml, I left listen_address as the private IP.
also, set rpc_address to 0.0.0.0
uncomment broadcast_address and set it to the public IP (like you did).
i set up dc_suffix in the cassandra-rackdc.properties file and uncommented prefer_local=true (inside the region, Cassandra will prefer to use private IPs).
I opened the security groups for Cassandra so that tcp ports 7000 and 7001 could talk between the nodes in the 2 different regions. Opscenter uses ports 61620 and 61621.
All nodes have the same cluster name.
seed IPs are set to the public IPs. I didn't use all the nodes as seeds, that's not recommended.
Start the seeds first, followed by the other nodes.
This provided a working cluster. Now I am working on ssl node-to-node communication between regions.

database cluster fail over

I planed to develop a HA program for MSSQL, but I got a question when reading some articles about cluster.
When master is down, then slaves vote for a new master, and the new master will take over virtual IP address of the old one.
What is virtual IP address here ?
For example:
A: master 192.168.1.100
B: slave 192.168.1.101
C: slave 192.168.1.102
Is there a another IP address assigned to A? or the IP 192.168.1.100 is the virtual IP ?
If an additional IP need to be assigned to master, what the kind of IP it is? public or local, and how it is assigned to server in C# code.
Windows system.
It is not possible to add extra network cards.
Thanks.
:D
In the MSSQL, every instance has an endpoint. In your case, each IP address should be the instance IP addresses.
When the master is down, no IP address should be resigned. It keeps as is.
During failover, Windows Failover Cluster will tell Secondary B or C which is voted to be new Primary.
When you create an Availability Group, you can create an AG listener for your HA. Usually this listener is the interface that client access to. By SQL Server, HA will route your traffic to the current primary.
Client -> AG endpoint (public IP / Private IP) -> Primary (192.168.1.100)
-> Secondary (192.168.1.101)
-> Secondary (192.168.1.102)
After Failover, Client still access same IP address, which is the AG listener.
Client -> AG endpoint (public IP / Private IP) -> Secondary (192.168.1.100)
-> Primary (192.168.1.100)
-> Secondary (192.168.1.102)
reference: https://msdn.microsoft.com/en-us/library/hh213080.aspx

Cassandra: what is the correct configuration for EC2 multi-region?

What is the correct configuration for a mulit-region setup in EC2 instances?
What should the listen_address, broadcast_address, rpc_address and seed ip/addresses be to work?
When do you use public IP address and when do you use private IP addresses?
According to the docs:
broadcast_address: (Default: listen_address) If your Cassandra cluster is deployed across multiple Amazon EC2 regions and you use the EC2MultiRegionSnitch, set the broadcast_address to public IP address of the node and the listen_address to the private IP.
listen_address: (Default: localhost) The IP address or hostname that other Cassandra nodes use to connect to this node. If left unset, the hostname must resolve to the IP address of this node using/etc/hostname, /etc/hosts, or DNS. Do not specify 0.0.0.0.
rpc_address: (Default: localhost) The listen address for client connections (Thrift remote procedure calls).
seed_provider: (Default: org.apache.cassandra.locator.SimpleSeedProvider) A list of comma-delimited hosts (IP addresses) to use as contact points when a node joins a cluster. Cassandra also uses this list to learn the topology of the ring. When running multiple nodes, you must change the - seeds list from the default value (127.0.0.1). In multiple data-center clusters, the - seeds list should include at least one node from each data center (replication group)
Trying to summarize:
the rpc_address is used for client connections and has nothing to do with multi-region EC2
the listen_address and broadcast_address are the 2 important options for multi-region EC2 configuration
in general when configuring any of these answer 2 questions:
who is connecting? (another nodes? clients?)
what IPs are accessible? (is this network interface accessible to who is connecting?)

Resources