Here is one deployment scenario wherein NAT exists between DC boundaries, following are the requirements:
Cassandra Version: 2.1.13
There are 2 DCs, two Cassandra nodes (dc1:node1 & dc2:node3) across DCs should communicate across NAT boundaries using public IP.
One of the DC which is behind NAT has 2 Cassandra nodes (dc1:node1 and dc1:node2) and both them should communicate within NAT using private IP.
All these 3 nodes (dc1:node1, dc1:node2 & dc2:node3) should form a ring and communicate with each other.
Looked into seeds, listen_address, broadcast_address & broadcast_rpc_address.
https://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html
If there are public IPs used in broadcast_address and seeds then across DC & NAT communication works, however the nodes which are within NAT not able to detect each other.
If there are private IPs used in broadcast_address and seeds then within DC & NAT communication works, however the nodes across DC & NAT not able to detect each other.
Looked into Ec2MultiRegionSnitch but that will not work for premise deployments: https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchEC2MultiRegion_c.html#architectureSnitchEC2MultiRegion_c__other-settings
What configuration settings will be required to achieve above 3 requirements?
Use gossiping property file snitch, set public ip as broadcast_address and private ip as listen_address. They will use the listen_address in same DC and broadcast address if in other DC.
Settings made with seeds=public address, listen_address=private and broadcast_address=public.
With these settings made on all 3 nodes:
dc1:node1 & dc2:node3 works but
dc1:node1 and dc1:node2 doesn't
Is it because of seeds have public address within DC behind NAT might nor work?
One of the observation to check listen on dc1:node1, private IP is listed:
node1# netstat -anp | grep -E "(7001)"
tcp 0 0 dc1:node1_privateIP:7001 0.0.0.0:* LISTEN 9999/java
Can dc1:node2 establish connection with dc1:node1_publicIP?
Is this https://issues.apache.org/jira/browse/CASSANDRA-9748 related here or will be only applicable in case of multiple NICS and not NAT environment?
Related
I can't seem to extend my pool for my additional users. I don't have any problem with my connection when the ip address handed is 192.168.10.xxx but when it reaches to 192.168.11.xxx to 192.168.16.xxx, it can no longer use the internet. What am I missing with my setup?
Updated:
If you want to add more IPs in your DHCP server, just increase the range of your current network. You currenlty have 253 clients (192.168.10.0/24)
Don't add more /24 networks, it's useless, just use for example 192.168.8.0/21 (range .8.1 to .15.254) to get 2046 IPs. See http://www.subnet-calculator.com/subnet.php to test various network ranges.
So I suggest this:
remove parasites /24 networks and NAT rules (192.168.11.0/24 to .16/24)
increase the range of your current network: change IP address from 192.168.10.1/24 to 192.168.10.1/21, subnet 255.255.248.0, network 192.168.8.0
change NAT/masquerade rule: src-address=192.168.8.0/21
change dhcp network range 192.168.8.0/21
change dhcp pool size with two segments: 192.168.8.1-192.168.9.254 and 192.168.10.100-192.168.15.254
and, normally, it should work
I'm new to Cassandra and EC2 configuration.
I have configured 3 nodes in AWS EC2 instances with Cassandra 3.0 and all the three nodes are connected to each other .
Following things have been configured in .yaml fie.
Broadcast_add: Private ip ec2 add of instance
seeds : public ip add of all the three nodes.
rpc_add : blank
When I try to connect to this cluster from Datastax dev centre it shows only connected to one node. When individually connecting to all the 3 ip's it gets connected to all the nodes. But when connecting to cluster with 3 ip's in connection file, it connects to only one node.
Could any one help with this issue ?
Thanks
Uttkarsh
open cassandra.yaml file and change the
1) listen_address :- private IP
2) broadcast_address :- blank
3) listen_on_broadcast_address:- true
4) rpc_address :- 0.0.0.0
5) broadcast_rpc_address :- public IP
6) seeds ip :- public IP for node.
it's working finally
Thanks Utpal
I'm trying to learn the ropes on packet queuing, so I thought I'd set up a limitation on traffic coming into port 80 from known Tor Exit nodes. This is on FreeBSD 9, so OpenBSD-specific solutions might not apply (syntax/etc).
# Snipped to mainly the relevant parts
table <torlist> persist file "/var/db/torlist"
# ...
set block-policy return
scrub in all
scrub out on $ext_if all
# What I *want* to do is create a cue for known tor exit nodes
# no single one IP should be able to do more than 56k/sec
# but the combined bandwidth of all tor visitors should not
# exceed 512k/sec, basically limiting Tor visitors to something
# like dialup
altq on $ext_if cbq bandwidth 512k queue { qin-tor }
queue qin-tor bandwidth 56Kb cbq ( default rio )
# ...
block in log all
antispoof for { $ext_if, $tun_if }
antispoof quick for $int_if inet
### inbound web rules
# Main Jail ($IP4_PUB3 is my webserver IP)
pass in on $ext_if inet proto tcp from <torlist> to $IP4_PUB3 port www synproxy state queue qin-tor
pass in on $ext_if inet proto tcp to $IP4_PUB3 port www synproxy state
The problem is, when the altq, queue, and pass line specific for torlist are enabled, all connections are extremely slow. I've even tested my own IP against pfctl -t torlist -T test , and got back "0/1 addresses match", and if I test one from the list it's "1/1 addresses match"
So I'm not really educated in the matter of what exactly I'm doing wrong, I was assuming the pass in line with in it would only be applied to the IPs listed in that table, as such my own IP wouldn't validate on that rule and would pass onto the next one.
Getting it working isn't urgent, but any help in understanding where I'm failing would be greatly appreciated.
Turns out that I didn't quite understand how altq works. When I created a queue on my external interface with only one queue I created a default for all connections. As a result I had to define my top speed plus create a default queue for everything else.
For example if my system has 100Mb top
altq on $ext_if cbq bandwidth 100Mb queue { qin-www, qin-tor }
queue qin-www bandwidth 98Mb priority 1 cbq ( default borrow )
queue qin-tor bandwidth 56Kb priority 7 cbq ( rio )
...
pass in on $ext_if inet proto tcp to $IP4_PUB3 port www synproxy state
pass in on $ext_if inet proto tcp from <torlist> to $IP4_PUB3 port www synproxy state queue qin-tor
(doesn't need to be on top since pf parses all the rules unless you use 'quick')
In this way only those IPs matching in gets throttled down to the qin-tor queue, everything else not defined defaults to the qin-www queue.
The FAQ on OpenBSD's pf didn't seem to make this clear to me until I thought about why there would be an error for a "default", then I figured maybe it applies to the whole interface, so need to define a default for rules not marked to a specific queue.
So there it is... the solution to my 'simple' problem. Hopefully anyone else who has this problem comes accross this.
This is the FAQ I was going by for packet queueing: http://www.openbsd.org/faq/pf/queueing.html
I am new to Cassandra and have been tasked with getting it up and running in the EC2 environment across multiple regions such that if an entire EC2 region goes belly up our app will continue on its merry way. I've read as much documentation as I could find regarding Ec2MultiRegionSnitch and have come to a dead stop. I am running cassandra 1.0.10.
My problems are as follows:
1) when I start bin/cassandra I get the error: Could not start register mbean in JMX. Though I can run bin/nodetool -h ring on any of the nodes and I get the display you would expect from a healthy system. I have added the mx4j library to my cassandra deployment. I could try removing that I suppose.
2) when I then start bin/cassandra-cli -h I am able to create the keyspace as follows:
CREATE KEYSPACE mykeyspace
WITH placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
and strategy_options = {us-east-1:2,us-west-1:2};
3) After I run 'use mykeyspace' I can create a column family as follows:
CREATE COLUMN FAMILY people
WITH comparator=UTF8Type AND key_validation_class=UTF8Type AND
default_validation_class=UTF8Type AND column_metadata=[{column_name:FIRST_NAME,validation_class:UTF8Type},
{column_name:LAST_NAME,validation_class:UTF8Type},
{column_name:EMAIL,validation_class:UTF8Type},
{column_name:LOGIN,validation_class:UTF8Type, index_type: KEYS}];
4) After I do this I can run bin/cassandra-cli -h on any of the 4 nodes, run use mykeyspace; describe; and each node correctly describes mykeyspace including the column family and seed list.
5) But when I try to perform a simple:
set people['1']['FIRST_NAME'] = 'John';
I get a stack trace as follows:
null
UnavailableException()
at org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:15206)
at org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:858)
at org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:830)
at org.apache.cassandra.cli.CliClient.executeSet(CliClient.java:901)
My configuration:
I have performed ec2-authorize for ports 22, 7000, 7199 and 9160
I have 4 nodes in my cluster: one node in each of the following regions:AvailabilityZones.
us-east-1:us-east-1a (initial_token: 0)
us-east-1:us-east-1c (initial_token: 85070591730234615865843651857942052864)
us-west-1:us-west-1a (initial_token: 1)
us-west-1:us-west-1c (initial_token: 85070591730234615865843651857942052865)
Each EC2 instance has been associated with a public IP address.
In each node I have configured cassandra.yaml as follows:
seeds: <set to the public ip address for the us-east-1a and us-west-1a nodes>
storage_port: 7000
listen_address: <private ip address of this node>
broadcast_address: <public ip address of this node>
rpc_address: 0.0.0.0
rpc_port: 9160
endpoint_snitch: Ec2MultiRegionSnitch
Additionally in each node's cassandra-env.sh I've included:
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<Node's local IP Address>"
My Plea
Hopefully I have provided someone with enough information to help me get this thing working as one would like.
Additional Information
Stack trace from first mx4j issue:
WARN 22:07:17,651 Could not start register mbean in JMX java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.cassandra.utils.Mx4jTool.maybeLoad(Mx4jTool.java:66)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:243)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:356)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107)
Caused by: java.net.BindException: Cannot assign requested address
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:353)
My cassandra-topology.properties
aaa.aaa.aaa.aaa=us-east-1:us-east-1a
bbb.bbb.bbb.bbb=us-east-1:us-east-1c
ccc.ccc.ccc.ccc=us-west-1:us-west-1a
ddd.ddd.ddd.ddd=us-west-1:us-west-1c
default=us-east-1:us-east-1a
My nodetool ring output __
Address DC Rack Status State Load Owns Token
85070591730234615865843651857942052865
aaa.aaa.aaa.aaa us-east 1a Up Normal 11.09 KB 50.00% 0
bbb.bbb.bbb.bbb us-west 1a Up Normal 6.68 KB 0.00% 1
ccc.ccc.ccc.ccc us-east 1c Up Normal 11.09 KB 50.00% 85070591730234615865843651857942052864
ddd.ddd.ddd.ddd us-west 1c Up Normal 15.5 KB 0.00% 85070591730234615865843651857942052865
I'm pretty certain I've added the regions/availability zone correctly. At least I think I matched what appears in the documentation. (Look at Ec2MultiRegionSnitch in this link)
http://www.datastax.com/docs/1.0/cluster_architecture/replication
I don't think I can just list the regions as us-west and us-east because there are two regions out west (us-west-1 is the California region and us-west-2 is the Oregon region). So I don't think just putting us-west would successfully differentiate regions.
My guess in my comment was right. Your replication settings and datacenter names don't match. A couple of things.
1) cassandra-topology.properties is only used by the PropertyFileSnitch. That file is irrelevant while using the ec2 snitch.
2) The reason the snitch is currently reporting 'us-west' instead of 'us-west-1' is due to a bug. https://issues.apache.org/jira/browse/CASSANDRA-4026. If you added nodes in 'us-west-2' they will correctly get reported as that.
So the solution here is to update your replication settings:
CREATE KEYSPACE mykeyspace
WITH placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
and strategy_options = {us-east:2,us-west:2};
Also, I unfortunately do not know what is wrong with mx4j. It isn't needed by cassandra though so unless you actually need it for something you can just remove it.
I am setting up a grid-enabled cluster. I plan to assign 2 IP to my head node: one for local connection (LAN for distributing jobs to compute nodes) and one for public (internet for user access). So, my /etc/hosts file looks something like this:
111.111.111.111 myserver.whatever.com myserver #for public IP
11.11.11.11 myserver.whatever.com myserver #for local LAN
22.22.22.22 computenode01
33.33.33.33 computenode03
My concern here is will the hostname of myserver get messed up since it is mapped to two IPs?
I fear the system will always choose the first entry (111.111.111.111) if you want to resolve "myserver" address.
It will simply ignore the second entry, as I guess. Choose different hostnames for each entry, e.g. myserver.local and myserver.remote.