Our current cassandra cluster is 26 nodes spread across four AWS EC2 regions. We use elastic IP's for all of our nodes, and we use security groups to allow all nodes to talk to each other.
There is a hard limit of 250 security group rules per network interface (http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html#vpc-limits-security-groups). From the documentation:
You can have 50 inbound and 50 outbound rules per security group (giving a total of 100 combined inbound and outbound rules). If you need to increase or decrease this limit, you can contact AWS Support — a limit change applies to both inbound and outbound rules. However, the multiple of the limit for inbound or outbound rules per security group and the limit for security groups per network interface cannot exceed 250
Since each node needs a security group entry for every other node in the cluster, that means we have a hard limit of a 250 node cluster.
I can think of some ways to mitigate the issue, like using two security groups, where one allows access from the 'local' nodes in the same region, and the other security group has the elastic IP's for the 'remote' nodes in other regions. This would help, but only a little bit.
I have been in touch with AWS technical support about this, and they suggested using contiguous blocks of elastic IP's (which I did not know was possible). This seems like it would solve the problem, but it turns out this is an involved process, and requires us (my company) to become the ARIN owner of these IP's. The AWS reps are pushing me towards alternatives, such as DynamoDB. I am open to other technologies to solve our problem, but I get the feeling they are just trying to get us to use AWS managed services. On top of that they are rather slow in getting back to me when I ask questions like "Is this typically how customers run multi region cassandra clusters in ec2?"
Does anyone have experience with this approach?
What are some alternatives to using security groups? Does anyone have experience running a large (>250 node), multi region cassandra cluster in EC2? Or even a cluster with >50 nodes (at which point a single security group isn't feasible any more)?
Related
In our application we have an elasticsearch cluster that is shared for all of our clients.
The types of requests made against this cluster are computationally intense and may take minutes to complete. Because of the type of data in the cluster, the types of requests we get, and the irregularity at which they are used, we can't predict when these requests will be made or do any caching before hand. If multiple clients make requests at the same time, they will experience slower response speeds than normal.
For most clients this isn't an issue. Their data isn't large enough to make a noticeable difference (3s -> 10s occasionally isn't a big deal). But for larger clients, the time difference may be minutes and is very noticeable.
What we'd like, above all else, is consistency- even if these operations were slower on average. To do that, we'd like to give these special clients dedicated nodes, while all other clients use the shared nodes. At a first glance, it seems the only way to do that is to create a dedicated cluster.
But this adds overhead in any application that interfaces with elasticsearch to first do a lookup for the cluster to route to. Ideally, we'd be able to create dedicated nodes within the cluster. This way, the application doesn't need to be aware of cluster routing, and the index configuration can be shared between these "virtual clusters".
Each document has a client id, which we can use to distribute across nodes using _routing. But this has its own problems. Firstly, this doesn't allow us to create a generic default cluster. Secondly, this may mean the dedicated nodes share client data with other clients- the goal is to get consistent speeds by removing node resource contention. And finally, this doesn't allow us to allocate multiple nodes for a given route.
Is there a way to create a routing rule that nodes can be added to explicitly. e.g. I want to add 3 nodes to routing key 582123.
Is there a way to create a default routing rule for nodes that don't match existing routes? If not we could always have an explicit default route. We'll still need to do a route lookup application side but it would still reduce complexity in a multi-cluster scenario.
Depending on the version of elasticsearch you are using and the setup of your indices, you could use per-index allocation. Basically you give a node an attribute and then you specify on index level settings where that index (more accurately, its shards) should end up. As you will read in the documentation, you need to make sure that other constraints are not being violated e.g.
Shards are only relocated if it is possible to do so without breaking another routing constraint, such as never allocating a primary and replica shard on the same node.
This means that you need to have different indices though.
What is the best way to deal with a surge in log messages being written to an ElasticSearch cluster in a standard ELK setup?
We use a standard ELK (ElasticSearch/Logstash/Kibana) set-up in AWS for our websites logging needs.
We have an autoscaling group of Logstash instances behind a load balancer, that log to an autoscaling group of ElasticSearch instances behind another load balancer. We then have a single instance serving Kibana.
For day to day business we run 2 Logstash instances and 2 ElasticSearch instances.
Our website experiences short periods of high level traffic during events - our traffic increases by about 2000% during these events. We know about these occurring events well in advance.
Currently we just increase the number of ElasticSearch instances temporarily during the event. However we have had issues where we have subsequently scaled down too quickly, meaning we have lost shards and corrupted our indexes.
I've been thinking of setting the auto_expand_replicas setting to "1-all" to ensure each node has a copy of all the data, so we don't need to worry about how quickly we scale up or down. How significant would the overhead of transferring all the data to new nodes be? We currently only keep about 2 weeks of log data - this works out around 50gb in all.
I've also seen people mention using a separate auto scaling group of non-data nodes to deal with increases of search traffic, while keep the number of data nodes the same. Would this help in a write heavy situation, such as the event I previously mentioned?
My Advice
Your best bet is using Redis as a broker in between Logstash and Elasticsearch:
This is described on some old Logstash docs but is still pretty relevant.
Yes, you will see a minimal delay between the logs being produced and them eventually landing in Elasticsearch, but it should be minimal as the latency between Redis and Logstash is relatively small. In my experience Logstash tends to work through the backlog on Redis pretty quickly.
This kind of setup also gives you a more robust setup where even if Logstash goes down, you're still accepting the events through Redis.
Just scaling Elasticsearch
As to your question on whether or not extra non-data nodes will help in write-heavy periods: I don't believe so, no. Non-data nodes are great when you're seeing lots of searches (reads) being performed, as they delegate the search to all the data nodes, and then aggregate the results before sending them back to the client. They take away the load of aggregating the results from the data nodes.
Writes will always involve your data nodes.
I don't think adding and removing nodes is a great way to cater for this.
You can try to tweak the thread pools and queues in your peak periods. Let's say normally you have the following:
threadpool:
index:
type: fixed
size: 30
queue_size: 1000
search
type: fixed
size: 30
queue_size: 1000
So you have an even amount of search and index threads available. Just before your peak time, you can change the setting (on the run) to the following:
threadpool:
index:
type: fixed
size: 50
queue_size: 2000
search
type: fixed
size: 10
queue_size: 500
Now you have a lot more threads doing indexing, allowing for a faster indexing throughput, while search is put on the backburner. For good measure I've also increased the queue_size to allow for more of a backlog to build up. This might not work as expected, though, and experimentation and tweaking is recommended.
as I understand it's technically possible to have redis cluster with nodes spreaded by different regions in amazon cloud (EC2) - so I will be able to obtain the same data in machines in each region.
But here is 2 questions I am not sure:
how it would impact on redis' speed? As guys measured (Speed from Different EC2 Regions) - there is about 4 times difference. What should it mean for redis?
how much would it cost for me? Or in other words - how much service traffic does reddis generate when work in cluster (for example, by one node in both of two regions)?
I have no any practical experience with redis, but seems it can be very useful for my purposes.
thanks a lot for help.
Costing to you will be the cost of bandwidth and instance hours.
For my application I am using auto scaling, without using elastic load balancing, is there any performance issue for directly using Auto scaling without ELB?
Adi,
David is right.
Autoscaling allows you to scale instances (based on cloudwatch metrics, a single event, or on a recurring schedule).
Suppose you have three instances running (scaled with Autoscaling): how is traffic going to reach them? You need to implement a Load Balancing somewhere, that's why Elastic Load Balancing is so useful.
Without that, your traffic can only be directed in a poorly-engineered manner.
See Slide #5 of this presentation on slideshare, to get a sense of the architecture: http://www.slideshare.net/harishganesan/scale-new-business-peaks-with-auto-scaling
Best,
Autoscaling determines, based on some measurement (CPU load is a common measurement), whether or not to increase/decrease the number of instances running.
Load balancing relates to how you distribute traffic to your instances based on domain name lookup, etc. Somewhere you must have knowledge of which IP addresses are those currently assigned to the instances that the autoscaling creates.
You can have multiple IP address entries for A records in the DNS settings and machines will be allocated in a roughly round-robin fashion from that pool. But, keeping the pool up to date in real-time is hard.
The load balancer gives you an easy mechanism to provide a single interface/IP address to the outside world and it has knowledge of which instances it is load balancing in real time.
If you are using autoscaling, unless you are going to create a fairly complex monitoring and DNS updating system, you can reasonably assume that you must use a load balancer as well.
Just a simple question, say I on a research team with 10 members and we want to deploy a project that will require on-demand leasing of 200 EC2 instances. Can we virtually bypass the 20 instances limit by creating 10 AWS accounts (one each) and then lease on demand up to 200 instances?
You can do that - you can have as many accounts, with their appropriate payment details, as you like. You then have the standard 20-instance limit associated with each account.
Bear in mind though, that these accounts are all separate and have their own sets of access keys. Any technology you deploy to span instances across these accounts will need to be aware of that, since your 'pool' manager will need the appropriate key for the account it is attempting to launch an instance in. You may also encounter complexities with EBS sharing, Security Group access (if, for example, you have a Domain Controller in one account, VMs in the other accounts will not be able to see it), and Load Balancing and VPC are likely to be difficult at best.
Amazon are usually fairly quick to respond to limit increase requests - after all, the more instances you're using, the more money they're making. They obviously want to ensure they have capacity for your requirement, which is why they manually 'vet' requests, but provided you have a reasonable business case they're fairly accomodating.
http://aws.amazon.com/ec2/faqs/#How_many_instances_can_I_run_in_Amazon_EC2
You are limited to running 20 On-Demand or Reserved Instances, and
running 100 Spot Instances per region. Cluster GPU Quadruple Extra
Large instances and High I/O Quadruple Extra Large instances are
limited to running 2 On-Demand instances per region [...].
Increase your limit with http://aws.amazon.com/contact-us/ec2-request
Bid the 'on-demand price' for a bunch of spot instances. You will never end up paying full price, but will rarely have your spot instances terminated. See
http://aws.amazon.com/ec2/spot-instances/ and deciding on your spot bidding strategy