1.Need to setup AWS DR which is best with respecting to 1 master and 23 agent machine
2.My Team need to get a proper ENV with best cost
3.If in case of DR need to start secondary side with the same workflow
Related
We have 2 datacenters where we have deployed 2 elasticsearch clusters and they are configured to use CCR (Cross Cluster Replication) in order to sync each-other.
Unfortunatelly CCR requires a License which we are not going to renew. There for we were thinking to convert the existing archiecture with CCR in a Primary/Secondary:
Applications connect to the Primary ELK cluster
In case of Disaster Recovery (e.g. Datacenter 1 goes down) we configure the applications to use the secondary ELK cluster located in the DC 2
Once the DC 1 is up and running again, the ELK cluster on DC 1 become the new secondary
Is this a valid approach? Is it achievable?
any tutorial on how to configure the PRimary/Seconday clusters in ELK
or "creating a recurrent backup of ELK1 and restore it on ELK2" is the only solution?
We have solved this creating a primary/secondary Elasticsearch instance and we forwards the events with Logastash to both cluster at the same time
Spot instances brings the posibility to use free resources in the cloud paying a lower price, however if the cloud demand is increased your resources will be dealocated. This is very usefull for non critical workloads whenever you can aford to loose some of the work done. More info 2 3
Databricks has the posibility to run spot instances on AWS but there is no documentation about how to do it on Azure.
Is it possible to run Databricks clusters on Azure Spot instances?
Yes, it is possible but not using Databricks UI. To use Azure spot instances on Databricks you need to use databricks cli.
Note
With the cli tool is it possible to administrate -create, edit, delete- clusters and instances-pools. However, to simplify the process, I'll focus on editing an existing cluster.
You can install databricks cli using pip install databricks-cli and configure your credentials with databricks configure --token. For more information, visit databricks documentation.
Run the command datbricks clusters list to know the ID of the cluster you want to modify:
$ datbricks clusters list
0422-112415-fifes919 Big Spark3 TERMINATED
0612-341234-jails230 Normal Spark3 TERMINATED
0212-623261-mopes727 Small 7.6 TERMINATED
In my case, I have 3 clusters. First column is the cluster ID, second one is the name of the cluster. Last column is the state.
The command databricks cluster get generates the cluster config in json format. Let's generate the json file to modify it:
databricks clusters get --cluster-id 0422-112415-fifes919 > /tmp/my_cluster.json
This file contains all the configuration related to the cluster like name, instance type, owner... In our case we are looking for the azure_attributes section. You will see something similar to:
...
"azure_attributes": {
"first_on_demand": 1,
"availability": "ON_DEMAND_AZURE",
"spot_bid_max_price": -1.0
},
...
We need to change the availability to SPOT_WITH_FALLBACK_AZURE and spot_bid_max_price with our bid price. Edit the file with your favorite tool. The result should be something like:
...
"azure_attributes": {
"first_on_demand": 1,
"availability": "SPOT_WITH_FALLBACK_AZURE",
"spot_bid_max_price": 0.4566
},
...
Once modified, just update the cluster with the new configuration file using databricks clusters edit:
databricks clusters edit --json-file /tmp/my_cluster.json
Now, everytime you start the cluster, the workers will be spot instances.To confirm this, you can go to the configuration tab inside the worker VM that is allocated in the resource group managed by databricks. You will see the Azure spot is active and with the price configured.
Databricks on AWS has more configuration options like SPOT for the availability field. However, until the documentation is released we'll need to wait or configure with try-error approach.
is there a way to 'inactivate' an EKS cluster? I'm aware that the worker can be suspended (EC2) but it would be nice to also suspend the EKS master since we will only use the cluster for dev/test and it would be great to be able to switch it off when not needed (to safe cost of 0.20 $ / hour).
There is the alternative to use kops (which would create a master node as EC2 instance) but maybe there is a way with EKS.
cheers
Currently, EKS master nodes cannot be scaled down to 0. There is an open request for the same. https://github.com/aws/containers-roadmap/issues/318
I'm new to Azure so please bear with me! I'm looking to create a HA (99.99%) node pool for AKS. I am more familiar with AWS and availability zones, whereby I'd split the auto scaling group over 3 AZs and that would be that.
It appears that Azure have picked up on AZs and do offer them (https://azure.microsoft.com/en-gb/blog/azure-availability-zones-now-available-for-the-most-comprehensive-resiliency-strategy/) however, I don't see anyway to specify these parameters when creating an AKS cluster - https://learn.microsoft.com/en-us/cli/azure/aks?view=azure-cli-latest#az-aks-create
Am I missing something here? If I use the availability set, there is only a 99.95% availability target which doesn't fulfill what I need. Basically I want to architect so that if an AZ fails in Azure my app keeps running...
Thanks!
Update: AKS with Availability Zone Support is now generally available: https://learn.microsoft.com/en-us/azure/aks/availability-zones
But note that availability zone configuration can only be set during the cluster creation!
Unfortunately，Azure Availability Zones does not support AKS currently. It is just available for some regions and services. For details, see support regions and services.
In all tutorials for ECS you need to create a cluster and after that an autoscaling group, that will spawn instances. Somehow in all these tutorials the instances magically show up in the cluster, but noone gives a hint what's connecting the autoscaling group and the cluster.
my autoscaling group spawns instances as expected, but they just dont show up on my ecs cluster, who holds my docker definitions.
Where is the connection I'm missing?
I was struggling with this for a while. The key to getting the instances in the autoscaling group associated with your ECS cluster is in the user data. When you are creating your launch config when you get to step 3 "Configure Details" hit the advanced tab and enter a simple bash script like the following for your user data.
#!/usr/bin/env bash
echo ECS_CLUSTER=your_cluster_name >> /etc/ecs/ecs.config
All the available parameters for agent configuration can be found here http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html
An autoscaling group is not strictly associated to a cluster. However, an autoscaling group can be configured such that each instance launched registers itself into a particular cluster.
Registering an instance into a cluster is the responsibility of the ECS Agent running on the instance. If you're using the Amazon ECS-optimized AMI, the ECS Agent will launch when the instance boots and register itself into the configured cluster. However, you can also use the ECS Agent on other Linux AMIs by following the installation instructions.
Well, i found out.
Its all about the ecs-agent and its config file /etc/ecs/ecs.config
(This file will be created through the Userdata field, when creating EC2 instances, even from an autoscaling configuration.)
Read about its configuration options here: http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html
But you can even copy a ecs.config stored on Amazon S3, do it like this (following lines go into Userdata field):
#!/bin/bash
yum install -y aws-cli
aws configure set default.s3.signature_version s3v4
aws configure set default.s3.addressing_style path
aws configure set default.region eu-central-1
aws s3 cp s3://<bucketname>/ecs.config /etc/ecs/ecs.config
note: Signature_version v4 is specific for some regions, like eu-central-1.
This ofc only works, if your IAM role for the instance (in my case its ecsInstanceRole) has the right AmazonS3ReadOnlyAccess
The AWS GUI console way for that would be:
Use the cluster wizard at https://console.aws.amazon.com/ecs/home#/firstRun .
It will create an autoscaling grou for your cluster, a loadbalancer in front of it, and connect it all nicely.
This question is old but the answer is not complete. There are 2 parts to getting your own auto-scaling group to show up in your cluster (as of Jan 2022).
You need to ensure your cluster name is set for ECS_CLUSTER variable in /etc/ecs/ecs.config as mentioned in this answer: https://stackoverflow.com/a/35324937/583875
You need to create a new capacity provider for the cluster and attach this auto scaling group. To do this, go to Cluster -> Capacity Provider -> Create -> Select your auto scaling group under Auto Scaling group.
Another tricky part is getting your service to use the instances (if you have a service running). You need to edit the Service, and change the Capacity provider strategy. Click on Add another provider and choose the new capacity provider you created in (2) above.
That's all! To ensure things are working properly: you should see your capacity provider under Graph -> Capacity Providers and you should see instances from your auto scaling group under Graph -> ECS Instances.