I’m researching my options to publish microservices under AWS lambda. As we will be billed by the microsecond and memory used, performance of our lambda functions is pretty important.
My DB is Cassandra or other NOSQL distributed on several AWS nodes, on several data centres.
How can I configure lambda to be sure that my lambda function, including DB access, is using a DB node on the same data centre?
I know that Cassandra drivers will redirect intelligently queries to the nearest DB node, but what I need is to the restrict lambda execution to just nodes on the same data centres where I have some DB node.
You can configure or create a lambda role using IAM roles which will have access to only a specific aws resource (you can be restrictive or open on access control by specifying that in the policies).
Then assign that role to your lambda in a VPC or default VPC.
Related
I'm trying to set up a way to monitor disk usage for instances belonging to an AutoScaling Group, and add an alarm when the volumes associated to the instances are almost full.
Since it seems there are no metrics normally offered by Amazon to do that, I resorted using the CloudWatch Agent to get what I wanted. So far so good, I can create graphs and alarms for the metrics I want using the CloudWatch console.
My issue is how to automate everything with CDK. How can I automate the creation of the metric for each instance, without knowing the instance id beforehand? Is there a solution for this issue?
You can install and config CloudWatch agent via EC2 user data and the auto scaling group uses launch template to launch EC2 instance. All of those things can be done by AWS CDK.
There is an example from this open source project for your reference.
Another approach you could take is using AWS Systems Manager. Essentially, you install an SSM agent for your instances, and create an SSM Document (think Shell/Python script) that will run your setup script/automation.
You then create a State Manager Association, tying the SSM Document with your instances based on EC2 tags e.g. Application=MyApp or Team=MyTeam. This way, you don't have to provide any resource ids, just the tag key value pair which could extend multiple instances and future instance replacements. You can schedule it to run at specific times (cron) or at a certain frequency (rate) to enforce state.
I have setup the Elasticsearch Certified by Bitnami on GCP
Which I would link to put behind the HTTP(S) Load Balancing on GCP for auto scaling propose. What I have done is create snapshot and use it to create image for instance template. But the Instance group continuous return "instance in being verified" and "Recreated instance" for long time do I don't know where the problem is so I design to use the default instance template from GCP instead.
My question is, when the new node created of when the data in elasticsearch updated how can I sync data between node in the GCP load balancer? Think about when there is high traffic and load balancer created the new coming node, and when the query come in from load balance how the new node have the exactly same data with the existing node or when the new index come in, all the node get the new index.
PS I dont mind for the delay if it less than 5 mins it is acceptable.
Thanks in advance for helping out.
In GCP, if you want to sync your data between nodes, we recommend using a centralized location to store your data. You can use Cloud Storage, Cloud SQL, Cloud File System etc. You can check this link to find more options for the data storage. Then you can create an instance template that specifies that when any instance is created it will use the custom image and has access to that centralized database. This is a recommended workaround rather than replicate new instances with data. You can find this link for the similar kind of thread.
For your Elasticsearch setup, I'll recommend deploying an Elasticsearch Cluster that provides multiple VMs that you can customize the configuration. If deploying cluster, this other Stackoverflow post suggest that is not not necessary to use a load balancer as Elasticsearch handles the load between the nodes.
Do any one know how to see the total space used and available in aws aurora schema, also the memory of the cluster within DB itself using queries. We are using AWS aurora psql and we don't have console, hence we want to try the same from queries like we will do for oracle.
This is most likely not doable using a SQL query in aurora, specifically the volume size. As an alternative, if you do have AWS CLI access, you should be able to query Cloudwatch metrics under "AWS/RDS" namespace to get these metrics in a more reliable and accurate manner. The ones you are interested in are "VolumeBytesUsed" and "CPUUtilization" if I'm not mistaken.
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Monitoring.html
If there are multiple read replicas, where load balancing related settings can be specified when using spring AWS libraries.
Read replicas have their own endpoint address similar to the original RDS instance. Your application will need to take care of using all the replicas and to switch between them. You'd need to introduce this algorithm into your application so it automatically detects which RDS instance it should connect to in turn. The following links can help:
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.Replication.html#Overview.ReadReplica
We're planning to move our Tomcat/MySQL app onto the Amazon cloud employing 2 EC2 instances (inst_1 & inst_2) running in different availability zones whereby inst_1 will contain the master RDS db and inst_2 the slave RDS db.
If we employ elastic load balancing to balance traffic between the two instances, will traffic directed to inst_2 that includes insert/update/delete db transactions first update the master RDS db in inst_1 followed by a synchronous update of the slave in inst_2; thereby ensuring that the two RDS instances are always synchronized?
Amazon's published info (whitepapers) suggests such, but doesn't explicitly state it. If not, how does one ensure that the two RDS instances remain synchronized?
Additional note: We're planning to employ Amazon's Elastic Beanstalk. Thanks!
You have to take a few things into consideration
AWS RDS instances are simple managed EC2 instances which run a MySQL server.
If you add a slave ( I think Amazon calls them read-replica) this is a read-only slave
Amazon doesn't manage the distribution of writing queries to the master server automatically.
Replication will ensure that your read slave always is up-to-date automatically ( with minimal delay which is increasing with write-load on the master )
This behavior is MySQL-specific
This means that you have to delegate manipulating queries to the master exclusively.
This can either be done by your application or by a MySQL proxy running on a extra machine.
The proxy then is the only interface your application servers will talk to. It is able to manage balancing between your RDS instances and the direction of any manipulation query to the master instance.
When RDS is used in multi-az mode you have no access to the secondary instance. There is only ever one instance that is visible to you, so most if your question doesn't apply. In case of failover the DNS address you are given will start resolving to a different ip. Amazon doesn't disclose how the two instances are kept in sync.
If instead of using a multi-az instance you use a single-az instance + a replica then it is up to you to direct queries appropriately - any attempt to alter data on the replica will fail. Since this is just standard MySQL replication, the replica can lag behind the master (in particular with current versions of MySQL the replica only runs a single thread)