EC2 check in Auto Scaling group - amazon-ec2

I have an ASG with min=2, max=4 configuration. In the boot up script of each EC2, I have a series of yum install and starting of 2 spring boot files. Now, when th e load increases and ASG spins up a new EC2 instance, it will perform all these in the boot up script.
Could anybody suggest a good method to validate whether all these yum installs have been successful and also whether the 2 spring boot files are running currently. If there is any problem with these, I dont want the EC2 instance to be attached to ELB.

I used cfn-signal to send information back to Cloudformation after performing my application and infrastructure level checks
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-signal.html

Related

Deploy spring boot microservices on AWS Appmesh with EC2

I am trying to deploy Spring Boot microservices using Docker using Appmesh and EC2. I have deployed two sample microservices (https://github.com/amitgct/appmesh-hello) namely: caller-service and called-service using docker on a single EC2 instance and configured appmesh accordingly by following guide https://docs.aws.amazon.com/app-mesh/latest/userguide/getting-started-ec2.html. Currently, my applications are running on ec2 but they cannot communicate with each other and getting error on calling called-service from caller-service i.e. Unknown host. Can anyone tell me how can I specify hostname and register service with that host on EC2 and App mesh. (Note: I don't want to use kubernetes, ECS, AWS cloud map, AWS Route53) . If can provide example also then very thankful to you. Please help.
https://www.appmeshworkshop.com/servicediscovery/
here's a step by step process shown, and this is for http protocol...
but if you change the listeners section in virtual routes to tcp then it should work for TCP messages as well - for those systems which works on tcp protocol - example Akka Clusters

How to check if docker Cassandra instance is ready to take connections

I have two docker instances that I launch with docker-compose.
One holds a Cassandra instance
One holds a Spring Boot application that tries to connect to that application.
However, the Spring Boot application will always fail, because it's trying to connect to a Cassandra instance that is not ready yet to take connections.
I have tried:
Using restart:always in Docker-compose
This still doesn't always work, because the Cassandra might be up 'enough' to no longer crash the Spring Boot application, but not up 'enough' to have successfully created the Table/Column family. On top of that, this is a very hacky solution.
Using healthcheck
It seems like healthcheck in compose doesn't have restart capabilities
Using a bash script as entrypoint
In the hope that I could use netstat,ping,... whatever to determine that readiness state of Cassandra
Right now the only thing that really works is using that same bash script and sleep the process for x seconds, then start the jar. This is even more hacky...
Does anyone have an idea on how to solve this?
Thanks!
Does the spring boot service defined in the docker-compose.yml depends_on the cassandara service? If yes then the service is started only if the cassandra service is ready.
https://docs.docker.com/compose/compose-file/#depends_on
Take a look at this github repository, to find a healthcheck for the cassandra service.
https://github.com/docker-library/healthcheck
CONCLUSION
After some discussion we found out that docker-compose seems not to provide a functionality for waiting until services are up and healthy, such as Kubernetes and Openshift provide (See comments below). They recommend to use wrapper script (docker-entrypoint.sh) which waits for the depending service to come up, which make binaries necessary, the actual service shouldn't use such as the cassandra client binary. Additionally the service depending on cassandra could never get up if cassandra doesn't, which shouldn't happen.
A main thing with microservices is that they have to be resilient for failures and are not supposed to die or not to come up if a depending service is currently not available or unexpectedly disappears. Therefore the microservice should be implemented in a way so that it retries to get connection after startup or an unexpected disappearance. Unexpected is a word actually wrongly used in this context, because you should always expect such issues in a distributed environment, and even with docker-compose you will face issues like that as discussed in this topic.
The following link points to a tutorial which helped to integrate cassandra properly into a spring boot application. It provides a way to implement the retrieval of a cassandra connection with a retry behavior, therefore the service is resilient to a non existing cassandra database and will not fail to start anymore. Hope this helps others as well.
https://dzone.com/articles/containerising-a-spring-data-cassandra-application

would it be a good design to deploy a Spring batch application using AWS Batch service?

I am doing a POC for batch application which is expected to be deployed in AWS cloud.
This batch is application is heavy expected to compute huge calculation for 2m-10M of records monthly. So number of EC2 instances has to grow dynamically based on the load.
Actually I was thinking of creating a spring-batch application that will be deployed in AWS ECS service.
However I see new AWS-Batch service which is also helping to deploy the job.
My question is can I deploy my spring-batch application using AWS-Batch assuming my batch application is containerized using docker. is it a good approach or should I go with deploying in ECS? Please suggest.
Thanks,
Mani.
From the information you have given, AWS Batch seems to be ideal for your application.
In general, ECS should rather be used for hosting services, while AWS Batch does computation. Also, AWS Batch is actually built on top of ECS for providing the functionality your application is likely to need. It also works well with Spot instances and will therefore save you money.
Further resources:
https://www.reddit.com/r/aws/comments/9syduu/how_does_aws_ecs_differ_from_aws_batch/
How is a batch processing system in AWS ECS different from AWS Batch?

Programmatic method of starting xd-admin and xd-container in spring XD

In a distributed setup, how do I programatically start containers?
More specifically, does there exist any API similar to deploying and undeploying streams for setting up and tearing down containers?
There is currently no way to do this via an API. Containers are only known to the cluster after they are started. Upon initialization, the container registers itself with ZooKeeper. Running a container requires XD to be installed on that host which is currently a manual process: download,unzip,configure, as is starting the container. Some automation of operations will likely be provided in a future release.

Spring XD Distributed Environment

I am working on Spring XD and GemFire XD. I want to understand how Spring XD's distributed environment works. I know spring xd uses either redis or rabittmq as the transport.
I am clear about this, I have install spring xd and rabittmq on one machine. I changed the redis.properties file and added hostnames.
Do I need to install spring xd on all the machines? If so, after installing, how to bring those up.
On the master machine, I will do ./xd-admin and ./xd-container
How do you start up the nodes (spring xd instances/workers) so that they can listen for instructions from xd-admin?
Please help me on this.
Thanks,
-Suyodhan
Redis is used for analytics as only supported platform. For transport, you need either Redis or Rabbit.
Basically you just need to install Redis and RabbitMQ per their respective documentation. They can be in same or different servers, Ideally you would use their high availability option. For example Redis Sentinal. YOu don't need RabbitMQ unless you want to change the default transport from Redis to Rabbit. Once you install Redis and Rabbit, bring them up and provide their host:port info (and any additional as applicable) to the servers.yml in XD install (in all nodes) and bring up admin and containers. Evrything should work automatically by using zookeeper as the means to manage the distributed runtime.
If you use Spring XD in distributed mode, I assume you have set up zookeeper as well. (If not check this http://docs.spring.io/spring-xd/docs/1.0.0.M7/reference/html/#_setting_up_zookeeper )
Admin and Container instances register themselves with Zookeeper as they come up. Admin queries zookeeper for available containers and assign tasks like deploying modules. Zookeeper is the trick behind Distributed mode.
Hope this helps.
You will install Spring xd one time on one machine, Spring XD will be connected to your hdfs distributed scaled out environment.
You need to start the followings:
1. redis or rappitMQ in your case
2. hsqldb server
3. container
4. admin
when you start spring xd, you need to register the name node firstly using the command:
hadoop config fs --name hdfs://serverip:8020
then you can use any module defined in spring xd (using stream or batch) by specifying its parameters directly without specifying those in the server.yml file.
Moha.

Resources