I have setup autoscaling using these steps...
$ elb-create-lb autoscalelb --headers --listener
"lb-port=80,instance-port=80,protocol=http" --listener
"lb-port=443,instance-port=443,protocol=tcp" --availability-zones
us-east-1d
$ elb-describe-lbs autoscalelb
$ elb-register-instances-with-lb autoscalelb --instances i-ee364697
$ elb-configure-healthcheck autoscalelb --headers --target "TCP:80"
--interval 5 --timeout 3 --unhealthy-threshold 2 --healthy-threshold 4
$ as-create-launch-config autoscalelc --image-id ami-baba68d3
--instance-type t1.micro
$ as-create-auto-scaling-group autoscleasg --availability-zones
us-east-1d --launch-configuration autoscalelc --min-size 1 --max-size
5 --desired-capacity 1 --load-balancers autoscalelb
$ as-describe-auto-scaling-groups autoscleasg
$ as-put-scaling-policy MyScaleUpPolicy --auto-scaling-group
autoscleasg --adjustment=1 --type ChangeInCapacity --cooldown 300
$ mon-put-metric-alarm MyHighCPUAlarm --comparison-operator
GreaterThanThreshold --evaluation-periods 1 --metric-name
CPUUtilization --namespace "AWS/EC2" --period 600 --statistic Average
--threshold 80 --alarm-actions arn:aws:autoscaling:us-east-1:616259365041:scalingPolicy:46c2d3b3-7f29-42b6-ab64-548f45de334f:autoScalingGroupName/autoscleasg:policyName/MyScaleUpPolicy
--dimensions "AutoScalingGroupName=autoscleasg"
$ as-put-scaling-policy MyScaleDownPolicy --auto-scaling-group
autoscleasg --adjustment=-1 --type ChangeInCapacity --cooldown 300
$ mon-put-metric-alarm MyLowCPUAlarm --comparison-operator
LessThanThreshold --evaluation-periods 1 --metric-name CPUUtilization
--namespace "AWS/EC2" --period 600 --statistic Average --threshold 50 --alarm-actions arn:aws:autoscaling:us-east-1:616259365041:scalingPolicy:30ccd42c-06fe-401a-8b8f-a4e49bbb9c7d:autoScalingGroupName/autoscleasg:policyName/MyScaleDownPolicy
--dimensions "AutoScalingGroupName=autoscleasg"
After this I'm running this command:
$ as-describe-auto-scaling-groups autoscleasg --headers
Response:
AUTO-SCALING-GROUP GROUP-NAME LAUNCH-CONFIG AVAILABILITY-ZONES
LOAD-BALANCERS MIN-SIZE MAX-SIZE DESIRED-CAPACITY
AUTO-SCALING-GROUP autoscleasg autoscalelc us-east-1d
autoscalelb 1 5 1 INSTANCE INSTANCE-ID
AVAILABILITY-ZONE STATE STATUS LAUNCH-CONFIG INSTANCE
i-acf48bd5 us-east-1d InService Healthy autoscalelc
And then:
$ elb-describe-instance-health autoscalelb --headers
It shows:
INSTANCE_ID INSTANCE_ID STATE DESCRIPTION
REASON-CODE INSTANCE_ID i-ee364697 InService N/A
N/A INSTANCE_ID i-acf48bd5 OutOfService Instance has failed at
least the UnhealthyThreshold number of health checks consecutively.
Instance
My first problem is:
It automatically creates One extra instance when there is no load on Main instance.
Secondly,
Newly created instance is always OutOfService.
if I change Min Size to 0 using following command:
$ as-update-auto-scaling-group autoscleasg --launch-configuration
autoscalelc --availability-zones us-east-1d --min-size 0 --max-size 5
And trying to put load on instance using xen:
hg clone http://xenbits.xensource.com/xen-unstable.hg
Autoscaling not creating any instance. Even if I'm running above command on upto 5 session, CPU Utilization reaches to 100% and still no instance is being created.
Please help me...
I am not sure what you want to achieve but if you want to use autoscaling capabilities to add more instances based on traffic increase or decrease , you need to use the load balancer parameters (i.e. Latency):
Change yours to:
--namespace='AWS/ELB'
--metric-name Latency
--period 60 (this is super quick)
--threshold 2.0 (this is very low)
To test if it works, I use Apache Bench, I run below command on multiple micro instances
$ ab -n 10000 -c 10 http://<your ELB>.us-east-1.elb.amazonaws.com/index.php
Related
I wish to run k6 in a container with some simple javascript load from local file system,
It seems the below had some syntax error
$ cat simple.js
import http from 'k6/http';
import { sleep } from 'k6';
export const options = {
vus: 10,
duration: '30s',
};
export default function () {
http.get('http://100.96.1.79:8080');
sleep(1);
}
$kubectl run k6 --image=grafana/k6 -- run - <simple.js
//OR
$kubectl run k6 --image=grafana/k6 run - <simple.js
in the k6 pod log, I got
│ time="2023-02-16T12:12:05Z" level=error msg="could not initialize '-': could not load JS test 'file:///-': no exported functions in s │
I guess this means the simple.js is not really passed to k6 this way?
thank you!
I think you can't pipe (host) files into Kubernetes containers this way.
One way that it should work is to:
Create a ConfigMap to represent your file
Apply a Pod config that mounts the ConfigMap file
NAMESPACE="..." # Or default
kubectl create configmap simple \
--from-file=${PWD}/simple.js \
--namespace=${NAMESPACE}
kubectl get configmap/simple \
--output=yaml \
--namespace=${NAMESPACE}
Yields:
apiVersion: v1
kind: ConfigMap
metadata:
name: simple
data:
simple.js: |
import http from 'k6/http';
import { sleep } from 'k6';
export default function () {
http.get('http://test.k6.io');
sleep(1);
}
NOTE You could just create e.g. configmap.yaml with the above YAML content and apply it.
Then with pod.yaml:
apiVersion: v1
kind: Pod
metadata:
name: simple
spec:
containers:
- name: simple
image: docker.io/grafana/k6
args:
- run
- /m/simple.js
volumeMounts:
- name: simple
mountPath: /m
volumes:
- name: simple
configMap:
name: simple
Apply it:
kubectl apply \
--filename=${PWD}/pod.yaml \
--namespace=${NAMESPACE}
Then, finally:
kubectl logs pod/simple \
--namespace=${NAMESPACE}
Yields:
/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/ \ | ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __________ \ |__| \__\ \_____/ .io
execution: local
script: /m/simple.js
output: -
scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
* default: 1 iterations for each of 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)
running (00m01.0s), 1/1 VUs, 0 complete and 0 interrupted iterations
default [ 0% ] 1 VUs 00m01.0s/10m0s 0/1 iters, 1 per VU
running (00m01.4s), 0/1 VUs, 1 complete and 0 interrupted iterations
default ✓ [ 100% ] 1 VUs 00m01.4s/10m0s 1/1 iters, 1 per VU
data_received..................: 17 kB 12 kB/s
data_sent......................: 542 B 378 B/s
http_req_blocked...............: avg=128.38ms min=81.34ms med=128.38ms max=175.42ms p(90)=166.01ms p(95)=170.72ms
http_req_connecting............: avg=83.12ms min=79.98ms med=83.12ms max=86.27ms p(90)=85.64ms p(95)=85.95ms
http_req_duration..............: avg=88.61ms min=81.28ms med=88.61ms max=95.94ms p(90)=94.47ms p(95)=95.2ms
{ expected_response:true }...: avg=88.61ms min=81.28ms med=88.61ms max=95.94ms p(90)=94.47ms p(95)=95.2ms
http_req_failed................: 0.00% ✓ 0 ✗ 2
http_req_receiving.............: avg=102.59µs min=67.99µs med=102.59µs max=137.19µs p(90)=130.27µs p(95)=133.73µs
http_req_sending...............: avg=67.76µs min=40.46µs med=67.76µs max=95.05µs p(90)=89.6µs p(95)=92.32µs
http_req_tls_handshaking.......: avg=44.54ms min=0s med=44.54ms max=89.08ms p(90)=80.17ms p(95)=84.62ms
http_req_waiting...............: avg=88.44ms min=81.05ms med=88.44ms max=95.83ms p(90)=94.35ms p(95)=95.09ms
http_reqs......................: 2 1.394078/s
iteration_duration.............: avg=1.43s min=1.43s med=1.43s max=1.43s p(90)=1.43s p(95)=1.43s
iterations.....................: 1 0.697039/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1
Tidy:
kubectl delete \
--filename=${PWD}/pod.yaml \
--namespace=${NAMESPACE}
kubectl delete configmap/simple \
--namespace=${NAMESPACE}
kubectl delete namespace/${NAMESPACE}
I have a command which runs pretty well when I just run it:
time aws sqs send-message --queue-url https://my_sqs_url --message-body "$(date "+%H:%M:%S_%N")"
{
"MD5OfMessageBody": "a19f365993d45d4885f7f15bce8aac97",
"MessageId": "30971fa7-d8ac-4540-9541-aebc38598856"
}
real 0m1.321s
user 0m1.174s
sys 0m0.117s
If I want to run in in background then the sqs message is sent however the process hangs infinitely (or at least I'm not patient enough to see when it eventually ends):
aws sqs send-message --queue-url https://my_sqs_url --message-body "$(date "+%H:%M:%S_%N")" &
[1] 9561
During that I see two processes instead of one:
ps -eFH | grep "aws sqs"
root 9561 2980 0 2210 912 1 09:29 pts/0 00:00:00 aws sqs send-message --queue-url https://my_sqs_url --message-body 09:29:30_009996044
root 9563 9561 0 63048 59172 1 09:29 pts/0 00:00:01 aws sqs send-message --queue-url https://my_sqs_url --message-body 09:29:30_009996044
The questions: why it hangs and how to do it properly?
This should work :
time aws sqs send-message --queue-url https://my_sqs_url --message-body "$(date "+%H:%M:%S_%N")" & wait $!
I wrote a readiness_probe for my pod by using a bash script. Readiness probe failed with Reason: Unhealthy but when I manually get in to the pod and run this command /bin/bash -c health=$(curl -s -o /dev/null --write-out "%{http_code}" http://localhost:8080/api/v2/ping); if [[ $health -ne 401 ]]; then exit 1; fi bash script exits with code 0.
What could be the reason? I am attaching the code and the error below.
Edit: Found out that the health variable is set to 000 which means timeout in for bash script.
readinessProbe:
exec:
command:
- /bin/bash
- '-c'
- |-
health=$(curl -s -o /dev/null --write-out "%{http_code}" http://localhost:8080/api/v2/ping);
if [[ $health -ne 401 ]]; then exit 1; fi
"kubectl describe pod {pod_name}" result:
Name: rustici-engine-54cbc97c88-5tg8s
Namespace: default
Priority: 0
Node: minikube/192.168.49.2
Start Time: Tue, 12 Jul 2022 18:39:08 +0200
Labels: app.kubernetes.io/name=rustici-engine
pod-template-hash=54cbc97c88
Annotations: <none>
Status: Running
IP: 172.17.0.5
IPs:
IP: 172.17.0.5
Controlled By: ReplicaSet/rustici-engine-54cbc97c88
Containers:
rustici-engine:
Container ID: docker://f7efffe6fc167e52f913ec117a4d78e62b326d8f5b24bfabc1916b5f20ed887c
Image: batupaksoy/rustici-engine:singletenant
Image ID: docker-pullable://batupaksoy/rustici-engine#sha256:d3cf985c400c0351f5b5b10c4d294d48fedfd2bb2ddc7c06a20c1a85d5d1ae11
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 12 Jul 2022 18:39:12 +0200
Ready: False
Restart Count: 0
Limits:
memory: 350Mi
Requests:
memory: 350Mi
Liveness: exec [/bin/bash -c health=$(curl -s -o /dev/null --write-out "%{http_code}" http://localhost:8080/api/v2/ping);
if [[ $health -ne 401 ]]; then exit 1; else exit 0; echo $health; fi] delay=10s timeout=5s period=10s #success=1 #failure=20
Readiness: exec [/bin/bash -c health=$(curl -s -o /dev/null --write-out "%{http_code}" http://localhost:8080/api/v2/ping);
if [[ $health -ne 401 ]]; then exit 1; else exit 0; echo $health; fi] delay=10s timeout=5s period=10s #success=1 #failure=10
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-whb8d (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-whb8d:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 24s default-scheduler Successfully assigned default/rustici-engine-54cbc97c88-5tg8s to minikube
Normal Pulling 23s kubelet Pulling image "batupaksoy/rustici-engine:singletenant"
Normal Pulled 21s kubelet Successfully pulled image "batupaksoy/rustici-engine:singletenant" in 1.775919851s
Normal Created 21s kubelet Created container rustici-engine
Normal Started 20s kubelet Started container rustici-engine
Warning Unhealthy 4s kubelet Readiness probe failed:
Warning Unhealthy 4s kubelet Liveness probe failed:
The probe could be failing because it is facing performance issues or slow startup. To troubleshoot this issue, you will need to check that the probe doesn’t start until the app is up and running in your pod. Perhaps you will need to increase the Timeout of the Readiness Probe, as well as the Timeout of the Liveness Probe, like in the following example:
readinessProbe:
initialDelaySeconds: 10
periodSeconds: 2
timeoutSeconds: 10
You can find more details about how to configure the Readlines Probe and Liveness Probe in this link.
I am running aws cli aws cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2010-02-20T12:00:00 --end-time 2010-02-20T15:00:00 --period 60 --namespace AWS/EC2 --extended-statistics p80 --dimensions Name=InstanceId,Value=i-0b123423423 in a
loop and extracting the metrics from json(using jq) in the below format for around 10-20 instances.
Illustrative JSON
{
"Label": "CPUUtilization",
"Datapoints": [
{
"Timestamp": "2020-02-20T12:15:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 0.16587132264856133
}
},
Instance-ABC
19.514049550078127
12.721997782508938
13.318820949213313
15.994192991030545
18.13096421299414
Instance-BCD
19.5140495
12.7219977
13.3188209
15.9941929
18.1309642
13.3188209
15.9941929
18.1309642
How can i calculate and run operations from the values in above output to get results like below in using Bash scripting
Instance above 70%
Instance-ABC
Instance-BCD
Instances below 20%
Instance-EFG
Instance-HIJ
I'm currently retrieving statistics with:
for i in $(aws ec2 describe-instances | jq -r '.["Reservations"]|.[]|.Instances|.[]| .InstanceId' | sort -n); do
echo "Instance $i"
aws cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2019-02-20T15:00:00T --end-time 2019-02-20T18:00:00 --period 60 --namespace AWS/EC2 --extended-statistics p80 --dimensions Name=InstanceId,Value=$i \
| jq '.Datapoints[].ExtendedStatistics[]'
done
I was able to setup localstack (https://github.com/atlassian/localstack) and also create lambda function in it (using create-function ... command). However, I couldnt find a way to create an APIGateway in localstack so that the lambda function can be called using it.
Basically, I need an APIGateway(and its arn), so that using that the lambda function can be called.
Walkthrough for creating a NodeJS Lambda together with API Gateway per CLI:
First we create a simple NodeJS Lambda:
const apiTestHandler = (payload, context, callback) => {
console.log(`Function apiTestHandler called with payload ${JSON.stringify(payload)}`);
callback(null, {
statusCode: 201,
body: JSON.stringify({
somethingId: payload.pathParameters.somethingId
}),
headers: {
"X-Click-Header": "abc"
}
});
}
module.exports = {
apiTestHandler,
}
Put that into a zip File called apiTestHandler.zip and upload it to localstack:
aws lambda create-function \
--region us-east-1 \
--function-name api-test-handler \
--runtime nodejs6.10 \
--handler index.apiTestHandler \
--memory-size 128 \
--zip-file fileb://apiTestHandler.zip \
--role arn:aws:iam::123456:role/role-name --endpoint-url=http://localhost:4574
Now we can create our Rest-Api:
aws apigateway create-rest-api --region us-east-1 --name 'API Test' --endpoint-url=http://localhost:4567
This gives the following response:
{
"name": "API Test",
"id": "487109A-Z548",
"createdDate": 1518081479
}
With the ID we got here, we can ask for its parent-ID:
aws apigateway get-resources --region us-east-1 --rest-api-id 487109A-Z548 --endpoint-url=http://localhost:4567
Response:
{
"items": [
{
"path": "/",
"id": "0270A-Z23550",
"resourceMethods": {
"GET": {}
}
}
]
}
Now we have everything to create our resource together with its path:
aws apigateway create-resource \
--region us-east-1 \
--rest-api-id 487109A-Z548 \
--parent-id 0270A-Z23550 \
--path-part "{somethingId}" --endpoint-url=http://localhost:4567
Response:
{
"resourceMethods": {
"GET": {}
},
"pathPart": "{somethingId}",
"parentId": "0270A-Z23550",
"path": "/{somethingId}",
"id": "0662807180"
}
The ID we got here is needed to create our linked GET Method:
aws apigateway put-method \
--region us-east-1 \
--rest-api-id 487109A-Z548 \
--resource-id 0662807180 \
--http-method GET \
--request-parameters "method.request.path.somethingId=true" \
--authorization-type "NONE" \
--endpoint-url=http://localhost:4567
We are almost there - one of the last things to do is to create our integration with the already uploaded lambda:
aws apigateway put-integration \
--region us-east-1 \
--rest-api-id 487109A-Z548 \
--resource-id 0662807180 \
--http-method GET \
--type AWS_PROXY \
--integration-http-method POST \
--uri arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:000000000000:function:api-test-handler/invocations \
--passthrough-behavior WHEN_NO_MATCH \
--endpoint-url=http://localhost:4567
Last but not least: Deploy our API to our desired stage:
aws apigateway create-deployment \
--region us-east-1 \
--rest-api-id 487109A-Z548 \
--stage-name test \
--endpoint-url=http://localhost:4567
Now we can test it:
curl http://localhost:4567/restapis/487109A-Z548/test/_user_request_/HowMuchIsTheFish
Response:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 34 100 34 0 0 9 0 0:00:03 0:00:03 --:--:-- 9
{"somethingId":"HowMuchIsTheFish"}
I hope this helps.
Hint 1: For easier use I recommend to install AWSCLI Local ( https://github.com/localstack/awscli-local ) - with this tool you can use the command "awslocal" and don't have to type "--endpoint-url= ..." for each command
Walkthrough for using Serverless Framework and Localstack:
You can also use the Serverless Framework (https://serverless.com/).
First install it via npm:
npm install serverless -g
Now you can create a sample application based on a nodejs-aws template:
serverless create --template aws-nodejs
In order to have an HTTP endpoint, you have to edit the serverless.yml and add the corresponding event :
functions:
hello:
handler: handler.hello
events:
- http:
path: ping
method: get
In order to run this against your localstack installation you have to use the serverless-localstack plugin ( https://github.com/temyers/serverless-localstack):
npm install serverless-localstack
Now you have to edit your serverless.yml again, add the plugin and adjust your endpoints. In my case localstack is running inside the Docker toolbox, so it's IP is 192.168.99.100 - you may have to change this to localhost, depending on your use:
plugins:
- serverless-localstack
custom:
localstack:
debug: true
stages:
- local
- dev
host: http://192.168.99.100
endpoints:
S3: http://192.168.99.100:4572
DynamoDB: http://192.168.99.100:4570
CloudFormation: http://192.168.99.100:4581
Elasticsearch: http://192.168.99.100:4571
ES: http://192.168.99.100:4578
SNS: http://192.168.99.100:4575
SQS: http://192.168.99.100:4576
Lambda: http://192.168.99.100:4574
Kinesis: http://192.168.99.100:4568
Now you can try to deploy it:
serverless deploy --verbose --stage local
This will create an S3 bucket, upload your lambda and create a cloudformation stack. However, the process will fail due to some inconsistencies of localstack when compared against AWS. Don't be dismayed though, the created cloudformation template works fine and you just need an additional request and you are done:
awslocal cloudformation update-stack --template-body file://.serverless/cloudformation-template-update-stack.json --stack-name aws-nodejs-local
Now your lambda is deployed and can be tested:
curl http://192.168.99.100:4567/restapis/75A-Z278430A-Z/local/_user_request_/ping
Response:
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 364 100 364 0 0 111 0 0:00:03 0:00:03 --:--:-- 111
{"message":"Go Serverless v1.0! Your function executed successfully!","input":{"body":null,"headers":{"host":"192.168.99.100:4567","accept":"*/*","user-agent":"curl/7.49.1"},"resource":"/restapis/75A-Z278430A-Z/local/_user_request_/ping","queryStringParameters":{},"httpMethod":"GET","stageVariables":{},"path":"/ping","pathParameters":{},"isBase64Encoded":false}}
Hope this helps.
Looks like there is an open issue related to setting up API Gateway with localstack on GitHub:
https://github.com/localstack/localstack/issues/129
You could try following the steps provided in the answer there.
Copied from the GitHub issue:
"""
One option would be to use the serverless framework (https://github.com/serverless/serverless). Otherwise, you can call the LocalStack services directly (via the CLI or an SDK) to create an API Gateway resource+method+integration, and connect them to your Lambda function.
Here are a few pointers that might be helpful:
https://ig.nore.me/2016/03/setting-up-lambda-and-a-gateway-through-the-cli/ (the "Creating a role" part can be skipped)
https://github.com/atlassian/localstack/issues/101
https://github.com/temyers/serverless-localstack
"""