Where does Ali Cloud keep a full list of availability zones? - alibaba-cloud

Alibaba Cloud's help page for regions gives you the full list of regions. That is helpful. It is also helpful that they tell you how many availability zones are in each region.
What is NOT helpful is that they fail to list the names of each availability zone. Twenty minutes of googling is way too long to spend on this. Where is the full list of availability zones for Ali Cloud/ Aliyun?

As of this writing there does not seem to be an easily accessible list of all availability zones by region for Aliyun. To get the full list you will need to pull it down with their Ali Cloud CLI tool.
It will spit out a wall of JSON, so it is also helpful to be on *nix and have the jq tool available.
Given all that, all you need is this short shell script:
#!/usr/bin/env bash
for region in $( aliyun ecs DescribeRegions | jq '.Regions.Region[].RegionId' )
do
echo $region
reg=$( echo $region | sed s/\"//g )
echo '---'
for zone in $( aliyun ecs DescribeZones --RegionId $reg | jq '.Zones.Zone[].ZoneId' | sort )
do
echo $zone
done
echo ''
done
The output of which is as follows.
"cn-qingdao"
---
"cn-qingdao-b"
"cn-qingdao-c"
"cn-beijing"
---
"cn-beijing-a"
"cn-beijing-b"
"cn-beijing-c"
"cn-beijing-d"
"cn-beijing-e"
"cn-beijing-f"
"cn-beijing-g"
"cn-zhangjiakou"
---
"cn-zhangjiakou-a"
"cn-zhangjiakou-b"
"cn-huhehaote"
---
"cn-huhehaote-a"
"cn-huhehaote-b"
"cn-hangzhou"
---
"cn-hangzhou-b"
"cn-hangzhou-c"
"cn-hangzhou-d"
"cn-hangzhou-e"
"cn-hangzhou-f"
"cn-hangzhou-g"
"cn-hangzhou-h"
"cn-shanghai"
---
"cn-shanghai-a"
"cn-shanghai-b"
"cn-shanghai-c"
"cn-shanghai-d"
"cn-shanghai-e"
"cn-shanghai-f"
"cn-shenzhen"
---
"cn-shenzhen-a"
"cn-shenzhen-b"
"cn-shenzhen-c"
"cn-shenzhen-d"
"cn-hongkong"
---
"cn-hongkong-a"
"cn-hongkong-b"
"cn-hongkong-c"
"ap-northeast-1"
---
"ap-northeast-1a"
"ap-southeast-1"
---
"ap-southeast-1a"
"ap-southeast-1b"
"ap-southeast-1c"
"ap-southeast-2"
---
"ap-southeast-2a"
"ap-southeast-2b"
"ap-southeast-3"
---
"ap-southeast-3a"
"ap-southeast-3b"
"ap-southeast-5"
---
"ap-southeast-5a"
"ap-south-1"
---
"ap-south-1a"
"ap-south-1b"
"us-east-1"
---
"us-east-1a"
"us-east-1b"
"us-west-1"
---
"us-west-1a"
"us-west-1b"
"eu-west-1"
---
"eu-west-1a"
"eu-west-1b"
"me-east-1"
---
"me-east-1a"
"eu-central-1"
---
"eu-central-1a"
"eu-central-1b"

Alibaba Cloud do have details about "No. of available zones under each region and services available in that particular region" in their website itself. Please have a look at Alibaba Cloud Website.

Agree, documentation is very poor and not readable, but i have found this - for those looking for a Python way:
https://www.alibabacloud.com/help/en/elastic-compute-service/latest/describezones
I search everything i need in this URL ( make sure to hit the tab "All products" )
Which you can request with either:
https://ecs.aliyuncs.com/?Action=DescribeZones
&RegionId=cn-hangzhou
&InstanceChargeType=PostPaid
&SpotStrategy=NoSpot
&<Common request parameters>
Python example
And if you're using Python, there's the SDK:
https://pypi.org/project/aliyun-python-sdk-vpc/
I have found inside the package the following file ( which isn't documented anywhere as well )
import json
from aliyunsdkcore.client import AcsClient
from aliyunsdkvpc.request.v20160428.DescribeZonesRequest import DescribeZonesRequest
region = "cn-shanghai"
client = AcsClient(access_key, secret_key, region)
request = DescribeZonesRequest()
request.set_accept_format('json')
zones_list = json.loads(client.do_action_with_exception(request).decode('utf8'))
print(zones_list)
And with the sdk above i'm also querying the regions like so:
aliyunsdkcore is being installed as a dependency for aliyunsdkvpc no need to install
from aliyunsdkcore.endpoint.local_config_regional_endpoint_resolver import LocalConfigRegionalEndpointResolver
resolver = LocalConfigRegionalEndpointResolver()
regions_list = resolver.get_valid_region_ids_by_product(product_code='ecs')
print(regions_list)

Related

CDK: How do I configure `userData` with values known only at deploy time?

I am using CDK v2, with Typescript.
I want my bastion machine to log stuff to Cloudwatch.
The specific LogGroup I want it to write to is also created via CDK (so that I can customise the retention).
How can I customise the userData script with knowledge about other AWS resources, which are also created by CDK - so I can't know their names?
My CDK stuff is being deployed via a CDK pipeline.
Here is my CDK script:
export class StoBastion extends cdk.Stack {
constructor(scope: Construct, id: string, props: cdk.StackProps){
super(scope, id, props);
// actual name: DemoStage-StoBastion-StoBastionLogGroup5EEB3DE8-AdkaWy0ELoeF
const logGroup = new LogGroup(this, "StoBastionLogGroup", {
retention: RetentionDays.TWO_WEEKS,
});
let initScriptPath = 'lib/script/bastion-linux2-asg-provision.sh';
const userDataText = readFileSync(initScriptPath, 'utf8');
const autoScalingGroup = new AutoScalingGroup(this, 'StoAsg', {
...
userData: UserData.custom(userDataText),
})
}
}
And the shell script I want to use as the userData for the instance:
#!/bin/sh
### cloudwatch ###
# This goes as early as possible in the script, so we can see what's going
# on from Cloudwatch ASAP.
echo " >>bastion>> installing cloudwatch package $(date)"
yum install -y awslogs
echo " >>bastion>> configuring cloudwatch - ${TF_APP_LOG_GROUP} $(date)"
## overwrite awscli.conf ##
cat > /etc/awslogs/awscli.conf <<EOL
[plugins]
cwlogs = cwlogs
[default]
region = ${TF_APP_REGION}
EOL
## end of overwrite awscli.conf ##
## overwrite awslogs.conf ##
cat > /etc/awslogs/awslogs.conf <<EOL
[general]
state_file = /var/lib/awslogs/agent-state
[cloudinit]
datetime_format = %b %d %H:%M:%S
file = /var/log/cloud-init-output.log
buffer_duration = 5000
log_group_name = ${TF_APP_LOG_GROUP}
log_stream_name = linux2-cloud-init-output-{instance_id}
initial_position = start_of_file
EOL
## of overwrite awslogs.conf ##
echo " >>bastion>> start awslogs daemon $(date)"
systemctl start awslogsd
echo " >>bastion>> make sure awslogs starts up on boot"
systemctl enable awslogsd.service
### end cloudwatch ###
I want to somehow replace the variable references in the userData script like ${TF_APP_LOG_GROUP} with values populated at CDK deploy time so they have the correct values.
I'm doing cloudwatch stuff at the moment, but there will be other stuff I need to do like this, so this question isn't about cloudwatch - it's about "how can I configure my userData with values known only at CDK deploy time"?
You can do this with conventional string formatting tools, as if the log group were a regular string:
const userDataText = readFileSync(
initScriptPath,
'utf8'
).replaceAll(
'${TF_APP_LOG_GROUP}',
logGroup.logGroupName
);
What this will do behind the scenes is replace all occurences of ${TF_APP_LOG_GROUP} in the text string with a token (a special string that looks something like ${TOKEN[LogGroup.Name.1234]}), and CloudFormation will in turn replace it with the actual value during deployment.
For reference: https://docs.aws.amazon.com/cdk/v2/guide/tokens.html
This was all incorrect. Leaving it for posterity purposes, but not valid as it was the wrong direction
Wherever you are synthing your template - be that in a SimpleSynth
Pipelines step or in a CodeBuild if using a standard
CodePipeline/CodeBuild, you can include context variables in the
synth.
The command cdk deploy can be followed with any context variables
you want: cdk deploy -c VariableName=value - if your bash script is
returning to the shell the answers, you can store them as shell
variables and use them in the cdk deploy.
you can then reference these variables within the actual stacks with
const bucket_name = this.node.tryGetContext('VariableName');
See this documentation or this SO post for more information.

How to add service principal to azure databricks workspace using databricks cli from cloud shell

I tried adding service principal to azure databricks workspace using cloud shell but getting error. I am able to look at all the clusters in the work space and I was the one who created that workspace. Do I need to be in admin group if I want to add Service Principal to workspace?
curl --netrc -X POST \ https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.net/api/2.0/preview/scim/v2/ServicePrincipals \ --header 'Content-type: application/scim+json' \ --data #create-service-principal.json \ | jq .
file has following info:
{ "displayName": "sp-name", "applicationId": "a9217fxxxxcd-9ab8-dxxxxxxxxxxxxx", "entitlements": [ { "value": "allow-cluster-create" } ], "schemas": [ "urn:ietf:params:scim:schemas:core:2.0:ServicePrincipal" ], "active": true }
Here is the error I am getting: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 279 100 279 0 0 5166 0 --:--:-- --:--:-- --:--:-- 5264 parse error: Invalid numeric literal at line 2, column 0
Do I need to be in admin group if I want to add Service Principal to workspace?
Issue is with JSON file not with access to admin group.
You need to check double quotes in line number 2 of your JSON file.
You can refer this github link
Try this code in Python that you can run in a Databricks notebook:
import pandas
import json
import requests
# COMMAND ----------
# MAGIC %md ### define variables
# COMMAND ----------
pat = 'EnterPATHere' # paste PAT. Get it from settings > user settings
workspaceURL = 'EnterWorkspaceURLHere' # paste the workspace url in the format of 'https://adb-1234567.89.azuredatabricks.net'
applicationID = 'EnterApplicationIDHere' # paste ApplicationID / ClientID of Service Principal. Get it from Azure Portal
friendlyName = 'AddFriendlyNameHere' # paste FriendlyName of ServicePrincipal. Get it from Azure Portal
# COMMAND ----------
# MAGIC %md ### add service principal
# COMMAND ----------
payload_raw = {
'schemas':
['urn:ietf:params:scim:schemas:core:2.0:ServicePrincipal'],
'applicationId': applicationID,
'displayName': friendlyName,
'groups':[],
'entitlements':[]
}
payload = json.loads(json.dumps(payload_raw))
response = requests.post(workspaceURL + '/api/2.0/preview/scim/v2/ServicePrincipals',\
headers = {'Authorization' : 'Bearer '+ pat,\
'Content-Type': 'application/scim+json'},\
data=json.dumps(payload))
response.content
I have actually published a blog post where a Python script is provided to fully manage service principals and access control in Databricks workspaces.

Receiving error in AWS Secrets manager awscli for: Version "AWSCURRENT" not found when deployed via Terraform

Overview
Create a aws_secretsmanager_secret
Create a aws_secretsmanager_secret_version
Store a uniquely generated string as that above version
Use local-exec provisioner to store the actual secured string using bash
Reference that string using the secretsmanager resource in for example, an RDS instance deployment.
Objective
Keep all plain text strings out of remote-state residing in a S3 bucket
Use AWS Secrets Manager to store these strings
Set once, retrieve by calling the resource in Terraform
Problem
Error: Secrets Manager Secret
"arn:aws:secretsmanager:us-east-1:82374283744:secret:Example-rds-secret-fff42b69-30c1-df50-8e5c-f512464a4a11-pJvC5U"
Version "AWSCURRENT" not found
when running terraform apply
Question
Why isn't it moving the AWSCURRENT version automatically? Am I missing something? Is my bash command wrong? The value does not write to the secret_version, but it does reference it correctly.
Look in main.tf code, which actually performs the command:
provisioner "local-exec" {
command = "bash -c 'RDSSECRET=$(openssl rand -base64 16); aws secretsmanager put-secret-value --secret-id ${data.aws_secretsmanager_secret.secretsmanager-name.arn} --secret-string $RDSSECRET --version-stages AWSCURRENT --region ${var.aws_region} --profile ${var.aws-profile}'"
}
Code
main.tf
data "aws_secretsmanager_secret_version" "rds-secret" {
secret_id = aws_secretsmanager_secret.rds-secret.id
}
data "aws_secretsmanager_secret" "secretsmanager-name" {
arn = aws_secretsmanager_secret.rds-secret.arn
}
resource "random_password" "db_password" {
length = 56
special = true
min_special = 5
override_special = "!#$%^&*()-_=+[]{}<>:?"
keepers = {
pass_version = 1
}
}
resource "random_uuid" "secret-uuid" { }
resource "aws_secretsmanager_secret" "rds-secret" {
name = "DAL-${var.environment}-rds-secret-${random_uuid.secret-uuid.result}"
}
resource "aws_secretsmanager_secret_version" "rds-secret-version" {
secret_id = aws_secretsmanager_secret.rds-secret.id
secret_string = random_password.db_password.result
provisioner "local-exec" {
command = "bash -c 'RDSSECRET=$(openssl rand -base64 16); aws secretsmanager put-secret-value --secret-id ${data.aws_secretsmanager_secret.secretsmanager-name.arn} --secret-string $RDSSECRET --region ${var.aws_region} --profile ${var.aws-profile}'"
}
}
variables.tf
variable "aws-profile" {
description = "Local AWS Profile Name "
type = "string"
}
variable "aws_region" {
description = "aws region"
default="us-east-1"
type = "string"
}
variable "environment" {}
terraform.tfvars
aws_region="us-east-1"
aws-profile="Example-Environment"
environment="dev"
The error likely isn't occuring in your provisioner execution per se, because if you remove the provisioner block the error still occurs on apply--but confusingly only the first time after a destroy.
Removing the data "aws_secretsmanager_secret_version" "rds-secret" block as well "resolves" the error completely.
I'm guessing there is some sort of config delay issue here...but adding a 20 second delay provisioner to the aws_secretsmanager_secret.rds-secret resource block didn't help.
And the value from the data block can be successfully output on subsequent apply runs, so maybe it's not just timing.
Even if you resolve the above more basic issue, it's likely your provisioner will still be confusing things by modifying a resource that Terraform is trying to manage in the same run. I'm not sure there's a way to get around that except perhaps by splitting into two separate operations.
Update:
It turns out that on the first run the data sources are read before the aws_secretsmanager_secret_version resource is created. Just adding depends_on = [aws_secretsmanager_secret_version.rds-secret-version] to the data "aws_secretsmanager_secret_version" block resolves this fully and makes the interpolation for your provisioner work as well. I haven't tested the actual provisioner.
Also you may need to consider this (which I take to not always apply to 0.13):
NOTE: In Terraform 0.12 and earlier, due to the data resource behavior of deferring the read until the apply phase when depending on values that are not yet known, using depends_on with data resources will force the read to always be deferred to the apply phase, and therefore a configuration that uses depends_on with a data resource can never converge. Due to this behavior, we do not recommend using depends_on with data resources.

TextFSM Template for Netmiko for "inc" phrase

I am trying to create a textfsm template with the Netmiko library. While it works for most of the commands, it does not work when I try performing "inc" operation in the network device. The textfsm index file seems like it is not recognizing the same command for 2 different templates; for instance:
If I am giving the command - show running | inc syscontact
And give another command - show running | inc syslocation
in textfsm index; the textfsm template seems like it is recognizing only the first command; and not the second command.
I understand that I can get the necessary data by the regex expression for syscontact and syslocation for the commands( via the template ), however I want to achieve this by the "inc" command from the device itself. Is there a way this can be done?
you need to escape the pipe in the index file. e.g. sh[[ow]] ru[[nning]] \| inc syslocation
There is a different way to parse that you want all datas which is called TTP module. You can take the code I wrote below as an example. You can create your own templates.
from pprint import pprint
from ttp import ttp
import json
import time
with open("showSystemInformation.txt") as f:
data_to_parse = f.read()
ttp_template = """
<group name="Show_System_Information">
System Name : {{System_Name}}
System Type : {{System_Type}} {{System_Type_2}}
System Version : {{Version}}
System Up Time : {{System_Uptime_Days}} days, {{System_Uptime_HR_MIN_SEC}} (hr:min:sec)
Last Saved Config : {{Last_Saved_Config}}
Time Last Saved : {{Last_Time_Saved_Date}} {{Last_Time_Saved_HR_MIN_SEC}}
Time Last Modified : {{Last_Time_Modified_Date}} {{Last_Time_Modifed_HR_MIN_SEC}}
</group>
"""
parser = ttp(data=data_to_parse, template=ttp_template)
parser.parse()
# print result in JSON format
results = parser.result(format='json')[0]
print(results)
Example run:
[appadmin#ryugbz01 Nokia]$ python3 showSystemInformation.py
[
{
"Show_System_Information": {
"Last_Saved_Config": "cf3:\\config.cfg",
"Last_Time_Modifed_HR_MIN_SEC": "11:46:57",
"Last_Time_Modified_Date": "2022/02/09",
"Last_Time_Saved_Date": "2022/02/07",
"Last_Time_Saved_HR_MIN_SEC": "15:55:39",
"System_Name": "SR7-2",
"System_Type": "7750",
"System_Type_2": "SR-7",
"System_Uptime_Days": "17",
"System_Uptime_HR_MIN_SEC": "05:24:44.72",
"Version": "C-16.0.R9"
}
}
]

Cloudwatch to Elasticsearch parse/tokenize log event before push to ES

Appreciate your help in advance.
In my scenario - Cloudwatch multiline logs needs to be shipped to elasticsearch service.
ECS--awslog->Cloudwatch---using lambda--> ES Domain
(Basic flow though very open to change how data is shipped from CW to ES )
I was able to solve multi-line issue using multi_line_start_pattern BUT
The main issue I am experiencing now - is my logs have ODL format (following format)
[yyyy-mm-ddThh:mm:ss.SSS-Z][ProductName-Version][Log Level]
[Message ID][LoggerName][Key Value Pairs][[
Message]]
AND I will like to parse and tokenize log events before storing in ES (vs the complete log line ).
For example:
[2018-05-31T11:08:49.148-0400] [glassfish 4.1] [INFO] [] [] [tid: _ThreadID=43 _ThreadName=Thread-8] [timeMillis: 1527692929148] [levelValue: 800] [[
[] INFO : (DummyApplicationFunctionJPADAO) EntityManagerFactory located under resource lookup name [null], resource name=AuthorizationPU]]
Needs to be parsed and tokenize using format
timestamp 2018-05-31T11:08:49.148-0400
ProductName-Version glassfish 4.1
LogLevel INFO
MessageID
LoggerName
KeyValuePairs tid: _ThreadID=43 _ThreadName=Thread-8
Message [] INFO : (DummyApplicationFunctionJPADAO)
EntityManagerFactorylocated under resource lookup name
[null], resource name=AuthorizationPU
In above Key Value pairs repeat and are variable - for simplicity I can store all as one long string.
As far as what I gathered about Cloudwatch - It seems Subscription Filter Pattern reg ex support is very limited really not sure how to fit the above pattern. For lambda function that pushes the data to ES have not seen AWS doc or examples that support lambda as means to parse and push for ES.
Will appreciate if someone can please guide what/where will be best option to parse CW logs before it gets into ES => Subscription Filter -Pattern vs in lambda function or any other way.
Thank you .
From what I can see your best bet is what you're suggesting, a CloudWatch log triggered lambda that reformats the logged data into your ES prefered format and then posts it into ES.
You'll need to subscribe this lambda to your CloudWatch logs. You can do this on the lambda console, or the cloudwatch console (https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html).
The lambda's event payload will be: { "awslogs": { "data": "encoded-logs" } }. Where encoded-logs is a Base64 encoding of a gzipped JSON.
For example, the sample event (https://docs.aws.amazon.com/lambda/latest/dg/eventsources.html#eventsources-cloudwatch-logs) can be decoded in node, for example, using:
const zlib = require('zlib');
const data = event.awslogs.data;
const gzipped = Buffer.from(data, 'base64');
const json = zlib.gunzipSync(gzipped);
const logs = JSON.parse(json);
console.log(logs);
/*
{ messageType: 'DATA_MESSAGE',
owner: '123456789123',
logGroup: 'testLogGroup',
logStream: 'testLogStream',
subscriptionFilters: [ 'testFilter' ],
logEvents:
[ { id: 'eventId1',
timestamp: 1440442987000,
message: '[ERROR] First test message' },
{ id: 'eventId2',
timestamp: 1440442987001,
message: '[ERROR] Second test message' } ] }
*/
From what you've outlined, you'll want to extract the logEvents array, and parse this into an array of strings. I'm happy to give some help on this too if you need it (but I'll need to know what language you're writing your lambda in- there are libraries for tokenizing ODL- so hopefully it's not too hard).
At this point you can then POST these new records directly into your AWS ES Domain. Somewhat crypitcally the S3-to-ES guide gives a good outline of how to do this in python: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-aws-integrations.html#es-aws-integrations-s3-lambda-es
You can find a full example for a lambda that does all this (by someone else) here: https://github.com/blueimp/aws-lambda/tree/master/cloudwatch-logs-to-elastic-cloud

Resources