Trouble implementing chained variables in Makefile - bash

I'm looking to use the previous variable to look up the next variable (these should all be set as envvars) I have this working in a smaller extent with just profile, but when i try chaining variables together they fall flat...
Expected output is that make build has TF_VARS_* set as envvars and that they are able to be used as regular variables for the next variable input. I assume that there should be an export of TF_VAR_aws_profile TF_VAR_aws_vpc and TF_VAR_aws_net into global scope, this seemed to work fine for when I was JUST using TF_VAR_aws_profile and was able to confirm with echo $TF_VAR_aws_profile
export TF_VAR_aws_profile TF_VAR_aws_vpc TF_VAR_aws_net
# An implicit guard target, used by other targets to ensure
# that environment variables are set before beginning tasks
assert-%:
# if [ "${${*}}" = "" ] ; then \
echo "Environment variable $* not set" ; \
exit 1 ; \
fi
vault:
## This is the problem section here....
# read -p "Enter AWS Profile Name: " profile ; \
vpc=$(shell aws --profile "$${profile}" --region us-west-2 ec2 describe-vpcs |jq -r '.[] | first | .VpcId') ; \
net=$(shell aws --profile "$${profile}" --region us-west-2 ec2 describe-subnets --filters "Name=vpc-id,Values=$${vpc}" |jq -r '.[] | first | .SubnetId') ; \
TF_VAR_aws_profile=$$profile TF_VAR_aws_vpc=$$vpc TF_VAR_aws_net=$$net make build && \
TF_VAR_aws_profile=$$profile make keypair && \
TF_VAR_aws_profile=$$profile make plan && \
TF_VAR_aws_profile=$$profile make apply
build: require-packer
aws-vault exec $(TF_VAR_aws_profile) --assume-role-ttl=60m -- \
"/usr/local/bin/packer" "build" \
"-var" "builder_subnet_id=$(TF_VAR_aws_net)" \
"-var" "builder_vpc_id=$(TF_VAR_aws_vpc)" \
"packer/vault.json"
require-packer: assert-TF_VAR_aws_vpc assert-TF_VAR_aws_net
# echo "[info] VPC: $(TF_VAR_aws_vpc)" ## Not set
# echo "[info] NET: $(TF_VAR_aws_net)" ## Not set
packer --version &> /dev/null
UPDATE
Looks like it might be something to do with line termination on the vault expression, added ; \ to continue the lines

Got it, it was a combination of TWO things, one running as $(shell \$CMD) does not evaluate the inner expression variables, I needed to revert to ticks instead to evaluate variables (keep in mind, variables STILL need $$ in order to evaluate inside of Makefiles) as well as the lines needed to be continued, so terminating only with ; is not effective, we need to use ; \.
Please note that I updated my variable names from above, they are now $$subnet and $$prvnet because I'm a spacing nazi...
export TF_VAR_aws_profile TF_VAR_aws_prvnet TF_VAR_aws_subnet
# An implicit guard target, used by other targets to ensure
# that environment variables are set before beginning tasks
assert-%:
# if [ "${${*}}" = "" ] ; then \
echo "Environment variable $* not set" ; \
exit 1 ; \
fi
vault:
# read -p "Enter AWS Profile Name: " profile ; \
prvnet=`aws --profile "$${profile}" --region us-west-2 ec2 describe-vpcs |jq -r '.[] | first | .VpcId'` ; \
subnet=`aws --profile "$${profile}" --region us-west-2 ec2 describe-subnets --filters "Name=vpc-id,Values=$${prvnet}" |jq -r '.[] | first | .SubnetId'` ; \
TF_VAR_aws_profile=$$profile TF_VAR_aws_prvnet=$$prvnet TF_VAR_aws_subnet=$$subnet make build && \
TF_VAR_aws_profile=$$profile make keypair && \
TF_VAR_aws_profile=$$profile make plan && \
TF_VAR_aws_profile=$$profile make apply
build: require-packer
aws-vault exec $(TF_VAR_aws_profile) --assume-role-ttl=60m -- \
"/usr/local/bin/packer" "build" \
"-var" "builder_subnet_id=$(TF_VAR_aws_net)" \
"-var" "builder_vpc_id=$(TF_VAR_aws_vpc)" \
"packer/vault.json"
require-packer: assert-TF_VAR_aws_prvnet assert-TF_VAR_aws_subnet
# echo "[info] VPC: $(TF_VAR_aws_prvnet)" ;
# echo "[info] NET: $(TF_VAR_aws_subnet)" ;
packer --version &> /dev/null

Related

GCP Dataproc - cluster creation failing when using connectors.sh in initialization-actions

I'm creating a Dataproc cluster, and it is timing out when i'm adding the connectors.sh in the initialization actions.
here is the command & error
NUM_WORKER=2
TYPE=n1-highmem-8
CNAME=dataproc-poc
BUCKET=dataproc-spark-karan
REGION=us-central1
ZONE=us-central1-c
IMG_VERSION=2.0.29-debian10
PROJECT=versa-kafka-poc
NUM_WORKER=2
Karans-MacBook-Pro:dataproc-versa-sase karanalang$ gcloud beta dataproc clusters create $CNAME \
> --enable-component-gateway \
> --bucket $BUCKET \
> --region $REGION \
> --zone $ZONE \
> --no-address --master-machine-type $TYPE \
> --master-boot-disk-size 100 \
> --master-boot-disk-type pd-ssd \
> --num-workers $NUM_WORKER \
> --worker-machine-type $TYPE \
> --worker-boot-disk-type pd-ssd \
> --worker-boot-disk-size 100 \
> --image-version $IMG_VERSION \
> --scopes 'https://www.googleapis.com/auth/cloud-platform' \
> --project $PROJECT \
> --initialization-actions 'gs://dataproc-kafka/config/pip_install.sh','gs://dataproc-kafka/config/connectors.sh' \
> --metadata 'gcs-connector-version=2.0.0' \
> --metadata 'bigquery-connector-version=1.2.0' \
> --properties 'dataproc:dataproc.logging.stackdriver.job.driver.enable=true,dataproc:job.history.to-gcs.enabled=true,spark:spark.dynamicAllocation.enabled=false,spark:spark.executor.instances=6,spark:spark.executor.cores=2,spark:spark.eventLog.dir=gs://dataproc-spark-karan/joblogs,spark:spark.history.fs.logDirectory=gs://dataproc-spark-karan/joblogs'
Waiting on operation [projects/versa-kafka-poc/regions/us-central1/operations/8aa13a77-30a8-3a84-a949-16b4d8907c45].
Waiting for cluster creation operation...
WARNING: This cluster is configured to use network 'https://www.googleapis.com/compute/v1/projects/versa-kafka-poc/global/networks/default' and its associated firewall rules '[prometheus-nodeport]' which contains the following potential security vulnerability: 'port 8088 is open to the internet, this may allow arbitrary code execution via the YARN REST API. Use Component Gateway for secure remote access to the YARN UI and other cluster UIs instead: https://cloud.google.com/dataproc/docs/concepts/accessing/dataproc-gateways.'
Waiting for cluster creation operation...done.
ERROR: (gcloud.beta.dataproc.clusters.create) Operation [projects/versa-kafka-poc/regions/us-central1/operations/8aa13a77-30a8-3a84-a949-16b4d8907c45] timed out.
connectors.sh
#!/bin/bash
set -euxo pipefail
VM_CONNECTORS_HADOOP_DIR=/usr/lib/hadoop/lib
VM_CONNECTORS_DATAPROC_DIR=/usr/local/share/google/dataproc/lib
declare -A MIN_CONNECTOR_VERSIONS
MIN_CONNECTOR_VERSIONS=(
["bigquery"]="0.11.0"
["gcs"]="1.7.0")
# Starting from these versions connectors name changed:
# "...-<version>-hadoop2.jar" -> "...-hadoop2-<version>.jar"
declare -A NEW_NAME_MIN_CONNECTOR_VERSIONS
NEW_NAME_MIN_CONNECTOR_VERSIONS=(
["bigquery"]="0.13.5"
["gcs"]="1.9.5")
BIGQUERY_CONNECTOR_VERSION=$(/usr/share/google/get_metadata_value attributes/bigquery-connector-version || true)
GCS_CONNECTOR_VERSION=$(/usr/share/google/get_metadata_value attributes/gcs-connector-version || true)
UPDATED_GCS_CONNECTOR=false
is_worker() {
local role
role="$(/usr/share/google/get_metadata_value attributes/dataproc-role || true)"
if [[ $role != Master ]]; then
return 0
fi
return 1
}
min_version() {
echo -e "$1\n$2" | sort -r -t'.' -n -k1,1 -k2,2 -k3,3 | tail -n1
}
validate_version() {
local name=$1 # connector name: "bigquery" or "gcs"
local version=$2 # connector version
local min_valid_version=${MIN_CONNECTOR_VERSIONS[$name]}
if [[ "$(min_version "$min_valid_version" "$version")" != "$min_valid_version" ]]; then
echo "ERROR: $name-connector version should be greater than or equal to $min_valid_version, but was $version"
return 1
fi
}
update_connector() {
local name=$1 # connector name: "bigquery" or "gcs"
local version=$2 # connector version
if [[ $version ]]; then
if [[ $name == gcs ]]; then
UPDATED_GCS_CONNECTOR=true
fi
# validate new connector version
validate_version "$name" "$version"
if [[ -d ${VM_CONNECTORS_DATAPROC_DIR} ]]; then
local vm_connectors_dir=${VM_CONNECTORS_DATAPROC_DIR}
else
local vm_connectors_dir=${VM_CONNECTORS_HADOOP_DIR}
fi
# remove old connector
rm -f "${vm_connectors_dir}/${name}-connector-"*
# download new connector
# connector name could be in one of 2 formats:
# 1) gs://hadoop-lib/${name}/${name}-connector-hadoop2-${version}.jar
# 2) gs://hadoop-lib/${name}/${name}-connector-${version}-hadoop2.jar
local new_name_min_version=${NEW_NAME_MIN_CONNECTOR_VERSIONS[$name]}
if [[ "$(min_version "$new_name_min_version" "$version")" == "$new_name_min_version" ]]; then
local jar_name="${name}-connector-hadoop2-${version}.jar"
else
local jar_name="${name}-connector-${version}-hadoop2.jar"
fi
gsutil cp "gs://hadoop-lib/${name}/${jar_name}" "${vm_connectors_dir}/"
# Update or create version-less connector link
ln -s -f "${vm_connectors_dir}/${jar_name}" "${vm_connectors_dir}/${name}-connector.jar"
fi
}
if [[ -z $BIGQUERY_CONNECTOR_VERSION ]] && [[ -z $GCS_CONNECTOR_VERSION ]]; then
echo "ERROR: None of connector versions are specified"
exit 1
fi
# because connectors from 1.7 branch are not compatible with previous connectors
# versions (they have the same class relocation paths) we need to update both
# of them, even if only one connector version is set
if [[ -z $BIGQUERY_CONNECTOR_VERSION ]] && [[ $GCS_CONNECTOR_VERSION == "1.7.0" ]]; then
BIGQUERY_CONNECTOR_VERSION="0.11.0"
fi
if [[ $BIGQUERY_CONNECTOR_VERSION == "0.11.0" ]] && [[ -z $GCS_CONNECTOR_VERSION ]]; then
GCS_CONNECTOR_VERSION="1.7.0"
fi
update_connector "bigquery" "$BIGQUERY_CONNECTOR_VERSION"
update_connector "gcs" "$GCS_CONNECTOR_VERSION"
if [[ $UPDATED_GCS_CONNECTOR != true ]]; then
echo "GCS connector wasn't updated - no need to restart any services"
exit 0
fi
# Restart YARN NodeManager service on worker nodes so they can pick up updated GCS connector
if is_worker; then
systemctl kill -s KILL hadoop-yarn-nodemanager
fi
# Restarts Dataproc Agent after successful initialization
# WARNING: this function relies on undocumented and not officially supported Dataproc Agent
# "sentinel" files to determine successful Agent initialization and not guaranteed
# to work in the future. Use at your own risk!
restart_dataproc_agent() {
# Because Dataproc Agent should be restarted after initialization, we need to wait until
# it will create a sentinel file that signals initialization competition (success or failure)
while [[ ! -f /var/lib/google/dataproc/has_run_before ]]; do
sleep 1
done
# If Dataproc Agent didn't create a sentinel file that signals initialization
# failure then it means that initialization succeded and it should be restarted
if [[ ! -f /var/lib/google/dataproc/has_failed_before ]]; then
systemctl kill -s KILL google-dataproc-agent
fi
}
export -f restart_dataproc_agent
# Schedule asynchronous Dataproc Agent restart so it will use updated connectors.
# It could not be restarted sycnhronously because Dataproc Agent should be restarted
# after its initialization, including init actions execution, has been completed.
bash -c restart_dataproc_agent &
disown
From what i understand, the connectors.sh just ensures that correct version of the connectors is included in the cluster.
Also - without the connectors.sh, the installation is going through fine.
How do i debug/fix this ?
tia!
It seems you are using an old version of the init action script. Based on the documentation from the Dataproc GitHub repo, you can set the version of the Hadoop GCS connector without the script in the following manner:
gcloud dataproc clusters create ${CLUSTER_NAME} \
--region ${REGION} \
--metadata GCS_CONNECTOR_VERSION=2.2.2
For the BigQuery connectors (Spark or Hadoop MR), please use the up-to-date init action in the following manner:
--initialization-actions gs://${BUCKET}/connectors.sh \
--metadata bigquery-connector-version=1.2.0 \
--metadata spark-bigquery-connector-version=0.23.2
Notice that the same repo contains the updated pip-install init action as well.

bash problem with single quotes expansion

I got the following script:
#!/usr/bin/env bash
# break on error
set -ex
## USAGE: ./open <type> <instance-name>
if [[ $# -ne 2 ]]; then
echo "## USAGE: ${0} <type> <instance-name>"
echo "type: team or project"
echo "instance-name: name of the instance to fetch the URL from"
exit 1
fi
ORGANIZATION="redacted_org"
PROJECT="redacted_project"
TYPE="redacted_type"
if [[ $1 == "team" ]]; then
TYPE="redacted_type_team"
fi
NAME=$2
BUILD_ID=$(az pipelines build definition list \
--organization "${ORGANIZATION}" \
--project "${PROJECT}" \
--query "\"[? contains(path, '\\\\stacks\\\\${TYPE}\\\\instances\\\\${NAME}') && name=='apply'].{id:id}\"" \
--output tsv)
echo "https://redacted_org.visualstudio.com/redacted_project/_build?definitionId=${BUILD_ID}"
The problem is that the BUILD_ID variable is being executed as (printed by the set -x):
++ az pipelines build definition list --organization redacted_org --project redacted_project --query '"[? contains(path, \'\''\\stacks\\redacted_type\\instances\\redacted_2\'\'') && name==\'\''apply\'\''].{id:id}"' --output tsv
Note the '" instead of " just after the --query parameter and in the end of it, and the \'\' everywhere I have a single quote instead of '
I need this BUILD_ID to be executed as follows, this command works fine when typed in the terminal itself:
$(az pipelines build definition list --organization "redacted_org" --project "redacted_project" --query "[? contains(path, '\\stacks\\redacted_type\\instances\\redacted_2') && name=='apply'].{id:id}" --output tsv)
What am I doing wrong?
In the stanza:
BUILD_ID=$(az pipelines build definition list \
--organization "${ORGANIZATION}" \
--project "${PROJECT}" \
--query "\"[? contains(path, \'\\\\stacks\\\\${TYPE}\\\\instances\\\\${NAME}\') && name==\'apply\'].{id:id}\"" \
--output tsv)
you've used some bizarre quoting. You want that to be just the way you expect:
BUILD_ID=$(az pipelines build definition list \
--organization "${ORGANIZATION}" \
--project "${PROJECT}" \
--query "[? contains(path, '\\stacks\\${TYPE}\\instances\\${NAME}) && name=='apply'].{id:id}" \
--output tsv)

Azure CLI - check existence of frontend port with Bash

I'm using bash to check if an application gateway frontend port exists using the following code.
I'm new to bash and it's giving me some syntax errors. What is the create syntax?
line 79: =: command not found
line 80: [: ==: unary operator expected
echo "Creating frontend port if it doesnt exist"
$frontendportExists = $(az network application-gateway frontend-port list -g $resourceGroupName
--gateway-name $appGatewayName --query "[?name=='appGatewayFrontendPort'] | length(#)")
if [ $frontendportExists == 0 ]; then
az network application-gateway frontend-port create \
--gateway-name $appGatewayName \
--name appGatewayFrontendPort \
--port 443 \
--resource-group $resourceGroupName
fi
There are a number of errors in your posted Bash code.
$ is not used as a prefix when assigning a value to a variable and spaces are not allowed surrounding the assignment = operator.
Wrong:
$frontendportExists = $(az network...
Should be:
frontendportExists=$(az network...
== is incorrect syntax with single bracket conditionals.
Wrong:
if [ $frontendportExists == 0 ]; then
Replace with (Bash only):
if [[ $frontendportExists == 0 ]]; then
Replace with (Posix):
if [ "$frontendportExists" = "0" ]; then
To prevent word splitting it is generally a good idea to double quote variables. e.g.
az network application-gateway frontend-port create \
--gateway-name "$appGatewayName" \
--name appGatewayFrontendPort \
--port 443 \
--resource-group "$resourceGroupName"
Please check your scripts using ShellCheck before posting in future.

Using option flag with condition (shell script)

I have this command inside my shell script file:
docker exec dev-wordpress $phpunitPath \
--configuration $configurationPath \
--testsuit $testsuit \
--group $group \
--testdox
It is working if I set the 'testsuit' and the 'group' as command line options.
The 'testsuit' and 'group' options should be used only if those variables has value.
Same issue with 'testdox' is solved with 'if-else' but it is not a good way when I want to do the same with 3 different options.
How can I avoid the '--group' option if I don't have value in $group variable?
#!/bin/zsh
phpunit="/var/www/html/wp-content/plugins/irea/api/src/vendor/bin/phpunit"
configuration="/var/www/html/wp-content/plugins/irea/api/tests/phpunit.xml"
testdox=
filter=
testsuite=
group=
while [ "$1" != "" ]; do
case $1 in
--group ) shift
group="--group $1"
;;
--testsuite ) shift
testsuite="--testsuite $1"
;;
--filter ) shift
filter="--filter $1"
;;
--testdox ) testdox="--testdox"
;;
esac
shift
done
docker exec irea-wordpress $phpunit \
--configuration $configuration \
$testsuite \
$group \
$filter \
$testdox
You can use parameter expansion :
docker exec dev-wordpress "$phpunitPath" \
--configuration "$configurationPath" \
${testsuit:+--testsuit "$testsuit"} \
${group:+--group "$group"} \
--testdox
This script should work well for $testsuit and $group.
I didn't notice you may have problem for the other two variable.
I updated the script, maybe you can try again.
I need to build up the command as a string first then run it with eval.
eval "docker exec irea-wordpress $phpunit --configuration $configuration $testsuite $group $filter $testdox"

Conditional in Docker argument in Bash

In my bash file I've something like this
docker run -d \
--network=host \
--name my-service \
--log-driver="$LOGGING" \
if [[ "$LOGGING" == 'splunk' ]]; then
echo "--log-opt tag={{.ImageName}}/{{.Name}}/{{.ID}} \\";
echo "--log-opt env=NODE_ENV \\";
fi
But shellcheck complains by showing the following result. Any idea?
https://github.com/koalaman/shellcheck/wiki/SC1089
Build the argument list first (in an array), then call docker. This has the additional benefit of getting rid of the ugly line continuation characters.
docker_opts=(
-d
--network=host
--name my-service
--log-driver="$LOGGING"
--log-opt="$log_opt"
)
if [[ $LOGGING == splunk ]]; then
docker_opts+=(
--log-opt "tag={{.ImageName}}/{{.Name}}/{{.ID}} \\"
--log-opt "env=NODE_ENV \\"
)
fi
docker run "${docker_opts[#]}"
The main idea, though, is to keep the conditional code as small as possible and keep it separate from the unconditional code.
I suggest to use $(if ..; then ...; fi):
docker run -d \
--network=host \
--name my-service \
--log-driver="$LOGGING" \
$(if [[ "$LOGGING" == 'splunk' ]]; then
echo "--log-opt tag={{.ImageName}}/{{.Name}}/{{.ID}}"
echo "--log-opt env=NODE_ENV"
fi)

Resources