Indirect reference to hashmap in bash - bash

I am writing a script to install packages from .deb files, but first, I would like to check if each package is already installed. I have a config file that contains the information for the packages as hashmaps, like this:
declare -A package_a=(
[name]="utility-blah"
[ver]="1.2"
[arch]="amd64"
)
declare -A package_b=(
[name]="tool-bleh"
[ver]="3.4"
[arch]="all"
)
#and so on and so forth
My install script sources the config file, and I would like it to iterate over the packages, checking if they are installed, and installing them if they are not, like this:
source packages.config
declare -a packageList=("package_a" "package_b" "package_d")
for package in ${packageList[#]}; do
# Check if the specific version is installed already
if apt show ${package[name]}=${package[ver]}; then
echo ${package[name]} ${package[ver]} is already installed.
else
echo Installing ${package[name]}
sudo apt install path/to/files/${package[name]}_${package[ver]}_${package[arch]}.deb
fi
done
How can I have package point to the hashmap containing the information about the package and use it in the following commands?
I'm using Bash 4.4.20 on Ubuntu 18.04

One idea using a nameref:
source packages.config
declare -a packageList=("package_a" "package_b" "package_d")
for pkg in "${packageList[#]}"; do # change variable name
declare -n package="${pkg}" # declare nameref; rest of code remains the same ...
# Check if the specific version is installed already
if apt show ${package[name]}=${package[ver]}; then
echo ${package[name]} ${package[ver]} is already installed.
else
echo Installing ${package[name]}
sudo apt install path/to/files/${package[name]}_${package[ver]}_${package[arch]}.deb
fi
done
Or (as M. Nejat Aydin and Benjamin W. have pointed out) the declare -n can go before the while loop, eg:
declare -n package
for package in "${packageList[#]}"; do
# Check if the specific version is installed already
if apt show ${package[name]}=${package[ver]}; then
echo ${package[name]} ${package[ver]} is already installed.
else
echo Installing ${package[name]}
sudo apt install path/to/files/${package[name]}_${package[ver]}_${package[arch]}.deb
fi
done
Simple test:
declare -n package
for package in ${packageList[#]}; do
echo "${!package} : ${package[name]}"
done
This generates:
package_a : utility-blah
package_b : tool-bleh
package_d :

This kind of input data is better suited for JSON rather than using bash associative arrays and indirection.
Lets say you have a packages.json:
{
"packages": [
{
"package": "package_a",
"name": "utility-blah",
"ver": "1.2",
"arch": "amd64"
},
{
"package": "package_b",
"name": "utility-bleh",
"ver": "3.4",
"arch": "all"
},
{
"package": "apache2",
"name": "Apache2 http server",
"ver": "2.4.52-1ubuntu4.1",
"arch": "all"
}
]
}
Such simple POSIX-shell script is able to process it as you need:
#! /bin/sh
# Fields are tab-delimited, records end with newline
IFS=$(printf '\t')
# Parses json input into record lines
jq -r '.packages[]|(.package + "\t" + .name + "\t" + .ver)' packages.json |
# Iterates records, reading fields
while read -r package name ver; do
{
# Query package for installed status and version
# formatted into two fields
dpkg-query -W --showformat='${db:Status-Abbrev}\t${Version}' "${package}" || :
} 2>/dev/null | {
# Reads status and installed version
read -r status installed_ver _
# If status is installed 'ii ' and installed version matches'
if [ "${status}x" = 'ii x' ] && [ "${ver}x" = "${installed_ver}x" ]; then
printf '%s %s is already installed.\n' "${name}" "${ver}"
else
printf 'Installing %s.\n' "${name}"
fi
}
done
Example output:
nstalling utility-blah.
nstalling utility-bleh.
Apache2 http server 2.4.52-1ubuntu4.1 is already installed.

Related

How to parse a Terraform file (versions.tf) in Bash?

I'd like to parse a Terraform file named versions.tf which give provider requirements for the project. This should be in bash script.
This file is formatted like this :
terraform {
required_providers {
archive = {
source = "hashicorp/archive"
version = "~> 1.3.0"
}
aws = {
source = "hashicorp/aws"
version = "~> 3.0.0"
}
local = {
source = "hashicorp/local"
version = "~> 2.0.0"
}
template = {
source = "hashicorp/template"
version = "~> 2.2.0"
}
random = {
source = "hashicorp/random"
version = "~> 3.0.1"
}
}
required_version = ">= 0.13"
}
The goal is to have 3 variables likes so in a for loop :
$name = "archive"
$source = "hashicorp/archive"
$version = "1.3.0" # we don't take care of the version constraints
The structure is almost the same for different projects we have.
I've already tried to parse it, but as I'm a noob in text parsing, nothing concrete.
I also tried to use the command terraform providers schema -json but it doesn't work while you don't have initialized the terraform script (it need to be not initialized).
As I will use the script in my corporate Jenkins pipeline, I can't access Internet and I don't have access to binaries like jq or something not "red hat" native.
Thanks you for your help !
This script scans for the lines with the name, which are identifiable by the third field being {, and after each of these reads the two lines with source and version, assigning to variables with the same name:
while read name _ rest
do if [ "$rest" = "{" ]
then read var _ val; eval $var=$val
read var _ val; eval $var=$val
echo $name $source ${version#~> }
fi
done <versions.tf
Might do what you wanted.
#!/usr/bin/env bash
while read -r line; do
[[ $line != *'='* ]] && continue
if [[ $line == *'= {' ]]; then
line=${line%=*\{}
printf '$name = "%s"\n' "${line% }"
else
printf '$%s\n' "${line//~> }"
fi
done < versions.tf
If the required_version line needs to be omitted add
[[ $line == 'required_version'* ]] && continue
Below the line that has continue

AWS CodeBuild buildspec bash syntax error: bad substitution with if statement

Background:
I'm using an AWS CodeBuild buildspec.yml to iterate through directories from a GitHub repo. Before looping through the directory path $TF_ROOT_DIR, I'm using a bash if statement to check if the GitHub branch name $BRANCH_NAME is within an env variable $LIVE_BRANCHES. As you can see in the error screenshot below, the bash if statement outputs the error: syntax error: bad substitution. When I reproduce the if statement within a local bash script, the if statement works as it's supposed to.
Here's the env variables defined in the CodeBuild project:
Here's a relevant snippet from the buildspec.yml:
version: 0.2
env:
shell: bash
phases:
build:
commands:
- |
if [[ " ${LIVE_BRANCHES[*]} " == *"$BRANCH_NAME"* ]]; then
# Iterate only through BRANCH_NAME directory
TF_ROOT_DIR=${TF_ROOT_DIR}/*/${BRANCH_NAME}/
else
# Iterate through both dev and prod directories
TF_ROOT_DIR=${TF_ROOT_DIR}/*/
fi
- echo $TF_ROOT_DIR
Here's the build log that shows the syntax error:
Here's the AWS CodeBuild project JSON to reproduce the CodeBuild project:
{
"projects": [
{
"name": "terraform_validate_plan",
"arn": "arn:aws:codebuild:us-west-2:xxxxx:project/terraform_validate_plan",
"description": "Perform terraform plan and terraform validator",
"source": {
"type": "GITHUB",
"location": "https://github.com/marshall7m/sparkify_end_to_end.git",
"gitCloneDepth": 1,
"gitSubmodulesConfig": {
"fetchSubmodules": false
},
"buildspec": "deployment/CI/dev/cfg/buildspec_terraform_validate_plan.yml",
"reportBuildStatus": false,
"insecureSsl": false
},
"secondarySources": [],
"secondarySourceVersions": [],
"artifacts": {
"type": "NO_ARTIFACTS",
"overrideArtifactName": false
},
"cache": {
"type": "NO_CACHE"
},
"environment": {
"type": "LINUX_CONTAINER",
"image": "hashicorp/terraform:0.12.28",
"computeType": "BUILD_GENERAL1_SMALL",
"environmentVariables": [
{
"name": "TF_ROOT_DIR",
"value": "deployment",
"type": "PLAINTEXT"
},
{
"name": "LIVE_BRANCHES",
"value": "(dev, prod)",
"type": "PLAINTEXT"
}
Here's the associated buildspec file content: (buildspec_terraform_validate_plan.yml)
version: 0.2
env:
shell: bash
parameter-store:
AWS_ACCESS_KEY_ID_PARAM: TF_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY_PARAM: TF_AWS_SECRET_ACCESS_KEY_ID
phases:
install:
commands:
# install/incorporate terraform validator?
pre_build:
commands:
# CodeBuild environment variables
# BRANCH_NAME -- GitHub branch that triggered the CodeBuild project
# TF_ROOT_DIR -- Directory within branch ($BRANCH_NAME) that will be iterated through for terraform planning and testing
# LIVE_BRANCHES -- Branches that represent a live cloud environment
- export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_PARAM
- export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_PARAM
- bash -version || echo "${BASH_VERSION}" || bash --version
- |
if [[ -z "${BRANCH_NAME}" ]]; then
# extract branch from github webhook
BRANCH_NAME=$(echo $CODEBUILD_WEBHOOK_HEAD_REF | cut -d'/' -f 3)
fi
- "echo Triggered Branch: $BRANCH_NAME"
- |
if [[ " ${LIVE_BRANCHES[*]} " == *"$BRANCH_NAME"* ]]; then
# Iterate only through BRANCH_NAME directory
TF_ROOT_DIR=${TF_ROOT_DIR}/*/${BRANCH_NAME}/
else
# Iterate through both dev and prod directories
TF_ROOT_DIR=${TF_ROOT_DIR}/*/
fi
- "echo Terraform root directory: $TF_ROOT_DIR"
build:
commands:
- |
for dir in $TF_ROOT_DIR; do
#get list of non-hidden directories within $dir/
service_dir_list=$(find "${dir}" -type d | grep -v '/\.')
for sub_dir in $service_dir_list; do
#if $sub_dir contains .tf or .tfvars files
if (ls ${sub_dir}/*.tf) > /dev/null 2>&1 || (ls ${sub_dir}/*.tfvars) > /dev/null 2>&1; then
cd $sub_dir
echo ""
echo "*************** terraform init ******************"
echo "******* At directory: ${sub_dir} ********"
echo "*************************************************"
terraform init
echo ""
echo "*************** terraform plan ******************"
echo "******* At directory: ${sub_dir} ********"
echo "*************************************************"
terraform plan
cd - > /dev/null
fi
done
done
Given this is just a side project, all files that could be relevant to this problem are within a public repo here.
UPDATES
Tried adding #!/bin/bash shebang line but resulted in the CodeBuild error:
Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: #!/bin/bash
version: 0.2
env:
shell: bash
phases:
build:
commands:
- |
#!/bin/bash
if [[ " ${LIVE_BRANCHES[*]} " == *"$BRANCH_NAME"* ]]; then
# Iterate only through BRANCH_NAME directory
TF_ROOT_DIR=${TF_ROOT_DIR}/*/${BRANCH_NAME}/
else
# Iterate through both dev and prod directories
TF_ROOT_DIR=${TF_ROOT_DIR}/*/
fi
- echo $TF_ROOT_DIR
Solution
As mentioned by #Marcin, I used an AWS managed image within Codebuild (aws/codebuild/standard:4.0) and downloaded Terraform within the install phase.
phases:
install:
commands:
- wget https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip -q
- unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip && mv terraform /usr/local/bin/
I tried to reproduce your issue, but it all works fine for me.
The only thing I've noticed is that you are using $BRANCH_NAME but its not defined anywhere. But even with missing $BRANCH_NAME the buildspec.yml you've posted runs fine.
Update using hashicorp/terraform:0.12.28 image

Merge two json in bash (no jq)

I have two jsons :
env.json
{
"environment":"INT"
}
roles.json
{
"run_list":[
"recipe[splunk-dj]",
"recipe[tideway]",
"recipe[AlertsSearch::newrelic]",
"recipe[AlertsSearch]"
]
}
expected output should be some thing like this :
{
"environment":"INT",
"run_list":[
"recipe[splunk-dj]",
"recipe[tideway]",
"recipe[AlertsSearch::newrelic]",
"recipe[AlertsSearch]"
]
}
I need to merge these two json (and other like these two) into one single json using only available inbuilt bash commands.
only have sed, cat, echo, tail, wc at my disposal.
Tell whoever put the constraint "bash only" on the project that bash is not sufficient for processing JSON, and get jq.
$ jq --slurp 'add' env.json roles.json
I couldn't use jq either as I was limited due to client's webhost jailing the user on the command line with limited binaries as most discount/reseller web hosting companies do. Luckily they usually have PHP available and you can do a oneliner command like this which something like what I would place in my install/setup bash script for example.
php -r '$json1 = "./env.json";$json2 = "./roles.json";$data = array_merge(json_decode(file_get_contents($json1), true),json_decode(file_get_contents($json2),true));echo json_encode($data, JSON_PRETTY_PRINT);'
For clarity php -r accepts line feeds as well so using this also works.
php -r '
$json1 = "./env.json";
$json2 = "./roles.json";
$data = array_merge(json_decode(file_get_contents($json1), true), json_decode(file_get_contents($json2), true));
echo json_encode($data, JSON_PRETTY_PRINT);'
Output
{
"environment": "INT",
"run_list": [
"recipe[splunk-dj]",
"recipe[tideway]",
"recipe[AlertsSearch::newrelic]",
"recipe[AlertsSearch]"
]
}
A little bit hacky, but hopefully will do.
env_lines=`wc -l < $1`
env_output=`head -n $(($env_lines - 1)) $1`
roles_lines=`wc -l < $2`
roles_output=`tail -n $(($roles_lines - 1)) $2`
echo "$env_output" "," "$roles_output"

bash script , how to server architecture from ansible setup

I have a bash script to build a rub .deb package in which I would like to use the server architecture ...
I installed ansible and run ansible localhost -m setup , which gives me the ansible facts ...
> ansible localhost -m setup
localhost | success >> {
"ansible_facts": {
"ansible_all_ipv4_addresses": [
"10.10.5.200"
],
"ansible_all_ipv6_addresses": [
"fe80::4c2:ccff:fe82:8d8c"
],
"ansible_architecture": "x86_64",
"ansible_bios_date": "06/02/2014",
..
I would like to use detected architecture "ansible_architecture": "x86_64" in my shell script within my fpm command :
currently :
fpm -s dir -t deb -n ruby$version -v $rubyversion -C $destdir \
-p ruby-VERSION_ARCH~trusty.deb -d "libstdc++6 (>= 4.4.3)" \
-d "libc6 (>= 2.6)" -d "libffi6 (>= 3.0.10)" -d "libgdbm3 (>= 1.8.3)" \
-d "libncurses5 (>= 5.7)" -d "libreadline6 (>= 6.1)" \
-d "libssl1.0.0 (>= 1.0.1)" -d "zlib1g (>= 1:1.2.2)" \
-d "libyaml-0-2 (>= 0.1.4-2)" \
usr/local/bin usr/local/lib usr/local/share/man usr/local/include
# VERSION_ARCH is giving me 'amd64' and I would like to use
# ansible_architecture ( which is detecting "x86_64"....
I may not have to use ansible facts to detect it ...
# I tried to write at the beginning of my script
$architecture=uname -m
# but it does gives me "x86_64" as a variable...
thanks for your suggestions
got it :
inserting
architecture="$(uname -m)"
echo "architecture: $architecture"
at the beginning of my script and using $architecture as a parameter

Go-Sublime-build configuration

Im having issues trying to set up go to run the current file from Sublime text 2.
Here's what I have in my go.sublime-build file
{
"cmd": [ "go", "run", "${file}" ]
}
When I try to run build on a go source file, I get the error
[Error 6] The handle is invalid
[cmd: [u'go run', u'C:\\Users\\gprasant\\Documents\\GitHub\\programming_pearls\\src\\go\\quicksort.go']]
[dir: C:\Users\gprasant\Documents\GitHub\programming_pearls\src\go]
Is there any way to get this fixed ? Or is there another plugin in Sublime text for Go development?
Installing GoSublime should get this working for you. After installing and restarting ST2: do ctrl-B, type "run" and hit enter.
I got by with
{
"cmd": "go run $file",
"shell" : true
}
In ST3: it is changed to be:
{
"shell_cmd": "go run ${file}"
}
On my mac, I needed the following code in:
/Users/your_user_name/Library/Application Support/Sublime Text 2/Packages/User/go.sublime-build
go.sublime-build
{
"cmd": ["go run '${file}'"],
"selector": "source.go",
"path": "/usr/local/go/bin",
"shell": true
}
"cmd" line quoting is to correctly handle file paths with spaces.
"shell" line is needed since commenting it out breaks it.
"path" line is needed because the basic shell, doesn't have access to my .zshrc file include the export GOPATH statement defining the go path.
After that any .go file should build and run with command+B, leaving the stdout message in a console built into sublime text 2.
what about:
{
"cmd": ["go", "run", "${file}"],
"path": "/user/local/go/bin"
}
I like GoSublime, just hate to type run each time when click Command + B
SublimeText 2
build-system for golang, making F4/shift-F4 work (next error/prev error)
1st, create a file: ~/gosublime_build.sh
GOPATH=~/go
export GOPATH
echo "GOPATH:$GOPATH"
if [ "$3." = "RUN." ]
then
EXENAME=${1##*/}
EXENAME=$GOPATH/bin/$EXENAME
echo $EXENAME
$($EXENAME)
echo "code: $?"
exit
fi
echo "go build $2"
cd /usr/local/go/bin
./go build -o ~/temp.go.compiled $2
if [ $? -eq 0 ]
then
cd $1
echo "Project: " $1
/usr/local/go/bin/go install
echo "go install exit code: $?"
else
echo "go build exit code: $?"
fi
2nd:
chmod 777 ~/gosublime_build.sh
3rd: create a new sublime2 build-system for "go" (Tools/Build System/New)
{
"cmd": ["~/gosublime_build.sh $file_path $file"]
,"shell": true
,"selector": "source.go"
,"file_regex": "([\\w/_-]+[.]{1}[\\w./_-]+?):([0-9]+):?([0-9]+)?(.*)?"
}
4th: select your new build-system (Tools/Build System)
5th: build with Ctrl-B, F4/Shift-F4: next/prev error
If anybody knows how to instruct the go compiler to inform FULL PATH of file and line for each error, this process can be simplified

Resources