This question already has answers here:
Usage of :- (colon dash) in bash
(2 answers)
Closed 5 years ago.
What means these declarations in bash script:
PREFIX=${1:-daily}
PROFILE=${2:-backup}
The key here is to understand what Parameter-Expansion ${PARAMETER:-WORD} syntax means here,
${PARAMETER:-WORD}
If the parameter PARAMETER is unset (never was defined) or null (empty), this one
expands to WORD, otherwise it expands to the value of PARAMETER, as if it
just was ${PARAMETER}
In your case the ${PARAMETER} being the positional arguments passed to a function or script,
Use the below script for a better understanding,
myTestFunction() {
PREFIX=${1:-daily}
PROFILE=${2:-backup}
printf "PREFIX value %s PROFILE value %s\n" "$PREFIX" "$PROFILE"
}
myTestFunction "some" "junk"
myTestFunction
which produces a result as
$ bash script.sh
PREFIX value some PROFILE value junk
PREFIX value daily PROFILE value backup
Also see the expanded debugger version of the script as
$ bash -x script.sh
+ myTestFunction some junk
+ PREFIX=some
+ PROFILE=junk
+ printf 'PREFIX value %s PROFILE value %s\n' some junk
PREFIX value some PROFILE value junk
+ myTestFunction
+ PREFIX=daily
+ PROFILE=backup
+ printf 'PREFIX value %s PROFILE value %s\n' daily backup
PREFIX value daily PROFILE value backup
how shell substitutes the values to the variables when $1 or $2 is not passed.
The syntax is generally used when by default you want to configure a variable with a certain value at the same time make it dynamically configurable also.
Assigns value of first argument to PREFIX variable if this first argument exist, "daily" if it does not.
It means, assign the first argument (if present), else daily to variable PREFIX and assign the second argument (if present), else backup to variable PROFILE
eg:
$ cat file.sh
#!/bin/bash
PREFIX=${1:-daily}
PROFILE=${2:-backup}
echo $PREFIX
echo $PROFILE
For the following command line args given, output would be like:
$ ./file.sh
daily
backup
$ ./file.sh abc
abc
backup
$ ./file.sh abc xyz
abc
xyz
Related
this is probably a very simple question. I looked at other answers but couldn't come up with a solution. I have a 365 line date file. file as below,
01-01-2000
02-01-2000
I need to read this file line by line and assign each day to a separate variable. like this,
d001=01-01-2000
d002=02-01-2000
I tried while read commands but couldn't get them to work.It takes a lot of time to shoot one by one. How can I do it quickly?
Trying to create named variable out of an associative array, is time waste and not supported de-facto. Better use this, using an associative array:
#!/bin/bash
declare -A array
while read -r line; do
printf -v key 'd%03d' $((++c))
array[$key]=$line
done < file
Output
for i in "${!array[#]}"; do echo "key=$i value=${array[$i]}"; done
key=d001 value=01-01-2000
key=d002 value=02-01-2000
Assumptions:
an array is acceptable
array index should start with 1
Sample input:
$ cat sample.dat
01-01-2000
02-01-2000
03-01-2000
04-01-2000
05-01-2000
One bash/mapfile option:
unset d # make sure variable is not currently in use
mapfile -t -O1 d < sample.dat # load each line from file into separate array location
This generates:
$ typeset -p d
declare -a d=([1]="01-01-2000" [2]="02-01-2000" [3]="03-01-2000" [4]="04-01-2000" [5]="05-01-2000")
$ for i in "${!d[#]}"; do echo "d[$i] = ${d[i]}"; done
d[1] = 01-01-2000
d[2] = 02-01-2000
d[3] = 03-01-2000
d[4] = 04-01-2000
d[5] = 05-01-2000
In OP's code, references to $d001 now become ${d[1]}.
A quick one-liner would be:
eval $(awk 'BEGIN{cnt=0}{printf "d%3.3d=\"%s\"\n",cnt,$0; cnt++}' your_file)
eval makes the shell variables known inside your script or shell. Use echo $d000 to show the first one of the newly defined variables. There should be no shell special characters (like * and $) inside your_file. Remove eval $() to see the result of the awk command. The \" quoted %s is to allow spaces in the variable values. If you don't have any spaces in your_file you can remove the \" before and after %s.
This question already has answers here:
Dynamic variable names in Bash
(19 answers)
Lookup shell variables by name, indirectly [duplicate]
(5 answers)
Closed 12 months ago.
I have a CSV that says this:
Source,Target,Database,Table,Is_deletable,Action
DBServer1,DBServer2,DB,TBL1,true,Add
DBServer2,DBServer1,DB,TBL2,true,Add
I have shell script that does this:
while IFS=, read -r source target database table is_deletable action; do
echo "Building pipeline for ${database}.${table}";
total=$((total+1))
Shost="server1.myorganization.com"
Thost="server2.myorganization.com"
I need Shost to look like this:
Shost = ${'the value of the source column'}
Thost = ${'the value of the target column'}
How do I set this to evaluate dynamically the variable I need based on the value of the column data.
For example:
Shost=${DBServer1}
Thost=${DBServer2}
then on the next loop:
Shost=${DBServer2}
Thost=${DBServer1}
Thanks!
Something like this should work for you:
DBServer1='server1.myorganization.com'
DBServer2='server2.myorganization.com'
while IFS=, read -r source target database table is_deletable action; do
[[ $source = "Source" ]] && continue
((total++))
Shost="${!source}"
Thost="${!target}"
# check variables Shost Thost total
declare -p Shost Thost total
done < file.csv
declare -- Shost="server1.myorganization.com"
declare -- Thost="server2.myorganization.com"
declare -- total="1"
declare -- Shost="server2.myorganization.com"
declare -- Thost="server1.myorganization.com"
declare -- total="2"
We can also use an associative array for this problem:
declare -A servers=(
[DBServer1]='server1.myorganization.com'
[DBServer2]='server2.myorganization.com'
)
{
read header
while IFS=, read -r source target database table is_deletable action; do
Shost=${servers[$source]}
Thost=${servers[$target]}
...
done
} < file.csv
In Bash -- but not necessarily in other shells -- you can reference a variable indirectly, exactly as you seem to want to do:
$ DBServer1=db1.my.com
$ source=DBServer1
$ echo ${source}
DBServer1
$ echo ${!source}
db1.my.com
As the Bash manual puts it (ref):
If [in a parameter expansion of the
form ${parameter}] the first character of parameter is an exclamation point (!), and parameter is
not a nameref, it introduces a level of variable indirection. Bash
uses the value of the variable formed from the rest of parameter as
the name of the variable; this variable is then expanded and that
value is used in the rest of the substitution, rather than the
value of parameter itself.
Applying that to your sample code and data, we get
DBServer1=server1.myorganization.com
DBServer2=server2.myorganization.com
# ...
while IFS=, read -r source target database table is_deletable action; do
echo "Building pipeline for ${database}.${table}";
total=$((total+1))
Shost="${!source}"
Thost="${!target}"
Basically what foo(**bar) does in python, here I’d want something like
foo **bar.yaml
and that would become
foo --bar1=1 --bar2=2
Where bar.yaml would be
bar1: 1
bar2: 2
You could use a combination of sed and xargs:
sed -E 's/^(.+):[[:space:]]+(.+)$/--\1=\2/' bar.yaml | xargs -d '\n' foo
sed converts the format of bar.yaml lines (e.g. bar1: 1 -> --bar1=1) and xargs feeds the converted lines as arguments to foo.
You could of course modify/extend the sed part to support other formats or single-dash options like -v.
To test if this does what you want, you can run this Bash script instead of foo:
#!/usr/bin/env bash
echo "Arguments: $#"
for ((i=1; i <= $#; i++)); do
echo "Argument $i: '${!i}'"
done
Here's a version for zsh. Run this code or add it to ~/.zshrc:
function _yamlExpand {
setopt local_options extended_glob
# 'words' array contains the current command line
# yaml filename is the last value
yamlFile=${words[-1]}
# parse 'key : value' lines from file, create associative array
typeset -A parms=("${(#s.:.)${(f)"$(<${yamlFile})"}}")
# trim leading and trailing whitespace from keys and values
# requires extended_glob
parms=("${(kv#)${(kv#)parms##[[:space:]]##}%%[[:space:]]##}")
# add -- and = to create flags
typeset -a flags
for key val in "${(#kv)parms}"; do
flags+=("--${key}='${val}'")
done
# replace the value on the command line
compadd -QU -- "$flags"
}
# add the function as a completion and map it to ctrl-y
compdef -k _yamlExpand expand-or-complete '^Y'
At the zsh shell prompt, type in the command and the yaml file name:
% print -l -- ./bar.yaml▃
With the cursor immediately after the yaml file name, hit ctrl+y. The yaml filename will be replaced with the expanded parameters:
% print -l -- --bar1='1' --bar2='2' ▃
Now you're set; you can hit enter, or add parameters, just like any other command line.
Notes:
This only supports the yaml subset in your example.
You can add more yaml parsing to the function, possibly with yq.
In this version, the cursor must be next to the yaml filename - otherwise the last value in words will be empty. You can add code to detect that case and then alter the words array with compset -n.
compadd and compset are described in the zshcompwid man page.
zshcompsys has details on compdef; the section on autoloaded files describes another way to deploy something like this.
Below is the sample scenario :-
There is a sample function defined that is echoing some text, as well as setting some exit code using return. There is another script that is calling this function. Below is the simplified code :-
~/playground/octagon/bucket/test/sample $
pwd
/Users/mogli/playground/octagon/bucket/test/sample
~/playground/octagon/bucket/test/sample $
cat functions.sh
myfunc() {
echo "This is output $1"
return 3
}
~/playground/octagon/bucket/test/sample $
cat example.sh
. functions.sh
function example(){
myfunc_out=$(myfunc $1); myfunc_rc=$?
echo "myfunc_out is: $myfunc_out"
echo "myfunc_rc is: $myfunc_rc"
}
example $1
~/playground/octagon/bucket/test/sample $
sh example.sh 44
myfunc_out is: This is output 44
myfunc_rc is: 3
Now, if I use local for variables that are used to store function return value and exit code in example.sh, then exit code is not coming correctly. Please find below modified example.sh :-
~/playground/octagon/bucket/test/sample $
cat example.sh
. functions.sh
function example(){
local myfunc_out=$(myfunc $1); local myfunc_rc=$?
echo "myfunc_out is: $myfunc_out"
echo "myfunc_rc is: $myfunc_rc"
}
example $1
~/playground/octagon/bucket/test/sample $
sh example.sh 44
myfunc_out is: This is output 44
myfunc_rc is: 0
When you write:
var=$(cmd)
the output of cmd is assigned to var (without word splitting) and $? is set to the value returned by cmd.
When you write:
var=$(cmd1) cmd2
cmd1 is executed and its output (without word splitting) is assigned to the variable var in the environment of cmd2, which is then executed. $? is set to the value returned by cmd2.
When you write:
cmd1 var=$(cmd2)
cmd2 is executed, the string var=output of cmd2 is subject to word splitting and passed as argument(s) to cmd1, and $? is set to the value returned by cmd1. (In almost all cases you want to supress the word splitting and instead write cmd1 var="$(cmd2)", which will guarantee that only one argument is passed.)
local is a command, and local myfunc_out=$(myfunc $1) is of the 3rd form (with a caveat), so it sets $? to the value returned by local. Note that if the output of myfunc $1 contains whitespace, word splitting does not take place. To quote the manpage: Assignment statements may also appear as arguments to the alias, declare, typeset, export, readonly, and local builtin commands, so the string counts as a variable assignment and word splitting is not done.
In short, local has an exit value, and it is being used to set $?
Instead of making the assignment an argument to local, you could use:
local myfunc_out myfunc_rc
myfunc_out="$(myfunc $1)"; myfunc_rc=$?
Note that the double quotes are not strictly necessary here, as word splitting does not take place in an assignment, but using them is definitely good practice.
I have a function that needs to change IFS for its logic:
my_func() {
oldIFS=$IFS; IFS=.; var="$1"; arr=($var); IFS=$oldIFS
# more logic here
}
Can I declare IFS as local IFS inside the function so that I don't need to worry about backing its current value and restore later?
It appears to work as you desire.
#!/bin/bash
changeIFSlocal() {
local IFS=.
echo "During local: |$IFS|"
}
changeIFSglobal() {
IFS=.
echo "During global: |$IFS|"
}
echo "Before: |$IFS|"
changeIFSlocal
echo "After local: |$IFS|"
changeIFSglobal
echo "After global: |$IFS|"
This prints:
Before: |
|
During local: |.|
After local: |
|
During global: |.|
After global: |.|
Yes it can be defined!
As long as you define it local, setting of the value in the function does not affect the global IFS value. See the difference between the snippets below
addNumbers () {
local IFS='+'
printf "%s\n" "$(( $* ))"
}
when called in command-line as,
addNumbers 1 2 3 4 5 100
115
and doing
nos=(1 2 3 4 5 100)
echo "${nos[*]}"
from the command line. The hexdump on the above echo output wouldn't show the IFS value defined in the function
echo "${nos[*]}" | hexdump -c
0000000 1 2 3 4 5 1 0 0 \n
000000e
See in one of my answers, how I've used the localized IFS to do arithmetic - How can I add numbers in a bash script
I got confused because I changed the value of IFS to : inside the function (without using local) and then tried to display the value of IFS with this command, after calling the function:
echo $IFS
which displayed an empty line that made me feel the function wasn't changing IFS. After posting the question, I realized that word splitting was at play and I should have used
echo "$IFS"
or
printf '%s\n' "$IFS"
or, even better
set | grep -w IFS=
to accurately display the IFS value.
Coming back to the main topic of local variables, yes, any variable can be declared as local inside a function to limit the scope, except for variables that have been declared readonly (with readonly or declare -r builtin commands). This includes Bash internal variables like BASH_VERSINFO etc.
From help local:
local: local [option] name[=value] ...
Define local variables.
Create a local variable called NAME, and give it VALUE. OPTION can
be any option accepted by `declare'.
Local variables can only be used within a function; they are visible
only to the function where they are defined and its children.
Exit Status:
Returns success unless an invalid option is supplied, a variable
assignment error occurs, or the shell is not executing a function.
You can designate IFS as a local variable; the local version is still used as the field separator string.
Sometimes it is useful to run a function in a completely isolated environment, where no changes are permanent. (For example, if the function needs to change shell options.) This can be achieved by making the function run in a subshell; just change the {} in the function definition to ():
f() (
shopt -s nullglob
IFS=.
# Commands to run in local environment
)