How to create files in the shell where variables are not replaced - bash

I need to create a file for subsequent nohup execution, the file contains some variables, I don't want the variables to be replaced when I generate through the file.
When I executed the code with the following code, all the variables were replaced in the generated script, which was not what I expected.
#!/bin/bash
gen_script() {
filepath=$1
if [ ! -f "$filepath" ]; then
cat >${filepath} <<EOF
#!/bin/bash
# Code generated by main.sh; DO NOT EDIT.
test(){
ip=$1
port=$2
restorefile=$3
redis-cli -h $ip -p $port --pipe < $restorefile
}
test "$#"
EOF
fi
}
main(){
gen_script exec.sh
nohup bash exec.sh $1 $2 > nohup.out 2>&1 &
}
main "$#"
How can I change my code please? I really appreciate any help with this.

To disable expansions in here document, quote the delimieter:
cat <<'EOF'
... $not_expanded
EOF
Instead, let bash serialize the function.
#!/bin/bash
work() {
ip=$1
port=$2
restorefile=$3
redis-cli -h $ip -p $port --pipe < $restorefile
}
gen_script() {
filepath=$1
if [ ! -f "$filepath" ]; then
cat >${filepath} <<EOF
#!/bin/bash
# Code generated by main.sh; DO NOT EDIT.
$(declare -f work)
work "\$#"
EOF
fi
}
main() {
gen_script exec.sh
nohup bash exec.sh "$1" "$2" > nohup.out 2>&1 &
}
main "$#"
Check your script with shellcheck. Do not define a function named test, there is already a super standard command test which is an alias for command [.

Related

Iterating array in declared function of bash shell script

I've been working through creating a script to move some files from a local machine to a remote server. As part of that process I have a function that can either be called directly or wrapped with 'declare -fp' and sent along to an ssh command. The code I have so far looks like this:
export REMOTE_HOST=myserver
export TMP=eyerep-files
doTest()
{
echo "Test moving files from $TMP with arg $1"
declare -A files=(["abc"]="123" ["xyz"]="789")
echo "Files: ${!files[#]}"
for key in "${!files[#]}"
do
echo "$key => ${files[$key]}"
done
}
moveTest()
{
echo "attempting move with wrapped function"
ssh -t "$REMOTE_HOST" "$(declare -fp doTest|envsubst); doTest ${1#Q}"
}
moveTest $2
If I run the script with something like
./myscript.sh test dev
I get the output
attempting move with wrapped function
Test moving files from eyerep-files with arg dev
Files: abc xyz
bash: line 7: => ${files[]}: bad substitution
It seems like the string expansion for the for loop is not working correctly. Is this expected behaviour? If so, is there an alternative way to loop through an array that would avoid this issue?
If you're confident that your remote account's default shell is bash, this might look like:
moveTest() {
ssh -t "$REMOTE_HOST" "$(declare -f doTest; declare -p $(compgen -e)); doTest ${1#Q}"
}
If you aren't, it might instead be:
moveTest() {
ssh -t "$REMOTE_HOST" 'exec bash -s' <<EOF
set -- ${##Q}
$(declare -f doTest; declare -p $(compgen -e))
doTest \"\$#\"
EOF
}
I managed to find an answer here: https://unix.stackexchange.com/questions/294378/replacing-only-specific-variables-with-envsubst/294400
Since I'm exporting the global variables, I can get a list of them using compgen and use that list with envsubst to specify which variables I want to replace. My finished function ended up looking like:
moveTest()
{
echo "attempting move with wrapped function"
ssh -t "$REMOTE_HOST" "$(declare -fp doTest|envsubst "$(compgen -e | awk '$0="${"$0"}"') '${1}'"); doTest ${1#Q}"
}

Bash script while loop breaks after running another script

Running another script in bash script while loop runs but the loop breaks!
N.B. The script I mentioned just loops over files in current directory and just run mpirun.
Here's my bash script:
#!/bin/bash
np="$1"
bin="$2"
ref="$3"
query="$4"
word_size="$5"
i=1;
input="$query"
while read line; do
echo $line
if [[ "${line:0:1}" == ">" ]] ; then
header="$line"
echo "$header" >> seq_"${i}".fasta
else
seq="$line"
echo "$seq" >> seq_"${i}".fasta
if ! (( i % 5)) ; then
./run.sh $np $bin $ref $word_size
^^^^^^^^
#for filename in *.fasta; do
# mpirun -np "${np}" "${bin}" -d "${ref}" -ql "${filename}" -k "${word_size}" -b > log
# rm $filename
#done
fi
((i++))
fi
done < $input
The problem is that your run.sh script is passing no parameters to mpirun. That script passes the variables ${np} ${bin} ${ref} ${filename} ${word_size} to mpirun, but those variables are local to your main script and are undefined in run.sh. You could export those variables in the main script so that they are available to all child processes, but a better solution would be to use positional parameters in run.sh:
for filename in *.fasta; do
mpirun -np "${1}" "${2}" -d "${3}" -ql "${4}" -k "${5}" -b > log
rm $filename
done
I don't know about mpirun, but if you have anything inside your loop that reads from stdin, the loop will break.

bash: script to identify specific alias causing a bug

[Arch Linux v5.0.7 with GNU bash 5.0.3]
Some .bashrc aliases seem to conflict with a bash shell-scripts provided by pyenv and pyenv-virtualenvwrapper.I tracked down the problem running the script, using set -x and with all aliases enabled, and saw finally that the script exits gracefully with exit code is 0 only when aliases are disabled with unalias -a. So this has to do with aliases... but which one ?
To try to automate that, I wrote the shell-script below:
It un-aliases one alias at a time, reading iteratively from the complete list of aliases,
It tests the conflicting shell script test.sh against that leave-one-out alias configuration, and prints something in case an error is detected,
It undoes the previous un-aliasing,
It goes on to un-aliasing the next alias.
But the two built-ins alias and unalias do not fare well in the script cac.sh below:
#! /usr/bin/bash
[ -e aliases.txt ] && rm -f aliases.txt
alias | sed 's/alias //' | cut -d "=" -f1 > aliases.txt
printf "File aliases.txt created with %d lines.\n" \
"$(wc -l < <(\cat aliases.txt))"
IFS=" "
n=0
while read -r line || [ -n "$line" ]; do
n=$((n+1))
aliasedAs=$( alias "$line" | sed 's/alias //' )
printf "Line %2d: %s\n" "$n" "$aliasedAs"
unalias "$line"
[ -z $(eval "$*" 1> /dev/null) ] \ # check output to stderr only
&& printf "********** Look up: %s\n" "$line"
eval "${aliasedAs}"
done < <(tail aliases.txt) # use tail + proc substitution for testing only
Use the script like so: $ cac.sh test.sh [optional arguments to test.sh] Any test.sh will do. It just needs to return some non-empty string to stderr.
The first anomaly is that the file aliases.txt is empty as if the alias builtin was not accessible from within the script. If I start the script from its 3rd line, using an already populated aliases.txt file, the script fails at the second line within the while block, again as if alias could not be called from within the script. Any suggestions appreciated.
Note: The one liner below works in console:
$ n=0;while read -r line || [ -n "$line" ]; do n=$((n+1)); printf "alias %d : %s\n" "$n" "$(alias "$line" | sed 's/alias //')"; done < aliases.txt
I would generally advise against implementing this as an external script at all -- it makes much more sense as a function that can be evaluated directly in your interactive shell (which is, after all, where all the potentially-involved aliases are defined).
print_result() {
local prior_retval=$? label=$1
if (( prior_retval == 0 )); then
printf '%-30s - %s\n' "$label" WORKS >&2
else
printf '%-30s - %s\n' "$label" BROKEN >&2
fi
}
test_without_each_alias() {
[ "$#" = 1 ] || { echo "Usage: test_without_each_alias 'code here'" >&2; return 1; }
local alias
(eval "$1"); print_result "Unchanged aliases"
for alias in "${!BASH_ALIASES[#]}"; do
(unalias "$alias" && eval "$1"); print_result "Without $alias"
done
}
Consider the following:
rm_in_home_only() { [[ $1 = /home/* ]] || return 1; rm -- "$#"; }
alias rm=rm_in_home_only # alias actually causing our bug
alias red_herring=true # another alias that's harmless
test_without_each_alias 'touch /tmp/foobar; rm /tmp/foobar; [[ ! -e /tmp/foobar ]]'
...which emits something like:
Unchanged aliases - BROKEN
Without rm - WORKS
Without red_herring - BROKEN
Note that if the code you pass executes a function, you'll want to be sure that the function is defined inside the eval'd code; since aliases are parser behavior, they take place when functions are defined, not when functions are run.
#Kamil_Cuk, #Benjamin_W and #cdarke all pointed to the fact that a noninteractive shell (as that spawned from a bash script) does not have access to aliases.
#CharlesDuffy pointed to probable word splitting and glob expansion resulting in something that could be invalid test syntax in the original [ -z $(eval "$*" 1> /dev/null) ] block above, or worse yet in the possibility of $(eval "$*" 1> /dev/null) being parsed as a glob resulting in unpredictable script behavior. Block corrected to: [ -z "$(eval "$*" 1> /dev/null)" ].
Making the shell spawned by cac.sh interactive, with #! /usr/bin/bash -i. make the two built-ins alias and unalias returned non-null result when invoked, and BASH_ALIASES[#] became accessible from within the script.
#! /usr/bin/bash -i
[ -e aliases.txt ] && rm -f aliases.txt
alias | sed 's/alias //' | cut -d "=" -f1 > aliases.txt
printf "File aliases.txt created with %d lines.\n" \
"$(wc -l < <(\cat aliases.txt))"
IFS=" "
while read -r line || [ -n "$line" ]; do
aliasedAs=$( alias "$line" | sed 's/alias //' )
unalias "$line"
[ -z "$(eval "$*" 2>&1 1>/dev/null)" ] \ # check output to stderr only
&& printf "********** Look up: %s\n" "$line"
eval "${aliasedAs}"
done < aliases.txt
Warning: testing test.sh resorts to the eval built-in. Arbitrary code can be executed on your system if test.sh and optional arguments do not come from a trusted source.

Avoid $var to be interpreted as a variable inside inverted commas

I have tried in many ways but couldn't get it the right way.
fun="
mkcdo ()
{
mkdir -p -- \"'$1'\" && cd -P -- \"'$1'\"
}"
echo "$fun" >> ~/.bashrc
What I want is to append this in .bashrc
mkcd ()
{
mkdir -p -- "$1" && cd -P -- "$1"
}
Could that be done? Is there a way in bash like there is in python: r'whatever\you\$write' so that it is completely ignored as simple text?
Using a variable to store a bash function code sounds much of anti-pattern. For multi-line formatted strings, I would recommend using here-doc and quote them to avoid expanding the variable,
cat >> ~/.bashrc << 'EOF'
mkcd () {
mkdir -p -- "$1" && cd -P -- "$1"
}
EOF
Further reading - Bash - Here documents
Alternative1: just use single quotes.
fun='
mkcdo ()
{
mkdir -p -- "$1" && cd -P -- "$1"
}'
echo "$fun" >> ~/.bashrc
Alternative2: escape the $ sign.
fun="
mkcdo ()
{
mkdir -p -- \"\$1\" && cd -P -- \"\$1\"
}"
echo "$fun" >> ~/.bashrc

How to avoid output from called script within wrapper

I have written a wrapper in bash, which calls other shell scripts. However, I need to print only the output from the wrapper, avoiding the output from called scripts, which I am basically logging into a log file.
Elaborating…..
Basically I am using a function as
start_logging ${LOGFILE}
{
Funtion1
Funtion2
} 2>&1 | tee -a ${LOGFILE}
Where start logging is define as:- (I could only understand this function partially)
start_logging()
{
## usage: start_logging
## start a new log or append to existing log file
declare -i rc=0
if [ ! "${LOGFILE}" ];then
## display error and bail
fi
local TIME_STAMP=$(date +%Y%m%d:%H:%M:%S)
## open ${LOGFILE} or append to existing ${LOGFILE} with timestamp and actual command line
if [ ${DRY_RUN} ]; then
echo "DRY_RUN set..."
echo "${TIME_STAMP} Starting $(basename ${0}) run: '${0} ${ORIG_ARGS}'" { I}
echo "DRY_RUN set..."
echo "Please ignore \"No such file or directory\" from tee..."
else
echo "${TIME_STAMP} Starting $(basename ${0}) run: '${0} ${ORIG_ARGS}'"
echo "${TIME_STAMP} Starting $(basename ${0}) run: '${0} ${ORIG_ARGS}'"
fi
return ${rc}
}
LOGFILE is defined in the wrapper as
{
TMPDIR ="$/tmp"
LOGFILE="${TMPDIR}/${$}/${BASENAME%.*}.log
}
Now when its calling funtion1, funtion2 which basically calls other bash scripts its logging all the output in the file .i.e. { TMPDIR}/${$}/${BASENAME%.*}.log } as well as on the bash terminal.
I wanted that it should only echo what I have written in the wrapper on to the bash terminal and rest should be recorded in the log.
PleaseNote:- the called scripts from wrapper have echo function within but I don’t wanted that there output should be displayed on the terminal
Is it possible to achieve....
You need to redirect both stdout + stderr of your called scripts into your logfile.
./your_other_script.sh 2&>1 >> /var/log/mylogfile.txt
You get output to the terminal, because of tee. So you can:
start_logging ${LOGFILE}
{
Funtion1
Funtion2
} 2>&1 | tee -a ${LOGFILE} >/dev/null
^^^^^^^^^^ - redirect the output from a tee to /dev/null
or simply remove the tee and all logs redirect only into the file
start_logging ${LOGFILE}
{
Funtion1
Funtion2
} 2>&1 >${LOGFILE}
or for bigger parts of script, enclose the part into ( ) pair, will executed in a subshell and redirect the output to /dev/null, so:
(
start_logging ${LOGFILE}
{
Funtion1
Funtion2
} 2>&1 | tee -a ${LOGFILE}
) >/dev/null

Resources