I know I can do something like
cat <(cat somefile)
But I want to build up a string of <().
So:
for file in *.file; do
mySubs="${mySubs} <(cat ${file})"
done
cat ${mySubs} #cat <(cat 1.file) <(cat 2.file) ... <(cat something.file)
Without having to use eval.
Use named pipes directly. Use mktemp to create temporary file names for each pipe so that you can remove them after you are done.
fifos=()
for f in file1 file2 file3; do
t=$(mktemp)
mkfifo "$t"
pipes+=("$t")
someCommand "$f" > "$t" &
done
someOtherCommand "${pipes[#]}"
rm "${pipes[#]}"
I'm assuming cat is a standin for a more complicated command. Here, I'm explicitly wrapping it to show that:
#!/usr/bin/env bash
someCommand() { echo "Starting file $1"; cat "$1"; echo "Ending file $1"; }
wrap_all() {
## STAGE 1: Assemble the actual command we want to run
local fd cmd_len retval
local -a cmd fds fd_args
cmd_len=$1; shift
while (( cmd_len > 0 )); do
cmd+=( "$1" )
cmd_len=$((cmd_len - 1))
shift
done
## STAGE 2: Open an instance of someCommand for each remaining argument
local fd; local -a fds
fds=( )
for arg; do
exec {fd}< <(someCommand "$arg")
fds+=( "$fd" )
fd_args+=( "/dev/fd/$fd" )
done
## STAGE 3: Actually run the command
"${cmd[#]}" "${fd_args[#]}"; retval=$?
## STAGE 4: Close all the file descriptors
for fd in "${fds[#]}"; do
exec {fd}>&-
done
return "$retval"
}
Invocation as:
echo "one" >one.txt; echo "two" >two.txt
wrap_all 1 cat one.txt two.txt
...which outputs:
Starting file one.txt
one
Ending file one.txt
Starting file two.txt
two
Ending file two.txt
Note that this requires bash 4.1 for automatic FD allocation support (letting us avoid the need for named pipes).
Related
Running another script in bash script while loop runs but the loop breaks!
N.B. The script I mentioned just loops over files in current directory and just run mpirun.
Here's my bash script:
#!/bin/bash
np="$1"
bin="$2"
ref="$3"
query="$4"
word_size="$5"
i=1;
input="$query"
while read line; do
echo $line
if [[ "${line:0:1}" == ">" ]] ; then
header="$line"
echo "$header" >> seq_"${i}".fasta
else
seq="$line"
echo "$seq" >> seq_"${i}".fasta
if ! (( i % 5)) ; then
./run.sh $np $bin $ref $word_size
^^^^^^^^
#for filename in *.fasta; do
# mpirun -np "${np}" "${bin}" -d "${ref}" -ql "${filename}" -k "${word_size}" -b > log
# rm $filename
#done
fi
((i++))
fi
done < $input
The problem is that your run.sh script is passing no parameters to mpirun. That script passes the variables ${np} ${bin} ${ref} ${filename} ${word_size} to mpirun, but those variables are local to your main script and are undefined in run.sh. You could export those variables in the main script so that they are available to all child processes, but a better solution would be to use positional parameters in run.sh:
for filename in *.fasta; do
mpirun -np "${1}" "${2}" -d "${3}" -ql "${4}" -k "${5}" -b > log
rm $filename
done
I don't know about mpirun, but if you have anything inside your loop that reads from stdin, the loop will break.
[Arch Linux v5.0.7 with GNU bash 5.0.3]
Some .bashrc aliases seem to conflict with a bash shell-scripts provided by pyenv and pyenv-virtualenvwrapper.I tracked down the problem running the script, using set -x and with all aliases enabled, and saw finally that the script exits gracefully with exit code is 0 only when aliases are disabled with unalias -a. So this has to do with aliases... but which one ?
To try to automate that, I wrote the shell-script below:
It un-aliases one alias at a time, reading iteratively from the complete list of aliases,
It tests the conflicting shell script test.sh against that leave-one-out alias configuration, and prints something in case an error is detected,
It undoes the previous un-aliasing,
It goes on to un-aliasing the next alias.
But the two built-ins alias and unalias do not fare well in the script cac.sh below:
#! /usr/bin/bash
[ -e aliases.txt ] && rm -f aliases.txt
alias | sed 's/alias //' | cut -d "=" -f1 > aliases.txt
printf "File aliases.txt created with %d lines.\n" \
"$(wc -l < <(\cat aliases.txt))"
IFS=" "
n=0
while read -r line || [ -n "$line" ]; do
n=$((n+1))
aliasedAs=$( alias "$line" | sed 's/alias //' )
printf "Line %2d: %s\n" "$n" "$aliasedAs"
unalias "$line"
[ -z $(eval "$*" 1> /dev/null) ] \ # check output to stderr only
&& printf "********** Look up: %s\n" "$line"
eval "${aliasedAs}"
done < <(tail aliases.txt) # use tail + proc substitution for testing only
Use the script like so: $ cac.sh test.sh [optional arguments to test.sh] Any test.sh will do. It just needs to return some non-empty string to stderr.
The first anomaly is that the file aliases.txt is empty as if the alias builtin was not accessible from within the script. If I start the script from its 3rd line, using an already populated aliases.txt file, the script fails at the second line within the while block, again as if alias could not be called from within the script. Any suggestions appreciated.
Note: The one liner below works in console:
$ n=0;while read -r line || [ -n "$line" ]; do n=$((n+1)); printf "alias %d : %s\n" "$n" "$(alias "$line" | sed 's/alias //')"; done < aliases.txt
I would generally advise against implementing this as an external script at all -- it makes much more sense as a function that can be evaluated directly in your interactive shell (which is, after all, where all the potentially-involved aliases are defined).
print_result() {
local prior_retval=$? label=$1
if (( prior_retval == 0 )); then
printf '%-30s - %s\n' "$label" WORKS >&2
else
printf '%-30s - %s\n' "$label" BROKEN >&2
fi
}
test_without_each_alias() {
[ "$#" = 1 ] || { echo "Usage: test_without_each_alias 'code here'" >&2; return 1; }
local alias
(eval "$1"); print_result "Unchanged aliases"
for alias in "${!BASH_ALIASES[#]}"; do
(unalias "$alias" && eval "$1"); print_result "Without $alias"
done
}
Consider the following:
rm_in_home_only() { [[ $1 = /home/* ]] || return 1; rm -- "$#"; }
alias rm=rm_in_home_only # alias actually causing our bug
alias red_herring=true # another alias that's harmless
test_without_each_alias 'touch /tmp/foobar; rm /tmp/foobar; [[ ! -e /tmp/foobar ]]'
...which emits something like:
Unchanged aliases - BROKEN
Without rm - WORKS
Without red_herring - BROKEN
Note that if the code you pass executes a function, you'll want to be sure that the function is defined inside the eval'd code; since aliases are parser behavior, they take place when functions are defined, not when functions are run.
#Kamil_Cuk, #Benjamin_W and #cdarke all pointed to the fact that a noninteractive shell (as that spawned from a bash script) does not have access to aliases.
#CharlesDuffy pointed to probable word splitting and glob expansion resulting in something that could be invalid test syntax in the original [ -z $(eval "$*" 1> /dev/null) ] block above, or worse yet in the possibility of $(eval "$*" 1> /dev/null) being parsed as a glob resulting in unpredictable script behavior. Block corrected to: [ -z "$(eval "$*" 1> /dev/null)" ].
Making the shell spawned by cac.sh interactive, with #! /usr/bin/bash -i. make the two built-ins alias and unalias returned non-null result when invoked, and BASH_ALIASES[#] became accessible from within the script.
#! /usr/bin/bash -i
[ -e aliases.txt ] && rm -f aliases.txt
alias | sed 's/alias //' | cut -d "=" -f1 > aliases.txt
printf "File aliases.txt created with %d lines.\n" \
"$(wc -l < <(\cat aliases.txt))"
IFS=" "
while read -r line || [ -n "$line" ]; do
aliasedAs=$( alias "$line" | sed 's/alias //' )
unalias "$line"
[ -z "$(eval "$*" 2>&1 1>/dev/null)" ] \ # check output to stderr only
&& printf "********** Look up: %s\n" "$line"
eval "${aliasedAs}"
done < aliases.txt
Warning: testing test.sh resorts to the eval built-in. Arbitrary code can be executed on your system if test.sh and optional arguments do not come from a trusted source.
title: bash parameter expansion within a scalar variable
I have a bash script which runs a diff between two files.
If there is a diff, I want it to print statement1 and statement2
They are long so i put them into variables, but the echo statement
will not expand the parameter.
Can this be done in bash?
#!/bin/bash
set -x
source="/home/casper"
target="data/scripts"
statement1="There is a change in ${i}, please check the file"
statement2="or cp /home/casper/${i} /data/scripts/$i"
for i in file1 file2l file3 file4 file5 ; do
sleep 1 ;
if diff $source/$i $target/$i 2>&1 > /dev/null ; then
echo " "
else
echo "$statement1 "
echo "$statement2 "
fi
done
exit 0
The script seems to work - it finds a diff when it needs to find one.
However this is what it prints out.
There is a change in , please check the file
or cp /home/casper/ data/scripts/
I want it to say
There is a change in file2, please check the file
or cp /home/casper/file2 /data/scripts/file2
The problem is that $i is expanded when you define statement1 and statement2, not when you expand them. Use a shell function to output the text.
notification () {
echo "There is a change in $1, please check the file"
echo "or cp /home/casper/$1 /data/scripts/$1"
}
source="/home/casper"
target="data/scripts"
for i in file1 file2l file3 file4 file5 ; do
sleep 1 ;
if diff "$source/$i" "$target/$i" 2>&1 > /dev/null ; then
echo " "
else
notification "$i"
fi
done
exit 0
This can be done using eval:
TEMPLATE_MSG="aaa \${VALUE} ccc"
...
VALUE="bbb"
eval echo "${TEMPLATE_MSG}"
But I don't recommend it, because eval is evil :-) Other option is using pattern substitution:
TEMPLATE_MSG="aaa #1# ccc"
...
VALUE="bbb"
echo "${TEMPLATE_MSG/#1#/${VALUE}}"
So you put some unique pattern in your message (e.g. #1#) and then, when you print the message, you replace it with the content of variable.
I have two bash script.
One script write in a fifo. The second one read from the fifo, but AFTER the first one end to write.
But something does not work. I do not understand where the problem is. Here the code.
The first script is (the writer):
#!/bin/bash
fifo_name="myfifo";
# Se non esiste, crea la fifo;
[ -p $fifo_name ] || mkfifo $fifo_name;
exec 3<> $fifo_name;
echo "foo" > $fifo_name;
echo "bar" > $fifo_name;
The second script is (the reader):
#!/bin/bash
fifo_name="myfifo";
while true
do
if read line <$fifo_name; then
# if [[ "$line" == 'ar' ]]; then
# break
#fi
echo $line
fi
done
Can anyone help me please?
Thank you
Replace the second script with:
#!/bin/bash
fifo_name="myfifo"
while true
do
if read line; then
echo $line
fi
done <"$fifo_name"
This opens the fifo only once and reads every line from it.
The problem with your setup is that you have fifo creation in the wrong script if you wish to control fifo access to time when the reader is actually running. In order to correct the problem you will need to do something like this:
reader: fifo_read.sh
#!/bin/bash
fifo_name="/tmp/myfifo" # fifo name
trap "rm -f $fifo_name" EXIT # set trap to rm fifo_name at exit
[ -p "$fifo_name" ] || mkfifo "$fifo_name" # if fifo not found, create
exec 3< $fifo_name # redirect fifo_name to fd 3
# (not required, but makes read clearer)
while :; do
if read -r -u 3 line; then # read line from fifo_name
if [ "$line" = 'quit' ]; then # if line is quit, quit
printf "%s: 'quit' command received\n" "$fifo_name"
break
fi
printf "%s: %s\n" "$fifo_name" "$line" # print line read
fi
done
exec 3<&- # reset fd 3 redirection
exit 0
writer: fifo_write.sh
#!/bin/bash
fifo_name="/tmp/myfifo"
# Se non esiste, exit :);
[ -p "$fifo_name" ] || {
printf "\n Error fifo '%s' not found.\n\n" "$fifo_name"
exit 1
}
[ -n "$1" ] &&
printf "%s\n" "$1" > "$fifo_name" ||
printf "pid: '%s' writing to fifo\n" "$$" > "$fifo_name"
exit 0
operation: (start reader in 1st terminal)
$ ./fifo_read.sh # you can background with & at end
(launch writer in second terminal)
$ ./fifo_write.sh "message from writer" # second terminal
$ ./fifo_write.sh
$ ./fifo_write.sh quit
output in 1st terminal:
$ ./fifo_read.sh
/tmp/myfifo: message from writer
/tmp/myfifo: pid: '28698' writing to fifo
/tmp/myfifo: 'quit' command received
The following script should do the job:
#!/bin/bash
FIFO="/tmp/fifo"
if [ ! -e "$FIFO" ]; then
mkfifo "$FIFO"
fi
for script in "$#"; do
echo $script > $FIFO &
done
while read script; do
/bin/bash -c $script
done < $FIFO
Given two script a.sh and b.sh where both scripts pass "a" and "b" to stdout, respectively, one will get the following result (given that the script above is called test.sh):
./test.sh /tmp/a.sh /tmp/b.sh
a
b
Best,
Julian
I have a script that prints in a loop. I want the loop to print differently the first time from all other times (i.e., it should print differently if anything has been printed at all). I am thinking a simple way would be to check whether anything has been printed yet (i.e., stdout has been written to). Is there any way to determine that?
I know I could also write to a variable and test whether it's empty, but I'd like to avoid a variable if I can.
I think that will do what you need. If you echo something between # THE SCRIPT ITSELF and # END, THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT will be printed STDOUT HAS NOT BEEN TOUCHED else...
#!/bin/bash
readonly TMP=$(mktemp /tmp/test_XXXXXX)
exec 3<> "$TMP" # open tmp file as fd 3
exec 4>&1 # save current value of stdout as fd 4
exec >&3 # redirect stdout to fd 3 (tmp file)
# THE SCRIPT ITSELF
echo Hello World
# END
exec >&4 # restore save stdout
exec 3>&- # close tmp file
TMP_SIZE=$(stat -f %z "$TMP")
if [ $TMP_SIZE -gt 0 ]; then
echo "THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT"
echo
cat "$TMP"
else
echo "STDOUT HAS NOT BEEN TOUCHED"
fi
rm "$TMP"
So, output of the script as is:
THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT
Hello World
and if you remove the echo Hello World line:
STDOUT HAS NOT BEEN TOUCHED
And if you really want to test that while running the script itself, you can do that, too :-)
#!/bin/bash
#FIRST ELSE
function echo_fl() {
TMP_SIZE=$(stat -f %z "$TMP")
if [ $TMP_SIZE -gt 0 ]; then
echo $2
else
echo $1
fi
}
TMP=$(mktemp /tmp/test_XXXXXX)
exec 3 "$TMP" # open tmp file as fd 3
exec 4>&1 # save current value of stdout as fd 4
exec >&3 # redirect stdout to fd 3 (tmp file)
# THE SCRIPT ITSELF
for f in fst snd trd; do
echo_fl "$(echo $f | tr a-z A-Z)" "$f"
done
# END
exec >&4 # restore save stdout
exec 3>&- # close tmp file
TMP_SIZE=$(stat -f %z "$TMP")
if [ $TMP_SIZE -gt 0 ]; then
echo "THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT"
echo
cat "$TMP"
else
echo "STDOUT HAS NOT BEEN TOUCHED"
fi
rm "$TMP"
output is:
THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT
FST
snd
trd
as you can see: Only the first line (FST) has caps on. That's what the echo_fl function does for you: If it's the first line of output, if echoes the first argument, if it's not it echoes the second argument :-)
It's hard to tell what you are trying to do here, but if your script is printing to stdout, you could simply pipe it to perl:
yourcommand | perl -pe 'if ($. == 1) { print "First line is: $_" }'
It all depends on what kind of changes you are attempting to do.
You cannot use the -f option with %z. The line TMP_SIZE=$(stat -f %z "$TMP") produces a long string that fails the test in if [ $TMP_SIZE -gt 0 ].