I have a bash script as shown below. I run this in a directory containing files such as input1.inp and other files like coords_i.xyz and submission.sub in order to make some simple modifications to them:
#!/bin/bash
sed -i -e '25d' *.inp
echo "*xyz -2 2" >> *.inp
sed -n '3,7p' *_i.xyz >> *.inp
echo "Q -1 0 0 3" >> *.inp
echo "Q +1 0 0 -3" >> *.inp
echo "*" >> *.inp
sed -i -e s/"replace1"/"replace2"/g *.sub
rm *.out
If I am in this directory, and I run all the commands individually in the terminal (line by line in the script), everything works fine. However, when I try to group all these commands into the script as shown above, I get an error - essentially after the line sed -i -e '25d' *.inp, the script stops and a file called *.inp is created in my directory. If I try to run the echo command separately after that, it says the command is ambiguous (presumably because of the existence of this *.inp file).
Why don't my wildcards work the same way in the script as they did when I ran them separately and sequentially in the terminal, and what can I do so that they work properly in the script?
Using wildcards this way is hazardous; the easy advice is "don't". Evaluate them only once, and then you can check their outputs before trying to use them.
In the below, we define an assert_only_one function that stops your script when an array -- assigned from a glob -- contains less or more than exactly one element. Consequently, we're able to write code that more clearly and explicitly describes our desired behavior.
#!/usr/bin/env bash
shopt -s nullglob # Stop *.xyz evaluating to '*.xyz' if no such files exist
assert_only_one() {
local glob; glob=$1; shift
case $# in
0) echo "ERROR: No files matching $glob exist" >&2; exit 1;;
1) return 0;;
*) echo "ERROR: More than one file matching $glob exists:" >*2
printf ' %q\n' "$#" >&2
exit 1;;
esac
}
inp_files=( *.inp ); assert_only_one '*.inp' "${inp_files[#]}"
sub_files=( *.sub ); assert_only_one '*.sub' "${sub_files[#]}"
xyz_files=( *_i.xyz )
sed -i -e '25d' "${inp_files[0]}"
{
echo "*xyz -2 2"
sed -n '3,7p' "${xyz_files[#]}"
echo "Q -1 0 0 3"
echo "Q +1 0 0 -3"
echo "*"
} >>"${inp_files[0]}"
sed -i -e s/"replace1"/"replace2"/g -- "${sub_files[#]}"
rm -- *.out
Related
I know I can do something like
cat <(cat somefile)
But I want to build up a string of <().
So:
for file in *.file; do
mySubs="${mySubs} <(cat ${file})"
done
cat ${mySubs} #cat <(cat 1.file) <(cat 2.file) ... <(cat something.file)
Without having to use eval.
Use named pipes directly. Use mktemp to create temporary file names for each pipe so that you can remove them after you are done.
fifos=()
for f in file1 file2 file3; do
t=$(mktemp)
mkfifo "$t"
pipes+=("$t")
someCommand "$f" > "$t" &
done
someOtherCommand "${pipes[#]}"
rm "${pipes[#]}"
I'm assuming cat is a standin for a more complicated command. Here, I'm explicitly wrapping it to show that:
#!/usr/bin/env bash
someCommand() { echo "Starting file $1"; cat "$1"; echo "Ending file $1"; }
wrap_all() {
## STAGE 1: Assemble the actual command we want to run
local fd cmd_len retval
local -a cmd fds fd_args
cmd_len=$1; shift
while (( cmd_len > 0 )); do
cmd+=( "$1" )
cmd_len=$((cmd_len - 1))
shift
done
## STAGE 2: Open an instance of someCommand for each remaining argument
local fd; local -a fds
fds=( )
for arg; do
exec {fd}< <(someCommand "$arg")
fds+=( "$fd" )
fd_args+=( "/dev/fd/$fd" )
done
## STAGE 3: Actually run the command
"${cmd[#]}" "${fd_args[#]}"; retval=$?
## STAGE 4: Close all the file descriptors
for fd in "${fds[#]}"; do
exec {fd}>&-
done
return "$retval"
}
Invocation as:
echo "one" >one.txt; echo "two" >two.txt
wrap_all 1 cat one.txt two.txt
...which outputs:
Starting file one.txt
one
Ending file one.txt
Starting file two.txt
two
Ending file two.txt
Note that this requires bash 4.1 for automatic FD allocation support (letting us avoid the need for named pipes).
title: bash parameter expansion within a scalar variable
I have a bash script which runs a diff between two files.
If there is a diff, I want it to print statement1 and statement2
They are long so i put them into variables, but the echo statement
will not expand the parameter.
Can this be done in bash?
#!/bin/bash
set -x
source="/home/casper"
target="data/scripts"
statement1="There is a change in ${i}, please check the file"
statement2="or cp /home/casper/${i} /data/scripts/$i"
for i in file1 file2l file3 file4 file5 ; do
sleep 1 ;
if diff $source/$i $target/$i 2>&1 > /dev/null ; then
echo " "
else
echo "$statement1 "
echo "$statement2 "
fi
done
exit 0
The script seems to work - it finds a diff when it needs to find one.
However this is what it prints out.
There is a change in , please check the file
or cp /home/casper/ data/scripts/
I want it to say
There is a change in file2, please check the file
or cp /home/casper/file2 /data/scripts/file2
The problem is that $i is expanded when you define statement1 and statement2, not when you expand them. Use a shell function to output the text.
notification () {
echo "There is a change in $1, please check the file"
echo "or cp /home/casper/$1 /data/scripts/$1"
}
source="/home/casper"
target="data/scripts"
for i in file1 file2l file3 file4 file5 ; do
sleep 1 ;
if diff "$source/$i" "$target/$i" 2>&1 > /dev/null ; then
echo " "
else
notification "$i"
fi
done
exit 0
This can be done using eval:
TEMPLATE_MSG="aaa \${VALUE} ccc"
...
VALUE="bbb"
eval echo "${TEMPLATE_MSG}"
But I don't recommend it, because eval is evil :-) Other option is using pattern substitution:
TEMPLATE_MSG="aaa #1# ccc"
...
VALUE="bbb"
echo "${TEMPLATE_MSG/#1#/${VALUE}}"
So you put some unique pattern in your message (e.g. #1#) and then, when you print the message, you replace it with the content of variable.
I am writing a script that will take in 3 outputs and then search all files within a predefined path. However, my grep command seems to be breaking the script with error code 123. I have been staring at it for a while and cannot really seem the error so I was hoping someone could point out my error. Here is the code:
#! /bin/bash -e
#Check if path exists
if [ -z $ARCHIVE ]; then
echo "ARCHIVE NOT SET, PLEASE SET TO PROCEED."
echo "EXITING...."
exit 1
elif [ $# -ne 3 ]; then
echo "Illegal number of arguments"
echo "Please enter the date in yyyy mm dd"
echo "EXITING..."
exit 1
fi
filename=output.txt
#Simple signal handler
signal_handler()
{
echo ""
echo "Process killed or interrupted"
echo "Cleaning up files..."
rm -f out
echo "Finsihed"
exit 1
}
trap 'signal_handler' KILL
trap 'signal_handler' TERM
trap 'signal_handler' INT
echo "line 32"
echo $1 $2 $3
#Search for the TimeStamp field and replace the / and : characters
find $ARCHIVE | xargs grep -l "TimeStamp: $2/$3/$1"
echo "line 35"
fileSize=`wc -c out.txt | cut -f 1 -d ' '`
echo $fileSize
if [ $fileSize -ge 1 ]; then
echo "no"
xargs -n1 basename < $filename
else
echo "NO FILES EXIST"
fi
I added the echo's to know where it was breaking. My program prints out line 32 and the args but never line 35. When I check the exit code I get 123.
Thanks!
Notes:
ARCHIVE is set to a test directory, i.e. /home/'uname'/testDir
$1 $2 $3 == yyyy mm dd (ie a date)
In testDir there are N number of directories. Inside these directories there are data files that have contain data as well as a time tag. The time tag is of the following format: TimeStamp: 02/02/2004 at 20:38:01
The scripts goal is to find all files that have the date tag you are searching for.
Here's a simpler test case that demonstrates your problem:
#!/bin/bash -e
echo "This prints"
true | xargs false
echo "This does not"
The snippet exits with code 123.
The problem is that xargs exits with code 123 if any command fails. When xargs exits with non-zero status, -e causes the script to exit.
The quickest fix is to use || true to effectively ignore xargs' status:
#!/bin/bash -e
echo "This prints"
true | xargs false || true
echo "This now prints too"
The better fix is to not rely on -e, since this option is misleading and unpredictable.
xargs makes the error code 123 when grep returns a nonzero code even just once. Since you're using -e (#!/bin/bash -e), bash would exit the script when one of its commands return a nonzero exit code. Not using -e would allow your code to continue. Just disabling it on that part can be a solution too:
set +e ## Disable
find "$ARCHIVE" | xargs grep -l "TimeStamp: $2/$1/$3" ## If one of the files doesn't match the pattern, `grep` would return a nonzero code.
set -e ## Enable again.
Consider placing your variables around quotes to prevent word splitting as well like "$ARCHIVE".
-d '\n' may also be required if one of your files' filename contain spaces.
find "$ARCHIVE" | xargs -d '\n' grep -l "TimeStamp: $2/$1/$3"
How do I check if file exists in bash?
When I try to do it like this:
FILE1="${#:$OPTIND:1}"
if [ ! -e "$FILE1" ]
then
echo "requested file doesn't exist" >&2
exit 1
elif
<more code follows>
I always get following output:
requested file doesn't exist
The program is used like this:
script.sh [-g] [-p] [-r FUNCTION_ID|-d FUNCTION_ID] FILE
Any ideas please?
I will be glad for any help.
P.S. I wish I could show the entire file without the risk of being fired from school for having a duplicate. If there is a private method of communication I will happily oblige.
My mistake. Fas forcing a binary file into a wrong place. Thanks for everyone's help.
Little trick to debugging problems like this. Add these lines to the top of your script:
export PS4="\$LINENO: "
set -xv
The set -xv will print out each line before it is executed, and then the line once the shell interpolates variables, etc. The $PS4 is the prompt used by set -xv. This will print the line number of the shell script as it executes. You'll be able to follow what is going on and where you may have problems.
Here's an example of a test script:
#! /bin/bash
export PS4="\$LINENO: "
set -xv
FILE1="${#:$OPTIND:1}" # Line 6
if [ ! -e "$FILE1" ] # Line 7
then
echo "requested file doesn't exist" >&2
exit 1
else
echo "Found File $FILE1" # Line 12
fi
And here's what I get when I run it:
$ ./test.sh .profile
FILE1="${#:$OPTIND:1}"
6: FILE1=.profile
if [ ! -e "$FILE1" ]
then
echo "requested file doesn't exist" >&2
exit 1
else
echo "Found File $FILE1"
fi
7: [ ! -e .profile ]
12: echo 'Found File .profile'
Found File .profile
Here, I can see that I set $FILE1 to .profile, and that my script understood that ${#:$OPTIND:1}. The best thing about this is that it works on all shells down to the original Bourne shell. That means if you aren't running Bash as you think you might be, you'll see where your script is failing, and maybe fix the issue.
I suspect you might not be running your script in Bash. Did you put #! /bin/bash on the top?
script.sh [-g] [-p] [-r FUNCTION_ID|-d FUNCTION_ID] FILE
You may want to use getopts to parse your parameters:
#! /bin/bash
USAGE=" Usage:
script.sh [-g] [-p] [-r FUNCTION_ID|-d FUNCTION_ID] FILE
"
while getopts gpr:d: option
do
case $option in
g) g_opt=1;;
p) p_opt=1;;
r) rfunction_id="$OPTARG";;
d) dfunction_id="$OPTARG";;
[?])
echo "Invalid Usage" 1>&2
echo "$USAGE" 1>&2
exit 2
;;
esac
done
if [[ -n $rfunction_id && -n $dfunction_id ]]
then
echo "Invalid Usage: You can't specify both -r and -d" 1>&2
echo "$USAGE" >2&
exit 2
fi
shift $(($OPTIND - 1))
[[ -n $g_opt ]] && echo "-g was set"
[[ -n $p_opt ]] && echo "-p was set"
[[ -n $rfunction_id ]] && echo "-r was set to $rfunction_id"
[[ -n $dfunction_id ]] && echo "-d was set to $dfunction_id"
[[ -n $1 ]] && echo "File is $1"
To (recap) and add to #DavidW.'s excellent answer:
Check the shebang line (first line) of your script to ensure that it's executed by bash: is it #!/bin/bash or #!/usr/bin/env bash?
Inspect your script file for hidden control characters (such as \r) that can result in unexpected behavior; run cat -v scriptFile | fgrep ^ - it should produce NO output; if the file does contain \r chars., they would show as ^M.
To remove the \r instances (more accurately, to convert Windows-style \r\n newline sequences to Unix \n-only sequences), you can use dos2unix file to convert in place; if you don't have this utility, you can use sed 's/'$'\r''$//' file > outfile (CAVEAT: use a DIFFERENT output file, otherwise you'll destroy your input file); to remove all \r instances (even if not followed by \n), use tr -d '\r' < file > outfile (CAVEAT: use a DIFFERENT output file, otherwise you'll destroy your input file).
In addition to #DavidW.'s great debugging technique, you can add the following to visually inspect all arguments passed to your script:
i=0; for a; do echo "\$$((i+=1))=[$a]"; done
(The purpose of enclosing the value in [...] (for example), is to see the exact boundaries of the values.)
This will yield something like:
$1=[-g]
$2=[input.txt]
...
Note, though, that nothing at all is printed if no arguments were passed.
Try to print FILE1 to see if it has the value you want, if it is not the problem, here is a simple script (site below):
#!/bin/bash
file="${#:$OPTIND:1}"
if [ -f "$file" ]
then
echo "$file found."
else
echo "$file not found."
fi
http://www.cyberciti.biz/faq/unix-linux-test-existence-of-file-in-bash/
Instead of plucking an item out of "$#" in a tricky way, why don't you shift off the args you've processed with getopts:
while getopts ...
done
shift $(( OPTIND - 1 ))
FILE1=$1
I'm writing a very simple shell scripts that would looked at the log of all failed tests, and print out all the name of all files in the current directory that are in the log
1 #! /bin/sh
2 for file in *
3 do
4 echo "checking: $file"
5 if [$(grep $file failed.txt -c) -ne 0]
6 then
7 echo "$file FAILED"
8 fi
9 done
When I execute it, I get this error:
line 6: [0: command not found
Does anyone have any idea why?
Thanks!!
[ is actually a command in linux (like bash or cat or grep).
$(grep $file failed.txt -c) is a command substitution which in your case evaluated to 0. Thus the line now reads [0 -ne 0], which is interpreted as run a program called [0 with arguments -ne 0].
What you should write instead is [ $(grep $file failed.txt -c) -ne 0 ]. Shell scripts require that there be spaces between the opening and closing square braces. Otherwise you change the command that is executed (the closing ] indicates that there are no more arguments to be read.
So now the command evaluates to [ 0 -ne 0 ]. You can try executing this in your shell to see what happens. [ exits with a value of 0 if the expression is true and 1 if it is false. You can see the exit value by echoing $? (the exit value of the last command to be run).
Instead of testing the count, you can test the return code of grep:
if grep -q $file failed.txt &>/dev/null
The script can be
#!/bin/sh
for file in *; do
echo "checking: $file"
grep failed.txt $file && echo "$file FAILED"
done
or, as an one-liner in user shell command history:
for file in *; do { echo "checking: $file" && grep failed.txt $file && echo "$file FAILED"; done
in man grep
EXIT STATUS
The exit status is 0 if selected lines are found, and 1 if not found. If an error occurred the exit status is 2. (Note: POSIX error handling code should check for '2' or greater.)