Problem Description
I am trying to filter directories that we don't want to ignore for the test coverage. For this purpose we are using Lcov.
When I try to put the directories that are to be ignore in a variable __ignoreinput the command #${__lcov} ${__gcovopts} --remove MYCODE.info "${__ignoreinput}" -o MYCODE_filtered.info > /dev/null 2> /dev/null doesn't work, it doesn't filter anything. Whereas when I use the command without the __ignoreinput as in
${__lcov} ${__gcovopts} --remove MYCODE.info '/opt/*' '/usr/include/*' '*3rdParty/*' '*Input_API/*' '*Grammars/*' -o MYCODE_filtered.info > /dev/null 2> /dev/null
if [[ ${?} -ne 0 ]] ;then echo "Error *** lcov filtrering failed" && exit 1 ;fi
The filter works ok. What am I doing wrong. I don't understand.
Script
#!/bin/bash
__orc=/home/anybody/workspace/project
__buildtype="local"
__output=/home/anybody/workspace/lcov
#doe not work
#__ignoreinput="'/opt/*' '/usr/include/*' '*3rdParty/*' '*Input_API/*' '*Grammars/*'"
#__ignoreinput="/opt/* /usr/include/* *3rdParty/* *Input_API/* *Grammars/* "
#__ignoreinput="\"/opt/*\" \"/usr/include/*\" \"*3rdParty/*\" \"*Input_API/*\" \"*Grammars/*\""
__gcovopts=--gcov-tool=/opt/1A/x86_64-2.6.32-v2/bin/gcov
__lcov=lcov
if [[ "${__buildtype}" == "docker" ]] ;then
__build=MYCODE/build_x86_64-2.6.32-v2_Gcov
else
__build=MYCODE/cmake-build-coverage
fi
echo "Filter lcov tracefile"
cd ${__orc}/${__build}
#does not work
#${__lcov} ${__gcovopts} --remove MYCODE.info "${__ignoreinput}" -o MYCODE_filtered.info > /dev/null 2> /dev/null
#works
${__lcov} ${__gcovopts} --remove MYCODE.info '/opt/*' '/usr/include/*' '*3rdParty/*' '*Input_API/*' '*Grammars/*' -o MYCODE_filtered.info > /dev/null 2> /dev/null
if [[ ${?} -ne 0 ]] ;then echo "Error *** lcov filtrering failed" && exit 1 ;fi
echo "Generate HTML reports"
cd ${__orc}/${__build}
genhtml --ignore-errors source -o ${__output}/lcov_"$(git rev-list HEAD -n 1)" MYCODE_filtered.info > /dev/null 2> /dev/null
if [[ ${?} -ne 0 ]] ;then echo "Error *** lcov reports failed" && exit 1 ;fi
One simple way to pass the __ignoreinput would be to store the glob expressions in the array under single quote expand them while passing to the lcov command. Write your ignore input as
__ignoreinput=( '/opt/*' '/usr/include/*' '*3rdParty/*' '*Input_API/*' '*Grammars/*' )
and doing below should work as expected.
"${__lcov}" "${__gcovopts}" --remove MYCODE.info "${__ignoreinput[#]}" -o MYCODE_filtered.info 2>&1 > /dev/null
For all the failure cases in your description __ignoreinput is set as one whole-string under ".." but the command expects words split up one expression for each. The array expansion "${__ignoreinput[#]}" though puts each word defined in the array as a separate word as expected by the command.
Also carefully single/double quote the words in the array during the definition, because with lack of quotes * could undergo path-name expansion and could expand to the list of filenames under each of those paths.
Also see how > /dev/null 2> /dev/null could be minimized to > /dev/null 2> /dev/null or simply &> /dev/null in bash.
Also naming variable names prefixed with __ is a bad practice. Like most of the languages out there, the character can itself could be a valid variable identifier. As shown above enclosing the variable name completely around {..} is the recommended way.
Related
I have written a shell script (bash) which runs some commands. It has an option to not to run the commands but to echo them to the screen. By default, the output of these commands is redirected to /dev/null but there is another option to show the output on the screen.
I use a function to check for the value of these variables and run the commands or simulate them:
runmaybe() {
if [[ true = $dry_run ]]; then
echo "Simulating '$#'"
else
if [[ true = $verbose ]]; then
$#
else
$# > /dev/null
fi
fi
}
The function is working but I had some issues with complex commands such as:
runmaybe eval svn cp $url $root/tags/$ntag -m \"Tagging revision with $ntag\"
I had to add the eval to prevent wordsplitting so svn gets the right value for the -m option.
I have some other complex commands in that script such as:
runmaybe vzctl exec 1 "( cd /var/wwww/vhosts/myhost ; php cron.php )"
runmaybe ssh -t user#$host "vzctl exec $vmid \"( /usr/local/bin/myscript )\"" 2>/dev/null
runmaybe rsync --delete --exclude=\"**/.svn/\" --exclude=\"**/.git/\" --include=*.exe --numeric-ids -a $vOpt -H $LOCAL_VM$dir $host:$REMOTE_VM$dir
Although the script is working right now, I wonder if there is a better way of doing this task.
The problem is in unquoted expansion of $#. As a rule of a thumb, if you see $, you should put it inside ". Unquoted expansions undergo word splitting and filename expansions - to prevent them, quote the expansion.
I had to add the eval
eval is evil. Do not use it.
runmaybe() {
if [[ true = $dry_run ]]; then
echo "Simulating '$*'"
# or better quote with printf in some corner cases
printf "Simulating:"
printf " %q" "$#"
printf "\n"
elif [[ true = $verbose ]]; then
"$#"
else
"$#" > /dev/null
fi
}
runmaybe svn cp "$url" "$root/tags/$ntag" -m "Tagging revision with $ntag"
runmaybe rsync --delete --exclude="**/.svn/" --exclude="**/.git/" --include="*.exe" --numeric-ids -a "$vOpt" -H "$LOCAL_VM$dir" "$host:$REMOTE_VM$dir"
I have a bash script as shown below. I run this in a directory containing files such as input1.inp and other files like coords_i.xyz and submission.sub in order to make some simple modifications to them:
#!/bin/bash
sed -i -e '25d' *.inp
echo "*xyz -2 2" >> *.inp
sed -n '3,7p' *_i.xyz >> *.inp
echo "Q -1 0 0 3" >> *.inp
echo "Q +1 0 0 -3" >> *.inp
echo "*" >> *.inp
sed -i -e s/"replace1"/"replace2"/g *.sub
rm *.out
If I am in this directory, and I run all the commands individually in the terminal (line by line in the script), everything works fine. However, when I try to group all these commands into the script as shown above, I get an error - essentially after the line sed -i -e '25d' *.inp, the script stops and a file called *.inp is created in my directory. If I try to run the echo command separately after that, it says the command is ambiguous (presumably because of the existence of this *.inp file).
Why don't my wildcards work the same way in the script as they did when I ran them separately and sequentially in the terminal, and what can I do so that they work properly in the script?
Using wildcards this way is hazardous; the easy advice is "don't". Evaluate them only once, and then you can check their outputs before trying to use them.
In the below, we define an assert_only_one function that stops your script when an array -- assigned from a glob -- contains less or more than exactly one element. Consequently, we're able to write code that more clearly and explicitly describes our desired behavior.
#!/usr/bin/env bash
shopt -s nullglob # Stop *.xyz evaluating to '*.xyz' if no such files exist
assert_only_one() {
local glob; glob=$1; shift
case $# in
0) echo "ERROR: No files matching $glob exist" >&2; exit 1;;
1) return 0;;
*) echo "ERROR: More than one file matching $glob exists:" >*2
printf ' %q\n' "$#" >&2
exit 1;;
esac
}
inp_files=( *.inp ); assert_only_one '*.inp' "${inp_files[#]}"
sub_files=( *.sub ); assert_only_one '*.sub' "${sub_files[#]}"
xyz_files=( *_i.xyz )
sed -i -e '25d' "${inp_files[0]}"
{
echo "*xyz -2 2"
sed -n '3,7p' "${xyz_files[#]}"
echo "Q -1 0 0 3"
echo "Q +1 0 0 -3"
echo "*"
} >>"${inp_files[0]}"
sed -i -e s/"replace1"/"replace2"/g -- "${sub_files[#]}"
rm -- *.out
So I have been struggling with this task for eternity and still don't get what went wrong. This program doesn't seem to download ANY pdfs. At the same time I checked the file that stores final links - everything stored correctly. The $PDFURL also checked, stores correct values. Any bash fans ready to help?
#!/bin/sh
#create a temporary directory where all the work will be conducted
TMPDIR=`mktemp -d /tmp/chiheisen.XXXXXXXXXX`
echo $TMPDIR
#no arguments given - error
if [ "$#" == "0" ]; then
exit 1
fi
# argument given, but wrong format
URL="$1"
#URL regex
URL_REG='(https?|ftp|file)://[-A-Za-z0-9\+&##/%?=~_|!:,.;]*[-A-Za-z0-9\+&##/%=~_|]'
if [[ ! $URL =~ $URL_REG ]]; then
exit 1
fi
# go to directory created
cd $TMPDIR
#download the html page
curl -s "$1" > htmlfile.html
#grep only links into temp.txt
cat htmlfile.html | grep -o -E 'href="([^"#]+)\.pdf"' | cut -d'"' -f2 > temp.txt
# iterate through lines in the file and try to download
# the pdf files that are there
cat temp.txt | while read PDFURL; do
#if this is an absolute URL, download the file directly
if [[ $PDFURL == *http* ]]
then
curl -s -f -O $PDFURL
err="$?"
if [ "$err" -ne 0 ]
then
echo ERROR "$(basename $PDFURL)">&2
else
echo "$(basename $PDFURL)"
fi
else
#update url - it is always relative to the first parameter in script
PDFURLU="$1""/""$(basename $PDFURL)"
curl -s -f -O $PDFURLU
err="$?"
if [ "$err" -ne 0 ]
then
echo ERROR "$(basename $PDFURLU)">&2
else
echo "$(basename $PDFURLU)"
fi
fi
done
#delete the files
rm htmlfile.html
rm temp.txt
P.S. Another minor problem I have just spotted. Maybe the problem is with the if in regex? I pretty much would like to see something like that there:
if [[ $PDFURL =~ (https?|ftp|file):// ]]
but this doesn't work. I don't have unwanted parentheses there, so why?
P.P.S. I also ran this script on URLs beginning with http, and the program gave the desired output. However, it still doesn't pass the test.
I have been busy this week trying to wrap my head around a little Bash program to migrate a CMS from one server to another. The reasopn for this is because I have more tha 40 of these to do, and need to get it done in a timely manner, thus the Bash idea.
Needless to say, I have run into a couple of problems so far, but one of them has halted my development completetly, directory checking.
No I have tried a couple of methods and none of them seem to work really. The catch is that I have to check the folder on a remote server via ssh. Here my example:
ExSshRsa=~/.ssh/id_rsa
ExSshPort=22
ExSshHost=localhost
ExRoot=/var/www/
echo -n "Verifying Root access $ExRoot..."
SSHRoot='ssh -i $ExSshRsa -p $ExSshPort $ExSshHost [ -d $ExRoot ] || exit 1 '
echo $SSHRoot
if [ "$SSHRoot" -eq 0 ]
then
echo "OK"
else
echo "FAIL"
fi
I get the Error: [: : integer expression expected
Does the [ or test not resturn a 0 which is numerical. ?
Passing strings as arguments to a remote host is not trivial; you need to use arrays. A test example:
declare -a cmd=(touch "file name with spaces")
printf -v escaped_cmd_str '%q ' "${cmd[#]}"
ssh localhost $escaped_cmd
ssh localhost ls # Should return "file name with spaces" on a separate line
So your case should be:
ExSshRsa=~/.ssh/id_rsa
ExSshPort=22
ExSshHost=localhost
ExRoot=/var/www/
echo -n "Verifying Root access $ExRoot..."
declare -a cmd=( '[' -d "$ExRoot" ']' ) # Need to quote "[" since it's a Bash-specific symbol
printf -v escaped_cmd_str '%q ' "${cmd[#]}"
if ssh -i "$ExSshRsa" -p "$ExSshPort" "$ExSshHost" $escaped_cmd
then
echo "OK"
else
echo "FAIL"
fi
This is a rare case where using unquoted variable expansion is perfectly fine.
change the shebang to #!/bin/bash -x and look at the output...
you are storing a string in variable SSHRoot using single quotes, meaning that no variables will be expanded, i.e. a $ is still a $. Use double quotes instead, i.e. "
to store the output from a command in bash, use
var=$(cmd)
the exist status of a command is stored in the variable $?. Do a check on that after the ssh-command
you are never executing the ssh-command in your code
Great link here for bash-programming
Try the following:
ExSshRsa=~/.ssh/id_rsa
ExSshPort=22
ExSshHost=localhost
ExRoot=/var/www/
echo -n "Verifying Root access $ExRoot..."
cmd="bash -c \"[ -d $ExRoot ] || exit 1\""
SSHRoot="ssh -i $ExSshRsa -p $ExSshPort $ExSshHost ${cmd}"
$SSHRoot
if [ $? -eq 0 ]
then
echo "OK"
else
echo "FAIL"
fi
The variables weren't being replaced in your SSHRoot variable as it's in single quotes. Also, you weren't passing an executable command, so that's why I use bash -c above. It will run the bash commands inside the quoted string.
$? stores the exit value of the last command, in this case the SSHRoot one.
#!/bin/bash
ExSshRsa=~/.ssh/id_rsa
ExSshPort=22
ExSshHost=localhost
ExBase='/tmp/'
ExRoot='one space/'
declare -a AExRoot
for argR in "${ExRoot[#]}"
do
ExRoot+=($(printf %q "$argR"))
done
clear
FRoot=( $ExBase${ExRoot[#]} )
echo -n "Verifying Root access $FRoot..."
SSHRootTest="bash -c \"[ -d $FRoot ] && echo 0 && exit 0 || echo 1 && exit 1\""
SSHRoot=$( ssh -i $ExSshRsa -p $ExSshPort $ExSshHost ${SSHRootTest})
if [ $? -eq 0 ]
then
echo -en "\e[1;32mOK\e[0;37;m..."
else
echo -en "\e[1;31mFAIL\e[0;37;m..."
fi
sleep 1
if [ -w $FRoot ]
then
echo -e "\e[1;32mwritable\e[0;37;m"
else
echo -e "\e[1;31mNOT writeable\e[0;37;m"
fi
echo -e "\e[0;m"
exit 0
So I have incorporated all of the suggestions so far and have one last problem, the FRoot is not getting populated by the complete array values. Other than that I think it now has the subjective approach as suggested #john-keyes, the proper expansion #frederik and the crazy space escapes #l0b0
How can I check in bash and csh if commands are builtin? Is there a method compatible with most shells?
You can try using which in csh or type in bash. If something is a built-in command, it will say so; otherwise, you get the location of the command in your PATH.
In csh:
# which echo
echo: shell built-in command.
# which parted
/sbin/parted
In bash:
# type echo
echo is a shell builtin
# type parted
parted is /sbin/parted
type might also show something like this:
# type clear
clear is hashed (/usr/bin/clear)
...which means that it's not a built-in, but that bash has stored its location in a hashtable to speed up access to it; (a little bit) more in this post on Unix & Linux.
In bash, you can use the type command with the -t option. Full details can be found in the bash-builtins man page but the relevant bit is:
type -t name
If the -t option is used, type prints a string which is one of alias, keyword, function, builtin, or file if name is an alias, shell reserved word, function, builtin, or disk file, respectively. If the name is not found, then nothing is printed, and an exit status of false is returned.
Hence you can use a check such as:
if [[ "$(type -t read)" == "builtin" ]] ; then echo read ; fi
if [[ "$(type -t cd)" == "builtin" ]] ; then echo cd ; fi
if [[ "$(type -t ls)" == "builtin" ]] ; then echo ls ; fi
which would result in the output:
read
cd
For bash, use type command
For csh, you can use:
which command-name
If it's built-in, it will tell so.
Not sure if it works the same for bash.
We careful with aliases, though. There may be options for that.
The other answers here are close, but they all fail if there is an alias or function with the same name as the command you're checking.
Here's my solution:
In tcsh
Use the where command, which gives all occurrences of the command name, including whether it's a built-in. Then grep to see if one of the lines says that it's a built-in.
alias isbuiltin 'test \!:1 != "builtin" && where \!:1 | egrep "built-?in" > /dev/null || echo \!:1" is not a built-in"'
In bash/zsh
Use type -a, which gives all occurrences of the command name, including whether it's a built-in. Then grep to see if one of the lines says that it's a built-in.
isbuiltin() {
if [[ $# -ne 1 ]]; then
echo "Usage: $0 command"
return 1
fi
cmd=$1
if ! type -a $cmd 2> /dev/null | egrep '\<built-?in\>' > /dev/null
then
printf "$cmd is not a built-in\n" >&2
return 1
fi
return 0
}
In ksh88/ksh93
Open a sub-shell so that you can remove any aliases or command names of the same name. Then in the subshell, use whence -v. There's also some extra archaic syntax in this solution to support ksh88.
isbuiltin() {
if [[ $# -ne 1 ]]; then
echo "Usage: $0 command"
return 1
fi
cmd=$1
if (
#Open a subshell so that aliases and functions can be safely removed,
# allowing `whence -v` to see the built-in command if there is one.
unalias "$cmd";
if [[ "$cmd" != '.' ]] && typeset -f | egrep "^(function *$cmd|$cmd\(\))" > /dev/null 2>&1
then
#Remove the function iff it exists.
#Since `unset` is a special built-in, the subshell dies if it fails
unset -f "$cmd";
fi
PATH='/no';
#NOTE: we can't use `whence -a` because it's not supported in older versions of ksh
whence -v "$cmd" 2>&1
) 2> /dev/null | grep -v 'not found' | grep 'builtin' > /dev/null 2>&1
then
#No-op
:
else
printf "$cmd is not a built-in\n" >&2
return 1
fi
}
Using the Solution
Once you applied the aforementioned solution in the shell of your choice, you can use it like this...
At the command line:
$ isbuiltin command
If the command is a built-in, it prints nothing; otherwise, it prints a message to stderr.
Or you can use it like this in a script:
if isbuiltin $cmd 2> /dev/null
then
echo "$cmd is a built-in"
else
echo "$cmd is NOT a built-in"
fi