I wrote a small bash script that looks round about like this:
VAR1="test"
VAR2="test2"
letsDoSomeStuff() {
echo $1
echo $2
echo $3`
}
fswatch -0 . | xargs -0 -n 1 -I {} bash -c 'letsDoSomeStuff {} "$VAR1 $VAR2"'
What I want to do, is to look for changes in a folder (via the swatch) and then do some stuff with the changed files in a function in the bash script. Unfortunately, I need to pass on the $VAR1 and $VAR2 as parameters as they don't "survive" the xargs call.
The bash scripts works fine. However, $VAR1 and $VAR2 are not properly passed on to the function. When I start the script, it outputs for every changed file:
filename
empty line
empty line
Can anybody here help me out with this call? I guess I'm messing up the single and double quotes but can't find the right way.
Thanks in advance
Norbert
You need to pass in parameters to your function call:
fswatch -0 . |
xargs -0 -n 1 -I {} bash -c 'letsDoSomeStuff "$1" "$2" "$3"' - {} "$VAR1" "$VAR2"
Related
I'm trying to kick off multiple processes to work through some test suites. In my bash script I have the following
printf "%s\0" "${SUITE_ARRAY[#]}" | xargs -P 2 -0 bash -c 'run_test_suite "$#" ${EXTRA_ARG}'
Below is the defined script, cut down to it's basics.
SUITE_ARRAY will be a list of suites that may have 1 or more, {Suite 1, Suite 2, ..., Suite n}
EXTRA_ARG will be like a specific name to store values in another script
#!/bin/bash
run_test_suite(){
suite=$1
someArg=$2
someSaveDir=someArg"/"suite
# some preprocess work happens here, but isn't relevant to running
runSomeScript.sh suite someSaveDir
}
export -f run_test_suite
SUITES=$1
EXTRA_ARG=$2
IFS=','
SUITECOUNT=0
for csuite in ${SUITES}; do
SUITE_ARRAY[$SUITECOUNT]=$csuite
SUITECOUNT=$(($SUITECOUNT+1))
done
unset IFS
printf "%s\0" "${SUITE_ARRAY[#]}" | xargs -P 2 -0 bash -c 'run_test_suite "$#" ${EXTRA_ARG}'
The issue I'm having is how to get the ${EXTRA_ARG} passed into xargs. From how I've come to understand it, xargs will take whatever is piped into it, so the way I have it doesn't seem correct.
Any suggestions on how to correctly pass the values? Thanks in advance
If you want EXTRA_ARG to be available to the subshell, you need to export it. You can do that either explicitly, with the export keyword, or by putting the var=value assignment in the same simple command as xargs itself:
#!/bin/bash
run_test_suite(){
suite=$1
someArg=$2
someSaveDir=someArg"/"suite
# some preprocess work happens here, but isn't relevant to running
runSomeScript.sh suite someSaveDir
}
export -f run_test_suite
# assuming that the "array" in $1 is comma-separated:
IFS=, read -r -a suite_array <<<"$1"
# see the EXTRA_ARG="$2" just before xargs on the same line; this exports the variable
printf "%s\0" "${suite_array[#]}" | \
EXTRA_ARG="$2" xargs -P 2 -0 bash -c 'run_test_suite "$#" "${EXTRA_ARG}"' _
The _ prevents the first argument passed from xargs to bash from becoming $0, and thus not included in "$#".
Note also that I changed "${suite_array[#]}" to be assigned by splitting $1 on commas. This or something like it (you could use IFS=$'\n' to split on newlines instead, for example) is necessary, as $1 cannot contain a literal array; every shell command-line argument is only a single string.
This is something of a guess:
#!/bin/bash
run_test_suite(){
suite="$1"
someArg="$2"
someSaveDir="${someArg}/${suite}"
# some preprocess work happens here, but isn't relevant to running
runSomeScript.sh "${suite}" "${someSaveDir}"
}
export -f run_test_suite
SUITE_ARRAY="$1"
EXTRA_ARG="$2"
printf "%s\0" "${SUITE_ARRAY[#]}" |
xargs -n 1 -I '{}' -P 2 -0 bash -c 'run_test_suite {} '"${EXTRA_ARG}"
Using GNU Parallel it looks like this:
#!/bin/bash
run_test_suite(){
suite="$1"
someArg="$2"
someSaveDir="$someArg"/"$suite"
# some preprocess work happens here, but isn't relevant to running
echo runSomeScript.sh "$suite" "$someSaveDir"
}
export -f run_test_suite
EXTRA_ARG="$2"
parallel -d, -q run_test_suite {} "$EXTRA_ARG" ::: "$1"
Called as:
mytester 'suite 1,suite 2,suite "three"' 'extra "quoted" args here'
If you have the suites in an array:
parallel -q run_test_suite {} "$EXTRA_ARG" ::: "${SUITE_ARRAY[#]}"
Added bonus: Any output from the jobs will not be mixed, so you will not have to deal with http://mywiki.wooledge.org/BashPitfalls#Using_output_from_xargs_-P
I'm trying to change the filename from
prod.test.PVSREGPLUS20170915-6777.DAT.gpg
to
PVSREGPLUS20170915-0003.DAT.gpg
I used this
DTE=$(date +%I);ls prod.test* |cut -f 3,4,5 -d .|sed "s/\-/-00$DTE/" |cut -c 1-23,28-35
My problem is I need this command in a shell script
"#! /bin/bash
DTE=$(date +%I)
newfile=$(ls prod.test* |cut -f 3,4,5 -d .|sed "s/-*./$DTE/"|cut -c 1-23,28-35
The sed can't do expansion, would awk be able to do this? Any help would be greatly appreciated. Thanks
The simplest way to do this is with a for-loop over a glob pattern, then use paramater expansion to remove the prefix
prefix="prod.test."
hour=$(date '+%I')
for f in "$prefix"*; do
new=$( echo "${f#$prefix}" | sed 's/-[[:digit:]]\+/-00'"$hour"'/' )
echo mv "$f" "$new"
done
We really don't need sed: extended patterns and more parameter expansion
shopt -s extglob
for f in "$prefix"*; do
new=${f#$prefix}
new=${new/-+([0-9])/-00$hour}
echo mv "$f" "$new"
done
Remove "echo" if it looks good.
Or, with the perl rename as suggested in the comments:
rename -v -n 's/prod\.test\.//; use Time::Piece; s{-\d+}{"-00" . (localtime)->strftime("%I") }e' prod.test.*
Remove "-n" if it looks good.
I have a directory with lots of files. I want to keep only the 6 newest. I guess I can look at their creation date and run rm on all those that are too old, but is the a better way for doing this? Maybe some linux command I could use?
Thanks!
:)
rm -v $(ls -t mysvc-*.log | tail -n +7)
ls -t, list sorted by time
tail -n +7, +7 here means length-7, so all but first 7 lines
$() makes a list of strings from the enclosed command output
rm to remove the files, of course
Beware files with space in their names, $() splits on any white-space!
Here's my take on it, as a script. It does handle spaces in file names even if it is a bit of a hack.
#!/bin/bash
eval set -- $(ls -t1 | sed -e 's/.*/"&"/')
if [[ $# -gt 6 ]] ; then
shift 6
while [[ $# -gt 0 ]] ; do
echo "remove this file: $1" # rm "$1"
shift
done
fi
The second option to ls up there is a "one" for one file name per line. Doesn't actually seem to matter, though, since that appears to be the default when ls isn't feeding a tty.
I have a short bash script running samtools mpileup. It works fine locally, but when I try run it on SGE, I get an "illegal variable name" feedback.
#!/bin/bash
for f in $(find /bed_files -name '*.bed' )
do
name=$(echo $f | awk 'gsub("/", "_")')
name2=$(echo $name | awk 'gsub("_bed_files_", "")')
name3=$(echo $name2 | awk 'gsub(".bed", "")')
samtools runs here
done
Is SGE variable syntax different to bash?
Yes, I needed to change the first line to:
#$ -S /bin/sh
The following almost works:
#!/bin/bash
/* 2>&1 >/dev/null
script_dir=$(dirname "$0")
export GROOVY_HOME=${script_dir}/../../../../Tools/groovy/groovy-2.0.2
exec ${GROOVY_HOME}/bin/groovy -cp "${script_dir}:$(ls ${script_dir}/build/lib/runtime/*.jar | xargs echo | sed -e 's| |:|g')" "$0"
*/ // 2>&1 >/dev/null
println("aoeu")
The only problem is that the shell globs /* and tries to execute it. In the end, all I really want to do is to be able to build the Groovy script's classpath without having to have two separate scripts.
You can also try the technique below, which is independent of Groovy syntax (and does not normally produce output on stderr):
#!/bin/sh
script_dir=$(dirname "$0")
export GROOVY_HOME="$script_dir/../../../../Tools/groovy/groovy-2.0.2"
awk 'mark_on{print}/^__END__$/{mark_on=1}' "$0" >/tmp/$$.groovy
"$GROOVY_HOME/bin/groovy" -cp "$script_dir:$(echo "$script_dir"/build/lib/runtime/*.jar | tr " " :)" /tmp/$$.groovy
status=$?
rm -f /tmp/$$.groovy
exit $status
__END__
println("aoeu")
Also notice the simplification in the classpath calculation; remember that globbing (wildcard expansion) is performed by the shell, not by the command that takes the arguments, so you do not have to (nor do you want to) use ls in this case.
The curly brackets ${} in your original code are technically superfluous in this case (they are purely stylistic); they would be needed if you had for example to append a string directly after a variable substitution where there is no clear break between the variable name and what follows, e.g. you cannot say $my_varsome_string but you can write ${my_var}some_string, or any of $my_var"some_string" or $my_var'some_string' or "$my_var"some_string or "$my_var""some_string". I removed the braces for "minimalistic" purposes and in order to illustrate the above, but again, it's perfectly fine to keep them for stylistic reasons.
The quotes I added consistently in the code above protect you from potential blanks and certain other special characters inside $GROOVY_HOME. Feel free to remove them in order to simplify quoting (and be minimalistic) if you know $GROOVY_HOME will not contain blanks.
It seems you want to write a groovy / shell polyglot.
I don't know groovy, but from the documentation it seems [] is a valid groovy command, creating an empty list.
Then you could write it as follow:
#!/bin/bash
[ /* 2> /dev/null > /dev/null
script_dir=$(dirname "$0")
export GROOVY_HOME=${script_dir}/../../../../Tools/groovy/groovy-2.0.2
exec ${GROOVY_HOME}/bin/groovy -cp "${script_dir}:$(ls ${script_dir}/build/lib/runtime/*.jar | xargs echo | sed -e 's| |:|g')" "$0"
*/
]
println("aoeu")
Groovy will read [] and ignore it, and bash will call [ with /*, which will cause an error that is ignored. But it will not run any unexpected programs.
#!/bin/bash
script_dir="$(cd $(dirname $0) >/dev/null; pwd -P)"
function after-bangshe() {
sed -e '1,/^!#$/d' "$1"
}
if [ -z "${GROOVY_HOME}" ]
then
echo 'GROOVY_HOME must be defined.' >&2
exit 1
fi
CLASSPATH="${script_dir}" "${GROOVY_HOME}/bin/groovy" -e "$(after-bangshe $0)" "$#"
exit
!#
println 'aoeu'
The output of cd is redirected to /dev/null in case CDPATH is set (which makes cd noisy).
CLASSPATH is set to the script directory so that any support classes can be found.
The sed command strips out everything after the !# line.