I'm trying to use ts/tsp to schedule idle tasks that I need done from time to time, which is OK if they don't complete due to a crash.
So far, I'm trying with a script like this:
the_args=(--long-arg /usr/share/lib --long-arg2 -j $j -o "'$path_o/'" -i "'$path_i'")
tsp -m -L "$jobname" bash -c '
echo task "$#"
cgexec -g cpu,freezer:execting exector "$#"
' "${the_args[#]}"
I want to run executor with the args given by the_args
I've tried many alternatives, including:
tsp -m -L "$jobname" bash -c "
echo task ${the_args[#]}
cgexec -g cpu,freezer:execting exector ${the_args[#]}
"
I've also tried with heredocs with different configurations.... None of them worked.
Unfortunately, none of those allow me to call the command with all the args. Some methods pass only the first element in the list (the ones shown), others don't even work.
What is the correct way to pass parameters to inside the deferred script?
The first argument following -c's argument is used to set $0 in the shell; it is not included in $#. You need to provide some dummy argument (since you probably don't care what $0 actually is).
Your quoting inside the_args also needs to be simpilfied.
the_args=(
--long-arg /usr/share/lib
--long-arg2
-j "$j"
-o "$path_o/"
-i "$path_i"
)
tsp -m -L "$jobname" bash -c '
echo task "$#"
cgexec -g cpu,freezer:execting exector "$#"
' "" "${the_args[#]}"
Related
How do I remove a single "command" from Bash's auto complete command suggestions? I'm asking about the very first argument, the command, in auto complete, not asking "How to disable bash autocomplete for the arguments of a specific command"
For example, if I have the command ls and the system path also finds ls_not_the_one_I_want_ever, and I type ls and then press tab, I want a way to have removed ls_not_the_one_I_want_ever from every being a viable option.
I think this might be related to the compgen -c list, as this seems to be the list of commands available.
Background: WSL on Windows is putting all the .dll files on my path, in addition to the .exe files that should be there, and so I have many dlls I would like to remove in my bash environment, but I'm unsure how to proceed.
Bash 5.0's complete command added a new -I option for this.
According to man bash —
complete -pr [-DEI] [name ...]
[...] The -I option indicates that other supplied options and actions should apply to completion on the initial non-assignment word on the line, or after a command delimiter such as ; or |, which is usually command name completion. [...]
Example:
function _comp_commands()
{
local cur=$2
if [[ $cur == ls* ]]; then
COMPREPLY=( $(compgen -c "$cur" | grep -v ls_not_wanted) )
fi
}
complete -o bashdefault -I -F _comp_commands
Using #pynexj's answer, I came up with the following example that seems to work well enough:
if [ "${BASH_VERSINFO[0]}" -ge "5" ]; then
function _custom_initial_word_complete()
{
if [ "${2-}" != "" ]; then
if [ "${2::2}" == "ls" ]; then
COMPREPLY=($(compgen -c "${2}" | \grep -v ls_not_the_one_I_want_ever))
else
COMPREPLY=($(compgen -c "${2}"))
fi
fi
}
complete -I -F _custom_initial_word_complete
fi
I have a script that was kicking off ~200 jobs for each sub-analysis. I realized that a job array would probably be much better for this for several reasons. It seems simple enough but is not quite working for me. My input files are not numbered so I've following examples I've seen I do this first:
INFILE=`sed -n ${SGE_TASK_ID}p <pathto/listOfFiles.txt`
My qsub command takes in quite a few variables as it is both pulling and outputting to different directories. $res does not change, however $INFILE is what I am looping through.
qsub -q test.q -t 1-200 -V -sync y -wd ${res} -b y perl -I /master/lib/ myanalysis.pl -c ${res}/${INFILE}/configFile-${INFILE}.txt -o ${res}/${INFILE}/
Since this was not working, I was curious as to what exactly was being passed. So I did an echo on this and saw that it only seems to expand up to the first time $INFILE is used. So I get:
perl -I /master/lib/ myanalysis.pl -c mydirectory/fileABC/
instead of:
perl -I /master/lib/ myanalysis.pl -c mydirectory/fileABC/configFile-fileABC.txt -o mydirectory/fileABC/
Hoping for some clarity on this and welcome all suggestions. Thanks in advance!
UPDATE: It doesn't look like $SGE_TASK_ID is set on the cluster. I looked for any variable that could be used for an array ID and couldn't find anything. If I see anything else I will update again.
Assuming you are using a grid engine variant then SGE_TASK_ID should be set within the job. It looks like you are expecting it to be set to some useful variable before you use qsub. Submitting a script like this would do roughly what you appear to be trying to do:
#!/bin/bash
INFILE=$(sed -n ${SGE_TASK_ID}p <pathto/listOfFiles.txt)
exec perl -I /master/lib/ myanalysis.pl -c ${res}/${INFILE}/configFile-${INFILE}.txt -o ${res}/${INFILE}/
Then submit this script with
res=${res} qsub -q test.q -t 1-200 -V -sync y -wd ${res} myscript.sh
`
I compile my project with:
debug=yes make -j4 or debug=no make -j4
The debug variable changes some compiler flags in the Makefile
Instead of typing this repeatedly in the shell, I wrote this script (lets call it daemon):
#!/bin/bash
inotifywait -q -m -e close_write `ls *.c* *.h*` |
while read; do
make -j4
done
so I just do ./daemon which automatically builds whenever a file is written to.
However, I would like to be able to pass the debug=no make -j4 to the ./daemon script like this:
./daemon debug=no make -j4
So I modified the script:
#!/bin/bash
if [ $# -lt 1 ]; then
echo "Usage `basename $0` [COMMAND]"
exit 1;
fi
inotifywait -q -m -e close_write `ls *.c* *.h*` |
while read; do
"$#"
done
This works with ./daemon make -j4 but when I say daemon debug=no make -j4 I get the following error:
./daemon: line 9: debug=no: command not found
How can I make it so debug=no is recognized as a variable and not a command in the daemon script?
Thanks
The expansion of "$#" is parsed after any pre-command assignments are recognized. All you need to do is ensure that debug=... is in the environment of the command that runs make, which is your daemon script.
debug=no ./daemon make -j4
Variable expansions will only ever become arguments (including the zeroth argument: the command name).
They will never become:
Redirections, so you can't var='> file'; cmd $var
Shell keywords or operators, so you can't var='&'; mydaemon $var
Assignments, including prefix assignments, so you can't var='debug=yes'; $var make as you discovered
Command expansions, loops, process substitutions, &&/||, escape sequences, or anything else.
If you want to do this though, you're in luck: there's a standard POSIX tool that will turn leading key=value pairs into environment variables and run the program you want.
It's called env. Here's an example:
run() {
env "$#"
}
run debug=yes make -j 4
Though TBH I'd use chepner's solution
You always need to put the (env) variable settings at the beginning of the command, i.e. before "daemon".
I have the most simple issue and I am so not certain what I am doing wrong.
I have a simple shell script using /bin/sh
inside the script I have the following:
exec_as_wwwdata() {
if [ $(whoami) = ${WWW_USER} ]]; then
$#
else
su -s /bin/sh -c '$#' ${WWW_USER}
fi
}
I am calling it with
exec_as_wwwdata composer config -g github-oauth.github.com $COMPOSER_GITHUB_TOKEN
it just does nothing, no error message nothing.
If I call the following directly inside the script
su -s /bin/sh -c 'composer config -g github-oauth.github.com $COMPOSER_GITHUB_TOKEN' ${WWW_USER}
it works.
What am I doing wrong here?
Based on feedback I have changed it to this
exec_as_wwwdata() {
if [ $(whoami) = ${WWW_USER} ]]; then
$#
else
su -s /bin/sh -c '"$#"' "$WWW_USER" _ "$#"
fi
}
Although when I am calling it with the following arguments
exec_as_wwwdata /usr/local/bin/php /usr/local/bin/composer create-project --repository-url=xxxx .
I receive the following error message
su: unrecognized option '--repository-url=
I think there is issue with -- in the string. How can I escape that ?
There are two overlapping uses of $# here, and you've inadvertently stumbled on a partial correct solution. -c expects a single word, while "$#" would produce multiple distinct words. The correct solution would be
su -s /bin/sh -c '"$#"' "$WWW_USER" _ "$#"
The short version: you don't want to build a command string from the current parameters; you want to pass them as arguments to a hard-coded command string to let the new shell expand things appropriately.
A breakdown:
-s /bin/sh - use /bin/sh instead of the appropriate users's login shell
-c '"$#"' run the command "$#", as desired. Note this is a hard-coded value; the new shell will expand its positional parameters correctly once it has started.
"$WWW_USER" - specify the user to run the shell as
_ - specify the value of $0 in the shell being run. You probably don't care what this value is; you just need some placeholder to prevent your first real argument from being treated as the value for $0.
"$#" pass the current positional parameters as arguments to the new shell, which will expand its "$#" to these values.
When I run the following script in bash:
#!/bin/bash
names=("ALL" "no_C" "no_R" "no_Q")
for name in $names; do
export name=$name
mkdir -p $name
( echo 'selection' 'System' | gmx cluster -f ${name}_protein_only.trr -s ${name}_protein_only.pdb -n ${name}_index.ndx -g ${name}/cluster.log -cutoff 0.2 -fit -method gromos -o ${name}/cluster.output -dist ${name}/rmsd-dist.xvg -av -cl ${name}/clusters.pdb ) &
done
wait
The for loop won't loop until the subshell has completed, even though I've put it into the background with '&'. If I run this same script in zsh, it runs as expected (4 parallel tasks). Is this a bug or am I missing something?
You need to use a different notation for all the elements of an array in Bash (see Arrays and Shell parameter expansion):
for name in "${name[#]}"; do
When you specify $name and name is an array, Bash treats it as ${name[0]}.
I used this variant on your code to demonstrate:
#!/bin/bash
names=("ALL" "no_C" "no_R" "no_Q")
for name in "${names[#]}"
do
export name=$name
mkdir -p $name
( echo $name 'selection' 'System' | sed s/s/z/g; sleep 3 ) &
done
wait
I believe what Jonathan Leffler wrote is correct in terms of the definition of a Bash array and how to loop through it.
But, if you wanted to define names as a series of strings like this:
names="ALL no_C no_R no_Q"
Then you could still loop through it using:
for name in $names; do
.... do something with name
done