I want to submit a bash script to my university's Sungrid computing cluster to run an executable in a loop. When I log in to the server, I'm in bash:
$ echo $SHELL
/bin/bash
And I include a bash shebang at the top of the script that I pass to qsub:
$ cat shell_sub
#!/bin/bash
#$ -N bSS_s13
#$ -o logs/bSS_s13.log
#$ -j y
#$ -cwd
echo $SHELL > shell.txt
But when I submit the above script:
qsub shell_sub
It instead executes in csh:
$ cat shell.txt
/bin/csh
How can I force qsub to execute my script with bash instead of csh?
Most likely your queue is configured with shell_start_mode as posix_compliant and the defined shell is listed as /bin/csh (which is the default). To check this:
$ qconf -sq <name-of-queue> | grep shell
shell /bin/bash
shell_start_mode unix_behavior
If you don't know the name of your queue, it's probably all.q.
If shell_start_mode is posix_compliant, then the shebang line is ignored and the job (if it's not submitted as binary: -b y) is started with the shell defined by the shell setting.
Why? From the man page: "POSIX does not consider first script line comments such a ‘#!/bin/csh’ as being significant. The POSIX standard for batch queuing systems (P1003.2d) therefore requires a compliant queuing system to ignore such lines but to use user specified or configured default command interpreters instead."
If shell_start_mode is unix_behavior, then the shebang line is used to determine the shell for the job.
You can ask your administrator to consider changing the queue settings.
You can set the shell for a submitted job (at least in Torque) using -S.
For example: qsub shell_sub -S /bin/bash
Related
According to bash man page, it says -i is for interactive mode of shell.
I tried example code to find out what -i option does.
interactive.sh is script that needs user input, which means interactive script.
The default bash option is non-interactive mode.
But the interactive.sh runs without any problem with non-interactive mode.
It also runs well with interactive mode. It confuses me.
What is the exact usage of -i option in bash?
What is the difference between interactive and non-interactive mode in shell?
$cat interactive.sh
#!/bin/bash
echo 'Your name ?'
read name
echo "Your name is $name"
$ bash interactive.sh
Your name ?
ABC
Your name is ABC
$ bash -i interactive.sh
Your name ?
DEF
Your name is DEF
With bash -i script you are running interactive non-login shell.q
What is the exact usage of -i option in bash?
From man bash:
-i If the -i option is present, the shell is interactive.
What is the difference between interactive and non-interactive mode in shell?
There are some differences. Look at man bash | grep -i -C5 interactive | less:
An interactive shell is one started without non-option arguments (unless -s is specified) and without the -c option whose standard input and er‐
ror are both connected to terminals (as determined by isatty(3)), or one started with the -i option. PS1 is set and $- includes i if bash is
interactive, allowing a shell script or a startup file to test this state.
When an interactive login shell exits, or a non-interactive login shell executes the exit builtin command, bash reads and executes commands from
the file ~/.bash_logout, if it exists.
When an interactive shell that is not a login shell is started, bash reads and executes commands from ~/.bashrc, if that file exists. This may
be inhibited by using the --norc option. The --rcfile file option will force bash to read and execute commands from file instead of ~/.bashrc.
[...]
When bash is interactive, in the absence of any traps, it ignores SIGTERM (so that kill 0 does not kill an interactive shell), and SIGINT is
caught and handled (so that the wait builtin is interruptible). In all cases, bash ignores SIGQUIT. If job control is in effect, bash ignores
SIGTTIN, SIGTTOU, and SIGTSTP.
etc. For example bash -i -c 'echo $PS1'.
I am running Sun Grid Engine for submitting jobs, and I want to have a bash script that sends in any file I need to run, instead of having to run a different qsub command with a different bash file for each of the jobs. I have been capable of generating output and error files that share the name of the input file, but now I am struggling with setting a different name for each file. My approach has been the following:
#!/bin/bash
#
#$ -cwd
#$ -S /bin/bash
#$ -N $1
#
python -u $1 >/output_dir/$1.out 2>/error_dir/$1.error
This way, running qsub send_to_sge.sh foo executes the program, and creates the files foo.error and foo.out with the errors and printouts, respectively. However, the job appears with the name $1 in the SGE queue. Instead, I would like to have foo as the job name. Is there any way to achieve what I am seeking?
Suppose that I have the following simple bash script which I want to submit to a batch server through SLURM:
#!/bin/bash
#SBATCH -o "outFile"$1".txt"
#SBATCH -e "errFile"$1".txt"
hostname
exit 0
In this script, I simply want to write the output of hostname on a textfile whose full name I control via the command-line, like so:
login-2:jobs$ sbatch -D `pwd` exampleJob.sh 1
Submitted batch job 203775
Unfortunately, it seems that my last command-line argument (1) is not parsed through sbatch, since the files created do not have the suffix I'm looking for and the string "$1" is interpreted literally:
login-2:jobs$ ls
errFile$1.txt exampleJob.sh outFile$1.txt
I've looked around places in SO and elsewhere, but I haven't had any luck. Essentially what I'm looking for is the equivalent of the -v switch of the qsub utility in Torque-enabled clusters.
Edit: As mentioned in the underlying comment thread, I solved my problem the hard way: instead of having one single script that would be submitted several times to the batch server, each with different command line arguments, I created a "master script" that simply echoed and redirected the same content onto different scripts, the content of each being changed by the command line parameter passed. Then I submitted all of those to my batch server through sbatch. However, this does not answer the original question, so I hesitate to add it as an answer to my question or mark this question solved.
I thought I'd offer some insight because I was also looking for the replacement to the -v option in qsub, which for sbatch can be accomplished using the --export option. I found a nice site here that shows a list of conversions from Torque to Slurm, and it made the transition much smoother.
You can specify the environment variable ahead of time in your bash script:
$ var_name='1'
$ sbatch -D `pwd` exampleJob.sh --export=var_name
Or define it directly within the sbatch command just like qsub allowed:
$ sbatch -D `pwd` exampleJob.sh --export=var_name='1'
Whether this works in the # preprocessors of exampleJob.sh is also another question, but I assume that it should give the same functionality found in Torque.
Using a wrapper is more convenient. I found this solution from this thread.
Basically the problem is that the SBATCH directives are seen as comments by the shell and therefore you can't use the passed arguments in them. Instead you can use a here document to feed in your bash script after the arguments are set accordingly.
In case of your question you can substitute the shell script file with this:
#!/bin/bash
sbatch <<EOT
#!/bin/bash
#SBATCH -o "outFile"$1".txt"
#SBATCH -e "errFile"$1".txt"
hostname
exit 0
EOT
And you run the shell script like this:
bash [script_name].sh [suffix]
And the outputs will be saved to outFile[suffix].txt and errFile[suffix].txt
If you pass your commands via the command line, you can actually bypass the issue of not being able to pass command line arguments in the batch script. So for instance, at the command line :
var1="my_error_file.txt"
var2="my_output_file.txt"
sbatch --error=$var1 --output=$var2 batch_script.sh
The lines starting with #SBATCH are not interpreted by bash but are replaced with code by sbatch.
The sbatch options do not support $1 vars (only %j and some others, replacing $1 by %1 will not work).
When you don't have different sbatch processes running in parallel, you could try
#!/bin/bash
touch outFile${1}.txt errFile${1}.txt
rm link_out.sbatch link_err.sbatch 2>/dev/null # remove links from previous runs
ln -s outFile${1}.txt link_out.sbatch
ln -s errFile${1}.txt link_err.sbatch
#SBATCH -o link_out.sbatch
#SBATCH -e link_err.sbatch
hostname
# I do not know about the background processing of sbatch, are the jobs still running
# at this point? When they are, you can not delete the temporary symlinks yet.
exit 0
Alternative:
As you said in a comment yourself, you could make a masterscript.
This script can contain lines like
cat exampleJob.sh.template | sed -e 's/File.txt/File'$1'.txt/' > exampleJob.sh
# I do not know, is the following needed with sbatch?
chmod +x exampleJob.sh
In your template the #SBATCH lines look like
#SBATCH -o "outFile.txt"
#SBATCH -e "errFile.txt"
This is an old question but I just stumbled into the same task and I think this solution is simpler:
Let's say I have the variable $OUT_PATH in the bash script launch_analysis.bash and I want to pass this variable to task_0_generate_features.sl which is my SLURM file to send the computation to a batch server. I would have the following in launch_analysis.bash:
`sbatch --export=OUT_PATH=$OUT_PATH task_0_generate_features.sl`
Which is directly accessible in task_0_generate_features.sl
In #Jason case we would have:
sbatch -D `pwd` --export=hostname=$hostname exampleJob.sh
Reference: Using Variables in SLURM Jobs
Something like this works for me and Torque
echo "$(pwd)/slurm.qsub 1" | qsub -S /bin/bash -N Slurm-TEST
slurm.qsub:
#!/bin/bash
hostname > outFile${1}.txt 2>errFile${1}.txt
exit 0
I'm writing a bash script that starts the tcsh interpreter as a login shell and has it execute my_command. The tcsh man page says that there are two ways to start a login shell. The first is to use /bin/tcsh -l with no other arguments. Not an option, because I need the shell to execute my_command. The second is to specify a dash (-) as the zeroeth argument.
Now the bash exec command with the -l option does exactly this, and in fact the following works perfectly:
#!/bin/bash
exec -l /bin/tcsh -c my_command
Except... I can't use exec because I need the script to come back and do some other things afterwards! So how can I specify - as the zeroeth argument to /bin/tcsh without using exec?
You can enclose the exec command into a sub-shell of your script.
#!/bin/bash
(exec -l /bin/tcsh -c my_command)
# ... whatever else you need to do after the command is done
You can write a wrapper (w.sh) script that contains:
#!/bin/bash
exec -l /bin/tcsh -c my_command
and execute w.sh in your main script.
I can't seem to find the difference between a script run two different ways.
Here's the script (named test.sh):
#! /bin/bash
printf "%b\n" "\u5A"
When the script is sourced:
. test.sh
> Z ## Result I want ##
When the script is run:
./test.sh
> \u5A ## Result I get ##
I want the run script to give the results of the sourced script... what setting do I need to set/change?
You are probably getting different versions of printf; the script you are sourcing from is probably a /bin/sh script, not a Bash script proper?
Shouldn't you be using \x instead of \u? printf "%b\n" "\x5A" works fine in both cases for me.
(Totally different idea here, so I'm posting it as another answer.)
Try running these at the command line:
builtin printf "%b\n" "\u5A"
/usr/bin/env printf "%b\n" "\u5A"
printf is both a shell builtin and an executable, and you may be getting different ones depending on whether you source or run the script. To find out, insert this in the script and run it each way:
type printf
While you're at it, you may as well insert this line too:
echo $SHELL
That will reveal if you're getting different shells, per tripleee.
HAHA!!! I finally traced down the problem! Read ahead if interested (leave the page if not).
These are the only command that will translate \u properly:
. ./test.sh ## Sourcing the script, hash-bang = #! /bin/sh
. ./test.bash ## Sourcing the script, hash-bang = #! /bin/bash
./test ## Running the script with no hash-bang
All of the following produce identical results in that they do NOT translate \u:
./test.sh ## Script is run from an interactive shell but in a non-interactive shell
## test.sh has first line: #! /bin/sh
/bin/sh -c "./test.sh" ## Running the script in a non-interactive sh shell
/bin/sh -lc "./test.sh" ## Running the script in a non-interactive, login sh shell
/bin/sh -c ". ./test.sh" ## Sourcing the file in a non-interactive sh shell
/bin/sh -lc ". ./test.sh" ## Sourcing the file in a non-interactive, login sh shell
## test.bash has first line: #! /bin/bash
/bin/bash -c "./test.bash" ## Running the script in a non-interactive bash shell
/bin/bash -lc "./test.bash" ## Running the script in a non-interactive, login bash shell
/bin/bash -c ". ./test.bash" ## Sourcing the file in a non-interactive bash shell
/bin/bash -lc ". ./test.bash" ## Sourcing the file in a non-interactive, login bash shell
## And from ***tripleee*** (thanks btw):
/bin/sh --norc; . ./test.sh ## Sourcing from an interactive sh shell without the ~/.bashrc file read
/bin/bash --norc; . ./test.bash ## Sourcing from an interactive bash shell without the ~/.bashrc file read
The only way to get proper translation is to run the script without a hash-bang... and I finally figured out why! Without a hash-bang my system chooses the default shell, which btw is NOT /bin/bash... it turns out to be /opt/local/bin/bash... two different versions of bash!
Finally, I removed the OSX /bin/bash [v3.2.48(1)] and replaced it with the MacPorts /opt/local/bin/bash [v4.2.10(2)] and now running the script works! It actually solved about 10-15 other problems I've had (like ${var,,}, read sN1 char, complete -EC "echo ' '", and a host of other commands I have scattered throughout my scripts, ~/.bashrc amd ~/.profile). Honestly, I really should have noticed when my scripts using associative arrays suddenly crapped out on me... how stupid can I get!?
I've been using bash v4 for a looong time now, and my Lion upgrade went and down-graded bash back to v3 (get with the program Apple!)... ugh, I feel so ashamed! Everyone still using bash v3, upgrade!! bash v4 is has many, many beautiful upgrades over version 3. Type bash --version to see what version you are running. One advantage is now bash can translate \uHEX into Unicode!
Try removing the space in the first line, I seem to recall that can cause problems. Offhand I'd guess that because of that space, you're not getting bash, but sh.
Glad you solved it. Still, you might be looking for a portable solution.
Assuming you are always using the same formatting string, we can just discard it, and use something like this;
printf () {
# Discard format string
shift
perl -CSD -le '
print map { s/^\\u//; chr(hex($_)) } #ARGV' "$#"
}
Edit to add: You would simply add this function definition at the beginning of your existing script, overriding the builtin printf. Obviously, if you also use printf for other stuff, this special-purpose replacement isn't good enough.
You could rename the function to uprintf or something, still. It merely translates a sequence of hex codes to the corresponding Unicode characters, discarding any \u prefix.