I am suppose to submit a job in parallel, but I am new to this and it is not really working. This is what I have until now:
#!/bin/bash
#PBS -q s48
#PBS -l walltime=2:00:00
#PBS -l nodes=1:ppn=1
#PBS -o localhost:${PBS_O_WORKDIR}/
#PBS -e localhost:${PBS_O_WORKDIR}/
cd $PBS_O_WORKDIR
PRM=$(awk "NR==$PBS_ARRAYID {print;exit}" 89418.txt)
echo "Start"
./Hysto $PRM
echo "Done"
So 89418.txt is the file I read from and Hysto is the program I want to run. So I submit it using qsub -t 1-10 run.sh (run.sh is the name of the file I submit) but it is not working. What am I doing wrong? Thank you!
Related
I want to submit a work to our school's cluster, but it seems that something went error with the environment variable. Here is a test PBS file
#PBS -N test
#PBS -l nodes=1:ppn=1
#PBS -q ser
#PBS -V
#PBS -S /bin/bash
echo ${PBS_NODEFILE}
echo ${PBS_JOBID}
env
I can see the two variable in "env" but "echo" does not print this two variables, and I also can't use these variables in my script. For example "cat ${PBS_NODEFILE} > nodes" will output nothing. Why is this happen???
Well actually I found later that not only this two variables but all variables that is not passed from the current environment(by using -V), such as PBS_O_SHELL , cannot be used in the script. And the other environmental variables such as PATH can be printed normally
This question already has answers here:
How to avoid heredoc expanding variables? [duplicate]
(2 answers)
Closed 5 years ago.
I'm trying to create a script that creates scripts (and do other things). My problem is that these scripts created contain environment variables, and when actually running my script, they do not appear in the script.
#!/bin/bash
for i in {001..116}
do
rm job
cat > job <<!
#PBS -S /bin/bash
#PBS -N $i-Fe63S
#PBS -j oe
#PBS -q default
#PBS -l nodes=1:ppn=24
#PBS -l walltime=48:00:00
#PBS -V
cd $PBS_O_WORKDIR
/opt/openmpi/bin/mpirun -np 24 /opt/vasp/5.4/vasp.5.4.1/bin/vasp_std > log
!
mkdir $i
cp job $i/
done
In the resulting "job" file created, it attempts, but fails to find the $PBS_O_WORKDIR, so the resulting script is
#PBS -S /bin/bash
#PBS -N 116-Fe63S
#PBS -j oe
#PBS -q default
#PBS -l nodes=1:ppn=24
#PBS -l walltime=48:00:00
#PBS -V
cd
/opt/openmpi/bin/mpirun -np 24 /opt/vasp/5.4/vasp.5.4.1/bin/vasp_std > log
How do i modify the script, so the line in the result script is written "cd $PBS_O_WORKDIR" rather than "cd"?
Go through the variables/expressions in your here document and escape the ones you don't want expanded at the point when you write the file.
In this case you want $i expanded now, and $PBS_O_WORKDIR expanded later, so you escape the latter:
#!/bin/bash
for i in {001..116}
do
rm job
cat > job <<!
#PBS -S /bin/bash
#PBS -N $i-Fe63S
#PBS -j oe
#PBS -q default
#PBS -l nodes=1:ppn=24
#PBS -l walltime=48:00:00
#PBS -V
cd \$PBS_O_WORKDIR # <- Escaped
/opt/openmpi/bin/mpirun -np 24 /opt/vasp/5.4/vasp.5.4.1/bin/vasp_std > log
!
mkdir $i
cp job $i/
done
Quote your heredoc sigil to prevent expansions from taking place inside the contents:
cat >job <<'!'
...or...
cat >job <<\!
...not...
cat >job <<!
I can submit a job to PBS using both approaches of Non-interactive Batch Jobs and/or Interactive Batch Jobs. However, I need to use the pbs commands in a function. In other world I need a structure like this:
#!/bin/sh
pbs_setup () {
#PBS -l $1
#PBS -N $2
#PBS -q normal
#PBS -A $USER
#PBS -m ae
#PBS -M $USER"#gmail.com"
#PBS -q normal
#PBS -l nodes=1:ppn=8
#PBS
}
pbs_setup "walltime=6:00:00" "step3";
echo " "
echo "Job started
echo " "
echo "Job Ended
When I am submitting this job it is not working.
In fact my final goal is separating the commands of job from the main body of code. So when HPC will be changed I just edit a shell file which is included this function instead of editing all the shells. I appreciate if you give me some suggestions.
You could create your custom submission command that collects the job options and sends them as command line parameters to actual qsub call.
Here is a rather basic example of this. In real usage I would add more sophisticated parameter handling tailored to the type of jobs, and more consistent with qsub interface. Also handling interactive jobs needs additional work.
submit.sh
#!/bin/bash
walltime="${2:-06:00:00}"
name="${3:-step3}"
queue="normal"
acct="$USER"
mailevents="ae"
mailaddress="$USER#gmail.com"
resources="nodes=1:ppn=8"
if [ $# -lt 1 ] ; then
echo "Usage: submit.sh script [walltime [name]]" >
exit 1
fi
script="$1"
qsub -l "$walltime" -N "$name" -q "$queue" -A "$acct" \
-m "$mailevents" -M "$mailaddress" -l "$resources" "$script"
script.sh
#!/bin/bash
echo " "
echo "Job started"
echo " "
echo "Job Ended"
This is supposed to be used as
submit.sh script.sh 06:00:00 step3
The issue with that job script is that the #PBS lines need to be first non-comment lines in the script file.
In my attempt to do this same concept, I used the same type of function you have, but cat the results and the actual commands into another file. i.e. An overarching script creates the 'job' script. You can put the HPC requirements in a separate file, then source it from the creation script.
Edit in response to comment:
e.g.
To specify a path to start the job from:
#PBS - d init_path
"working directory path to be used for the job, PBS_O_INITDIR"
Or
#PBS -D root_path
"root directory to be used for the job, PBS_O_ROOTDIR."
Or
#PBS -w working_path
"If the -w option is not specified, the default working directory is the current directory. This option sets the environment variable PBS_O_WORKDIR."
So the default PBS_O_WORKDIR is the current directory you are IN when you call the script to submit the script to qsub.
Thus, if you set the specific options (d, D, w) for paths relative to the actual script running environment, you'll be able to use the paths you intend.
For specifics including default values of these and other options, you can check out the man page for your app. If using the Torque version of the PBS system, it's available at linux.die.net - qsub
We create a torque pbs file "testpbs" as follows:
#!/bin/sh
#PBS -N T1272_flt
#PBS -q batch
#PBS -l nodes=1:ppn=1
#PBS -o /data/software/torque-4.2.6.1/testpbs.sh.out
#PBS -e /data/software/torque-4.2.6.1/testpbs.sh.err
sleep 20
Then submitted the file testbps.
qsub testpbs
We got error messages:
more testpbs.sh.err
/var/spool/torque/mom_priv/jobs/8.centos64.SC: line 9: sleep: command
not found
However, we ran sleep 20 in command line. No error occurs.
$sleep 20
Thanks in advance.
We ran echo $PATH in shell and got the following:
echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/X11R6/bin:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/bin:/data/software/cufflinks-2.0.2.Linux_x86_64:/home/amin/bin/blast-2.2.19:/root/bin:/home/amin/bin
We use qstat -f jobid to review the details of this job.
PBS_O_LOGNAME=amin,
PBS_O_PATH= /usr/lib64/qt-3.3/bin: /usr/local/sbin: /usr/local/bin:
/sbin: /bin: /usr/sbin: /usr/bin: /sbin:/bin: /usr/sbin: /usr/bin:
/usr/X11R6/bin: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/bin:
/data/software/cufflinks-2.0.2.Linux_x86_64:
/home/amin/bin/blast-2.2.19: /root/bin: /home/aimin/bin,
PBS_O_MAIL=/var/spool/mail/root,
PBS_O_SHELL=/bin/bash,
PBS_O_LANG=en_US.UTF-8,
PBS_O_WORKDIR=/data/software/torque-4.2.6.1,
PBS_O_HOST=centos64,
PBS_O_SERVER=centos64
Thank larsks's great help. The following works:
#!/bin/sh
#PBS -N T1272_flt
#PBS -q batch
#PBS -l nodes=1:ppn=1
#PBS -o /data/software/torque-4.2.6.1/testpbs.sh.out
#PBS -e /data/software/torque-4.2.6.1/testpbs.sh.err
export PATH=$PBS_O_PATH
sleep 20
Try replacing sleep with the full path to the command (possibly /usr/bin/sleep) and see if that changes the behavior. If it does, then your script, when run under Torque, simply has a different (or empty) $PATH variable.
You can either (a) continue to use explicit paths, or (b) set $PATH explicitly in your script, e.g:
PATH=/bin:/usr/bin:/usr/local/bin
Is there a way to directly pass parameters to a .pbs script before submitting a job? I need to loop over a list of files indicated by different numbers and apply a script to analyze each file.
The best I've been able to come up with is the following:
#!/bin/sh
for ((i= 1; i<= 10; i++))
do
export FILENUM=$i
qsub pass_test.pbs
done
where pass_test.pbs is the following script:
#!/bin/sh
#PBS -V
#PBS -S /bin/sh
#PBS -N pass_test
#PBS -l nodes=1:ppn=1,walltime=00:02:00
#PBS -M XXXXXX#XXX.edu
cd /scratch/XXXXXX/pass_test
./run_test $FILENUM
But this feels a bit wonky. Particularly, I want to avoid having to create an environment variable to handle this.
The qsub utility can read the script from the standard input, so by using a here document you can create scripts on the fly, dynamically:
#!/bin/sh
for i in `seq 1 10`
do
cat <<EOS | qsub -
#!/bin/sh
#PBS -V
#PBS -S /bin/sh
#PBS -N pass_test
#PBS -l nodes=1:ppn=1,walltime=00:02:00
#PBS -M XXXXXX#XXX.edu
cd /scratch/XXXXXX/pass_test
./run_test $i
EOS
done
Personally, I would use a more compact version:
#!/bin/sh
for i in `seq 1 10`
do
cat <<EOS | qsub -V -S /bin/sh -N pass_test -l nodes=1:ppn=1,walltime=00:02:00 -M XXXXXX#XXX.edu -
cd /scratch/XXXXXX/pass_test
./run_test $i
EOS
done
You can use the -F option, as described here:
-F
Specifies the arguments that will be passed to the job script when the script is launched. The accepted syntax is:
qsub -F "myarg1 myarg2 myarg3=myarg3value" myscript2.sh
Note: Quotation marks are required. qsub will fail with an error
message if the argument following -F is not a quoted value. The
pbs_mom server will pass the quoted value as arguments to the job
script when it launches the script.
See also this answer
If you just need to pass numbers and run a list of jobs with the same command except the input file number, it's better to use a job array instead of a for loop as job array would have less burden on the job scheduler.
To run, you specify the file number with PBS_ARRAYID like this in the pbs file:
./run_test ${PBS_ARRAYID}
And to invoke it, on command line, type:
qsub -t 1-10 pass_test.pbs
where you can specify what array id to use after -t option