I launch a pbs script once others are completed. For that I use this commands:
$ job1=$(qsub job1.pbs)
$ jobN=$(qsub jobN.pbs)
$ qsub -W depend=afterok:$job1:$jobN join.pbs
This works, in most cases. However if I run the joining script when job1 and jobN are already finished, it will go idle indefinitely as it is waiting for the already-finished-jobs to finish. That sounds insane, but this is what happens. If I run qstat I can clearly see that my joining job is being held ('H')
$ qstat -u me
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
1990613 me workq join.pbs -- 1 1 -- -- H --
However if at least one of the jobs is still running, while the other is already finished, then the joining script will not go idle and will finish.
So what are the solutions to deal with jobs that are already over? We clearly need this job to finish.
When the join job starts, the server still needs to know about the depended-upon jobs; if either of those is gone from qstat, then you'll need to increase keep_completed in qmgr. Otherwise, when the join job is ready to run, the dependency will never get satisfied, and the hold will never get released.
To check: $ qmgr -c 'print server keep_completed'
To add/modify: $ qmgr -c 'set server keep_completed=300'
(I also believe you can set keep_completed on queues.)
Related
How do I get the slurm job status (e.g. COMPLETED, FAILED, TIMEOUT, ...) on job completion (within the submission script)?
I.e. I want to write to separately keep track of jobs which are timed out / failed.
Currently I work with the exit code, however jobs which TIMEOUT also get exit code 0.
For future reference, here is how I finally do it.
To retrieve the jobid at the beginning of the job and write some information (e.g. "${SLURM_JOB_ID} ${PWD}") to a summary file.
Then process this file and use something like sacct -X -n -o State --j ${jid} to get the job status.
I'm running a numerical model which parameters are in a "parameter.input" file. I use sbatch to submit multiple iterations of the model, with one parameter in the parameter file changing every time. Here is the loop I use:
#!/bin/bash -l
for a in {01..30}
do
sed -i "s/control_[0-9][0-9]/control_${a}/g" parameter.input
sbatch --time=21-00:00:00 run_model.sh
sleep 60
done
The sed line changes a parameter in the parameter file. The
run_model.sh file runs the model.
The problem: depending on the resources available, a job might run immediately or stay pending for a few hours. With my default loop, if 60 seconds is not enough time to find resources for job n to run, the parameter file will be modified while job n is pending, meaning job n will run with the wrong parameters. (and I can't wait for job n to complete before submitting job n+1 because each job takes several days to complete)
How can I force batch to wait to submit job n+1 until job n is running?
I am not sure how to create an until loop that would grab the status of job n and wait until it changes to 'running' before submitting job n+1. I have experimented with a few things, but the server I use also hosts another 150 people's jobs, and I'm afraid too much experimenting might create some issues...
Use the following to grab the last submitted job's ID and its status, and wait until it isn't pending anymore to start the next job:
sentence=$(sbatch --time=21-00:00:00 run_model.sh) # get the output from sbatch
stringarray=($sentence) # separate the output in words
jobid=(${stringarray[3]}) # isolate the job ID
sentence="$(squeue -j $jobid)" # read job's slurm status
stringarray=($sentence)
jobstatus=(${stringarray[12]}) # isolate the status of job number jobid
Check that the job status is 'running' before submitting the next job with:
if [ "$jobstatus" = "R" ];then
# insert here relevant code to run next job
fi
You can insert that last snippet in an until loop that checks the job's status every few seconds.
After using the Slurm cluster manager to sbatch a job with multiple processes, is there a way to know the status (running or finishing) of each process? Can it be implemented in a python script?
Just use the command sacct that comes with Slurm.
Given this code (my.sh):
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=2
srun -n1 sleep 10 &
srun -n1 sleep 3
wait
I run it:
sbatch my.sh
And then check on it with sacct:
sacct
Which gives me per-step info:
JobID JobName Partition Account AllocCPUS State ExitCode
---------- ---------- ---------- ---------- ---------- ---------- --------
8021 my.sbatch CLUSTER me 2 RUNNING 0:0
8021.0 sleep me 1 RUNNING 0:0
8021.1 sleep me 1 COMPLETED 0:0
sacct has a lot of options to customize its output. For example,
sacct --format='JobID%6,State'
Will just give you the IDs (up to 6 characters) and the current state of jobs:
JobID State
------ ----------
8021 RUNNING
8021.0 RUNNING
8021.1 COMPLETED
If the processes you mention are distincts steps, then sacct can give you the information as explained by #Christopher Bottoms.
But if the processes are different tasks in a single step, then you can use this script that uses parallel SSH to run 'ps' commands on the compute nodes and offer a summarised view, as #Tom de Geus suggests.
I run a lot of small computing jobs in remote cluster where job submission is managed by PBS. Normally in a PBS (bash) script I specify the queue that I would like to submit the job through the command
#PBS -q <queue_name>
The job queue that I need to chose depends on the load on a specific queue. Every time before I submit a job, I analyze this through the command on terminal
qstat -q
which provides me an output which looks like as follows
Queue Memory CPU Time Walltime Node Run Que Lm State
---------------- ------ -------- -------- ---- --- --- -- -----
queue1 -- -- 03:00:00 -- 0 2 -- E R
queue2 -- -- 06:00:00 -- 8 6 -- E R
I would like to automate the queue selection by the job script based on two constraints
The queue selected must have a walltime more than the job time specified. The job time is specified through command #PBS -l walltime=02:30:00.
The queue must have the least no. of jobs in Que as shown in the above output.
I'm having trouble in identifying which tools that I need to use in terminal to help me automate the queue selection
It is possible that you could wrap your qsub submission in another script which would run qstat -q, parse the output, and then select the queue based on the walltime requested and how many active jobs are in each queue. The script could then submit the job and add -q <name of desired queue> to the end of the qsub command.
However, it seems that you are manually trying to do some of what a scheduler - with appropriate policies - does for you. Why do you need to dynamically switch queues? A better setup would be for the queues to essentially categorize the jobs - like you are already doing with walltime - and then allowing the scheduler to run the jobs appropriately. Any setup where a user needs to carefully select the queue seems a little suspect to me.
I have a question.
I want to run more instance of same job in parallel from within a script: I have a loop in which I invoke jobs with dsjob and without option "-wait" and "-jobstatus".
I want that jobs completed before script termination, but I don't know how to verify if job instance terminated.
I though to use wait command but it is not appropriate.
Thanks in advance
First,you should assure job compile option "Allow Multiple Instance" choose.
Second:
#!/bin/bash
. /home/dsadm/.bash_profile
INVOCATION=(1 2 3 4 5)
cd $DSHOME/bin
for id in ${INVOCATION[#]}
do
./dsjob -run -mode NORMAL -wait test demo.$id
done
project -- test
job -- demo
$id -- invocation id
the two line in shell scipt:guarantee the environment path can work.
Run the jobs like you say without the -wait, and then loop around running dsjob -jobinfo and parse the output for a job status of 1 or 2. When all jobs return this status, they are all finished.
You might find, though, that you check the status of the job before it actually starts running and you might pick up an old status. You might be able to fix this by first resetting the job instance and waiting for a status of "Not running", prior to running the job.
Invoke the jobs in loop without wait or job-status option
after your loop , check the jobs status by dsjob command
Example - dsjob -jobinfo projectname jobname.invocationid
you can code one more loop for this also and use sleep command under that
write yours further logic as per status of the jobs
but its good to create Job Sequence to invoke this multi-instance job simultaneously with the help of different invoaction-ids
create a sequence job if these are in same process
create different sequences or directly create different scripts to trigger these jobs simultaneously with invocation- ids and schedule in same time.
Best option create a standard generalized script where each thing will be getting created or getting value as per input command line parameters
Example - log files on the basis of jobname + invocation-id
then schedule the same script for different parameters or invocations .