Problem with submitting dependent job in slurm - bash

I'm using the following .sh script to submit some Slurm jobs.
# Set hardware resource requests
nb_partitions=200 # number of distinct partitions to which you'll send sbatch statements
nb_threads=7 # number of CPUs requested per partition
# Make directory in which you'll store results
mkdir results
# Make a separate folder for each partition
for j in $(seq 1 $nb_partitions)
do
mkdir results/partition$j
done
# Copy MainFolder which has code for execution to each partition folder and change partition number stored in each input file to correspond to its number
for i in $(seq 1 $nb_partitions)
do
cp -r MainFolder results/partition$i
sed -i "1s/.*/${i[#]}/g" results/partition$i/MainFolder/partition_nb.txt
done
echo "Files and folders have been copied and created."
# Loop to submit simulations job for each partition
slurmids="" # storage of slurm job ids
for k in $(seq 1 $nb_partitions)
do
cd results/partition$k/MainFolder
# Write parameters to files within the main folder so they can be read into FORTRAN
echo "$nb_partitions" >> nb_partitions.txt
echo "$nb_threads" >> nb_threads.txt
slurmids="$slurmids:$(sbatch --parsable --cpus-per-task=$nb_threads run.sh)"
cd ..
cd ..
cd ..
done
echo "Jobs are now running."
# Submit job to combine files
ID=$(sbatch --dependency=afterok=$slurmids combine_results.sh)
The final line of this code is supposed to submit a batch job that will execute only after all of the other jobs submitted in the loop are done. However, when I run this shell script the job combine_results in the final line of code never runs. The Slurm manager simply says the job is waiting because the dependencies are not satisifed.
Any ideas on what I'm doing wrong here? Thanks!

Related

How to get original location of script used for SLURM job?

I'm starting the SLURM job with script and script must work depending on it's location which is obtained inside of script itself with SCRIPT_LOCATION=$(realpath $0). But SLURM copies script to slurmd folder and starts job from there and it screws up further actions.
Are there any option to get location of script used for slurm job before it has been moved/copied?
Script is located in network shared folder /storage/software_folder/software_name/scripts/this_script.sh and it must to:
get it's own location
return the software_name folder
copy the software_name folder to a local folder /node_folder on node
run another script from copied folder /node_folder/software_name/scripts/launch.sh
My script is
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --partition=my_partition_name
# getting location of software_name
SHARED_PATH=$(dirname $(dirname $(realpath $0)))
# separating the software_name from path
SOFTWARE_NAME=$(basename $SHARED_PATH)
# target location to copy project
LOCAL_SOFTWARE_FOLDER='/node_folder'
# corrected path for target
LOCAL_PATH=$LOCAL_SOFTWARE_FOLDER/$SOFTWARE_NAME
# Copying software folder from network storage to local
cp -r $SHARED_PATH $LOCAL_SOFTWARE_FOLDER
# running the script
sh $LOCAL_PATH/scripts/launch.sh
It runs perfectly, when I run it on the node itself (without using SLURM) via: sh /storage/software/scripts/this_script.sh.
In case of running it with SLURM as
sbatch /storage/software/scripts/this_script.sh it is assigned to one of nodes, but:
before run it is copied to /var/spool/slurmd/job_number/slurm_script and it screws everything up since $(dirname $(dirname $(realpath $0))) returns /var/spool/slurmd
Is it possible to get original location (/storage/software_folder/software_name/) inside of script when it is started with SLURM?
P.S. All machines are running Fedora 30 (x64)
UPDATE 1
There was a suggestion to run as sbatch -D /storage/software_folder/software_name ./scripts/this_script.sh and use the SHARED_PATH="${SLURM_SUBMIT_DIR}" inside of script itself.
But it raise the error sbatch: error: Unable to open file ./scripts/this_script.sh.
Also, I tried to use absolute paths:
sbatch -D /storage/software_folder/software_name /storage/software_folder/software_name/scripts/this_script.sh. It tries to run, but:
in such case it uses specified folder for creating output file only
software still doesn't want to run
attempt to use echo "${SLURM_SUBMIT_DIR}" inside of script prints /home/username_who_started_script instead of /storage/software_folder/software_name
Any other suggestions?
UPDATE 2:
Also tried to use #SBATCH --chdir=/storage/software_folder/software_name inside of script, but in such case echo "${SLURM_SUBMIT_DIR}" returns /home/username_who_started_scriptor / (if run as root)
UPDATE 3
Approach with ${SLURM_SUBMIT_DIR} worked only if task is ran as:
cd /storage/software_folder/software_name
sbatch ./scripts/this_script.sh
But it doesn't seem to be a proper solution. Are there any other ways?
SOLUTION
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --partition=my_partition_name
# check if script is started via SLURM or bash
# if with SLURM: there variable '$SLURM_JOB_ID' will exist
# `if [ -n $SLURM_JOB_ID ]` checks if $SLURM_JOB_ID is not an empty string
if [ -n $SLURM_JOB_ID ]; then
# check the original location through scontrol and $SLURM_JOB_ID
SCRIPT_PATH=$(scontrol show job $SLURM_JOBID | awk -F= '/Command=/{print $2}')
else
# otherwise: started with bash. Get the real location.
SCRIPT_PATH=$(realpath $0)
fi
# getting location of software_name
SHARED_PATH=$(dirname $(dirname $(SCRIPT_PATH)))
# separating the software_name from path
SOFTWARE_NAME=$(basename $SHARED_PATH)
# target location to copy project
LOCAL_SOFTWARE_FOLDER='/node_folder'
# corrected path for target
LOCAL_PATH=$LOCAL_SOFTWARE_FOLDER/$SOFTWARE_NAME
# Copying software folder from network storage to local
cp -r $SHARED_PATH $LOCAL_SOFTWARE_FOLDER
# running the script
sh $LOCAL_PATH/scripts/launch.sh
You can get the initial (i.e. at submit time) location of the submission script from scontrol like this:
scontrol show job $SLURM_JOBID | awk -F= '/Command=/{print $2}'
So you can replace the realpath $0 part with the above. This will only work within a Slurm allocation of course. So if you want the script to work in any situation, you will need some logic like:
if [ -n $SLURM_JOB_ID ] ; then
THEPATH=$(scontrol show job $SLURM_JOBID | awk -F= '/Command=/{print $2}')
else
THEPATH=$(realpath $0)
fi
and then proceed with
SHARED_PATH=$(dirname $(dirname "${THEPATH}"))
I had to do the same in an array job, the accepted answer from #damienfrancois works well for all jobs except the jobid which is same as ArrayJobId. Just piping awk command to head command would do the trick
scontrol show job $SLURM_JOBID | awk -F= '/Command=/{print $2}' | head -n 1
In script, get SHARED_PATH as SHARED_PATH="${SLURM_SUBMIT_DIR}"
Submit script as sbatch -D /storage/software ./scripts/this_script.sh
See here.
From referred page:
-D
Set the working directory of the batch script to directory before it
is executed. The path can be specified as full path or relative path
to the directory where the command is executed.
SLURM_SUBMIT_DIR
The directory from which sbatch was invoked or, if applicable, the directory specified by the -D, --chdir option.
P.S. Above is from Version 19.05 doc.
While looking in to archive, referring Ver. 18.x (esp. 18.08), it doesn't mention the same. See this
SLURM_SUBMIT_DIR.
The directory from which sbatch was invoked.

whether a shell script can be executed if another instance of the same script is already running

I have a shell script which usually runs nearly 10 mins for a single run,but i need to know if another request for running the script comes while a instance of the script is running already, whether new request need to wait for existing instance to compplete or a new instance will be started.
I need a new instance must be started whenever a request is available for the same script.
How to do it...
The shell script is a polling script which looks for a file in a directory and execute the file.The execution of the file takes nearly 10 min or more.But during execution if a new file arrives, it also has to be executed simultaneously.
the shell script is below, and how to modify it to execute multiple requests..
#!/bin/bash
while [ 1 ]; do
newfiles=`find /afs/rch/usr8/fsptools/WWW/cgi-bin/upload/ -newer /afs/rch/usr$
touch /afs/rch/usr8/fsptools/WWW/cgi-bin/upload/.my_marker
if [ -n "$newfiles" ]; then
echo "found files $newfiles"
name2=`ls /afs/rch/usr8/fsptools/WWW/cgi-bin/upload/ -Art |tail -n 2 |head $
echo " $name2 "
mkdir -p -m 0755 /afs/rch/usr8/fsptools/WWW/dumpspace/$name2
name1="/afs/rch/usr8/fsptools/WWW/dumpspace/fipsdumputils/fipsdumputil -e -$
$name1
touch /afs/rch/usr8/fsptools/WWW/dumpspace/tempfiles/$name2
fi
sleep 5
done
When writing scripts like the one you describe, I take one of two approaches.
First, you can use a pid file to indicate that a second copy should not run. For example:
#!/bin/sh
pidfile=/var/run/$(0##*/).pid
# remove pid if we exit normally or are terminated
trap "rm -f $pidfile" 0 1 3 15
# Write the pid as a symlink
if ! ln -s "pid=$$" "$pidfile"; then
echo "Already running. Exiting." >&2
exit 0
fi
# Do your stuff
I like using symlinks to store pid because writing a symlink is an atomic operation; two processes can't conflict with each other. You don't even need to check for the existence of the pid symlink, because a failure of ln clearly indicates that a pid cannot be set. That's either a permission or path problem, or it's due to the symlink already being there.
Second option is to make it possible .. nay, preferable .. not to block additional instances, and instead configure whatever it is that this script does to permit multiple servers to run at the same time on different queue entries. "Single-queue-single-server" is never as good as "single-queue-multi-server". Since you haven't included code in your question, I have no way to know whether this approach would be useful for you, but here's some explanatory meta bash:
#!/usr/bin/env bash
workdir=/var/tmp # Set a better $workdir than this.
a=( $(get_list_of_queue_ids) ) # A command? A function? Up to you.
for qid in "${a[#]}"; do
# Set a "lock" for this item .. or don't, and move on.
if ! ln -s "pid=$$" $workdir/$qid.working; then
continue
fi
# Do your stuff with just this $qid.
...
# And finally, clean up after ourselves
remove_qid_from_queue $qid
rm $workdir/$qid.working
done
The effect of this is to transfer the idea of "one at a time" from the handler to the data. If you have a multi-CPU system, you probably have enough capacity to handle multiple queue entries at the same time.
ghoti's answer shows some helpful techniques, if modifying the script is an option.
Generally speaking, for an existing script:
Unless you know with certainty that:
the script has no side effects other than to output to the terminal or to write to files with shell-instance specific names (such as incorporating $$, the current shell's PID, into filenames) or some other instance-specific location,
OR that the script was explicitly designed for parallel execution,
I would assume that you cannot safely run multiple copies of the script simultaneously.
It is not reasonable to expect the average shell script to be designed for concurrent use.
From the viewpoint of the operating system, several processes may of course execute the same program in parallel. No need to worry about this.
However, it is conceivable, that a (careless) programmer wrote the program in such a way that it produces incorrect results, when two copies are executed in parallel.

How to know when PBS batch jobs are complete

I have a BASH script that submits multiple serial jobs to the PBS queueing system. Once the jobs are submitted the script ends. The jobs then run on a cluster and when they are all finished I can move on to the next step. A typical workflow might involve several of these steps.
My question:
Is there a way for my script not to exit upon completion of the submission, but rather to sleep until ALL jobs submitted by that script have completed on the cluster, only then exiting?
You are trying to establish a workflow, correct? The best way to do what you're attempting to accomplish would be to use job dependencies. Essentially, what you are trying to do is submit X number of jobs, and then submit more jobs that depend on the first set of jobs, and you can do this with job dependencies. There are different ways to do dependencies that you can read about in the previous link, but here's an example of submitting 3 jobs and then submitting 3 more that won't execute until after the first 3 have exited.
#first batch
jobid1=`qsub ...`
jobid2=`qsub ...`
jobid3=`qsub ...`
#next batch
depend_str="-W after:${jobid1} -W after:${jobid2} -W after:${jobid3}"
qsub ... $depend_str
qsub ... $depend_str
qsub ... $depend_str
One way to do this would be using GNU Parallel command 'sem'
I learnt about this doing queue stuff as well. It acts as a timer allowing commands to be executed after exiting etc.
Edit: I know the example here is really basic but there is a lot that can be achieved running tasks using parallel --sem or even just parallel in general. Have a look at the tutorial, I'm certain you will be able to find a relevant example that will help.
There is a great tutorial here
An example from a tutorial:
sem 'sleep 1; echo The first finished' &&
echo The first is now running in the background &&
sem 'sleep 1; echo The second finished' &&
echo The second is now running in the background
sem --wait
Output:
The first is now running in the background
The first finished
The second is now running in the background
The second finished
See Man Page
To actually check if a job is done, we need to use qstat and the job ID to get the job status and then grep the status for a status code. As long as your username or job name are not "C", the following should work:
#!/bin/bash
# SECTION 1: Launch all jobs and store their job IDs in a variable
myJobs="job1.qsub job2.qsub job3.qsub" # Your job names here
numJobs=$(echo "$myJobs" | wc -w) # Count the jobs
myJobIDs="" # Initialize an empty list of job IDs
for job in $myJobs; do
jobID_full=$(qsub $job)
# jobID_full will look like "12345.machinename", so use sed
# to get just the numbers
jobID=$(echo "$jobID_full" | sed -e 's|\([0-9]*\).*|\1|')
myJobIDs="$myJobIDs $jobID" # Add this job ID to our list
done
# SECTION 2: Check the status of each job, and exit while loop only
# if they are all complete
numDone=0 # Initialize so that loop starts
while [ $numDone -lt $numJobs ]; do # Less-than operator
numDone=0 # Zero since we will re-count each time
for jobID in $myJobIDs; do # Loop through each job ID
# The following if-statement ONLY works if qstat won't return
# the string ' C ' (a C surrounded by two spaces) in any
# situation besides a completed job. I.e. if your username
# or jobname is 'C' then this won't work!
# Could add a check for error (grep -q ' E ') too if desired
if qstat $jobID | grep -q ' C '
then
(( numDone++ ))
else
echo $numDone jobs completed out of $numJobs
sleep 1
fi
done
done
echo all jobs complete

cront not working with hadoop command in shell script

I'm trying to schedule a cronjob using crontab to execute a shell script which executes a list of hadoop commands sequentially, but when i look at the hadoop folder the folders are not created or dropped. The hadoop connectivity on our cluster is pretty slow. so these hadoop command might take sometime to execute due to number of retries.
Cron expression
*/5 * * * * sh /test1/a/bin/ice.sh >> /test1/a/run.log
shell script
#!/bin/sh
if [ $# == 1 ]
then
TODAY=$1
else
TODAY=`/bin/date +%m%d%Y%H%M%S`
fi
# define seed folder here
#filelist = "ls /test1/a/seeds/"
#for file in $filelist
for file in `/bin/ls /test1/a/seeds/`
do
echo $file
echo $TODAY
INBOUND="hadoop fs -put /test1/a/seeds/$file /apps/hdmi-set/inbound/$file.$TODAY/$file"
echo $INBOUND
$INBOUND
SEEDDONE="hadoop fs -put /test1/a/seedDone /apps/hdmi-set/inbound/$file.$TODAY/seedDone"
echo $SEEDDONE
$SEEDDONE
done
echo "hadoop Inbound folders created for job1 ..."
Since there are no output that has been captured that could be used to debug the output, I can only speculate.
But from my past experience, one of the common reason hadoop jobs fail when they are spawned through scripts is that HADOOP_HOME is not available when these commands are executed.
Usually that is not the case when working directly from the terminal. Try adding the following to both ".bashrc" and ".bash_profile" or ".profile":
export HADOOP_HOME=/usr/lib/hadoop
You may have to change the path based on your specific installation.
And yes as comment says, don't just redirect standard output but error too in the file.

How to process the output folder created by one script

I have a script (first.sh) which creates a output folder at a particular location.Now a second script (second.sh) need the location of this output folder as it need to process the data into it.
Keep in mind that the first.sh script will generate the output folder as per the user needs.(as per path set by the user).
Now i need the second.sh script to find the path of the output folder generated by first.sh script and then it should enter into it for processing some data?
I need the logic for the above problem. Also remember that the path for output folder always changes and the second.sh script should be able to find the exact path and output folder.
first.sh /add/sub/output_folder (user decides the output path)
second.sh
It should find the "/add/sub/output_folder" location
Hope it helps.Plz help me.
More details of what the both scripts do would be helpfull.
I assume you can not/ don't lunch second.sh form first.sh because this would be the easiest option.
One way of approaching this would be to output the folder location to a property file by first.sh that second.sh has access to.
Depending on the manner in which you are calling your code, you can probably use $PPID to generate a primitive means by which script 1 and script 2 can exchange information.
$PPID is the parent-process-id of your script. In the following example, both scripta.sh and scriptb.sh see the same value:
scripta.sh:
#!/bin/sh
datadir=/tmp/$PPID.tmp
mkdir -p $datadir
sleep $(( RANDOM % 60 ))
echo Some data from $PPID/$$ on $(tty) >$datadir/testfile.txt_tmp
mv $datadir/testfile.txt_tmp $datadir/testfile.txt
scriptb.sh
#!/bin/sh
datadir=/tmp/$PPID.tmp
timeout=60
while (( (i=i+1) < timeout )); do
cat $datadir/testfile.txt 2>&- && break
sleep 1
done
This example will hold for up to 60 seconds for the data to appear and will work no matter how many users are active.
You should test this by running the following on two or more separate terminals.
scripta.sh &
scriptb.sh &
You'll need another script (or overnight process) to clean up the datadir directories.
BTW, I do not recommend /tmp as a holding zone for these files. Create a buffer directory somewhere else where there is lots of space.
If you're allowed to call second.sh from first.sh you can do
first.sh $YOUR_OUTPUT_PATH
You can get access to that path by using $1, e.g.
echo $1

Resources