I'm trying to schedule a cronjob using crontab to execute a shell script which executes a list of hadoop commands sequentially, but when i look at the hadoop folder the folders are not created or dropped. The hadoop connectivity on our cluster is pretty slow. so these hadoop command might take sometime to execute due to number of retries.
Cron expression
*/5 * * * * sh /test1/a/bin/ice.sh >> /test1/a/run.log
shell script
#!/bin/sh
if [ $# == 1 ]
then
TODAY=$1
else
TODAY=`/bin/date +%m%d%Y%H%M%S`
fi
# define seed folder here
#filelist = "ls /test1/a/seeds/"
#for file in $filelist
for file in `/bin/ls /test1/a/seeds/`
do
echo $file
echo $TODAY
INBOUND="hadoop fs -put /test1/a/seeds/$file /apps/hdmi-set/inbound/$file.$TODAY/$file"
echo $INBOUND
$INBOUND
SEEDDONE="hadoop fs -put /test1/a/seedDone /apps/hdmi-set/inbound/$file.$TODAY/seedDone"
echo $SEEDDONE
$SEEDDONE
done
echo "hadoop Inbound folders created for job1 ..."
Since there are no output that has been captured that could be used to debug the output, I can only speculate.
But from my past experience, one of the common reason hadoop jobs fail when they are spawned through scripts is that HADOOP_HOME is not available when these commands are executed.
Usually that is not the case when working directly from the terminal. Try adding the following to both ".bashrc" and ".bash_profile" or ".profile":
export HADOOP_HOME=/usr/lib/hadoop
You may have to change the path based on your specific installation.
And yes as comment says, don't just redirect standard output but error too in the file.
Related
I'm setting a cron job that is a bash script containing the below:
#!/bin/bash
NUM_CONTAINERS=$(docker ps -q | wc -l)
if [ $NUM_CONTAINERS -lt 40 ]
then
echo "Time: $(date). Restart containers."
cd /opt
pwd
sudo docker kill $(docker ps -q)
docker-compose up -d
echo "Completed."
else
echo Nothing to do
fi
The output is appended to a log file:
>> cron.log
However the output in the cron file only shows:
Time: Sun Aug 15 10:50:01 UTC 2021. Restart containers.
/opt
Completed.
Both command do not seem to execute as I don't see any change in my containers either.
These 2 non working commands work well in a standalone .sh script without condition though.
What am I doing wrong?
User running the cron has sudo privileges, and we can see the second echo printing.
Lots of times, things that work outside of cron don't work within cron because the environment is not set up in the same way.
You should generally capture standard output and error, to see if something going wrong.
For example, use >> cron.log 2>&1 in your crontab file, this will capture both.
There's at least the possibility that docker is not in your path or, even if it is, the docker commands are not working for some other reason (that you're not seeing since you only capture standard output).
Capturing standard error should help out with that, if it is indeed the issue.
As an aside, I tend to use full path names inside cron scripts, or set up very limited environments at the start to ensure everything works correctly (once I've established why it's not working correctly).
I'm starting the SLURM job with script and script must work depending on it's location which is obtained inside of script itself with SCRIPT_LOCATION=$(realpath $0). But SLURM copies script to slurmd folder and starts job from there and it screws up further actions.
Are there any option to get location of script used for slurm job before it has been moved/copied?
Script is located in network shared folder /storage/software_folder/software_name/scripts/this_script.sh and it must to:
get it's own location
return the software_name folder
copy the software_name folder to a local folder /node_folder on node
run another script from copied folder /node_folder/software_name/scripts/launch.sh
My script is
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --partition=my_partition_name
# getting location of software_name
SHARED_PATH=$(dirname $(dirname $(realpath $0)))
# separating the software_name from path
SOFTWARE_NAME=$(basename $SHARED_PATH)
# target location to copy project
LOCAL_SOFTWARE_FOLDER='/node_folder'
# corrected path for target
LOCAL_PATH=$LOCAL_SOFTWARE_FOLDER/$SOFTWARE_NAME
# Copying software folder from network storage to local
cp -r $SHARED_PATH $LOCAL_SOFTWARE_FOLDER
# running the script
sh $LOCAL_PATH/scripts/launch.sh
It runs perfectly, when I run it on the node itself (without using SLURM) via: sh /storage/software/scripts/this_script.sh.
In case of running it with SLURM as
sbatch /storage/software/scripts/this_script.sh it is assigned to one of nodes, but:
before run it is copied to /var/spool/slurmd/job_number/slurm_script and it screws everything up since $(dirname $(dirname $(realpath $0))) returns /var/spool/slurmd
Is it possible to get original location (/storage/software_folder/software_name/) inside of script when it is started with SLURM?
P.S. All machines are running Fedora 30 (x64)
UPDATE 1
There was a suggestion to run as sbatch -D /storage/software_folder/software_name ./scripts/this_script.sh and use the SHARED_PATH="${SLURM_SUBMIT_DIR}" inside of script itself.
But it raise the error sbatch: error: Unable to open file ./scripts/this_script.sh.
Also, I tried to use absolute paths:
sbatch -D /storage/software_folder/software_name /storage/software_folder/software_name/scripts/this_script.sh. It tries to run, but:
in such case it uses specified folder for creating output file only
software still doesn't want to run
attempt to use echo "${SLURM_SUBMIT_DIR}" inside of script prints /home/username_who_started_script instead of /storage/software_folder/software_name
Any other suggestions?
UPDATE 2:
Also tried to use #SBATCH --chdir=/storage/software_folder/software_name inside of script, but in such case echo "${SLURM_SUBMIT_DIR}" returns /home/username_who_started_scriptor / (if run as root)
UPDATE 3
Approach with ${SLURM_SUBMIT_DIR} worked only if task is ran as:
cd /storage/software_folder/software_name
sbatch ./scripts/this_script.sh
But it doesn't seem to be a proper solution. Are there any other ways?
SOLUTION
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --partition=my_partition_name
# check if script is started via SLURM or bash
# if with SLURM: there variable '$SLURM_JOB_ID' will exist
# `if [ -n $SLURM_JOB_ID ]` checks if $SLURM_JOB_ID is not an empty string
if [ -n $SLURM_JOB_ID ]; then
# check the original location through scontrol and $SLURM_JOB_ID
SCRIPT_PATH=$(scontrol show job $SLURM_JOBID | awk -F= '/Command=/{print $2}')
else
# otherwise: started with bash. Get the real location.
SCRIPT_PATH=$(realpath $0)
fi
# getting location of software_name
SHARED_PATH=$(dirname $(dirname $(SCRIPT_PATH)))
# separating the software_name from path
SOFTWARE_NAME=$(basename $SHARED_PATH)
# target location to copy project
LOCAL_SOFTWARE_FOLDER='/node_folder'
# corrected path for target
LOCAL_PATH=$LOCAL_SOFTWARE_FOLDER/$SOFTWARE_NAME
# Copying software folder from network storage to local
cp -r $SHARED_PATH $LOCAL_SOFTWARE_FOLDER
# running the script
sh $LOCAL_PATH/scripts/launch.sh
You can get the initial (i.e. at submit time) location of the submission script from scontrol like this:
scontrol show job $SLURM_JOBID | awk -F= '/Command=/{print $2}'
So you can replace the realpath $0 part with the above. This will only work within a Slurm allocation of course. So if you want the script to work in any situation, you will need some logic like:
if [ -n $SLURM_JOB_ID ] ; then
THEPATH=$(scontrol show job $SLURM_JOBID | awk -F= '/Command=/{print $2}')
else
THEPATH=$(realpath $0)
fi
and then proceed with
SHARED_PATH=$(dirname $(dirname "${THEPATH}"))
I had to do the same in an array job, the accepted answer from #damienfrancois works well for all jobs except the jobid which is same as ArrayJobId. Just piping awk command to head command would do the trick
scontrol show job $SLURM_JOBID | awk -F= '/Command=/{print $2}' | head -n 1
In script, get SHARED_PATH as SHARED_PATH="${SLURM_SUBMIT_DIR}"
Submit script as sbatch -D /storage/software ./scripts/this_script.sh
See here.
From referred page:
-D
Set the working directory of the batch script to directory before it
is executed. The path can be specified as full path or relative path
to the directory where the command is executed.
SLURM_SUBMIT_DIR
The directory from which sbatch was invoked or, if applicable, the directory specified by the -D, --chdir option.
P.S. Above is from Version 19.05 doc.
While looking in to archive, referring Ver. 18.x (esp. 18.08), it doesn't mention the same. See this
SLURM_SUBMIT_DIR.
The directory from which sbatch was invoked.
I have shell script " test.sh " as below
#!/bin/bash
export UDR_START_DT=default.test_tab_$(date +"%Y%m%d" -d "yesterday")
echo "Start date : "$UDR_START_DT
hive -f tst_tab.hql
the above shell script is saved in a folder in hadoop
/scripts/Linux/test.sh
the tst_tab.hql contains a simple create table statement, as I am just testing to get the hive working. This hive file is saved in the My documents folder in hue (same folder where my workflow is saved)
I have created an Oozie workflow that calls test.sh in a shell action.
Issue I am facing:
the above shell script runs successfully until line 3.
but when I add line 4 (hive -f tst_tab.hql), it generates the error
Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
I verified YARN logs too. Nothing helpful there.
I have this script that parses images from an external address.
I need to make sure that the script is running in a specific directory. I also want to log the time.
This is what I have done for the cronjob:
*/2 * * * * cd /path/to/script; (time /path/to/script/code.sh arg) &>> /path/to/log/time.log)
And my shell script is pretty simple:
#!/bin/sh
timestamp()
{
date +"%Y-%m-%d %T"
}
echo "###########################"
echo $(timestamp)
python /path/to/script/parser1.py "-s$1"
python /path/to/script/parser2.py "-s$1"
I get mail to my user, but it is just of the $(timestamp) echo. I should have the timestamp and also have output from both parser1 and parser2 updating in my time.log file. I don't know why this isn't working! I can run the cronjob command from the command line and it works.
So, I think the problem is that cron is running your script, but not from the directory you want to run it FROM. I am not sure, but I believe cron executes scripts from either the /root or / directories.
Try this - Add this right below your timestamp function - modified with the proper path, of course.
DIRNAME=/path/to/directory
LOGFILE=logfile.log
cd ${DIRNAME}
echo $(timestamp) >> ${LOGFILE}
And then remove the current echo $(timestamp) line.
Also, make sure the log file exists, and you have write access to the file.
I have squence of Pig scripts in a file and I want to execute it from Shell script
which execute pig scripts sqeuenciatly.
For Ex:
sh script.sh /it/provider/file_name PIGddl.txt
Suppose PIGddl.txt has Pig scripts like
Record Count
Null validation e.t.c
If all the Pig queries are in one file then how to execute the pig scripts from Shell scripts?
below idea works ,but if you want sequential process like if 1 execute then execute 2 else execute 3 kind of flow,you may go with Oozie for running and scheduling the jobs.
#!/bin/sh
x=1
while [ $x -le 3 ]
do
echo "pig_dcnt$x.pig will be run"
pig -f /home/Scripts/PigScripts/pig_dcnt$x.pig --param timestamp=$timestamp1
x=$(( $x + 1 ))
done
I haven't tested this but I'm pretty sure this will work fine.
Lets assume you have two pig files which you want to run using shell script then you would write a shell script file with following:
#!/bin/bash
pig
exec pig_script_file1.pig
exec pig_script_file2.pig
so when you run this shell script, initially it will execute pig command and goes into grunt shell and there it will execute your pig files in the order that you have mentioned
Update:
The above solution doesn't work. Please refer the below one which is
tested
Update your script file with the following so that it can run your pig files in the order that you have defined
#!/bin/bash
pig pig_script_file1.pig
pig pig_script_file2.pig
Here is what you have to do
1. Keep xxx.pig file at some location #
2. to execute this pig script from shell use the below command
pig -p xx=date(if you have some arguments to pass) -p xyz=value(if there is another arguments to be passed) -f /path/xxx.pig
-f is used to execute the pig lines of code from .pig file.