I am trying to schedule a spark job using cron.
I have made a shell script and it executes well on the terminal.
However, when I execute the script using cron it gives me insufficient memory to start JVM thread error.
Every time I start the script using terminal there is no issue. This issue comes when the script starts with cron.
Kindly if you could suggest something.
Related
When i try to execute the jobs from terminal it works for one hour without any issue. when i try to execute shell from Jenkins it works for just one minute and stops the execution. The output from Jenkins console output as follows :
Creating folder path in /jenkins/workspace/load_test/scripts/loadtest/loadtest1
PWD is : /jenkins/workspace/load_test/scripts/loadtest
Running /jenkins/workspace/load_test/scripts/loadtest/loadtest1/testRestApi.sh
1495126268
1495129868
3600
Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information
Finished: SUCCESS
Any ideas/ suggestion to make the script run for one hour from Jenkins job ?
Have you tried with BUILD_ID=dontKillMe it is commonly used for daemons. https://wiki.jenkins-ci.org/display/JENKINS/ProcessTreeKiller however this should let you run your script
Im working with the LSF, running bsub commands.
I'm implementing the -Ep switch to run a post exec script. This works great until the Job is killed or hits a memory limit, run limit etc.
Is there any way for the job to detect its running out of resource and then run the script? or to force it to run the script even if its been killed?
I guess my other option is running job with a dependency on that job which will run the "post exec" script when it finishes.
Any thoughts?
Kind Regards,
TheBigPeeler
From the documentation, you should be seeing the behaviour that you want.
A post-execution command runs after the job finishes, regardless of
the exit state of the job. Once a post-execution command is associated
with a job, that command runs even if the job fails. You cannot
configure the post-execution command to run only under certain
conditions.
I thought that maybe the interaction with JOB_INCLUDE_POSTEXEC (lsb.params) could account for the difference, but from my test the post-exec still runs in both cases. I used runlimit (bsub -W) to trigger the job kill.
Is it possible that the post exec is running, but exits early?
What version of LSF are you using? (What's the output of mbatchd -V and sbatchd -V)
i am trying to run a cron job which will execute my shell script, my shell script is having hive & pig scripts. I am setting the cron job to execute after every 2 mins but before my shell script is getting finish my cron job starts again is it going to effect my result or once the script finishes its execution then only it will start. I am in a bit of dilemma here. Please help.
Thanks
I think there are two ways to better resolve this, a long way and a short way:
Long way (probably most correct):
Use something like Luigi to manage job dependencies, then run that with Cron (it won't run more than one of the same job).
Luigi will handle all your job dependencies for you and you can make sure that a particular job only executes once. It's a little more work to get set-up, but it's really worth it.
Short Way:
Lock files have already been mentioned, but you can do this on HDFS too, that way it doesn't depend on where you run the cron job from.
Instead of checking for a lock file, put a flag on HDFS when you start and finish the job, and have this as a standard thing in all of your cron jobs:
# at start
hadoop fs -touchz /jobs/job1/2016-07-01/_STARTED
# at finish
hadoop fs -touchz /jobs/job1/2016-07-01/_COMPLETED
# Then check them (pseudocode):
if(!started && !completed): run_job; add_completed; remove_started
At the start of the script, have a check:
#!/bin/bash
if [ -e /tmp/file.lock ]; then
rm /tmp/file.lock # removes the lock and continue
else
exit # No lock file exists, which means prev execution has not completed.
fi
.... # Your script here
touch /tmp/file.lock
There are many others ways of achieving the same. I am giving a simple example.
I currenty have a simple shell script that I created for a linux machine to be run using cron, but now I want to be able to run the file using windows task scheduler. I have tried to get it to work using cron for cygwin, but even after running cron-config successfully and ensuring that the shell script can be executed successfully, for some reason the cron task simply wasn't executing. So I decided to give in and use the windows task scheduler. In order to do this, I looked at the following posts about the issue:
Cgywin .sh file run as Windows Task Scheduler
http://www.davidjnice.com/cygwin_scheduled_tasks.html
in my case, the entry in the "actions" tab of the new task looks like this:
program/script: c:\cygwin64\bin\bash.exe
arguments: -l -c "/cygdrive/c/users/paul/bitcoinbot/download_all_data.sh >> cygdrive/c/users/paul/bitcoinbot/logfile.log 2>&1"
start in: c:\cygwin64\bin
Notice that I redirected the output of the shell script to a log file, so that I should be able to see there whether the program run. Other than that, I simply edited the "trigger" tab to run the task daily, and set the time to a couple of minutes in the fture to see whether it ran successfully.
Alas, when I look at the detailed event history for the task, nothing changes when the trigger time passes. And when I manually "run" the task, the event history seems to add a few different events, but the task is completed within seconds, whereas this task should take over an hour (and it does when the shell script is executed directly from the terminal). And when I look for the log file that should have been created, there is nothing.
Does anyone have any idea what might be the issue here? How can I get my task to run properly at the trigger time, and how can I make sure it does so?
Best,
Paul
EDIT:
here are the pictures showing event history, as per Ken White's request.
Please ignore the fact that it says there are 24 events. These are from multiple separate runs of the task. The events shown here are a complete list of the events triggered by a single run.
EDIT 2:
Regarding my attempts to get cron to work, I have run into the following problem when I try to start the cron service using cygrunsrv. First of all, I tried to start cron by typing
cygrunsrv -I cron -p /usr/sbin/cron.exe -a -D
Now when I type
$cygrunsrv -Q cron
Service: cron
Current State: stopped
Command: /usr/bin/cron.exe
Now, I tried to start the cron service by typing
cygrunsrv -S cron
Cygrunsrv: Error starting a service: QueryServiceStatus: Win32 error 1062:
The service has not been started.
Does anyone hae any idea what this error means? I tried googling it, but couldn't find any answers.
I am on shared hosting and I'm trying to schedule cronjob to run every now and then. Via cPanel I scheduled to execute my script but even though that according to my host support the cronjob runs, the script doesn't seem as doing anything. The cron job command I set via cPanel is:
/bin/sh /home1/myusername/public_html/somefolder/cronjob2.sh
and the cronjob2.sh
#!/bin/bash
/home1/myusername/public_html/somefolder/node_modules/forever/bin/forever stop 0
when via SSH I execute:
/home1/myusername/public_html/somefolder/cronjob2.sh
it stops forever process as needed. From cronjob doesn't do anything.
How can I get this working?
EDIT:
So I've tried:
/bin/sh /home1/username/public_html/somefolder/cronjob2.sh >> /tmp/mylog 2>&1
and mylog entries say:
/usr/bin/env: node: No such file or directory
It seems that forever needs to run node and this cannot be found. How would I possibly fix this?
EDIT2:
Accepted answer at superuser.com. Thank you all for help
https://superuser.com/questions/763261/simple-script-run-via-cronjob-doesnt-work-but-works-from-shell/763288#763288
For cron job lines in a crontab it's not required to specify kind of shell or e.g. of perl.
It's enough, that your script contains
shebang
line.
Therefore you should remove /bin/sh from your cron job line.
Another aspect, that might cause a different behavior of your script by interactive start and by cron daemon start is possible different environment, first of all the PATH variable. Therefore check, if you script is able to be executed in very restricted environment, that is provided by cron daemon. You can determine your cron job environment experimentally by start of temporary cron job, that executes "env" command and writes its output to a file.
Once more aspect: Have you redirected STDOUT and STDERR of the cron job to a log file and read its content to analyze the issue? You can do it as follows:
your_cron_job >/tmp/any_name.log 2>&1
According to what you wrote, when you run your script via SSH, you are using bash, because this line is the first of your script:
#!/bin/bash
However, in the crontab, you are forcing the use of sh instead of bash. Are you sure your script is fully compatible with sh? Otherwise, simply replace /bin/sh with /bin/bash in your cron command and test again.