I have some perl scripts which are scheduled using task scheduler in windows 2003 R2 and 2008. These scripts are called directly using perl.exe or via a batch file.
Sometimes these scripts fails to execute (crashes maybe) and we are not aware of these crashes.
Are there any ways a mail can be sent when these script crashes? more or less like monitoring of these scripts
Thanks in advance
Karthik
Why monitor the scripts from the outside when you can make the plugins to monitor theirself? First you can use eval in order to catch errors, and if an error occours you can send an email with the Net::SMTP module as rpg suggested. However I highly recommend you to use some kind of log file in order to keep trace of what happened right before the error and what caused the error. Your main goal should be to avoid the error. That ofcourse requires you to modify the scripts, if, for any reason, you cannot do that then the situation may be a little more complicated because you need another script.
With the Win32::Process::Info module you can retrieve running processes on Windows and check if your plugin is running or not.
while(1) {
my $found = false;
my $p = Win32::Process::Info->new;
foreach my $proc ($pi->GetProcInfo) {
if ($proc->{Name} =~ /yourscriptname/i ) {
found = true;
}
}
if ($found eq 'false') {
# send email
my $smtp = Net::SMTP->new("yoursmtpserver");
eval {
$smtp->mail("sender#test.it");
$smtp->recipient("recipient#test.it");
$smtp->data;
$smtp->datasend("From: sender#test.it");
$smtp->datasend("\n");
$smtp->datasend("To: recipient#test.it");
$smtp->datasend("\n");
$smtp->datasend("Subject: Plugin crashed!");
$smtp->datasend("\n");
$smtp->datasend("Plugin crashed!");
$smtp->dataend;
$smtp->quit;
};
}
sleep(300);
}
I did not test this code because I don't have Perl installed on Windows but the logic should be ok.
For monitoring - Please check the error code. This will help you for its failure.
For mail sending - You can use Net::SMTP module to send email. Let me know if you need a code snippet for it.
You can use PushMon to monitor your scripts. What you do is create PushMon URLs that matches the schedule of your Perl scripts. Then you should "ping" these URLs when your scripts run successfully. If these URLs are not accessed, maybe because your scripts crashed or there's a power failure, PushMon will notify you by email.
Disclaimer: I am associated with PushMon.
Related
In a project I'm working on, we are using shell scripts to execute different tasks. Some are sh/bash scripts that run rsync, and some are PHP scripts. One of the PHP scripts is running some integration tests that output to JUnit XML, code coverage reports, and similar.
Jenkins is able to mark the jobs as successful / failed based on exit status. In PHP, the script exits with 1 if it has detected that the tests failed during the run. The other shell scripts run commands and use the exit codes from those to mark a build as failed.
// :: End of PHP script:
// If any tests have failed, fail the build
if ($build_error) exit(1);
In Jenkins Terminology, an unstable build is defined as:
A build is unstable if it was built successfully and one or more publishers report it unstable. For example if the JUnit publisher is configured and a test fails then the build will be marked unstable.
How can I get Jenkins to mark a build as unstable instead of only success / failed when running shell scripts?
Modern Jenkins versions (since 2.26, October 2016) solved this: it's just an advanced option for the Execute shell build step!
You can just choose and set an arbitrary exit value; if it matches, the build will be unstable. Just pick a value which is unlikely to be launched by a real process in your build.
It can be done without printing magic strings and using TextFinder. Here's some info on it.
Basically you need a .jar file from http://yourserver.com/cli available in shell scripts, then you can use the following command to mark a build unstable:
java -jar jenkins-cli.jar set-build-result unstable
To mark build unstable on error, you can use:
failing_cmd cmd_args || java -jar jenkins-cli.jar set-build-result unstable
The problem is that jenkins-cli.jar has to be available from shell script. You can either put it in easy-to-access path, or download in via job's shell script:
wget ${JENKINS_URL}jnlpJars/jenkins-cli.jar
Use the Text-finder plugin.
Instead of exiting with status 1 (which would fail the build), do:
if ($build_error) print("TESTS FAILED!");
Than in the post-build actions enable the Text Finder, set the regular expression to match the message you printed (TESTS FAILED!) and check the "Unstable if found" checkbox under that entry.
You should use Jenkinsfile to wrap your build script and simply mark the current build as UNSTABLE by using currentBuild.result = "UNSTABLE".
stage {
status = /* your build command goes here */
if (status === "MARK-AS-UNSTABLE") {
currentBuild.result = "UNSTABLE"
}
}
you should also be able to use groovy and do what textfinder did
marking a build as un-stable with groovy post-build plugin
if(manager.logContains("Could not login to FTP server")) {
manager.addWarningBadge("FTP Login Failure")
manager.createSummary("warning.gif").appendText("<h1>Failed to login to remote FTP Server!</h1>", false, false, false, "red")
manager.buildUnstable()
}
Also see Groovy Postbuild Plugin
In my job script, I have the following statements (this job only runs on the Jenkins master):
# This is the condition test I use to set the build status as UNSTABLE
if [ ${PERCENTAGE} -gt 80 -a ${PERCENTAGE} -lt 90 ]; then
echo WARNING: disc usage percentage above 80%
# Download the Jenkins CLI JAR:
curl -o jenkins-cli.jar ${JENKINS_URL}/jnlpJars/jenkins-cli.jar
# Set build status to unstable
java -jar jenkins-cli.jar -s ${JENKINS_URL}/ set-build-result unstable
fi
You can see this and a lot more information about setting build statuses on the Jenkins wiki: https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+CLI
Configure PHP build to produce xml junit report
<phpunit bootstrap="tests/bootstrap.php" colors="true" >
<logging>
<log type="junit" target="build/junit.xml"
logIncompleteSkipped="false" title="Test Results"/>
</logging>
....
</phpunit>
Finish build script with status 0
...
exit 0;
Add post-build action Publish JUnit test result report for Test report XMLs. This plugin will change Stable build to Unstable when test are failing.
**/build/junit.xml
Add Jenkins Text Finder plugin with console output scanning and unchecked options. This plugin fail whole build on fatal error.
PHP Fatal error:
Duplicating my answer from here because I spent some time looking for this:
This is now possible in newer versions of Jenkins, you can do something like this:
#!/usr/bin/env groovy
properties([
parameters([string(name: 'foo', defaultValue: 'bar', description: 'Fails job if not bar (unstable if bar)')]),
])
stage('Stage 1') {
node('parent'){
def ret = sh(
returnStatus: true, // This is the key bit!
script: '''if [ "$foo" = bar ]; then exit 2; else exit 1; fi'''
)
// ret can be any number/range, does not have to be 2.
if (ret == 2) {
currentBuild.result = 'UNSTABLE'
} else if (ret != 0) {
currentBuild.result = 'FAILURE'
// If you do not manually error the status will be set to "failed", but the
// pipeline will still run the next stage.
error("Stage 1 failed with exit code ${ret}")
}
}
}
The Pipeline Syntax generator shows you this in the advanced tab:
I find the most flexible way to do this is by reading a file in the groovy post build plugin.
import hudson.FilePath
import java.io.InputStream
def build = Thread.currentThread().executable
String unstable = null
if(build.workspace.isRemote()) {
channel = build.workspace.channel;
fp = new FilePath(channel, build.workspace.toString() + "/build.properties")
InputStream is = fp.read()
unstable = is.text.trim()
} else {
fp = new FilePath(new File(build.workspace.toString() + "/build.properties"))
InputStream is = fp.read()
unstable = is.text.trim()
}
manager.listener.logger.println("Build status file: " + unstable)
if (unstable.equalsIgnoreCase('true')) {
manager.listener.logger.println('setting build to unstable')
manager.buildUnstable()
}
If the file contents are 'true' the build will be set to unstable. This will work on the local master and on any slaves you run the job on, and for any kind of scripts that can write to disk.
I thought I would post another answer for people that might be looking for something similar.
In our build job we have cases where we would want the build to continue, but be marked as unstable. For ours it's relating to version numbers.
So, I wanted to set a condition on the build and set the build to unstable if that condition is met.
I used the Conditional step (single) option as a build step.
Then I used Execute system Groovy script as the build step that would run when that condition is met.
I used Groovy Command and set the script the following
import hudson.model.*
def build = Thread.currentThread().executable
build.#result = hudson.model.Result.UNSTABLE
return
That seems to work quite well.
I stumbled upon the solution here
http://tech.akom.net/archives/112-Marking-Jenkins-build-UNSTABLE-from-environment-inject-groovy-script.html
In addition to all others answers, jenkins also allows the use of the unstable() method (which is in my opinion clearer).
This method can be used with a message parameter which describe why the build is unstable.
In addition of this, you can use the returnStatus of your shell script (bat or sh) to enable this.
For example:
def status = bat(script: "<your command here>", returnStatus: true)
if (status != 0) {
unstable("unstable build because script failed")
}
Of course, you can make something with more granularity depending on your needs and the return status.
Furthermore, for raising error, you can also use warnError() in place of unstable(). It will indicate your build as failed instead of unstable, but the syntax is same.
The TextFinder is good only if the job status hasn't been changed from SUCCESS to FAILED or ABORTED.
For such cases, use a groovy script in the PostBuild step:
errpattern = ~/TEXT-TO-LOOK-FOR-IN-JENKINS-BUILD-OUTPUT.*/;
manager.build.logFile.eachLine{ line ->
errmatcher=errpattern.matcher(line)
if (errmatcher.find()) {
manager.build.#result = hudson.model.Result.NEW-STATUS-TO-SET
}
}
See more details in a post I've wrote about it:
http://www.tikalk.com/devops/JenkinsJobStatusChange/
As a lighter alternative to the existing answers, you can set the build result with a simple HTTP POST to access the Groovy script console REST API:
curl -X POST \
--silent \
--user "$YOUR_CREDENTIALS" \
--data-urlencode "script=Jenkins.instance.getItemByFullName( '$JOB_NAME' ).getBuildByNumber( $BUILD_NUMBER ).setResult( hudson.model.Result.UNSTABLE )" $JENKINS_URL/scriptText
Advantages:
no need to download and run a huge jar file
no kludges for setting and reading some global state (console text, files in workspace)
no plugins required (besides Groovy)
no need to configure an extra build step that is superfluous in the PASSED or FAILURE cases.
For this solution, your environment must meet these conditions:
Jenkins REST API can be accessed from slave
Slave must have access to credentials that allows to access the Jenkins Groovy script REST API.
If you want to use a declarative approach I suggest you to use code like this.
pipeline {
stages {
// create separate stage only for problematic command
stage("build") {
steps {
sh "command"
}
post {
failure {
// set status
unstable 'rsync was unsuccessful'
}
always {
echo "Do something at the end of stage"
}
}
}
}
post {
always {
echo "Do something at the end of pipeline"
}
}
}
In case you want to keep everything in one stage use catchError
pipeline {
stages {
// create separate stage only for problematic command
stage("build") {
steps {
catchError(stageResult: 'UNSTABLE') {
sh "command"
}
sh "other command"
}
}
}
}
One easy way to set a build as unstable, is in your "execute shell" block, run exit 13
You can just call "exit 1", and the build will fail at that point and not continue. I wound up making a passthrough make function to handle it for me, and call safemake instead of make for building:
function safemake {
make "$#"
if [ "$?" -ne 0 ]; then
echo "ERROR: BUILD FAILED"
exit 1
else
echo "BUILD SUCCEEDED"
fi
}
This example routine generates two Throw::nocatch warning messages in the kernel window. Can they be handled somehow?
The example consists of this code in a file "test.m" created in C:\Temp:
Needs["JLink`"];
$FrontEndLaunchCommand = "Mathematica.exe";
UseFrontEnd[NotebookWrite[CreateDocument[], "Testing"]];
Then these commands pasted and run at the Windows Command Prompt:
PATH = C:\Program Files\Wolfram Research\Mathematica\8.0\;%PATH%
start MathKernel -noprompt -initfile "C:\Temp\test.m"
Addendum
The reason for using UseFrontEnd as opposed to UsingFrontEnd is that an interactive front end may be required to preserve output and messages from notebooks that are usually run interactively. For example, with C:\Temp\test.m modified like so:
Needs["JLink`"];
$FrontEndLaunchCommand="Mathematica.exe";
UseFrontEnd[
nb = NotebookOpen["C:\\Temp\\run.nb"];
SelectionMove[nb, Next, Cell];
SelectionEvaluate[nb];
];
Pause[10];
CloseFrontEnd[];
and a notebook C:\Temp\run.nb created with a single cell containing:
x1 = 0; While[x1 < 1000000,
If[Mod[x1, 100000] == 0,
Print["x1=" <> ToString[x1]]]; x1++];
NotebookSave[EvaluationNotebook[]];
NotebookClose[EvaluationNotebook[]];
this code, launched from a Windows Command Prompt, will run interactively and save its output. This is not possible to achieve using UsingFrontEnd or MathKernel -script "C:\Temp\test.m".
During the initialization, the kernel code is in a mode which prevents aborts.
Throw/Catch are implemented with Abort, therefore they do not work during initialization.
A simple example that shows the problem is to put this in your test.m file:
Catch[Throw[test]];
Similarly, functions like TimeConstrained, MemoryConstrained, Break, the Trace family, Abort and those that depend upon it (like certain data paclets) will have problems like this during initialization.
A possible solution to your problem might be to consider the -script option:
math.exe -script test.m
Also, note that in version 8 there is a documented function called UsingFrontEnd, which does what UseFrontEnd did, but is auto-configured, so this:
Needs["JLink`"];
UsingFrontEnd[NotebookWrite[CreateDocument[], "Testing"]];
should be all you need in your test.m file.
See also: Mathematica Scripts
Addendum
One possible solution to use the -script and UsingFrontEnd is to use the 'run.m script
included below. This does require setting up a 'Test' kernel in the kernel configuration options (basically a clone of the 'Local' kernel settings).
The script includes two utility functions, NotebookEvaluatingQ and NotebookPauseForEvaluation, which help the script to wait for the client notebook to finish evaluating before saving it. The upside of this approach is that all the evaluation control code is in the 'run.m' script, so the client notebook does not need to have a NotebookSave[EvaluationNotebook[]] statement at the end.
NotebookPauseForEvaluation[nb_] := Module[{},While[NotebookEvaluatingQ[nb],Pause[.25]]]
NotebookEvaluatingQ[nb_]:=Module[{},
SelectionMove[nb,All,Notebook];
Or##Map["Evaluating"/.#&,Developer`CellInformation[nb]]
]
UsingFrontEnd[
nb = NotebookOpen["c:\\users\\arnoudb\\run.nb"];
SetOptions[nb,Evaluator->"Test"];
SelectionMove[nb,All,Notebook];
SelectionEvaluate[nb];
NotebookPauseForEvaluation[nb];
NotebookSave[nb];
]
I hope this is useful in some way to you. It could use a few more improvements like resetting the notebook's kernel to its original and closing the notebook after saving it,
but this code should work for this particular purpose.
On a side note, I tried one other approach, using this:
UsingFrontEnd[ NotebookEvaluate[ "c:\\users\\arnoudb\\run.nb", InsertResults->True ] ]
But this is kicking the kernel terminal session into a dialog mode, which seems like a bug
to me (I'll check into this and get this reported if this is a valid issue).
I'm using Net::SSH2's scp_put method to place one file in my home directory on a Unix server from a Windows box. I am using Strawberry Perl 5.12 (portable version). I installed the libssh2 1.2.5 binaries and then Net::SSH2 from cpan.
Here's my code snippet:
sub uploadToHost{
my $file=#_[0];
my $host=#_[1];
my $user=#_[2];
my $pass=#_[3];
my $remotelocation=#_[4];
#makes a new SSH2 object
my $ssh=Net::SSH2->new() or die "couldn't make SSH object\n";
#prints proper error messages
$ssh->debug(1);
#nothing works unless I explicitly set blocking on
$ssh->blocking(1);
print "made SSH object\n";
#connect to host; this always works
$ssh->connect($host) or die "couldn't connect to host\n";
print "connected to host\n";
#authenticates with password
$ssh->auth_password($user, $pass) or die "couldn't authenticate $user\n";
print "authenticated $user\n";
#this is the tricky bit that hangs
$ssh->scp_put($file, $remotelocation") or die "couldn't put file in $remotelocation\n";
print "uploaded $file successfully\n";
$ssh->disconnect or die "couldn't disconnect\n";
} #ends sub
Output (edited for anonymity):
made SSH object\n
connected to host\n
authenticated \n
libssh2_scp_send_ex(ss->session, path, mode, size, mtime, atime) -> 0x377e61c\n
Net::SSH2::Channel::read(size = 1, ext = 0)\n
It then hangs forever (>40 minutes in one test) and needs to be killed.
What's strange is that it actually does scp the file to the remote server! It only hangs after it should have completed. I couldn't find references to this curious problem elsewhere on StackOverflow or elsewhere.
Can anyone point me in the right direction to either 1) stop it from hanging, or 2) implement (as a workaround) a timer that kills this one command after a few seconds, which is enough time to scp the file?
Thanks, everyone!
You can try using alarm() to prod your process into behaving, if you save this example as 'alarm.pl' you can see how it works:
use strict;
use warnings;
use 5.10.0;
# pretend to be a slow process if run as 'alarm.pl s'
if (#ARGV && $ARGV[0] eq 's') {
sleep(30);
exit();
}
# Otherwise set an alarm, then run myself with 's'
eval {
local $SIG{ALRM} = sub {die "alarmed\n"};
alarm(5);
system("perl alarm.pl s");
};
if ($#) {
die $# unless $# eq "alarmed\n";
say "Timed out slow process";
}
else {
say "Slow process finished";
}
Use Net::SFTP::Foreign with the Net::SSH2 backend, Net::SFTP::Foreign::Backend::Net_SSH2:
use Net::SFTP::Foreign;
my $sftp = Net::SFTP::Foreign->new($host, user => $user, password => $password, backend => Net_SSH2);
$sftp->die_on_error("Unable to connect to remote host");
$sftp->put($file, $remotelocation);
$sftp->die_on_error("Unable to copy file");
If that doesn't work either, you can try using plink (from the PuTTY project) instead of the Net::SSH2 backend.
I don't think it is hanging it is just REALLY SLOW. 10x slower than what it should be. The reason the file would appear to be there is that it allocates the file before it has finished transferring. This isn't really too unexpected, Perl finds new ways to disappoint and frustrate programmers on a daily basis. Sometimes I think I spend more time working around Perl's idiosyncrasies and learning 10 slightly different ways to do the same thing than doing real work.
I created a small shell script which logs all of it's input to a log file, with which I had thought I could replace the sendmail binary and thus achieve a simple way to simulate e-mail delivery without actually setting up a working sendmail.
This failed, however. For reasons I cannot understand.
I've looked at the PHP mail.c source and as far as I can understand (mind you, I'm not very experienced in C), PHP executes and talks directly to the binary (set in sendmail_path). But no log files are being created when I replace the sendmail binary with my script and the script replacing sendmail will always create a log file when it's executed, regardless of if there's input present or not.
The script itself works fine. Its' return codes should conform to that of sendmail's. With the difference that my script always returns 0, regardless of the input, since I'm not really interested in checking if the input is valid - just that I'm getting some.
Is it possible to achieve what I want, i.e. using a sendmail simulator?
The script source is provided below:
#!/bin/bash
LOGDIR=/tmp/sendmail-sim
NOW=$(date +%Y%m%dT%H%M)
CNT=1
FILENAME="$LOGDIR/$NOW.$CNT.log"
while [ -f $FILENAME ]; do
CNT=$(($CNT + 1))
FILENAME="$LOGDIR/$NOW.$CNT.log"
done
echo "$0 $*" > $FILENAME
while read BUF
do
echo $BUF >> $FILENAME
done
exit 0
PS. My current sendmail (or actually, postfix) does receive email from PHP, but I don't want to actually send any email or need to go digging in its' mail queue in development.
The problem was User Error, as usually. So, boys and girls, don't forget to check write permissions on all the relevant folders.
I've used Fakemail for this purpose in the past. It accepts SMTP connections but writes all mail to files rather than sending them along as email. There are both python and perl implementations.
http://www.lastcraft.com/fakemail.php
We setup apache to serve the directory that Fakemail was writing to. That was a quick and easy way for staff to view the messages that Fakemail was receiving and review for content, destination, etc. Formatting of HTML emails was a bit whacky for various reasons, so it was not so useful for vetting formatting of html emails.
If you need to to test you PHP application's ability to properly format and send email without actually sending them, I suggest you use the Pear Mail package. Fiddling with your system is not a good idea.
If you use Mail from Pear you could switch from sendmail to smtp to a mock implementation of the mail interface by changing the driver from 'sendmail' or 'smtp' to 'mock'.
http://pear.php.net/package/Mail/docs/latest/Mail/Mail.html
http://pear.php.net/package/Mail/docs/latest/Mail/Mail_mock.html
If your code looks like this:
mail('me#example.com', 'My Subject', $message);
Then change it to be testable using PEAR Mail:
include('Mail.php');
function sendEmail($recipient, $subject, $body, $driver = 'mail') {
$m = Mail::factory("mail");
$headers = array(
"From"=>"me#example.com",
"To" => $recipient,
"Subject"=> $subject);
$m->send($recipient, $headers, $body);
return $m;
}
// In Production:
sendEmail('me#example.com', 'My Subject', $message);
// During testing:
$m = sendEmail('me#example.com', 'My Subject', $message, 'mock');
var_dump($m->sentMessages);
This is very crude, since you should be using PHPUnit or SimpleTest, but this is a topic for another time and place :)
A note, if you just want to grab stdin and write it into a file, you don't need to loop one line at a time: you can write
cat - >> $FILENAME
Within a Oracle Forms trigger I am using the host command to make a directory on the file server. An example of this part of my code is below:
HOST ('mkdir'||:GLOBAL.DIRECTORY_PATH||'\FERTILIZER\'||ADDY);
I need to have the error code returned to me if the directory is not created on the server. Any suggestions of the code I need to add?
Thank you.
FORM_SUCCESS will return FALSE if the command fails for any reason (unless you're on Windows 95 in which case it will still return TRUE).
HOST('...');
IF NOT FORM_SUCCESS THEN
MESSAGE('something went wrong');
END IF;
If you are looking for the actual OS level error code, then you are out of luck. The aforementioned answer from Jeffrey Kemp
is the best you will get.
If you are having failures, keep in mind that the HOST built-in runs on the machine that actually runs the form (normally the application server). So, your command must be valid for the particular OS of the application server.
Also, and you may have figured this out already, in your example, 'mkdir'||:GLOBAL.DIRECTORY_PATH||'\FERTILIZER\'||ADDY
could result in your command having no space between the command - mkdir and the string, resulting in a failed command.
You still can get the error message by using this statement HOST(vCommand || ' > error.txt');