Should able to process larger log files and provide exception message reports
After completion of log analysis, report notification trigger to specific mail id's.
And also please suggest which framework is the best for processing large files.[eg: spring boot/batch]
I would suggest to go with ELK stack. Stream the logs to elastic search and set up alerts in Kibana.
Can use sendmail client on system and run script on that system to send alert on any Exception.
exception="Exception" # "Error", "HTTP 1.1 \" 500", etc
ignoredException="ValidationException"
# log file to scan
logFileToScan=/var/log/tomcat8/log/application.log
# file where we will keep log of this script
logFilePath=/home/ec2-user/exception.log
# a file where we store till what line the log file has been scanned
# initalize it with 0
countPath=/home/ec2-user/lineCount
# subject with which you want to receive the mail regading Exception
subject="[ALERT] Exception"
# from whom do you want to send the mail regarding Exception
from="abc#abc.com"
# to whom do you want to send the mail
to="xyz#xyz.com"
# number of lines, before the line containing the word to be scanned, to be sent in the mail
linesBefore=1
# number of lines, before the line containing the word to be scanned, to be sent in the mail
linesAfter=4
# start line
fromLine=`cat $countPath`
# current line count in the file
toLine=`wc -l $logFileToScan | awk '{print $1}'`
#logs are rolling so if fromLine has a value greater than toLine then fromLine has to be set to 0
if [ "$fromLine" == "" ]; then
fromLine=0
echo `date` fromLine values was empty, set to 0 >> $logFilePath
elif [ $fromLine -gt $toLine ]; then
echo `date` logfile was rolled, updating fromLine from $fromLine to 0 >> $logFilePath
fromLine=0
fi
# if from n to lines are equal then no logs has been generated since last scan
if [ "$fromLine" == "$toLine" ]; then
echo `date` no logs genetared after last scan >> $logFilePath
else
echo `date` updating linecount to $toLine >> $logFilePath
echo $toLine > $countPath
logContent=`tail -n +"$fromLine" $logFileToScan | head -n "$((toLine - fromLine))" | grep -v $ignoredException | grep -A $linesAfter -B $linesBefore $exception`
logContent=`echo $logContent | cut -c1-2000`
if [ "$logContent" == "" ]; then
echo `date` no exception found >> $logFilePath
else
/usr/sbin/sendmail $to <<EOF
subject: $subject
from: $from
logContent=$logContent
EOF
fi
fi
I am at my wits end....
All I want to do is set the From using a command line mail in a script.
I have tried mutt, mail, mailx...
Here is on command I tried:
echo "Once again" | mail -s "Maybe this time" -A file.csv stevec#isonas.com -r 'John <john#company.com>' -a From:John\<john#company.com\>
And another:
mutt -e "set from=john#company.com"...
And another:
mutt -e "set from=john#company.com" -e "set realname=John"
My /etc/mailname has the name I want too. I also tried configuring a muttrc file and that doesn't work
And another try:
mailx -r "DoNotReply#company.com" -a "From:DoNotReply#company.com" -s "Subject" -A "/opt/file.csv" "john#anothercompnay.com" <<< Some Text
Any ideas I can try?
I assume that you have configured john#company.com in an extra configuration file (~/.mutt/accounts/john) which you usually source as soon as you start using it in mutt. Simply load that configuration.
echo "mail body" | mutt -e "source ~/.mutt/accounts/john" -e "set content_type=text/plain" -s "subject" -- peter#company.com
An account specific configuration file may look like this (msmtp is used in this case to send mails):
set sendmail = "/usr/bin/msmtp -a Mailbox"
set realname = "John"
set from = "john#company.com"
set mbox_type = Maildir
set folder = "/home/john/.mails"
set spoolfile = "+Mailbox/INBOX"
set record = "+Mailbox/Sent"
set postponed = "+Mailbox/Drafts"
I have written this little script for getting multiple files from my remote server to my host computer:
#! /usr/bin/expect -f
spawn scp \
user#remote:/home/user/{A.txt,B.txt} \
/home/user_local/Documents
expect "password: "
send "somesecretpwd\r"
interact
This is working fine, but when I want to make new lines between the files like this:
user#remote:/home/user/{A.txt,\
B.txt} \
I am getting the following error(s):
scp: /home/user/{A.txt,: No such file or directory
scp: B.txt}: No such file or directory
I tried this:
user#remote:"/home/user/{A.txt,\
B.txt}" \
getting:
bash: -c: line 0: unexpected EOF while looking for matching `"'
bash: -c: line 1: syntax error: unexpected end of file
cp: cannot stat 'B.txt}"': No such file or directory
or this:
"user#remote:/home/user/{A.txt,\
B.txt}" \
getting the same error at the beginning.
How can I write the files in multiple lines but so that the program is working correctly? I need this for a better readability of the choosen files.
Edit:
Only changed the local user name to user_local
In Tcl (and so Expect) \<NEWLINE><SPACEs> will be converted into one single <SPACE> so you cannot write a string containing no spaces into multiple lines.
% puts "abc\
def"
abc def
% puts {abc\
def}
abc def
%
Assuming the filenames are really longer (not much point otherwise) you could use a couple of variables like this:
#! /usr/bin/expect -f
set A A.txt
set B B.txt
spawn scp \
user#remote:/home/user/{$A,$B} \
/home/user/Documents
expect "password: "
send "somesecretpwd"
interact
For anyone who want to solve a similar problem with only using expect:
You can write a list of files and then concat all files to one string.
Here is the code:
#! /usr/bin/expect -f
set files {\ # a list of files
A.txt\
B.txt\
C.txt\
}
# will return the concatenated string with all files
# in this example it would be: A.txt,B.txt,C.txt
set concat [join $files ,]
# self made version of concat
# set concat [lindex $files 0] # get the first file
# set last_idx [expr {[llength $files]-1}] # calc the last index from the list
# set rest_files [lrange $files 1 $last_idx] # get other files
# foreach file $rest_files {
# set concat $concat,$file # append the concat varibale with a comma and the other file
# }
# # puts "$concat" # only for testing the output
spawn scp \
user#remote:/home/doublepmcl/{$concat} \
/home/user_local/Documents
expect "password: "
send "somesecretpwd\r"
interact
I want to clean my mailbox from mails from specific address
I have thousands of messages, I want to do this in bash script, and run it from time to time (a receive SPAM from different addresses, and unfortunately my "spam filters" have only small effect on them)
To interact with a mail server through command line, you could use either telnet or openssl.
You can connect to your pop server using the following command (I've taken gmail as an example. You'll have to look for your email host pop3 address and socket.) :
openssl s_client -connect pop.gmail.com:995 -quiet
As this command is interactive, it will ask for a username, a password and a serie of commands.
expect is a tool that can automate interaction with interactive commands. The basic syntax is as follow : expect "somestring" action -> If the program we monitor displays "somestring", we execute the action.
Here is a script that would delete all the messages present on your email address :
#!/usr/bin/expect
#you can modify the timeout if the script fails
set timeout 1
#our connection variables
set ip "pop.gmail.com"
set socket "995"
set user "user.name"
set pass "password"
#we set the address we want to remove mails from here. Escape special regex characters such as dots.
set target_address "mail\.example#gmail\.com"
#we launch the subprocess we want to interact with
spawn openssl s_client -connect $ip:$socket -quiet
#if connection went all right, we try to login
expect -re ".OK.*" {send "user $user\r"}
expect -re ".OK.*" {send "pass $pass\r"}
#if login went alright, we try to count the messages on the server
#you will get the following output :
#+OK NB_MSG TOTAL_SIZE
expect -re ".OK.*" {send "stat\r"}
#if the stat command went alright ...
expect -re ".OK.*" {
#we extract the number of mail from the output of the stat command
set mail_count [lindex [split [lindex [split $expect_out(buffer) \n] 1] " "] 1]
#we iterate through every email...
for {set i 1} {$i <= $mail_count} {incr i 1} {
#we retrieve the header of the email
send "top $i 0\r"
#if the header contains "To: $target_address" (or "To: <$target_address>" or "To: Contact Name <$target_address>" ...)
#to filter according to the sender, change the regex to "\nFrom: ..."
expect -re "\nTo: \[^\n\]*$target_address" {
#we delete the email
send "dele $i\r"
}
}
}
expect default
You might need to alter your email account settings to allow the use of external programs
Here I have written a similar script in PHP, by use of the IMAP function of PHP. I did not have Linux, so I could not use the bash script. However. I thought that sharing the PHP-script would provide a solution for those running WINDOWS and having access
<?php
echo "My showmail.php script";
// This is the format for opening the email connection:
//
// domain prt foldr[.<.subfolder........>] my#blp.com xxxxxxxx
// $connection = imap_open('{imap.one.com:143}Inbox.<write subfolder here>','email-name','password');
// Execution: (write this in the browsers address bar)
http://digitalageinstitute.com/showmail.php?fromdate=01%20January%202009&todate=28%20August%202018&search=#weheartit.com&email=david.svarrer#digitalageinstitute.com&password=xxxxxxx&domain=imap.one.com&folder=Inbox
//
// This line above, entered into a browsers address bar will execute this script (named showmail.php),
// from the servers root (www.digitalageinstitute.com), and
//
// will delete emails from the mail folder "Inbox" (case sensitive) for the user david.svarrer#digitalageinstitute.com, by use of
// the IMAP domain (this is where you access emails from outside) imap.one.com and at port 143 (see this, further down)
//
$emailaddress = $_GET["email"];
$emailpassword= $_GET["password"];
$fromdate = $_GET["fromdate"];
$todate = $_GET["todate"];
$search = $_GET["search"];
$domain = $_GET["domain"];
$folder = $_GET["folder"];
echo $emailaddress." ".$emailpassword." ".$fromdate." ".$todate." ".$search." ".$domain." ".$folder;
$connection = imap_open('{'.$domain.':143}'.$folder,$emailaddress,$emailpassword);
$ccount = imap_num_msg($connection);
$maxtoexpunge=20000;
echo "Parameters = ".$fromdate.",".$todate.",".$search.":".'ALL FROM "'.$search.'" SINCE "'.$fromdate.'" BEFORE "'.$todate.'"';
// $some = imap_search($connection, 'ALL FROM "Twitter"');
$some = imap_search($connection, 'ALL FROM "'.$search.'" SINCE "'.$fromdate.'" BEFORE "'.$todate.'"');
print_r ($some);
echo "<br/>Elements: ".count($some)."<br/>";
$expunge = 0;
for($msgno = 0; $msgno < count($some) ; $msgno++) {
// echo "Fetching element ".$some[$msgno]."<br/>";
$headers = imap_headerinfo($connection, $some[$msgno]);
//$position = True;
$position = strpos(" ".strtolower($headers->fromaddress), strtolower($search));
$position2 = strpos(" ".strtolower($headers->subject), strtolower($search));
if ($position || $position2) {
if ($expunge<$maxtoexpunge) {
imap_delete ($connection, $some[$msgno]);
echo "<br/>Deleting:".$expunge." sequence [".$msgno."]=".$some[$msgno]." ".$headers->fromaddress.", subject:".$headers->subject;
$expunge++;
} else {
echo "<br/>Skipping:"." sequence [".$msgno."]=".$some[$msgno]." ".$headers->fromaddress.", subject:".$headers->subject;
}
}
}
// The expunge command deletes after all action has been taken. DON'T call it until very end, as it will otherwise mess up message numbers!!
imap_expunge ($connection);
echo "<br/>Expunged!!<br/>";
/* Here are the keywords to be used for the imap search order.
toaddress - full to: line, up to 1024 characters
to - an array of objects from the To: line, with the following properties: personal, adl, mailbox, and host
fromaddress - full from: line, up to 1024 characters
from - an array of objects from the From: line, with the following properties: personal, adl, mailbox, and host
ccaddress - full cc: line, up to 1024 characters
cc - an array of objects from the Cc: line, with the following properties: personal, adl, mailbox, and host
bccaddress - full bcc: line, up to 1024 characters
bcc - an array of objects from the Bcc: line, with the following properties: personal, adl, mailbox, and host
reply_toaddress - full Reply-To: line, up to 1024 characters
reply_to - an array of objects from the Reply-To: line, with the following properties: personal, adl, mailbox, and host
senderaddress - full sender: line, up to 1024 characters
sender - an array of objects from the Sender: line, with the following properties: personal, adl, mailbox, and host
return_pathaddress - full Return-Path: line, up to 1024 characters
return_path - an array of objects from the Return-Path: line, with the following properties: personal, adl, mailbox, and host
remail -
date - The message date as found in its headers
Date - Same as date
subject - The message subject
Subject - Same as subject
in_reply_to -
message_id -
newsgroups -
followup_to -
references -
Recent - R if recent and seen, N if recent and not seen, ' ' if not recent.
Unseen - U if not seen AND not recent, ' ' if seen OR not seen and recent
Flagged - F if flagged, ' ' if not flagged
Answered - A if answered, ' ' if unanswered
Deleted - D if deleted, ' ' if not deleted
Draft - X if draft, ' ' if not draft
Msgno - The message number
MailDate -
Size - The message size
udate - mail message date in Unix time
fetchfrom - from line formatted to fit fromlength characters
fetchsubject - subject line formatted to fit subjectlength characters
*/
?>
I was looking for a bash script and as I did not found one I made my own, only external programm needed is socat for SSL/TLS connect to server:
#!/bin/bash -
#===============================================================================
#
# FILE: delete-mail.sh
#
USAGE="./delete-mail.sh user pass [server] [port] [max_delete]"
#
# PARAMETER: user pass - login credentials, mandatory
# server - mail server, default: imap.1und1.de
# port - port on mail server, default: pop3s
# max_delete - maximum # of messages to delete, default: 1000
#
DESCRIPTION="delete max_delete messages in given pop3 mailbox"
#
# AUTHOR: KayM (), kay#rrr.de
# CREATED: 09.06.2021 18:03:58
# REVISION: ---
#===============================================================================
# check args
if [[ "$1" == "-h"* ]]; then
printf "usage: %s\ndescr: %s\n" "$USAGE" "$DESCRIPTION" 1>&2
exit
elif [ "$#" -lt 2 ]; then
printf "Missing args, usage: %s\n" "$USAGE"
exit 1
fi
# assign values
USER="$1"
PASS="$2"
SERVER="${3:-imap.1und1.de}"
PORT="${4:-pop3s}"
MAX_MESSAGE="${5:-100}"
# define actions
popserver() { socat - "OPENSSL:$SERVER:$PORT" 2>/dev/null; }
poplogin() {
printf "USER %s\n" "$USER"
sleep 0.5
printf "PASS %s\n" "$PASS"
sleep 0.5
}
deletemsg() {
for (( j = 1 ; j <= MAX_MESSAGE; j++ ))
do
printf "DELE %s\n" "$j"
sleep 0.5
done
}
popstat() { echo "STAT"; }
popquit() { echo "QUIT"; }
checkresult() {
local line
while read -r line
do
[[ "$line" == "-ERR "* ]] && MAX_MESSAGE=0
printf "%s\n" "$line"
done
}
# get message count, remove newline
RES="$({ poplogin; popstat; } | popserver | tail -n 1 )"
RES="${RES//[$'\r\n']}"
if [[ "$RES" == "+OK 0 "* ]]; then
printf "No messages to delete (%s)\n" "$RES"
exit
elif [[ "$RES" == "+OK "* ]]; then
: "${RES#+OK }"; NUM="${_%% *}"
printf "%s Messages found (%s)\n" "$NUM" "$RES"
[ "$NUM" -lt "$MAX_MESSAGE" ] && MAX_MESSAGE="$NUM"
else
printf "Can't get number of messages: %s\n" "$RES"
exit 1
fi
if [ "$MAX_MESSAGE" = "0" ]; then
printf "Keep all messages\n"
exit
fi
# delete messages
printf "Delete %s messages ...\n" "$MAX_MESSAGE"
{ poplogin; deletemsg; popstat; popquit;} | popserver | checkresult
Is it possible to send multiple attachments with uuencode and sendmail?
In a script I have a variable containing the files that need to be attached to a single e-mail like:
$attachments=attachment_1.pdf attachment_2.pdf attachment_3.pdf attachment_4.pdf
Also a $template variable like:
$template="Subject: This is the subject
From: no-reply#domain.com
To: %s
Content-Type: text/plain
This is the body.
"
I have come up with:
printf "$template" "$recipient" |
sendmail -oi -t
Somewhere within this I must attach everything in the $attachments variable?
uuencode attachemnts and send via sendmail
Sending MIME attachemnts is better.
uuencode is simpler to implement in scripts but email some clients DO NOT support it.
attachments="attachment_1.pdf attachment_2.pdf attachment_3.pdf attachment_4.pdf"
recipient='john.doe#example.net'
# () sub sub-shell should generate email headers and body for sendmail to send
(
# generate email headers and begin of the body asspecified by HERE document
cat - <<END
Subject: This is the subject
From: no-reply#domain.com
To: $recipient
Content-Type: text/plain
This is the body.
END
# generate/append uuencoded attachments
for attachment in $attachments ; do
uuencode $attachment $attachment
done
) | /usr/sbin/sendmail -i -- $recipient
For what its worth, mailx also works well.
mailx -s "Subject" -a attachment1 -a attachement2 -a attachment3 email.address#domain.com < /dev/null