I want to clean my mailbox from mails from specific address
I have thousands of messages, I want to do this in bash script, and run it from time to time (a receive SPAM from different addresses, and unfortunately my "spam filters" have only small effect on them)
To interact with a mail server through command line, you could use either telnet or openssl.
You can connect to your pop server using the following command (I've taken gmail as an example. You'll have to look for your email host pop3 address and socket.) :
openssl s_client -connect pop.gmail.com:995 -quiet
As this command is interactive, it will ask for a username, a password and a serie of commands.
expect is a tool that can automate interaction with interactive commands. The basic syntax is as follow : expect "somestring" action -> If the program we monitor displays "somestring", we execute the action.
Here is a script that would delete all the messages present on your email address :
#!/usr/bin/expect
#you can modify the timeout if the script fails
set timeout 1
#our connection variables
set ip "pop.gmail.com"
set socket "995"
set user "user.name"
set pass "password"
#we set the address we want to remove mails from here. Escape special regex characters such as dots.
set target_address "mail\.example#gmail\.com"
#we launch the subprocess we want to interact with
spawn openssl s_client -connect $ip:$socket -quiet
#if connection went all right, we try to login
expect -re ".OK.*" {send "user $user\r"}
expect -re ".OK.*" {send "pass $pass\r"}
#if login went alright, we try to count the messages on the server
#you will get the following output :
#+OK NB_MSG TOTAL_SIZE
expect -re ".OK.*" {send "stat\r"}
#if the stat command went alright ...
expect -re ".OK.*" {
#we extract the number of mail from the output of the stat command
set mail_count [lindex [split [lindex [split $expect_out(buffer) \n] 1] " "] 1]
#we iterate through every email...
for {set i 1} {$i <= $mail_count} {incr i 1} {
#we retrieve the header of the email
send "top $i 0\r"
#if the header contains "To: $target_address" (or "To: <$target_address>" or "To: Contact Name <$target_address>" ...)
#to filter according to the sender, change the regex to "\nFrom: ..."
expect -re "\nTo: \[^\n\]*$target_address" {
#we delete the email
send "dele $i\r"
}
}
}
expect default
You might need to alter your email account settings to allow the use of external programs
Here I have written a similar script in PHP, by use of the IMAP function of PHP. I did not have Linux, so I could not use the bash script. However. I thought that sharing the PHP-script would provide a solution for those running WINDOWS and having access
<?php
echo "My showmail.php script";
// This is the format for opening the email connection:
//
// domain prt foldr[.<.subfolder........>] my#blp.com xxxxxxxx
// $connection = imap_open('{imap.one.com:143}Inbox.<write subfolder here>','email-name','password');
// Execution: (write this in the browsers address bar)
http://digitalageinstitute.com/showmail.php?fromdate=01%20January%202009&todate=28%20August%202018&search=#weheartit.com&email=david.svarrer#digitalageinstitute.com&password=xxxxxxx&domain=imap.one.com&folder=Inbox
//
// This line above, entered into a browsers address bar will execute this script (named showmail.php),
// from the servers root (www.digitalageinstitute.com), and
//
// will delete emails from the mail folder "Inbox" (case sensitive) for the user david.svarrer#digitalageinstitute.com, by use of
// the IMAP domain (this is where you access emails from outside) imap.one.com and at port 143 (see this, further down)
//
$emailaddress = $_GET["email"];
$emailpassword= $_GET["password"];
$fromdate = $_GET["fromdate"];
$todate = $_GET["todate"];
$search = $_GET["search"];
$domain = $_GET["domain"];
$folder = $_GET["folder"];
echo $emailaddress." ".$emailpassword." ".$fromdate." ".$todate." ".$search." ".$domain." ".$folder;
$connection = imap_open('{'.$domain.':143}'.$folder,$emailaddress,$emailpassword);
$ccount = imap_num_msg($connection);
$maxtoexpunge=20000;
echo "Parameters = ".$fromdate.",".$todate.",".$search.":".'ALL FROM "'.$search.'" SINCE "'.$fromdate.'" BEFORE "'.$todate.'"';
// $some = imap_search($connection, 'ALL FROM "Twitter"');
$some = imap_search($connection, 'ALL FROM "'.$search.'" SINCE "'.$fromdate.'" BEFORE "'.$todate.'"');
print_r ($some);
echo "<br/>Elements: ".count($some)."<br/>";
$expunge = 0;
for($msgno = 0; $msgno < count($some) ; $msgno++) {
// echo "Fetching element ".$some[$msgno]."<br/>";
$headers = imap_headerinfo($connection, $some[$msgno]);
//$position = True;
$position = strpos(" ".strtolower($headers->fromaddress), strtolower($search));
$position2 = strpos(" ".strtolower($headers->subject), strtolower($search));
if ($position || $position2) {
if ($expunge<$maxtoexpunge) {
imap_delete ($connection, $some[$msgno]);
echo "<br/>Deleting:".$expunge." sequence [".$msgno."]=".$some[$msgno]." ".$headers->fromaddress.", subject:".$headers->subject;
$expunge++;
} else {
echo "<br/>Skipping:"." sequence [".$msgno."]=".$some[$msgno]." ".$headers->fromaddress.", subject:".$headers->subject;
}
}
}
// The expunge command deletes after all action has been taken. DON'T call it until very end, as it will otherwise mess up message numbers!!
imap_expunge ($connection);
echo "<br/>Expunged!!<br/>";
/* Here are the keywords to be used for the imap search order.
toaddress - full to: line, up to 1024 characters
to - an array of objects from the To: line, with the following properties: personal, adl, mailbox, and host
fromaddress - full from: line, up to 1024 characters
from - an array of objects from the From: line, with the following properties: personal, adl, mailbox, and host
ccaddress - full cc: line, up to 1024 characters
cc - an array of objects from the Cc: line, with the following properties: personal, adl, mailbox, and host
bccaddress - full bcc: line, up to 1024 characters
bcc - an array of objects from the Bcc: line, with the following properties: personal, adl, mailbox, and host
reply_toaddress - full Reply-To: line, up to 1024 characters
reply_to - an array of objects from the Reply-To: line, with the following properties: personal, adl, mailbox, and host
senderaddress - full sender: line, up to 1024 characters
sender - an array of objects from the Sender: line, with the following properties: personal, adl, mailbox, and host
return_pathaddress - full Return-Path: line, up to 1024 characters
return_path - an array of objects from the Return-Path: line, with the following properties: personal, adl, mailbox, and host
remail -
date - The message date as found in its headers
Date - Same as date
subject - The message subject
Subject - Same as subject
in_reply_to -
message_id -
newsgroups -
followup_to -
references -
Recent - R if recent and seen, N if recent and not seen, ' ' if not recent.
Unseen - U if not seen AND not recent, ' ' if seen OR not seen and recent
Flagged - F if flagged, ' ' if not flagged
Answered - A if answered, ' ' if unanswered
Deleted - D if deleted, ' ' if not deleted
Draft - X if draft, ' ' if not draft
Msgno - The message number
MailDate -
Size - The message size
udate - mail message date in Unix time
fetchfrom - from line formatted to fit fromlength characters
fetchsubject - subject line formatted to fit subjectlength characters
*/
?>
I was looking for a bash script and as I did not found one I made my own, only external programm needed is socat for SSL/TLS connect to server:
#!/bin/bash -
#===============================================================================
#
# FILE: delete-mail.sh
#
USAGE="./delete-mail.sh user pass [server] [port] [max_delete]"
#
# PARAMETER: user pass - login credentials, mandatory
# server - mail server, default: imap.1und1.de
# port - port on mail server, default: pop3s
# max_delete - maximum # of messages to delete, default: 1000
#
DESCRIPTION="delete max_delete messages in given pop3 mailbox"
#
# AUTHOR: KayM (), kay#rrr.de
# CREATED: 09.06.2021 18:03:58
# REVISION: ---
#===============================================================================
# check args
if [[ "$1" == "-h"* ]]; then
printf "usage: %s\ndescr: %s\n" "$USAGE" "$DESCRIPTION" 1>&2
exit
elif [ "$#" -lt 2 ]; then
printf "Missing args, usage: %s\n" "$USAGE"
exit 1
fi
# assign values
USER="$1"
PASS="$2"
SERVER="${3:-imap.1und1.de}"
PORT="${4:-pop3s}"
MAX_MESSAGE="${5:-100}"
# define actions
popserver() { socat - "OPENSSL:$SERVER:$PORT" 2>/dev/null; }
poplogin() {
printf "USER %s\n" "$USER"
sleep 0.5
printf "PASS %s\n" "$PASS"
sleep 0.5
}
deletemsg() {
for (( j = 1 ; j <= MAX_MESSAGE; j++ ))
do
printf "DELE %s\n" "$j"
sleep 0.5
done
}
popstat() { echo "STAT"; }
popquit() { echo "QUIT"; }
checkresult() {
local line
while read -r line
do
[[ "$line" == "-ERR "* ]] && MAX_MESSAGE=0
printf "%s\n" "$line"
done
}
# get message count, remove newline
RES="$({ poplogin; popstat; } | popserver | tail -n 1 )"
RES="${RES//[$'\r\n']}"
if [[ "$RES" == "+OK 0 "* ]]; then
printf "No messages to delete (%s)\n" "$RES"
exit
elif [[ "$RES" == "+OK "* ]]; then
: "${RES#+OK }"; NUM="${_%% *}"
printf "%s Messages found (%s)\n" "$NUM" "$RES"
[ "$NUM" -lt "$MAX_MESSAGE" ] && MAX_MESSAGE="$NUM"
else
printf "Can't get number of messages: %s\n" "$RES"
exit 1
fi
if [ "$MAX_MESSAGE" = "0" ]; then
printf "Keep all messages\n"
exit
fi
# delete messages
printf "Delete %s messages ...\n" "$MAX_MESSAGE"
{ poplogin; deletemsg; popstat; popquit;} | popserver | checkresult
Related
I have a bash script that should execute mongodump on multiple databases within a single mongo instance.
The script should dynamically inject the Mongo-URIs and backup filenames as below :
mongodump --uri=$endpoint --authenticationDatabase admin --gzip --archive=/tmp/$filename.gz
When the script runs its facing an error - it appears its not injecting the variables correctly :
dump.sh
Starting-BACKUP
dump.sh
platforms-2022-10-25-19:05:20
sports-2022-10-25-19:05:20
mongodb://root:mypassword#mongodb-prom/platforms
mongodb://root:mypassword#mongodb-prom/sports
2022-10-25T19:05:20.392+0000 Failed: error creating intents to dump: error getting collections for database `platformsmongodb://root:mypassword#mongodb-prom/sports`: (InvalidNamespace) Invalid database name: 'platformsmongodb://root:mypassword#mongodb-prom/sports'
2022-10-25T19:05:50.409+0000 Failed: can't create session: could not connect to server: server selection error: server selection timeout, current topology: { Type: Single, Servers: [{ Addr: localhost:27017, Type: Unknown, Last error: connection() error occurred during connection handshake: dial tcp [::1]:27017: connect: connection refused }, ] }
Database backup was successful
Both the Mongo-URIs and backup filenames are being populated from arrays as below :
#declare -a BACKUP_NAMES=()
#declare -a BACKUP_ENDPOINTS=()
#declare –a DATABASES=()
#declare –a RAW_DBNAMES=()
#declare –a DATABASES=()
DATABASES+=("platforms")
DATABASES+=("sports")
BACKUP_RETURN_CODE=${?}
MONGODB_URI="mongodb://root:mypassword#mongodb-prom/"
NOW="$(date +"%F")-$(date +"%T")"
#array for raw db names
for DATABASE in "${DATABASES[#]}";
do
RAW_DBNAMES+=("$DATABASE")
done
#array for constructing backup filenames
for DATABASE in "${DATABASES[#]}";
do
#echo $DATABASE
BACKUP_NAMES+=("$DATABASE")
done
#construct backup filenames
cnt=${#BACKUP_NAMES[#]}
for ((i=0;i<cnt;i++)); do
BACKUP_NAMES[i]="${BACKUP_NAMES[i]}-$NOW"
echo "${BACKUP_NAMES[i]}"
done
#construct db endpoints
for ((i=0;i<cnt;i++)); do
uri="$MONGODB_URI${RAW_DBNAMES[i]}"
echo "$uri"
BACKUP_ENDPOINTS+=$uri
done
#pass BACKUP_ENDPOINTS & BACKUP_FILENAMES to mongodump
for ((i=0; i<${#BACKUP_NAMES[#]}; i++));
do
mongodump --uri="${BACKUP_ENDPOINTS[$i]}" --authenticationDatabase admin --gzip --archive=/tmp/"${BACKUP_NAMES[$i]}".gz
done
#If the return code is non-zero then the backup was not successful
if [[ 0 != ${BACKUP_RETURN_CODE} ]]
then
echo "Database backup has failed"
exit ${BACKUP_RETURN_CODE}
else
echo "Database backup was successful"
fi
exit 0
As a check I used these two sample arrays and I have verified that I can indeed loop through two arrays at the same time although this particular example is concatenating the elements (Is this how it works in general?):
array1=( 1 2 3 )
array2=("toronto""new york")
for ((i=0; i<${#array1[#]}; i++));
do echo "${array1[$i]}${array2[$i]}";
done
but I am not sure why in my case the strings are concatenating so wrongly.
What am I missing ?
You don't need multiple arrays create just one and rearrange it like this:
arr=(
# id item product
1 apple juice
2 banana smuzi
3 'raw potato' chips
# ...
)
N=${#arr[*]} # number of items in array
C=3 # number of 'columns'
Then loop over it like so:
for ((i=0; i<$N; i+=$C)); {
read id item product <<< "${arr[#]:$i:$C}"
echo "id: $id"
echo "item: $item"
echo "product: $product"
echo '-----------------'
}
Result:
id: 1
item: apple
product: juice
-----------------
id: 2
item: banana
product: smuzi
-----------------
id: 3
item: raw
product: potato chips
-----------------
p.s. check out this backup script for example it's for psql but probably could be useful.
Yes, it turned i was instantiating the BACKUP_ENDPOINTS array the wrong way.
This is the correct way :
#construct db endpoints
for ((i=0;i<cnt;i++)); do
# uri="$MONGODB_URI${RAW_DBNAMES[i]}"
# echo "$uri"
# BACKUP_ENDPOINTS+=$uri THIS IS WRONG !!!
BACKUP_ENDPOINTS[i]="$MONGODB_URI${RAW_DBNAMES[i]}"
done
for ENDPOINT in "${BACKUP_ENDPOINTS[#]}"; do
echo "$ENDPOINT"
done
#pass BACKUP_ENDPOINTS & BACKUP_FILENAMES to mongodump
for ((i=0; i<${#BACKUP_NAMES[#]}; i++));
do
mongodump --uri="${BACKUP_ENDPOINTS[$i]}" --authenticationDatabase admin --gzip --archive=/tmp/"${BACKUP_NAMES[$i]}".gz
done
Should able to process larger log files and provide exception message reports
After completion of log analysis, report notification trigger to specific mail id's.
And also please suggest which framework is the best for processing large files.[eg: spring boot/batch]
I would suggest to go with ELK stack. Stream the logs to elastic search and set up alerts in Kibana.
Can use sendmail client on system and run script on that system to send alert on any Exception.
exception="Exception" # "Error", "HTTP 1.1 \" 500", etc
ignoredException="ValidationException"
# log file to scan
logFileToScan=/var/log/tomcat8/log/application.log
# file where we will keep log of this script
logFilePath=/home/ec2-user/exception.log
# a file where we store till what line the log file has been scanned
# initalize it with 0
countPath=/home/ec2-user/lineCount
# subject with which you want to receive the mail regading Exception
subject="[ALERT] Exception"
# from whom do you want to send the mail regarding Exception
from="abc#abc.com"
# to whom do you want to send the mail
to="xyz#xyz.com"
# number of lines, before the line containing the word to be scanned, to be sent in the mail
linesBefore=1
# number of lines, before the line containing the word to be scanned, to be sent in the mail
linesAfter=4
# start line
fromLine=`cat $countPath`
# current line count in the file
toLine=`wc -l $logFileToScan | awk '{print $1}'`
#logs are rolling so if fromLine has a value greater than toLine then fromLine has to be set to 0
if [ "$fromLine" == "" ]; then
fromLine=0
echo `date` fromLine values was empty, set to 0 >> $logFilePath
elif [ $fromLine -gt $toLine ]; then
echo `date` logfile was rolled, updating fromLine from $fromLine to 0 >> $logFilePath
fromLine=0
fi
# if from n to lines are equal then no logs has been generated since last scan
if [ "$fromLine" == "$toLine" ]; then
echo `date` no logs genetared after last scan >> $logFilePath
else
echo `date` updating linecount to $toLine >> $logFilePath
echo $toLine > $countPath
logContent=`tail -n +"$fromLine" $logFileToScan | head -n "$((toLine - fromLine))" | grep -v $ignoredException | grep -A $linesAfter -B $linesBefore $exception`
logContent=`echo $logContent | cut -c1-2000`
if [ "$logContent" == "" ]; then
echo `date` no exception found >> $logFilePath
else
/usr/sbin/sendmail $to <<EOF
subject: $subject
from: $from
logContent=$logContent
EOF
fi
fi
I have the following bash script for sending an email reports. I am trying to get it working correctly with mail command on bash and want to use variable substitution conditional which requires to use eval.
#!/bin/bash
LOG="/tmp/error.txt"
echo "test" > $LOG
INCLUDED="aaaaaaaaaaaaaaaaa"
EXCLUDED="bbbbbbbbbbbbbbbbb"
STR=INCLUDED
BODY="
Email Body:
\${!STR}"
#Set STR based on some condition
STR=EXCLUDED # set conditional with if statement
SUB="email subject"
PARAMS="-r \" Sender <ret#address.com>\""
PARAMS+=" -s \"$SUB\""
PARAMS+=" -A $LOG" # use conditional with if statement
eval echo \"MESSAGE: "${BODY}"\" | mail $(eval echo ${PARAMS}) "user#email.com"
While running the below code, I have to pass $PARAMS output without the quotes around the string as input to mail command, which does not recognise the diff options. The closest I can get to is using calling $(eval echo $PARAMS), but this garbels the subject since it remove the double quotes around the subject.
+ LOG=/tmp/error.txt
+ echo test
+ INCLUDED=aaaaaaaaaaaaaaaaa
+ EXCLUDED=bbbbbbbbbbbbbbbbb
+ STR=INCLUDED
+ BODY='
Email Body:
${!STR}'
+ STR=EXCLUDED
+ SUB='email subject'
+ PARAMS='-r " Sender <ret#address.com>"'
+ PARAMS+=' -s "email subject"'
+ PARAMS+=' -A /tmp/error.txt'
+ eval echo '"MESSAGE:' '
Email Body:
${!STR}"'
++ echo 'MESSAGE:
Email Body:
bbbbbbbbbbbbbbbbb'
++ eval echo -r '"' Sender '<ret#address.com>"' -s '$SUB' -A /tmp/error.txt
+++ echo -r ' Sender <ret#address.com>' -s email subject -A /tmp/error.txt
+ mail -r Sender '<ret#address.com>' -s email subject -A /tmp/error.txt user#email.com
How do I pass the $PARAMS to the mail command with passing the subject (-s) as quoted string ?
Use an array instead and put each string in an array element.
params=('-r')
params=("${params[#]}" ' Sender <ret#address.com>')
mail "${params[#]}"
Or simpler
params=('-r')
params+=(' Sender <ret#address.com>')
mail "${params[#]}"
See here: Bash: add value to array without specifying a key
Update, because I think you lack some fundamental shell programming knowledge.
Don't use eval! If you need to delay an evaluation, write a function, which performs the evaluation, when you call it.
Use single quotes for static strings and double quotes for strings containing variables.
Use arrays, if you need arrays.
Group commands in curly braces, if you need to group commands.
This is you example updated according to the rules above.
LOG=/tmp/error.txt
echo 'test' > $LOG
INCLUDED='aaaaaaaaaaaaaaaaa'
EXCLUDED='bbbbbbbbbbbbbbbbb'
STR=INCLUDED
body () { echo "
Email Body:
$STR"
}
#Set STR based on some condition
STR=EXCLUDED # set conditional with if statement
SUB='email subject'
PARAMS=('-r' 'Sender <ret#address.com>')
PARAMS+=('-s' "$SUB")
PARAMS+=('-A' "$LOG") # use conditional with if statement
{ echo -n 'MESSAGE: '; body; } | mail "${PARAMS[#]}" 'user#email.com'
You can test your code by putting a mail mock-up at the beginning of the program.
mail() { echo mail "$#"; cat; }
i tried to setup squid3 with multiple auth_param. Basically, the first choice should be basic_ldap_auth and if this doesnt return OK it should try basic_ncsa_auth with the same values. As far as i know squid doesnt support it however there is the possibility to use "external" ACL
auth_param basic program /usr/lib/squid3/basic_fake_auth
external_acl_type MultAuth %SRC %LOGIN %{Proxy-Authorization} /etc/squid3/multAuth.pl
acl extAuth external MultAuth
my "multAuth.pl"
use URI::Escape;
use MIME::Base64;
$|=1;
while (<>) {
($ip,$user,$auth) = split();
# Retrieve the password from the authentication header
$auth = uri_unescape($auth);
($type,$authData) = split(/ /, $auth);
$authString = decode_base64($authData);
($username,$password) = split(/:/, $authString);
# do the authentication and pass results back to Squid.
$ldap = `/bin/bash auth/ldap.sh`;
if ($ldap == "OK") {
print "OK";
}
$ncsa = `/bin/bash auth/ncsa.sh`;
if ($ncsa == "OK") {
print "OK";
} else {
print "ERR";
}
}
now i am trying to run with ncsa.sh and ldap.sh the "normal" shell command for these auth methods.
./basic_ldap_auth -R -b "dc=domain,dc=de" -D "CN=Administrator,CN=Users,DC=domain,DC=de" -w "password" -f sAMAccountName=%s -h domain.de
user password
and
./basic_ncsa_auth /etc/squid3/users
user password
Therefor i ran:
auth/ncsa.sh
#!/usr/bin/expect
eval spawn [lrange $argv 0 end]
expect ""
send [lindex $argv 1]
send '\r'
expect {
"OK" {
echo OK
exp_continue
}
"ERR" {
echo ERR
exp_continue
}
interact
with
./ncsa.sh "/usr/lib/squid3/basic_ncsa_auth /etc/squid3/users" "user password"
and i generate the following error:
couldn't execute "/usr/lib/squid3/basic_ncsa_auth /etc/squid3/users": no such file or directory
while executing
"spawn {/usr/lib/squid3/basic_ncsa_auth /etc/squid3/users} {user password}"
("eval" body line 1)
invoked from within
"eval spawn [lrange $argv 0 end]"
(file "./ncsa.sh" line 2)
Besides this error, i am not sure how to pass the variables (username & password) forward and i am also not sure how to answer the shell questions like for example the user & pw input for basic_ldap_auth .
Is there a nice way how to solve that? or any other good plan ?
thanks!
FWIW, the following script helped me transition from passwd based to LDAP based authentication.
Contrary to your requirements, my script acts the other way around: It first checks passwd, then LDAP.
#!/usr/bin/env bash
# multiple Squid basic auth checks
# originally posted here: https://github.com/HackerHarry/mSquidAuth
#
# credits
# https://stackoverflow.com/questions/24147067/verify-user-and-password-against-a-file-created-by-htpasswd/40131483
# https://stackoverflow.com/questions/38710483/how-to-stop-ldapsearch1-from-base64-encoding-userpassword-and-other-attributes
#
# requires ldap-utils, openssl and perl
# tested with Squid 4 using a "auth_param basic program /usr/lib/squid/mSquidAuth.sh" line
# authenticate first against squid password file
# if this fails, try LDAP (Active Directory) and also check group membership
# variables
# sLOGFILE=/var/log/squid/mSquidAuth.log
sPWDFILE="/etc/squid/passwd"
sLDAPHOST="ldaps://dc.domain.local:636"
sBASE="DC=domain,DC=local"
sLDS_OPTIONS="-o ldif-wrap=no -o nettimeout=7 -LLL -P3 -x "
sBINDDN="CN=LDAP-read-user,OU=Users,DC=domain,DC=local"
sBINDPW="read-user-password"
sGROUP="Proxy-Users"
# functions
function _grantAccess {
# echo "access granted - $sUSER" >>$sLOGFILE
echo "OK"
}
function _denyAccess {
# echo "access denied - $sUSER" >>$sLOGFILE
echo "ERR"
}
function _setUserAndPass {
local sAuth="$1"
local sOldIFS=$IFS
IFS=' '
set -- $sAuth
IFS=$sOldIFS
# set it globally
sUSER="$1"
sPASS="$2"
}
# loop
while (true); do
read -r sAUTH
sUSER=""
sPASS=""
sSALT=""
sUSERENTRY=""
sHASHEDPW=""
sUSERDN=""
iDNCOUNT=0
if [ -z "$sAUTH" ]; then
# echo "exiting" >>$sLOGFILE
exit 0
fi
_setUserAndPass "$sAUTH"
sUSERENTRY=$(grep -E "^${sUSER}:" "$sPWDFILE")
if [ -n "$sUSERENTRY" ]; then
sSALT=$(echo "$sUSERENTRY" | cut -d$ -f3)
if [ -n "$sSALT" ]; then
sHASHEDPW=$(openssl passwd -apr1 -salt "$sSALT" "$sPASS")
if [ "$sUSERENTRY" = "${sUSER}:${sHASHEDPW}" ]; then
_grantAccess
continue
fi
fi
fi
# LDAP is next
iDNCOUNT=$(ldapsearch $sLDS_OPTIONS -H "$sLDAPHOST" -D "$sBINDDN" -w "$sBINDPW" -b "$sBASE" "(|(sAMAccountName=${sUSER})(userPrincipalName=${sUSER}))" dn 2>/dev/null | grep -cE 'dn::? ')
if [ $iDNCOUNT != 1 ]; then
# user needs a unique account
_denyAccess
continue
fi
# get user's DN
# we need the extra grep in case we get lines back starting with "# refldap" :/
sUSERDN=$(ldapsearch $sLDS_OPTIONS -H "$sLDAPHOST" -D "$sBINDDN" -w "$sBINDPW" -b "$sBASE" "(|(sAMAccountName=${sUSER})(userPrincipalName=${sUSER}))" dn 2>/dev/null | perl -MMIME::Base64 -n -00 -e 's/\n +//g;s/(?<=:: )(\S+)/decode_base64($1)/eg;print' | grep -E 'dn::? ' | sed -r 's/dn::? //')
# try and bind using that DN to check password validity
# also test if that user is member of a particular group
# backslash in DN needs special treatment
if ldapsearch $sLDS_OPTIONS -H "$sLDAPHOST" -D "$sUSERDN" -w "$sPASS" -b "$sBASE" "name=${sGROUP}" member 2>/dev/null | perl -MMIME::Base64 -n -00 -e 's/\n +//g;s/(?<=:: )(\S+)/decode_base64($1)/eg;print' | grep -q "${sUSERDN/\\/\\\\}"; then
_grantAccess
continue
fi
_denyAccess
done
Hi I am new to shell scripting.
my requirement is:
There is one server & 3 clients. On each client error logs files are generated which are of 4 types.Say type 1 error , type 2 error like this type 4 error.
I want to write a script which read all the 3 clients from server & provide me number of times 4 different type of error logs are genereted on each client.
In short it should use ssh & grep command combination.
I have written the demo script but it's not providing me the number of times different type of logs occured on clients.
#error[1]='Exception: An application error occurred during an address lookup request, please contact IT'
#error[2]='SocketTimeoutException: Read timed out'
#error[3]='Exception: The search has produced too many matches to be returned'
#error[4]='Exception: No matching address found'
error_1='exception 1'
error_2='exception 2'
function get_list_of_clients()
{
NUM_OF_CLIENTS=$(wc -l ${CLIENT_IP_LIST} | awk -F " " '{ print $1 }' )
echo $NUM_OF_CLIENTS
if [ "${NUM_OF_CLIENTS}" -gt 0 ]
then
for ((row=2; row<=$NUM_OF_CLIENTS; row++))
do
CLIENTS_IP=$(sed -n ${row}p ${CLIENT_IP_LIST}| awk -F " " '{print $3 }')
echo ${CLIENTS_IP}
# get_number_of_errors
# copy_count_errors
echo ${$error_$row}
done
fi
}
function get_number_of_errors()
{
for((row_no=1; row_no<=4; row_no++))
do
{
/usr/bin/expect - <<- EndMark
spawn ssh root#${CLIENTS_IP} "grep $error[$row_no] var/error.log |wc -l" >> /tmp/${CLIENTS_IP}_error${row_no}.txt
match_max 50000
expect {
"*yes/no*" {
send -- "yes\r"
send -- "\r"
exp_continue
}
"*?assword:*" {
send -- "${CLIENT_PASSWORD}\r"
send -- "\r"
}
}
expect eof
EndMark
}
done
}
function copy_count_errors()
{
/usr/bin/expect - <<- EndMark
spawn scp root#${CLIENTS_IP}:/tmp/${CLIENTS_IP}* /tmp/
match_max 50000
expect {
"*yes/no*" {
send -- "yes\r"
send -- "\r"
exp_continue
}
"*?assword:*" {
send -- "${CLIENT_PASSWORD}\r"
send -- "\r"
}
}
expect eof
EndMark
}
get_list_of_clients
================================================================================
please help.
This is not really an answer, an attempt to help you getting your own.
The Problem
If I understand it correctly:
Your script runs on the server
You have three clients, each has log files
Your list of clients is in a file named $CLIENT_IP_LIST, where the IP is the third field and the first line is some sort of header, which you want to exclude.
Suggestions
It seems you need to ssh and scp to the clients. If possible, I suggest setting up SSH Public Key Authentication between your server and clients. This will greatly simplify your scripts.
Use the -C flag for compression with the scp and ssh commands.
You should copy the log files from the clients to the server and do processing on the server.
If you can choose a different scripting language such as Python, Tcl (You used Expect, which is really Tcl). Bash can handle your problem, but you will find other languages work better. This is my opinion, of course.
Get one piece of your puzzle to work before moving on to the next. For example, right now, your get_list_of_clients() function does not yet working.
That being said, here is a rewrite of get_list_of_clients, which I tested and it works within my assumptions about your $CLIENT_IP_LIST file:
function get_list_of_clients() {
let row=1
while read line
do
if (( row > 1 ))
then
set $line # Breaks line into pieces
CLIENT_IP="$3"
echo "$CLIENT_IP"
fi
let row=row+1
done < $CLIENT_IP_LIST
}