Need some help on awk/grep command on shell script - shell

This is in reference to Need some help on shell scipt (using grep command)
I have a log file and I am trying to write a shell script to find pause/play happening or not (Success /Failure).
The script should do the following:
if ("trickModeRequest":{"rate":0.0})
{
if ("status":"OK")
printf("Pause Successful");
else if ("status":"GENERAL_ERROR")
printf("Pause failed");
else if ("status":"UNRECOGNIZED REQUEST")
printf("Pause failed");
}
Similarly:
if ("trickModeRequest":{"rate":1.0})
{
if ("status":"OK")
printf("Play Successful");
else if ("status":"GENERAL_ERROR")
printf("Play failed");
else if ("status":"UNRECOGNIZED REQUEST")
printf("Play failed");
}
My script file:
logfile=$1
awk '/trickModeRequest/ && /status\"\:\"GENERAL_ERROR/{ print "trickModeRequestFAILED" } /trickModeRequest/ && /status\":\"OK/ {print "trickModeRequestSUCCESS"} ' $logfile
The problem is that {"trickModeRequest":{"rate":0.0} or {"trickModeRequest":{"rate":1.0} is not placed in same paragraph of logs, so I dont know how to distinguish between pause/play...
My log file (logs.txt):
160125-11:11:16.654574 [mod=CARDLESSCA, lvl=DEBUG] [tid=2343] [RECORD]NAGRA_API:vlhal_CasPvrRecordingStart-post handle=0x007288a8
160125-11:11:16.654617 [mod=CARDLESSCA, lvl=INFO] [tid=2343] Recording nagra request for Recording vlhal_CasPvrPlaybackStart 0x7288a8
160125-11:11:16.655113 [mod=DVR, lvl=ERROR] [tid=2343] VLTune::process_buffer:1088(this=0x58ca00) - is_recording=1(was 0), hal_recording=0x7555ddd8
160125-11:11:16.905656 [mod=SYS, lvl=TRACE] [tid=2343] vl_env_get_bool: key=NAGRA.PRM.SUPPORT.ENABLED, result=1
160125-11:11:16.910125 [mod=SYS, lvl=TRACE] [tid=2343] vl_env_get_bool: key=NAGRA.PRM.SUPPORT.ENABLED, result=1
160125-11:11:16.911879 [mod=DVR, lvl=INFO] [tid=2343] HAL_RECORD_GetData:2989(0x7555ddd8) - /data/data/OCAP_MSV/0/0/DEFAULT_RECORDING_VOLUME/dvr/1453720276418.BOTF_Marker(worte=376)
160125-11:11:16.949874 [mod=SYS, lvl=INFO] [tid=2332] Read message '{"event":1,"handler":1,"name":"onRPCCall","params":{"callGUID":"228dd4a5-6727-4525-abce-331c87d54c18","callParams":[**{"trickModeRequest":{"rate":0.0}}**],"class":"com.comcast.xre.events.XRERPCCallInfo","destinationSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d",**"method":"trickModeRequest"**,"sourceSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d"},"phase":"STANDARD","source":1}
'
160125-11:11:16.950180 [mod=SYS, lvl=INFO] [tid=2332] ======= Message is onRPCCall ======>
160125-11:11:16.950326 [mod=SYS, lvl=INFO] [tid=2332] Entering onRPCCallEvent for request ---> trickModeRequest
160125-11:11:16.950621 [mod=SYS, lvl=INFO] [tid=2332] Received json request = {"event":1,"handler":1,"name":"onRPCCall","params":{"callGUID":"228dd4a5-6727-4525-abce-331c87d54c18","callParams":[**{"trickModeRequest":{"rate":0.0}**}],"class":"com.comcast.xre.events.XRERPCCallInfo","destinationSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d","method":"trickModeRequest","sourceSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d"},"phase":"STANDARD","source":1}
160125-11:11:16.950689 [mod=SYS, lvl=INFO] [tid=2332] trickModeRequest: trickModeRequest
160125-11:11:16.950794 [mod=TUNERSOURCE, lvl=INFO] [tid=2332] Setting rate 0.0, skipping 0 ms
160125-11:11:16.950872 [mod=TUNERSOURCE, lvl=INFO] [tid=2332] trickModeRequest() - Sending the TrickMode event
160125-11:11:16.950959 [mod=SYS, lvl=INFO] [tid=2332] PlaybackResponse: entered 0
160125-11:11:16.950994 [mod=MEDIA_SESSION_MGR, lvl=DEBUG] [tid=2331] MediaSessionManager::trickMode() - Handling trick mode
160125-11:11:16.951059 [mod=MEDIA_SESSION_MGR, lvl=DEBUG] [tid=2331] MediaSessionManager::trickMode() - rate = 0.00000, skip = 0 ms
160125-11:11:16.951123 [mod=PLAYER, lvl=TRACE] [tid=2331] PlayerBase(id=56a602d3)::stop - Before event: state = %s
160125-11:11:16.951182 [mod=PLAYER, lvl=DEBUG] [tid=2331] DecodePlayer(id=56a602d3)::DecodePlayerStatePresenting::stop - entering state %s
160125-11:11:16.951295 [mod=PLAYER, lvl=DEBUG] [tid=2331] PlayerBase(id=56a602d3)::PlayerStateBase::waitForExistingWorkerThread - waiting for the previous worker thread to stop
160125-11:11:16.951361 [mod=PLAYER, lvl=DEBUG] [tid=2331] PlayerBase(id=56a602d3)::PlayerStateBase::waitForExistingWorkerThread - done joining
160125-11:11:16.951535 [mod=PLAYER, lvl=TRACE] [tid=2331] PlayerBase(id=56a602d3)::stop - After event: state = %s
160125-11:11:16.951639 [mod=SYS, lvl=INFO] [tid=2332] ====== Response sending is {"appId":1,"command":"CALL","commandIndex":5,"method":"generateAppEvent","params":[{"class":"com.comcast.xre.events.XREOutgoingEvent","name":"onRPCReturn","params":{"callGUID":"228dd4a5-6727-4525-abce-331c87d54c18","class":"com.comcast.xre.events.XRERPCReturnInfo","destinationSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d",**"method":"trickModeRequest"**,"returnVal":{"class":"com.comcast.parker.SelectServiceSyncResponse","selectServiceStatus":"AWAITING_ASYNC","startTime":1296590759,"**status":"OK"**,"statusMessage":"Tuning to service: "},"sourceSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d"}},"ab00ebea-5f63-4619-9877-273a2bceea1d"],"targetId":1,"targetPath":"","timestamp":0}
====
160125-11:11:18.002455 [mod=SYS, lvl=INFO] [tid=2332] Read message '{"event":1,"handler":1,"name":"onRPCCall","params":{"callGUID":"fe11a665-2aad-404c-a887-bd3d967bbea0","callParams":[{**"trickModeRequest":{"rate":1.0}**}],"class":"com.comcast.xre.events.XRERPCCallInfo","destinationSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d",**"method":"trickModeRequest"**,"sourceSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d"},"phase":"STANDARD","source":1}
'
160125-11:11:18.002756 [mod=SYS, lvl=INFO] [tid=2332] ======= Message is onRPCCall ======>
160125-11:11:18.002843 [mod=SYS, lvl=INFO] [tid=2332] Entering onRPCCallEvent for request ---> trickModeRequest
160125-11:11:18.003106 [mod=SYS, lvl=INFO] [tid=2332] Received json request = {"event":1,"handler":1,"name":"onRPCCall","params":{"callGUID":"fe11a665-2aad-404c-a887-bd3d967bbea0","callParams":[{"trickModeRequest":{"rate":1.0}}],"class":"com.comcast.xre.events.XRERPCCallInfo","destinationSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d",**"method":"trickModeRequest"**,"sourceSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d"},"phase":"STANDARD","source":1}
160125-11:11:18.003168 [mod=SYS, lvl=INFO] [tid=2332] trickModeRequest: trickModeRequest
160125-11:11:18.003264 [mod=TUNERSOURCE, lvl=INFO] [tid=2332] Setting rate 1.0, skipping 0 ms
160125-11:11:18.003315 [mod=TUNERSOURCE, lvl=INFO] [tid=2332] trickModeRequest() - Sending the TrickMode event
160125-11:11:18.003387 [mod=SYS, lvl=INFO] [tid=2332] PlaybackResponse: entered 0
160125-11:11:18.003952 [mod=SYS, lvl=INFO] [tid=2332] ====== Response sending is {"appId":1,"command":"CALL","commandIndex":5,"method":"generateAppEvent","params":[{"class":"com.comcast.xre.events.XREOutgoingEvent","name":"onRPCReturn","params":{"callGUID":"fe11a665-2aad-404c-a887-bd3d967bbea0","class":"com.comcast.xre.events.XRERPCReturnInfo","destinationSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d",**"method":"trickModeRequest"**,"returnVal":{"class":"com.comcast.parker.SelectServiceSyncResponse","selectServiceStatus":"AWAITING_ASYNC","startTime":1296590759,**"status":"OK"**,"statusMessage":"Tuning to service: "},"sourceSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d"}},"ab00ebea-5f63-4619-9877-273a2bceea1d"],"targetId":1,"targetPath":"","timestamp":0}
====
160125-11:11:18.004268 [mod=MEDIA_SESSION_MGR, lvl=DEBUG] [tid=2331] MediaSessionManager::trickMode() - Handling trick mode
160125-11:11:18.004370 [mod=MEDIA_SESSION_MGR, lvl=DEBUG] [tid=2331] MediaSessionManager::trickMode() - rate = 1.00000, skip = 0 ms
160125-11:11:18.004429 [mod=PLAYER, lvl=DEBUG] [tid=2331] PlayerBase(id=56a602d3)::PlayerStateBase::waitForExistingWorkerThread - waiting for the previous worker thread to stop
160125-11:11:18.010346 [mod=SYS, lvl=INFO] [tid=2332] ====== Response of 685 bytes sent ======
160125-11:11:18.010529 [mod=SYS, lvl=INFO] [tid=2332] ProcessJsonData: returned success
160125-11:11:18.010633 [mod=SYS, lvl=INFO] [tid=2332] === OnConnect returned PS_SUCCESS
160125-11:11:18.011512 [mod=DVR, lvl=INFO] [tid=2349] HAL_PLAYBACK_Start:3789(0x596290) - NEXUS_Playpump_Open(NEXUS_ANY_ID, 0xc7e66800) returns 0x54052014)
160125-11:11:18.011555 [mod=SYS, lvl=TRACE] [tid=2343] vl_env_get_bool: key=NAGRA.PRM.SUPPORT.ENABLED, result=1
160125-11:11:18.013621 [mod=SYS, lvl=TRACE] [tid=2349] vl_env_get_bool: key=FEATURE.TRICKPLAY.CHANNEL_CHANGE.SLOW_MOTION, result=0
160125-11:11:18.013727 [mod=SYS, lvl=TRACE] [tid=2349] vl_env_get_bool: key=FEATURE.PARENTALCONTROL.POLICY.STRICT, result=1
160125-11:11:18.013786 [mod=DVR, lvl=INFO] [tid=2349] HAL_Playback_Init:355() - Using FEATURE.PARENTALCONTROL.POLICY.STRICT = true

Following your logic, you could use a variable type to keep track of the calls. Indeed, in your logs, you always have a line listing the Received json: ... callParams before the response with the status. If you always have this sequence (hard to guess with your example), you could do:
awk 'BEGIN { type=""; } /Received json/ && /"rate":1.0/ {type="play"} /Received json/ && /"rate":0.0/ {type="pause"} <your code>' $logfile
You can then use type in your printed message, which should be play or pause.
Complete code:
awk 'BEGIN{ type=""; } /trickModeRequest/ && /status\"\:\"GENERAL_ERROR/{ print type, "FAILED" } /trickModeRequest/ && /status\":\"OK/ {print type, "SUCCESSFUL"} /Received json/ && /"rate":1.0/ {type="play"} /Received json/ && /"rate":0.0/ {type="pause"} ' $logfile
Output:
pause SUCCESSFUL
play SUCCESSFUL

Related

How to create a customized log monitoring job in java which provide reports of exception message happened on log file

Should able to process larger log files and provide exception message reports
After completion of log analysis, report notification trigger to specific mail id's.
And also please suggest which framework is the best for processing large files.[eg: spring boot/batch]
I would suggest to go with ELK stack. Stream the logs to elastic search and set up alerts in Kibana.
Can use sendmail client on system and run script on that system to send alert on any Exception.
exception="Exception" # "Error", "HTTP 1.1 \" 500", etc
ignoredException="ValidationException"
# log file to scan
logFileToScan=/var/log/tomcat8/log/application.log
# file where we will keep log of this script
logFilePath=/home/ec2-user/exception.log
# a file where we store till what line the log file has been scanned
# initalize it with 0
countPath=/home/ec2-user/lineCount
# subject with which you want to receive the mail regading Exception
subject="[ALERT] Exception"
# from whom do you want to send the mail regarding Exception
from="abc#abc.com"
# to whom do you want to send the mail
to="xyz#xyz.com"
# number of lines, before the line containing the word to be scanned, to be sent in the mail
linesBefore=1
# number of lines, before the line containing the word to be scanned, to be sent in the mail
linesAfter=4
# start line
fromLine=`cat $countPath`
# current line count in the file
toLine=`wc -l $logFileToScan | awk '{print $1}'`
#logs are rolling so if fromLine has a value greater than toLine then fromLine has to be set to 0
if [ "$fromLine" == "" ]; then
fromLine=0
echo `date` fromLine values was empty, set to 0 >> $logFilePath
elif [ $fromLine -gt $toLine ]; then
echo `date` logfile was rolled, updating fromLine from $fromLine to 0 >> $logFilePath
fromLine=0
fi
# if from n to lines are equal then no logs has been generated since last scan
if [ "$fromLine" == "$toLine" ]; then
echo `date` no logs genetared after last scan >> $logFilePath
else
echo `date` updating linecount to $toLine >> $logFilePath
echo $toLine > $countPath
logContent=`tail -n +"$fromLine" $logFileToScan | head -n "$((toLine - fromLine))" | grep -v $ignoredException | grep -A $linesAfter -B $linesBefore $exception`
logContent=`echo $logContent | cut -c1-2000`
if [ "$logContent" == "" ]; then
echo `date` no exception found >> $logFilePath
else
/usr/sbin/sendmail $to <<EOF
subject: $subject
from: $from
logContent=$logContent
EOF
fi
fi

bash script delete mails from specific address (pop3 account)

I want to clean my mailbox from mails from specific address
I have thousands of messages, I want to do this in bash script, and run it from time to time (a receive SPAM from different addresses, and unfortunately my "spam filters" have only small effect on them)
To interact with a mail server through command line, you could use either telnet or openssl.
You can connect to your pop server using the following command (I've taken gmail as an example. You'll have to look for your email host pop3 address and socket.) :
openssl s_client -connect pop.gmail.com:995 -quiet
As this command is interactive, it will ask for a username, a password and a serie of commands.
expect is a tool that can automate interaction with interactive commands. The basic syntax is as follow : expect "somestring" action -> If the program we monitor displays "somestring", we execute the action.
Here is a script that would delete all the messages present on your email address :
#!/usr/bin/expect
#you can modify the timeout if the script fails
set timeout 1
#our connection variables
set ip "pop.gmail.com"
set socket "995"
set user "user.name"
set pass "password"
#we set the address we want to remove mails from here. Escape special regex characters such as dots.
set target_address "mail\.example#gmail\.com"
#we launch the subprocess we want to interact with
spawn openssl s_client -connect $ip:$socket -quiet
#if connection went all right, we try to login
expect -re ".OK.*" {send "user $user\r"}
expect -re ".OK.*" {send "pass $pass\r"}
#if login went alright, we try to count the messages on the server
#you will get the following output :
#+OK NB_MSG TOTAL_SIZE
expect -re ".OK.*" {send "stat\r"}
#if the stat command went alright ...
expect -re ".OK.*" {
#we extract the number of mail from the output of the stat command
set mail_count [lindex [split [lindex [split $expect_out(buffer) \n] 1] " "] 1]
#we iterate through every email...
for {set i 1} {$i <= $mail_count} {incr i 1} {
#we retrieve the header of the email
send "top $i 0\r"
#if the header contains "To: $target_address" (or "To: <$target_address>" or "To: Contact Name <$target_address>" ...)
#to filter according to the sender, change the regex to "\nFrom: ..."
expect -re "\nTo: \[^\n\]*$target_address" {
#we delete the email
send "dele $i\r"
}
}
}
expect default
You might need to alter your email account settings to allow the use of external programs
Here I have written a similar script in PHP, by use of the IMAP function of PHP. I did not have Linux, so I could not use the bash script. However. I thought that sharing the PHP-script would provide a solution for those running WINDOWS and having access
<?php
echo "My showmail.php script";
// This is the format for opening the email connection:
//
// domain prt foldr[.<.subfolder........>] my#blp.com xxxxxxxx
// $connection = imap_open('{imap.one.com:143}Inbox.<write subfolder here>','email-name','password');
// Execution: (write this in the browsers address bar)
http://digitalageinstitute.com/showmail.php?fromdate=01%20January%202009&todate=28%20August%202018&search=#weheartit.com&email=david.svarrer#digitalageinstitute.com&password=xxxxxxx&domain=imap.one.com&folder=Inbox
//
// This line above, entered into a browsers address bar will execute this script (named showmail.php),
// from the servers root (www.digitalageinstitute.com), and
//
// will delete emails from the mail folder "Inbox" (case sensitive) for the user david.svarrer#digitalageinstitute.com, by use of
// the IMAP domain (this is where you access emails from outside) imap.one.com and at port 143 (see this, further down)
//
$emailaddress = $_GET["email"];
$emailpassword= $_GET["password"];
$fromdate = $_GET["fromdate"];
$todate = $_GET["todate"];
$search = $_GET["search"];
$domain = $_GET["domain"];
$folder = $_GET["folder"];
echo $emailaddress." ".$emailpassword." ".$fromdate." ".$todate." ".$search." ".$domain." ".$folder;
$connection = imap_open('{'.$domain.':143}'.$folder,$emailaddress,$emailpassword);
$ccount = imap_num_msg($connection);
$maxtoexpunge=20000;
echo "Parameters = ".$fromdate.",".$todate.",".$search.":".'ALL FROM "'.$search.'" SINCE "'.$fromdate.'" BEFORE "'.$todate.'"';
// $some = imap_search($connection, 'ALL FROM "Twitter"');
$some = imap_search($connection, 'ALL FROM "'.$search.'" SINCE "'.$fromdate.'" BEFORE "'.$todate.'"');
print_r ($some);
echo "<br/>Elements: ".count($some)."<br/>";
$expunge = 0;
for($msgno = 0; $msgno < count($some) ; $msgno++) {
// echo "Fetching element ".$some[$msgno]."<br/>";
$headers = imap_headerinfo($connection, $some[$msgno]);
//$position = True;
$position = strpos(" ".strtolower($headers->fromaddress), strtolower($search));
$position2 = strpos(" ".strtolower($headers->subject), strtolower($search));
if ($position || $position2) {
if ($expunge<$maxtoexpunge) {
imap_delete ($connection, $some[$msgno]);
echo "<br/>Deleting:".$expunge." sequence [".$msgno."]=".$some[$msgno]." ".$headers->fromaddress.", subject:".$headers->subject;
$expunge++;
} else {
echo "<br/>Skipping:"." sequence [".$msgno."]=".$some[$msgno]." ".$headers->fromaddress.", subject:".$headers->subject;
}
}
}
// The expunge command deletes after all action has been taken. DON'T call it until very end, as it will otherwise mess up message numbers!!
imap_expunge ($connection);
echo "<br/>Expunged!!<br/>";
/* Here are the keywords to be used for the imap search order.
toaddress - full to: line, up to 1024 characters
to - an array of objects from the To: line, with the following properties: personal, adl, mailbox, and host
fromaddress - full from: line, up to 1024 characters
from - an array of objects from the From: line, with the following properties: personal, adl, mailbox, and host
ccaddress - full cc: line, up to 1024 characters
cc - an array of objects from the Cc: line, with the following properties: personal, adl, mailbox, and host
bccaddress - full bcc: line, up to 1024 characters
bcc - an array of objects from the Bcc: line, with the following properties: personal, adl, mailbox, and host
reply_toaddress - full Reply-To: line, up to 1024 characters
reply_to - an array of objects from the Reply-To: line, with the following properties: personal, adl, mailbox, and host
senderaddress - full sender: line, up to 1024 characters
sender - an array of objects from the Sender: line, with the following properties: personal, adl, mailbox, and host
return_pathaddress - full Return-Path: line, up to 1024 characters
return_path - an array of objects from the Return-Path: line, with the following properties: personal, adl, mailbox, and host
remail -
date - The message date as found in its headers
Date - Same as date
subject - The message subject
Subject - Same as subject
in_reply_to -
message_id -
newsgroups -
followup_to -
references -
Recent - R if recent and seen, N if recent and not seen, ' ' if not recent.
Unseen - U if not seen AND not recent, ' ' if seen OR not seen and recent
Flagged - F if flagged, ' ' if not flagged
Answered - A if answered, ' ' if unanswered
Deleted - D if deleted, ' ' if not deleted
Draft - X if draft, ' ' if not draft
Msgno - The message number
MailDate -
Size - The message size
udate - mail message date in Unix time
fetchfrom - from line formatted to fit fromlength characters
fetchsubject - subject line formatted to fit subjectlength characters
*/
?>
I was looking for a bash script and as I did not found one I made my own, only external programm needed is socat for SSL/TLS connect to server:
#!/bin/bash -
#===============================================================================
#
# FILE: delete-mail.sh
#
USAGE="./delete-mail.sh user pass [server] [port] [max_delete]"
#
# PARAMETER: user pass - login credentials, mandatory
# server - mail server, default: imap.1und1.de
# port - port on mail server, default: pop3s
# max_delete - maximum # of messages to delete, default: 1000
#
DESCRIPTION="delete max_delete messages in given pop3 mailbox"
#
# AUTHOR: KayM (), kay#rrr.de
# CREATED: 09.06.2021 18:03:58
# REVISION: ---
#===============================================================================
# check args
if [[ "$1" == "-h"* ]]; then
printf "usage: %s\ndescr: %s\n" "$USAGE" "$DESCRIPTION" 1>&2
exit
elif [ "$#" -lt 2 ]; then
printf "Missing args, usage: %s\n" "$USAGE"
exit 1
fi
# assign values
USER="$1"
PASS="$2"
SERVER="${3:-imap.1und1.de}"
PORT="${4:-pop3s}"
MAX_MESSAGE="${5:-100}"
# define actions
popserver() { socat - "OPENSSL:$SERVER:$PORT" 2>/dev/null; }
poplogin() {
printf "USER %s\n" "$USER"
sleep 0.5
printf "PASS %s\n" "$PASS"
sleep 0.5
}
deletemsg() {
for (( j = 1 ; j <= MAX_MESSAGE; j++ ))
do
printf "DELE %s\n" "$j"
sleep 0.5
done
}
popstat() { echo "STAT"; }
popquit() { echo "QUIT"; }
checkresult() {
local line
while read -r line
do
[[ "$line" == "-ERR "* ]] && MAX_MESSAGE=0
printf "%s\n" "$line"
done
}
# get message count, remove newline
RES="$({ poplogin; popstat; } | popserver | tail -n 1 )"
RES="${RES//[$'\r\n']}"
if [[ "$RES" == "+OK 0 "* ]]; then
printf "No messages to delete (%s)\n" "$RES"
exit
elif [[ "$RES" == "+OK "* ]]; then
: "${RES#+OK }"; NUM="${_%% *}"
printf "%s Messages found (%s)\n" "$NUM" "$RES"
[ "$NUM" -lt "$MAX_MESSAGE" ] && MAX_MESSAGE="$NUM"
else
printf "Can't get number of messages: %s\n" "$RES"
exit 1
fi
if [ "$MAX_MESSAGE" = "0" ]; then
printf "Keep all messages\n"
exit
fi
# delete messages
printf "Delete %s messages ...\n" "$MAX_MESSAGE"
{ poplogin; deletemsg; popstat; popquit;} | popserver | checkresult

Expect send command is not sending - stops sending during script

The problem is the command send "xxxxx" just does not send. The command seem to be completely ignored.
In the script below I am able to log in to the server but no more send commands are transmitted to the server.
At the end there is an example of the output.
The timeout traps were for debugging.
#!/usr/bin/expect -f
## getver_t.sh - Use telnet to get ver from Cisco devices
## Could be asked for login ID or just pasword
set pass {Password1 Pass2}
set index 0
set timeout 5
set host [lindex $argv 0 ]
spawn ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null admin#$host
expect {
"assword:" {
send_user "\n--- sending [lindex $pass $index] ---\n"
send "[lindex $pass $index]\r"
incr index
exp_continue
}
"\>" {
send_user "\n--- I see a prompt ---\n"
send -- "ter len 0\r"
sleep 3
send -- "show inv\r"
sleep 3
send -- "show ver\r"
send_user "\n--- Time to end ---\n"
sleep 3
send -- "exit\r"
sleep 3
exit
}
timeout {
send -- "\r"
set timeout 5
send_user "\n--- TIMED OUT 1 ---\n"
send -- "ter len 0\r"
sleep 3
send -- "show inv\r"
sleep 3
send -- "show ver\r"
sleep 3
send -- "exit\r"
sleep 3
exit
}
}
send_user "\n--- out side land ---\n"
send "\r"
expect {
timeout {
send_user "\n--- TIMED OUT 2 ---\n"
}
}
This is what I see running the script.
$ ./getver_s.sh server1
spawn ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null admin#server1
Warning: Permanently added 'server1,10.1.1.3' (RSA) to the list of known hosts.
Password:
--- sending Password1 ---
Password:
--- sending Pass2 ---
------------------------------------------------------------
Welcome to server1
------------------------------------------------------------
server1>
--- I see a prompt ---
--- Time to end ---
$
I see the "--- I see a prompt ---" and "--- Time to end ---" messages
but the send commands between the send_user commands are not sent to the server.
I have rewritten this script changed the logic and run it off different versions of servers (Debian7 & 8) but the result is always the same.
Why are those send commands not being sent?
Added expect -d output
Login Banner
-------------- CUT ------------------
Password:"
--- sending Password1 ---
send: sending "Password1\r" to { exp7 }
expect: continuing expect
expect: does " " (spawn_id exp7) match glob pattern "assword:"? no
">"? no
expect: does " \r\n" (spawn_id exp7) match glob pattern "assword:"? no
">"? no
Password:
expect: does " \r\nPassword: " (spawn_id exp7) match glob pattern "assword:"? yes
expect: set expect_out(0,string) "assword:"
expect: set expect_out(spawn_id) "exp7"
expect: set expect_out(buffer) " \r\nPassword:"
--- sending Pass2 ---
send: sending "Pass2\r" to { exp7 }
expect: continuing expect
expect: does " " (spawn_id exp7) match glob pattern "assword:"? no
">"? no
expect: does " \r\n" (spawn_id exp7) match glob pattern "assword:"? no
">"? no
-------------- CUT ------------------
motd
-------------- CUT ------------------
-\r\nServer1>"
--- I see a prompt ---
send: sending "ter len 0\r" to { exp7 }
send: sending "show inv\r" to { exp7 }
send: sending "show ver\r" to { exp7 }
send: sending "exit\r" to { exp7 }
Server1$

rsync hangs when syncing VirtualBox directory in expect script

My expect script fails to proceed when r-syncing the VirtualBox directories.
Here is the script:
#!/usr/bin/expect -f
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/Documents/ username#server:/home/username/BAK/Documents/
expect "password:"
send "thepassword\r"
interact
sleep 1
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/AndroidDev/ username#server:/home/username/BAK/AndroidDev/
expect "password:"
send "thepassword\r"
interact
sleep 1
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/Android/ username#server:/home/username/BAK/Android/
expect "password:"
send "thepassword\r"
interact
sleep 1
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/Videos/ username#server:/home/username/BAK/Videos/ ;
expect "password:"
send "thepassword\r"
interact
sleep 1
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/VirtualBox\ VMs/ username#server:/home/username/BAK/"VirtualBox\ VMs/" ;
expect "password:"
send "thepassword\r"
interact
sleep 1
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/Pictures/ username#server:/home/username/BAK/Pictures ;
expect "password:"
send "thepassword\r"
interact
sleep 1
spawn scp -v /home/james/.vimrc username#server:/home/username/BAK/ ;
expect "password:"
send "thepassword\r"
interact
sleep 1
spawn scp -v /home/james/.bashrc username#server:/home/username/BAK/ ;
expect "password:"
send "thepassword\r"
interact
sleep 1
spawn scp -v /home/james/.profile username#server:/home/username/BAK/ ;
expect "password:"
send "thepassword\r"
interact
sleep 1
spawn scp -v /home/james/.xsession* username#server:/home/username/BAK/ ;
expect "password:"
send "thepassword\r"
interact
sleep 1
If I change spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/rschostag/VirtualBox\ VMs/ remote#server:/home/directory/BAK/VirtualBox to spawn scp -vr /home/rschostag/VirtualBox\ VMs/ remote#server:/home/directory/BAK/VirtualBox, it works. However, I do not want to do full backup every time.
What is the best way to incrementally backup this directory or otherwise make the rsync command work properly?
Thank you!
As requested, please find the output of expect -d script as follows:
$ sudo expect -d rsync_backup
expect version 5.45
argv[0] = expect argv[1] = -d argv[2] = rsync_backup
set argc 0
set argv0 "rsync_backup"
set argv ""
executing commands from command file rsync_backup
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/Documents/ username#server:/home/username/BAK/Documents/
parent: waiting for sync byte
parent: telling child to go ahead
parent: now unsynchronized from child
spawn: returns {5064}
expect: does "" (spawn_id exp6) match glob pattern "password:"? no
username#server password:
expect: does "username#server password: " (spawn_id exp6) match glob pattern "password:"? yes
expect: set expect_out(0,string) "password:"
expect: set expect_out(spawn_id) "exp6"
expect: set expect_out(buffer) "username#server password:"
send: sending "thepassword\r" to { exp6 }
tty_raw_noecho: was raw = 0 echo = 1
spawn id exp6 sent <\r\n>
spawn id exp6 sent <sending incremental file list\r\n>
sending incremental file list
spawn id exp6 sent <Scripts/rsync.log\r\n>
Scripts/rsync.log
spawn id exp6 sent <\r 2.29K 0% 0.00kB/s 0:00:00 >
2.29K 0% 0.00kB/s 0:00:00 spawn id exp6 sent <\r 5.26M 100% 78.42MB/s 0:00:00 (xfr#1, ir-chk=1561/3024)\r\n>
5.26M 100% 78.42MB/s 0:00:00 (xfr#1, ir-chk=1561/3024)
spawn id exp6 sent <projects/JOBS/jobsearch_production/\r\n>
projects/JOBS/jobsearch_production/
spawn id exp6 sent <projects/JOBS/jobsearch_production/db.sqlite3\r\n>
projects/JOBS/jobsearch_production/db.sqlite3
spawn id exp6 sent <\r 5.41K 0% 0.00kB/s 0:00:00 >
5.41K 0% 0.00kB/s 0:00:00 spawn id exp6 sent <\r 7.35M 100% 73.71MB/s 0:00:00 (xfr#2, ir-chk=1025/6252)\r\n>
7.35M 100% 73.71MB/s 0:00:00 (xfr#2, ir-chk=1025/6252)
spawn id exp6 sent <\r\nNumber of files: 10,529 (reg: 8,419, dir: 2,096, link: 14)>
Number of files: 10,529 (reg: 8,419, dir: 2,096, link: 14)spawn id exp6 sent <\r\nNumber of created files: 0>
Number of created files: 0spawn id exp6 sent <\r\nNumber of regular files transferred: 2>
Number of regular files transferred: 2spawn id exp6 sent <\r\nTotal file size: 1.73G bytes>
Total file size: 1.73G bytesspawn id exp6 sent <\r\nTotal transferred file size: 12.61M bytes>
Total transferred file size: 12.61M bytesspawn id exp6 sent <\r\nLiteral data: 62.70K bytes>
Literal data: 62.70K bytesspawn id exp6 sent <\r\nMatched data: 12.55M bytes>
Matched data: 12.55M bytesspawn id exp6 sent <\r\nFile list size: 0>
File list size: 0spawn id exp6 sent <\r\nFile list generation time: 0.001 seconds>
File list generation time: 0.001 secondsspawn id exp6 sent <\r\nFile list transfer time: 0.000 seconds>
File list transfer time: 0.000 secondsspawn id exp6 sent <\r\nTotal bytes sent: 333.42K>
Total bytes sent: 333.42Kspawn id exp6 sent <\r\nTotal bytes received: 40.66K>
Total bytes received: 40.66Kspawn id exp6 sent <\r\n\r\n>
spawn id exp6 sent <sent 333.42K bytes received 40.66K bytes 32.53K bytes/sec\r\ntotal size is 1.73G speedup is 4,616.35>
sent 333.42K bytes received 40.66K bytes 32.53K bytes/sec
total size is 1.73G speedup is 4,616.35spawn id exp6 sent <\r\n>
interact: received eof from spawn_id exp6
tty_set: raw = 0, echo = 1
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/AndroidDev/ username#server:/home/username/BAK/AndroidDev/
parent: waiting for sync byte
parent: telling child to go ahead
parent: now unsynchronized from child
spawn: returns {5069}
expect: does "" (spawn_id exp7) match glob pattern "password:"? no
username#server password:
expect: does "username#server password: " (spawn_id exp7) match glob pattern "password:"? yes
expect: set expect_out(0,string) "password:"
expect: set expect_out(spawn_id) "exp7"
expect: set expect_out(buffer) "username#server password:"
send: sending "thepassword\r" to { exp7 }
tty_raw_noecho: was raw = 0 echo = 1
spawn id exp7 sent <\r\n>
spawn id exp7 sent <sending incremental file list>
sending incremental file listspawn id exp7 sent <\r\n>
spawn id exp7 sent <\r\n>
spawn id exp7 sent <Number of files: 2,758 (reg: 2,273, dir: 485)>
Number of files: 2,758 (reg: 2,273, dir: 485)spawn id exp7 sent <\r\n>
spawn id exp7 sent <Number of created files: 0>
Number of created files: 0spawn id exp7 sent <\r\nNumber of regular files transferred: 0\r\nTotal file size: 1.00G bytes>
Number of regular files transferred: 0
Total file size: 1.00G bytesspawn id exp7 sent <\r\nTotal transferred file size: 0 bytes>
Total transferred file size: 0 bytesspawn id exp7 sent <\r\nLiteral data: 0 bytes\r\nMatched data: 0 bytes>
Literal data: 0 bytes
Matched data: 0 bytesspawn id exp7 sent <\r\nFile list size: 0\r\nFile list generation time: 0.001 seconds>
File list size: 0
File list generation time: 0.001 secondsspawn id exp7 sent <\r\nFile list transfer time: 0.000 seconds>
File list transfer time: 0.000 secondsspawn id exp7 sent <\r\nTotal bytes sent: 74.25K\r\n>
Total bytes sent: 74.25K
spawn id exp7 sent <Total bytes received: 2.43K\r\n\r\nsent 74.25K bytes received 2.43K bytes 30.67K bytes/sec>
Total bytes received: 2.43K
sent 74.25K bytes received 2.43K bytes 30.67K bytes/secspawn id exp7 sent <\r\ntotal size is 1.00G speedup is 13,102.55>
total size is 1.00G speedup is 13,102.55spawn id exp7 sent <\r\n>
interact: received eof from spawn_id exp7
tty_set: raw = 0, echo = 1
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/Android/ username#server:/home/username/BAK/Android/
parent: waiting for sync byte
parent: telling child to go ahead
parent: now unsynchronized from child
spawn: returns {5074}
expect: does "" (spawn_id exp8) match glob pattern "password:"? no
username#server password:
expect: does "username#server password: " (spawn_id exp8) match glob pattern "password:"? yes
expect: set expect_out(0,string) "password:"
expect: set expect_out(spawn_id) "exp8"
expect: set expect_out(buffer) "username#server password:"
send: sending "thepassword\r" to { exp8 }
tty_raw_noecho: was raw = 0 echo = 1
spawn id exp8 sent <\r\n>
spawn id exp8 sent <sending incremental file list\r\n>
sending incremental file list
spawn id exp8 sent <\r\nNumber of files: 3 (dir: 3)>
Number of files: 3 (dir: 3)spawn id exp8 sent <\r\n>
spawn id exp8 sent <Number of created files: 0>
Number of created files: 0spawn id exp8 sent <\r\nNumber of regular files transferred: 0>
Number of regular files transferred: 0spawn id exp8 sent <\r\nTotal file size: 0 bytes>
Total file size: 0 bytesspawn id exp8 sent <\r\nTotal transferred file size: 0 bytes>
Total transferred file size: 0 bytesspawn id exp8 sent <\r\n>
spawn id exp8 sent <Literal data: 0 bytes\r\nMatched data: 0 bytes\r\nFile list size: 0>
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0spawn id exp8 sent <\r\n>
spawn id exp8 sent <File list generation time: 0.001 seconds\r\nFile list transfer time: 0.000 seconds\r\nTotal bytes sent: 83>
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 83spawn id exp8 sent <\r\nTotal bytes received: 21>
Total bytes received: 21spawn id exp8 sent <\r\n\r\n>
spawn id exp8 sent <sent 83 bytes received 21 bytes 69.33 bytes/sec\r\ntotal size is 0 speedup is 0.00>
sent 83 bytes received 21 bytes 69.33 bytes/sec
total size is 0 speedup is 0.00spawn id exp8 sent <\r\n>
interact: received eof from spawn_id exp8
tty_set: raw = 0, echo = 1
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/Videos/ username#server:/home/username/BAK/Videos/
parent: waiting for sync byte
parent: telling child to go ahead
parent: now unsynchronized from child
spawn: returns {5079}
expect: does "" (spawn_id exp9) match glob pattern "password:"? no
username#server password:
expect: does "username#server password: " (spawn_id exp9) match glob pattern "password:"? yes
expect: set expect_out(0,string) "password:"
expect: set expect_out(spawn_id) "exp9"
expect: set expect_out(buffer) "username#server password:"
send: sending "thepassword\r" to { exp9 }
tty_raw_noecho: was raw = 0 echo = 1
spawn id exp9 sent <\r\n>
spawn id exp9 sent <sending incremental file list\r\n>
sending incremental file list
spawn id exp9 sent <\r\nNumber of files: 30 (reg: 29, dir: 1)\r\nNumber of created files: 0>
Number of files: 30 (reg: 29, dir: 1)
Number of created files: 0spawn id exp9 sent <\r\nNumber of regular files transferred: 0>
Number of regular files transferred: 0spawn id exp9 sent <\r\nTotal file size: 1.35G bytes>
Total file size: 1.35G bytesspawn id exp9 sent <\r\nTotal transferred file size: 0 bytes>
Total transferred file size: 0 bytesspawn id exp9 sent <\r\nLiteral data: 0 bytes>
Literal data: 0 bytesspawn id exp9 sent <\r\nMatched data: 0 bytes\r\nFile list size: 0>
Matched data: 0 bytes
File list size: 0spawn id exp9 sent <\r\nFile list generation time: 0.001 seconds>
File list generation time: 0.001 secondsspawn id exp9 sent <\r\nFile list transfer time: 0.000 seconds>
File list transfer time: 0.000 secondsspawn id exp9 sent <\r\nTotal bytes sent: 587\r\n>
Total bytes sent: 587
spawn id exp9 sent <Total bytes received: 11\r\n\r\nsent 587 bytes received 11 bytes 398.67 bytes/sec\r\n>
Total bytes received: 11
sent 587 bytes received 11 bytes 398.67 bytes/sec
spawn id exp9 sent <total size is 1.35G speedup is 2,263,533.89\r\n>
total size is 1.35G speedup is 2,263,533.89
interact: received eof from spawn_id exp9
tty_set: raw = 0, echo = 1
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/VirtualBox VMs/ username#server:/home/username/BAK/"VirtualBox VMs/"
parent: waiting for sync byte
parent: telling child to go ahead
parent: now unsynchronized from child
spawn: returns {5084}
expect: does "" (spawn_id exp10) match glob pattern "password:"? no
username#server password:
expect: does "username#server password: " (spawn_id exp10) match glob pattern "password:"? yes
expect: set expect_out(0,string) "password:"
expect: set expect_out(spawn_id) "exp10"
expect: set expect_out(buffer) "username#server password:"
send: sending "thepassword\r" to { exp10 }
tty_raw_noecho: was raw = 0 echo = 1
spawn id exp10 sent <\r\n>
spawn id exp10 sent <sending incremental file list\r\n>
sending incremental file list
This is where it hangs and does not provide an error message that I can identify.
I moved the remote VirtualBox directory and started the backup again, and it completed finally after running the backup for a few days. The total size of the backup was 71 GB. I ran it with expect -d which prints a very verbose output in the terminal, so that was helpful alone, because the script without -d may have looked like it was hanging only because it was not printing anything in the output. So, there was never anything wrong, I believe, other than no progress indication, which made me believe that it may not be working.
Thank you for advising me about the expect -d option.
#glennjackman answered my question in a comment but did not answer the question formally.
Here is the new script that works:
#!/usr/bin/expect -f
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/ryan/Documents/ username#server:/home/username/BAK/Documents/
expect "password:"
send "thepassword\r"
expect eof
sleep 1
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/ryan/AndroidDev/ username#server:/home/username/BAK/AndroidDev/
expect "password:"
send "thepassword\r"
expect eof
sleep 1
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/ryan/Android/ username#server:/home/username/BAK/Android/
expect "password:"
send "thepassword\r"
expect eof
sleep 1
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/ryan/Videos/ username#server:/home/username/BAK/Videos/ ;
expect "password:"
send "thepassword\r"
expect eof
sleep 1
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/ryan/VirtualBox\ VMs/ username#server:/home/username/BAK/"VirtualBox\ VMs/" ;
expect "password:"
send "thepassword\r"
expect eof
sleep 1
spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/ryan/Pictures/ username#server:/home/username/BAK/Pictures ;
expect "password:"
send "thepassword\r"
expect eof
sleep 1
spawn scp -v /home/ryan/.vimrc username#server:/home/username/BAK/ ;
expect "password:"
send "thepassword\r"
expect eof
sleep 1
spawn scp -v /home/ryan/.bashrc username#server:/home/username/BAK/ ;
expect "password:"
send "thepassword\r"
expect eof
sleep 1
spawn scp -v /home/ryan/.profile username#server:/home/username/BAK/ ;
expect "password:"
send "thepassword\r"
expect eof
sleep 1
spawn scp -v /home/ryan/.xsession* username#server:/home/username/BAK/ ;
expect "password:"
send "thepassword\r"
expect eof
sleep 1

LIST command in Pop3 server-like

I must do a pop3 server in bash and I have a problem with the LIST command.
I can connect to my server, do some command but when LIST are calling my client wait long time. He probably wait for another input but I can't see what I'm doing wrong.
I follow the [RFC 1939][1] :
C: LIST
S: +OK 2 messages (320 octets)
S: 1 120
S: 2 200
S: .
My minimal implementation :
[...]
"STAT"*)
echo "+OK 1 3"
;;
"LIST"*)
echo "+OK 1 messages (3 octets)"
echo "1 3"
echo "."
;;
[...]
Claws Mail Log :
* Account 'root#pop.kali': Connecting to POP3 server: localhost...
[18:02:42] POP3< +OK
[18:02:42] POP3> USER root
[18:02:42] POP3< +OK USER
[18:02:42] POP3> PASS ********
[18:02:42] POP3< +OK PASS
[18:02:42] POP3> STAT
[18:02:42] POP3< +OK 1 3
[18:02:42] POP3> UIDL
[18:02:42] POP3< -ERR 'UIDL
' n'est pas une commande valide
** command not supported
[18:02:42] POP3> LAST
[18:02:42] POP3< -ERR 'LAST
' n'est pas une commande valide
** command not supported
[18:02:42] POP3> LIST
[18:02:42] POP3< +OK 1 messages (3 octets)
// HERE WAIT A LONG LONG TIME //
** Session timed out. You may be able to recover by increasing the timeout value in Preferences/Other/Miscellaneous.
Thank you.
[1]: https://www.rfc-editor.org/rfc/rfc1939#page-6
Try making sure each line of your responses ends with CRLF, not just LF. It might be that just the commands expecting a multi-line response insist on seeing .CRLF, not just .LF.
[...]
"STAT"*)
printf "+OK 1 3\r\n"
;;
"LIST"*)
printf "+OK 1 messages (3 octets)\r\n"
printf "1 3\r\n"
printf ".\r\n"
;;
[...]

Resources