Need some help on awk/grep command on shell script - shell
This is in reference to Need some help on shell scipt (using grep command)
I have a log file and I am trying to write a shell script to find pause/play happening or not (Success /Failure).
The script should do the following:
if ("trickModeRequest":{"rate":0.0})
{
if ("status":"OK")
printf("Pause Successful");
else if ("status":"GENERAL_ERROR")
printf("Pause failed");
else if ("status":"UNRECOGNIZED REQUEST")
printf("Pause failed");
}
Similarly:
if ("trickModeRequest":{"rate":1.0})
{
if ("status":"OK")
printf("Play Successful");
else if ("status":"GENERAL_ERROR")
printf("Play failed");
else if ("status":"UNRECOGNIZED REQUEST")
printf("Play failed");
}
My script file:
logfile=$1
awk '/trickModeRequest/ && /status\"\:\"GENERAL_ERROR/{ print "trickModeRequestFAILED" } /trickModeRequest/ && /status\":\"OK/ {print "trickModeRequestSUCCESS"} ' $logfile
The problem is that {"trickModeRequest":{"rate":0.0} or {"trickModeRequest":{"rate":1.0} is not placed in same paragraph of logs, so I dont know how to distinguish between pause/play...
My log file (logs.txt):
160125-11:11:16.654574 [mod=CARDLESSCA, lvl=DEBUG] [tid=2343] [RECORD]NAGRA_API:vlhal_CasPvrRecordingStart-post handle=0x007288a8
160125-11:11:16.654617 [mod=CARDLESSCA, lvl=INFO] [tid=2343] Recording nagra request for Recording vlhal_CasPvrPlaybackStart 0x7288a8
160125-11:11:16.655113 [mod=DVR, lvl=ERROR] [tid=2343] VLTune::process_buffer:1088(this=0x58ca00) - is_recording=1(was 0), hal_recording=0x7555ddd8
160125-11:11:16.905656 [mod=SYS, lvl=TRACE] [tid=2343] vl_env_get_bool: key=NAGRA.PRM.SUPPORT.ENABLED, result=1
160125-11:11:16.910125 [mod=SYS, lvl=TRACE] [tid=2343] vl_env_get_bool: key=NAGRA.PRM.SUPPORT.ENABLED, result=1
160125-11:11:16.911879 [mod=DVR, lvl=INFO] [tid=2343] HAL_RECORD_GetData:2989(0x7555ddd8) - /data/data/OCAP_MSV/0/0/DEFAULT_RECORDING_VOLUME/dvr/1453720276418.BOTF_Marker(worte=376)
160125-11:11:16.949874 [mod=SYS, lvl=INFO] [tid=2332] Read message '{"event":1,"handler":1,"name":"onRPCCall","params":{"callGUID":"228dd4a5-6727-4525-abce-331c87d54c18","callParams":[**{"trickModeRequest":{"rate":0.0}}**],"class":"com.comcast.xre.events.XRERPCCallInfo","destinationSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d",**"method":"trickModeRequest"**,"sourceSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d"},"phase":"STANDARD","source":1}
'
160125-11:11:16.950180 [mod=SYS, lvl=INFO] [tid=2332] ======= Message is onRPCCall ======>
160125-11:11:16.950326 [mod=SYS, lvl=INFO] [tid=2332] Entering onRPCCallEvent for request ---> trickModeRequest
160125-11:11:16.950621 [mod=SYS, lvl=INFO] [tid=2332] Received json request = {"event":1,"handler":1,"name":"onRPCCall","params":{"callGUID":"228dd4a5-6727-4525-abce-331c87d54c18","callParams":[**{"trickModeRequest":{"rate":0.0}**}],"class":"com.comcast.xre.events.XRERPCCallInfo","destinationSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d","method":"trickModeRequest","sourceSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d"},"phase":"STANDARD","source":1}
160125-11:11:16.950689 [mod=SYS, lvl=INFO] [tid=2332] trickModeRequest: trickModeRequest
160125-11:11:16.950794 [mod=TUNERSOURCE, lvl=INFO] [tid=2332] Setting rate 0.0, skipping 0 ms
160125-11:11:16.950872 [mod=TUNERSOURCE, lvl=INFO] [tid=2332] trickModeRequest() - Sending the TrickMode event
160125-11:11:16.950959 [mod=SYS, lvl=INFO] [tid=2332] PlaybackResponse: entered 0
160125-11:11:16.950994 [mod=MEDIA_SESSION_MGR, lvl=DEBUG] [tid=2331] MediaSessionManager::trickMode() - Handling trick mode
160125-11:11:16.951059 [mod=MEDIA_SESSION_MGR, lvl=DEBUG] [tid=2331] MediaSessionManager::trickMode() - rate = 0.00000, skip = 0 ms
160125-11:11:16.951123 [mod=PLAYER, lvl=TRACE] [tid=2331] PlayerBase(id=56a602d3)::stop - Before event: state = %s
160125-11:11:16.951182 [mod=PLAYER, lvl=DEBUG] [tid=2331] DecodePlayer(id=56a602d3)::DecodePlayerStatePresenting::stop - entering state %s
160125-11:11:16.951295 [mod=PLAYER, lvl=DEBUG] [tid=2331] PlayerBase(id=56a602d3)::PlayerStateBase::waitForExistingWorkerThread - waiting for the previous worker thread to stop
160125-11:11:16.951361 [mod=PLAYER, lvl=DEBUG] [tid=2331] PlayerBase(id=56a602d3)::PlayerStateBase::waitForExistingWorkerThread - done joining
160125-11:11:16.951535 [mod=PLAYER, lvl=TRACE] [tid=2331] PlayerBase(id=56a602d3)::stop - After event: state = %s
160125-11:11:16.951639 [mod=SYS, lvl=INFO] [tid=2332] ====== Response sending is {"appId":1,"command":"CALL","commandIndex":5,"method":"generateAppEvent","params":[{"class":"com.comcast.xre.events.XREOutgoingEvent","name":"onRPCReturn","params":{"callGUID":"228dd4a5-6727-4525-abce-331c87d54c18","class":"com.comcast.xre.events.XRERPCReturnInfo","destinationSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d",**"method":"trickModeRequest"**,"returnVal":{"class":"com.comcast.parker.SelectServiceSyncResponse","selectServiceStatus":"AWAITING_ASYNC","startTime":1296590759,"**status":"OK"**,"statusMessage":"Tuning to service: "},"sourceSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d"}},"ab00ebea-5f63-4619-9877-273a2bceea1d"],"targetId":1,"targetPath":"","timestamp":0}
====
160125-11:11:18.002455 [mod=SYS, lvl=INFO] [tid=2332] Read message '{"event":1,"handler":1,"name":"onRPCCall","params":{"callGUID":"fe11a665-2aad-404c-a887-bd3d967bbea0","callParams":[{**"trickModeRequest":{"rate":1.0}**}],"class":"com.comcast.xre.events.XRERPCCallInfo","destinationSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d",**"method":"trickModeRequest"**,"sourceSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d"},"phase":"STANDARD","source":1}
'
160125-11:11:18.002756 [mod=SYS, lvl=INFO] [tid=2332] ======= Message is onRPCCall ======>
160125-11:11:18.002843 [mod=SYS, lvl=INFO] [tid=2332] Entering onRPCCallEvent for request ---> trickModeRequest
160125-11:11:18.003106 [mod=SYS, lvl=INFO] [tid=2332] Received json request = {"event":1,"handler":1,"name":"onRPCCall","params":{"callGUID":"fe11a665-2aad-404c-a887-bd3d967bbea0","callParams":[{"trickModeRequest":{"rate":1.0}}],"class":"com.comcast.xre.events.XRERPCCallInfo","destinationSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d",**"method":"trickModeRequest"**,"sourceSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d"},"phase":"STANDARD","source":1}
160125-11:11:18.003168 [mod=SYS, lvl=INFO] [tid=2332] trickModeRequest: trickModeRequest
160125-11:11:18.003264 [mod=TUNERSOURCE, lvl=INFO] [tid=2332] Setting rate 1.0, skipping 0 ms
160125-11:11:18.003315 [mod=TUNERSOURCE, lvl=INFO] [tid=2332] trickModeRequest() - Sending the TrickMode event
160125-11:11:18.003387 [mod=SYS, lvl=INFO] [tid=2332] PlaybackResponse: entered 0
160125-11:11:18.003952 [mod=SYS, lvl=INFO] [tid=2332] ====== Response sending is {"appId":1,"command":"CALL","commandIndex":5,"method":"generateAppEvent","params":[{"class":"com.comcast.xre.events.XREOutgoingEvent","name":"onRPCReturn","params":{"callGUID":"fe11a665-2aad-404c-a887-bd3d967bbea0","class":"com.comcast.xre.events.XRERPCReturnInfo","destinationSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d",**"method":"trickModeRequest"**,"returnVal":{"class":"com.comcast.parker.SelectServiceSyncResponse","selectServiceStatus":"AWAITING_ASYNC","startTime":1296590759,**"status":"OK"**,"statusMessage":"Tuning to service: "},"sourceSessionGUID":"ab00ebea-5f63-4619-9877-273a2bceea1d"}},"ab00ebea-5f63-4619-9877-273a2bceea1d"],"targetId":1,"targetPath":"","timestamp":0}
====
160125-11:11:18.004268 [mod=MEDIA_SESSION_MGR, lvl=DEBUG] [tid=2331] MediaSessionManager::trickMode() - Handling trick mode
160125-11:11:18.004370 [mod=MEDIA_SESSION_MGR, lvl=DEBUG] [tid=2331] MediaSessionManager::trickMode() - rate = 1.00000, skip = 0 ms
160125-11:11:18.004429 [mod=PLAYER, lvl=DEBUG] [tid=2331] PlayerBase(id=56a602d3)::PlayerStateBase::waitForExistingWorkerThread - waiting for the previous worker thread to stop
160125-11:11:18.010346 [mod=SYS, lvl=INFO] [tid=2332] ====== Response of 685 bytes sent ======
160125-11:11:18.010529 [mod=SYS, lvl=INFO] [tid=2332] ProcessJsonData: returned success
160125-11:11:18.010633 [mod=SYS, lvl=INFO] [tid=2332] === OnConnect returned PS_SUCCESS
160125-11:11:18.011512 [mod=DVR, lvl=INFO] [tid=2349] HAL_PLAYBACK_Start:3789(0x596290) - NEXUS_Playpump_Open(NEXUS_ANY_ID, 0xc7e66800) returns 0x54052014)
160125-11:11:18.011555 [mod=SYS, lvl=TRACE] [tid=2343] vl_env_get_bool: key=NAGRA.PRM.SUPPORT.ENABLED, result=1
160125-11:11:18.013621 [mod=SYS, lvl=TRACE] [tid=2349] vl_env_get_bool: key=FEATURE.TRICKPLAY.CHANNEL_CHANGE.SLOW_MOTION, result=0
160125-11:11:18.013727 [mod=SYS, lvl=TRACE] [tid=2349] vl_env_get_bool: key=FEATURE.PARENTALCONTROL.POLICY.STRICT, result=1
160125-11:11:18.013786 [mod=DVR, lvl=INFO] [tid=2349] HAL_Playback_Init:355() - Using FEATURE.PARENTALCONTROL.POLICY.STRICT = true
Following your logic, you could use a variable type to keep track of the calls. Indeed, in your logs, you always have a line listing the Received json: ... callParams before the response with the status. If you always have this sequence (hard to guess with your example), you could do:
awk 'BEGIN { type=""; } /Received json/ && /"rate":1.0/ {type="play"} /Received json/ && /"rate":0.0/ {type="pause"} <your code>' $logfile
You can then use type in your printed message, which should be play or pause.
Complete code:
awk 'BEGIN{ type=""; } /trickModeRequest/ && /status\"\:\"GENERAL_ERROR/{ print type, "FAILED" } /trickModeRequest/ && /status\":\"OK/ {print type, "SUCCESSFUL"} /Received json/ && /"rate":1.0/ {type="play"} /Received json/ && /"rate":0.0/ {type="pause"} ' $logfile
Output:
pause SUCCESSFUL
play SUCCESSFUL
Related
How to create a customized log monitoring job in java which provide reports of exception message happened on log file
Should able to process larger log files and provide exception message reports After completion of log analysis, report notification trigger to specific mail id's. And also please suggest which framework is the best for processing large files.[eg: spring boot/batch]
I would suggest to go with ELK stack. Stream the logs to elastic search and set up alerts in Kibana.
Can use sendmail client on system and run script on that system to send alert on any Exception. exception="Exception" # "Error", "HTTP 1.1 \" 500", etc ignoredException="ValidationException" # log file to scan logFileToScan=/var/log/tomcat8/log/application.log # file where we will keep log of this script logFilePath=/home/ec2-user/exception.log # a file where we store till what line the log file has been scanned # initalize it with 0 countPath=/home/ec2-user/lineCount # subject with which you want to receive the mail regading Exception subject="[ALERT] Exception" # from whom do you want to send the mail regarding Exception from="abc#abc.com" # to whom do you want to send the mail to="xyz#xyz.com" # number of lines, before the line containing the word to be scanned, to be sent in the mail linesBefore=1 # number of lines, before the line containing the word to be scanned, to be sent in the mail linesAfter=4 # start line fromLine=`cat $countPath` # current line count in the file toLine=`wc -l $logFileToScan | awk '{print $1}'` #logs are rolling so if fromLine has a value greater than toLine then fromLine has to be set to 0 if [ "$fromLine" == "" ]; then fromLine=0 echo `date` fromLine values was empty, set to 0 >> $logFilePath elif [ $fromLine -gt $toLine ]; then echo `date` logfile was rolled, updating fromLine from $fromLine to 0 >> $logFilePath fromLine=0 fi # if from n to lines are equal then no logs has been generated since last scan if [ "$fromLine" == "$toLine" ]; then echo `date` no logs genetared after last scan >> $logFilePath else echo `date` updating linecount to $toLine >> $logFilePath echo $toLine > $countPath logContent=`tail -n +"$fromLine" $logFileToScan | head -n "$((toLine - fromLine))" | grep -v $ignoredException | grep -A $linesAfter -B $linesBefore $exception` logContent=`echo $logContent | cut -c1-2000` if [ "$logContent" == "" ]; then echo `date` no exception found >> $logFilePath else /usr/sbin/sendmail $to <<EOF subject: $subject from: $from logContent=$logContent EOF fi fi
bash script delete mails from specific address (pop3 account)
I want to clean my mailbox from mails from specific address I have thousands of messages, I want to do this in bash script, and run it from time to time (a receive SPAM from different addresses, and unfortunately my "spam filters" have only small effect on them)
To interact with a mail server through command line, you could use either telnet or openssl. You can connect to your pop server using the following command (I've taken gmail as an example. You'll have to look for your email host pop3 address and socket.) : openssl s_client -connect pop.gmail.com:995 -quiet As this command is interactive, it will ask for a username, a password and a serie of commands. expect is a tool that can automate interaction with interactive commands. The basic syntax is as follow : expect "somestring" action -> If the program we monitor displays "somestring", we execute the action. Here is a script that would delete all the messages present on your email address : #!/usr/bin/expect #you can modify the timeout if the script fails set timeout 1 #our connection variables set ip "pop.gmail.com" set socket "995" set user "user.name" set pass "password" #we set the address we want to remove mails from here. Escape special regex characters such as dots. set target_address "mail\.example#gmail\.com" #we launch the subprocess we want to interact with spawn openssl s_client -connect $ip:$socket -quiet #if connection went all right, we try to login expect -re ".OK.*" {send "user $user\r"} expect -re ".OK.*" {send "pass $pass\r"} #if login went alright, we try to count the messages on the server #you will get the following output : #+OK NB_MSG TOTAL_SIZE expect -re ".OK.*" {send "stat\r"} #if the stat command went alright ... expect -re ".OK.*" { #we extract the number of mail from the output of the stat command set mail_count [lindex [split [lindex [split $expect_out(buffer) \n] 1] " "] 1] #we iterate through every email... for {set i 1} {$i <= $mail_count} {incr i 1} { #we retrieve the header of the email send "top $i 0\r" #if the header contains "To: $target_address" (or "To: <$target_address>" or "To: Contact Name <$target_address>" ...) #to filter according to the sender, change the regex to "\nFrom: ..." expect -re "\nTo: \[^\n\]*$target_address" { #we delete the email send "dele $i\r" } } } expect default You might need to alter your email account settings to allow the use of external programs
Here I have written a similar script in PHP, by use of the IMAP function of PHP. I did not have Linux, so I could not use the bash script. However. I thought that sharing the PHP-script would provide a solution for those running WINDOWS and having access <?php echo "My showmail.php script"; // This is the format for opening the email connection: // // domain prt foldr[.<.subfolder........>] my#blp.com xxxxxxxx // $connection = imap_open('{imap.one.com:143}Inbox.<write subfolder here>','email-name','password'); // Execution: (write this in the browsers address bar) http://digitalageinstitute.com/showmail.php?fromdate=01%20January%202009&todate=28%20August%202018&search=#weheartit.com&email=david.svarrer#digitalageinstitute.com&password=xxxxxxx&domain=imap.one.com&folder=Inbox // // This line above, entered into a browsers address bar will execute this script (named showmail.php), // from the servers root (www.digitalageinstitute.com), and // // will delete emails from the mail folder "Inbox" (case sensitive) for the user david.svarrer#digitalageinstitute.com, by use of // the IMAP domain (this is where you access emails from outside) imap.one.com and at port 143 (see this, further down) // $emailaddress = $_GET["email"]; $emailpassword= $_GET["password"]; $fromdate = $_GET["fromdate"]; $todate = $_GET["todate"]; $search = $_GET["search"]; $domain = $_GET["domain"]; $folder = $_GET["folder"]; echo $emailaddress." ".$emailpassword." ".$fromdate." ".$todate." ".$search." ".$domain." ".$folder; $connection = imap_open('{'.$domain.':143}'.$folder,$emailaddress,$emailpassword); $ccount = imap_num_msg($connection); $maxtoexpunge=20000; echo "Parameters = ".$fromdate.",".$todate.",".$search.":".'ALL FROM "'.$search.'" SINCE "'.$fromdate.'" BEFORE "'.$todate.'"'; // $some = imap_search($connection, 'ALL FROM "Twitter"'); $some = imap_search($connection, 'ALL FROM "'.$search.'" SINCE "'.$fromdate.'" BEFORE "'.$todate.'"'); print_r ($some); echo "<br/>Elements: ".count($some)."<br/>"; $expunge = 0; for($msgno = 0; $msgno < count($some) ; $msgno++) { // echo "Fetching element ".$some[$msgno]."<br/>"; $headers = imap_headerinfo($connection, $some[$msgno]); //$position = True; $position = strpos(" ".strtolower($headers->fromaddress), strtolower($search)); $position2 = strpos(" ".strtolower($headers->subject), strtolower($search)); if ($position || $position2) { if ($expunge<$maxtoexpunge) { imap_delete ($connection, $some[$msgno]); echo "<br/>Deleting:".$expunge." sequence [".$msgno."]=".$some[$msgno]." ".$headers->fromaddress.", subject:".$headers->subject; $expunge++; } else { echo "<br/>Skipping:"." sequence [".$msgno."]=".$some[$msgno]." ".$headers->fromaddress.", subject:".$headers->subject; } } } // The expunge command deletes after all action has been taken. DON'T call it until very end, as it will otherwise mess up message numbers!! imap_expunge ($connection); echo "<br/>Expunged!!<br/>"; /* Here are the keywords to be used for the imap search order. toaddress - full to: line, up to 1024 characters to - an array of objects from the To: line, with the following properties: personal, adl, mailbox, and host fromaddress - full from: line, up to 1024 characters from - an array of objects from the From: line, with the following properties: personal, adl, mailbox, and host ccaddress - full cc: line, up to 1024 characters cc - an array of objects from the Cc: line, with the following properties: personal, adl, mailbox, and host bccaddress - full bcc: line, up to 1024 characters bcc - an array of objects from the Bcc: line, with the following properties: personal, adl, mailbox, and host reply_toaddress - full Reply-To: line, up to 1024 characters reply_to - an array of objects from the Reply-To: line, with the following properties: personal, adl, mailbox, and host senderaddress - full sender: line, up to 1024 characters sender - an array of objects from the Sender: line, with the following properties: personal, adl, mailbox, and host return_pathaddress - full Return-Path: line, up to 1024 characters return_path - an array of objects from the Return-Path: line, with the following properties: personal, adl, mailbox, and host remail - date - The message date as found in its headers Date - Same as date subject - The message subject Subject - Same as subject in_reply_to - message_id - newsgroups - followup_to - references - Recent - R if recent and seen, N if recent and not seen, ' ' if not recent. Unseen - U if not seen AND not recent, ' ' if seen OR not seen and recent Flagged - F if flagged, ' ' if not flagged Answered - A if answered, ' ' if unanswered Deleted - D if deleted, ' ' if not deleted Draft - X if draft, ' ' if not draft Msgno - The message number MailDate - Size - The message size udate - mail message date in Unix time fetchfrom - from line formatted to fit fromlength characters fetchsubject - subject line formatted to fit subjectlength characters */ ?>
I was looking for a bash script and as I did not found one I made my own, only external programm needed is socat for SSL/TLS connect to server: #!/bin/bash - #=============================================================================== # # FILE: delete-mail.sh # USAGE="./delete-mail.sh user pass [server] [port] [max_delete]" # # PARAMETER: user pass - login credentials, mandatory # server - mail server, default: imap.1und1.de # port - port on mail server, default: pop3s # max_delete - maximum # of messages to delete, default: 1000 # DESCRIPTION="delete max_delete messages in given pop3 mailbox" # # AUTHOR: KayM (), kay#rrr.de # CREATED: 09.06.2021 18:03:58 # REVISION: --- #=============================================================================== # check args if [[ "$1" == "-h"* ]]; then printf "usage: %s\ndescr: %s\n" "$USAGE" "$DESCRIPTION" 1>&2 exit elif [ "$#" -lt 2 ]; then printf "Missing args, usage: %s\n" "$USAGE" exit 1 fi # assign values USER="$1" PASS="$2" SERVER="${3:-imap.1und1.de}" PORT="${4:-pop3s}" MAX_MESSAGE="${5:-100}" # define actions popserver() { socat - "OPENSSL:$SERVER:$PORT" 2>/dev/null; } poplogin() { printf "USER %s\n" "$USER" sleep 0.5 printf "PASS %s\n" "$PASS" sleep 0.5 } deletemsg() { for (( j = 1 ; j <= MAX_MESSAGE; j++ )) do printf "DELE %s\n" "$j" sleep 0.5 done } popstat() { echo "STAT"; } popquit() { echo "QUIT"; } checkresult() { local line while read -r line do [[ "$line" == "-ERR "* ]] && MAX_MESSAGE=0 printf "%s\n" "$line" done } # get message count, remove newline RES="$({ poplogin; popstat; } | popserver | tail -n 1 )" RES="${RES//[$'\r\n']}" if [[ "$RES" == "+OK 0 "* ]]; then printf "No messages to delete (%s)\n" "$RES" exit elif [[ "$RES" == "+OK "* ]]; then : "${RES#+OK }"; NUM="${_%% *}" printf "%s Messages found (%s)\n" "$NUM" "$RES" [ "$NUM" -lt "$MAX_MESSAGE" ] && MAX_MESSAGE="$NUM" else printf "Can't get number of messages: %s\n" "$RES" exit 1 fi if [ "$MAX_MESSAGE" = "0" ]; then printf "Keep all messages\n" exit fi # delete messages printf "Delete %s messages ...\n" "$MAX_MESSAGE" { poplogin; deletemsg; popstat; popquit;} | popserver | checkresult
Expect send command is not sending - stops sending during script
The problem is the command send "xxxxx" just does not send. The command seem to be completely ignored. In the script below I am able to log in to the server but no more send commands are transmitted to the server. At the end there is an example of the output. The timeout traps were for debugging. #!/usr/bin/expect -f ## getver_t.sh - Use telnet to get ver from Cisco devices ## Could be asked for login ID or just pasword set pass {Password1 Pass2} set index 0 set timeout 5 set host [lindex $argv 0 ] spawn ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null admin#$host expect { "assword:" { send_user "\n--- sending [lindex $pass $index] ---\n" send "[lindex $pass $index]\r" incr index exp_continue } "\>" { send_user "\n--- I see a prompt ---\n" send -- "ter len 0\r" sleep 3 send -- "show inv\r" sleep 3 send -- "show ver\r" send_user "\n--- Time to end ---\n" sleep 3 send -- "exit\r" sleep 3 exit } timeout { send -- "\r" set timeout 5 send_user "\n--- TIMED OUT 1 ---\n" send -- "ter len 0\r" sleep 3 send -- "show inv\r" sleep 3 send -- "show ver\r" sleep 3 send -- "exit\r" sleep 3 exit } } send_user "\n--- out side land ---\n" send "\r" expect { timeout { send_user "\n--- TIMED OUT 2 ---\n" } } This is what I see running the script. $ ./getver_s.sh server1 spawn ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null admin#server1 Warning: Permanently added 'server1,10.1.1.3' (RSA) to the list of known hosts. Password: --- sending Password1 --- Password: --- sending Pass2 --- ------------------------------------------------------------ Welcome to server1 ------------------------------------------------------------ server1> --- I see a prompt --- --- Time to end --- $ I see the "--- I see a prompt ---" and "--- Time to end ---" messages but the send commands between the send_user commands are not sent to the server. I have rewritten this script changed the logic and run it off different versions of servers (Debian7 & 8) but the result is always the same. Why are those send commands not being sent? Added expect -d output Login Banner -------------- CUT ------------------ Password:" --- sending Password1 --- send: sending "Password1\r" to { exp7 } expect: continuing expect expect: does " " (spawn_id exp7) match glob pattern "assword:"? no ">"? no expect: does " \r\n" (spawn_id exp7) match glob pattern "assword:"? no ">"? no Password: expect: does " \r\nPassword: " (spawn_id exp7) match glob pattern "assword:"? yes expect: set expect_out(0,string) "assword:" expect: set expect_out(spawn_id) "exp7" expect: set expect_out(buffer) " \r\nPassword:" --- sending Pass2 --- send: sending "Pass2\r" to { exp7 } expect: continuing expect expect: does " " (spawn_id exp7) match glob pattern "assword:"? no ">"? no expect: does " \r\n" (spawn_id exp7) match glob pattern "assword:"? no ">"? no -------------- CUT ------------------ motd -------------- CUT ------------------ -\r\nServer1>" --- I see a prompt --- send: sending "ter len 0\r" to { exp7 } send: sending "show inv\r" to { exp7 } send: sending "show ver\r" to { exp7 } send: sending "exit\r" to { exp7 } Server1$
rsync hangs when syncing VirtualBox directory in expect script
My expect script fails to proceed when r-syncing the VirtualBox directories. Here is the script: #!/usr/bin/expect -f spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/Documents/ username#server:/home/username/BAK/Documents/ expect "password:" send "thepassword\r" interact sleep 1 spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/AndroidDev/ username#server:/home/username/BAK/AndroidDev/ expect "password:" send "thepassword\r" interact sleep 1 spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/Android/ username#server:/home/username/BAK/Android/ expect "password:" send "thepassword\r" interact sleep 1 spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/Videos/ username#server:/home/username/BAK/Videos/ ; expect "password:" send "thepassword\r" interact sleep 1 spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/VirtualBox\ VMs/ username#server:/home/username/BAK/"VirtualBox\ VMs/" ; expect "password:" send "thepassword\r" interact sleep 1 spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/Pictures/ username#server:/home/username/BAK/Pictures ; expect "password:" send "thepassword\r" interact sleep 1 spawn scp -v /home/james/.vimrc username#server:/home/username/BAK/ ; expect "password:" send "thepassword\r" interact sleep 1 spawn scp -v /home/james/.bashrc username#server:/home/username/BAK/ ; expect "password:" send "thepassword\r" interact sleep 1 spawn scp -v /home/james/.profile username#server:/home/username/BAK/ ; expect "password:" send "thepassword\r" interact sleep 1 spawn scp -v /home/james/.xsession* username#server:/home/username/BAK/ ; expect "password:" send "thepassword\r" interact sleep 1 If I change spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/rschostag/VirtualBox\ VMs/ remote#server:/home/directory/BAK/VirtualBox to spawn scp -vr /home/rschostag/VirtualBox\ VMs/ remote#server:/home/directory/BAK/VirtualBox, it works. However, I do not want to do full backup every time. What is the best way to incrementally backup this directory or otherwise make the rsync command work properly? Thank you! As requested, please find the output of expect -d script as follows: $ sudo expect -d rsync_backup expect version 5.45 argv[0] = expect argv[1] = -d argv[2] = rsync_backup set argc 0 set argv0 "rsync_backup" set argv "" executing commands from command file rsync_backup spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/Documents/ username#server:/home/username/BAK/Documents/ parent: waiting for sync byte parent: telling child to go ahead parent: now unsynchronized from child spawn: returns {5064} expect: does "" (spawn_id exp6) match glob pattern "password:"? no username#server password: expect: does "username#server password: " (spawn_id exp6) match glob pattern "password:"? yes expect: set expect_out(0,string) "password:" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "username#server password:" send: sending "thepassword\r" to { exp6 } tty_raw_noecho: was raw = 0 echo = 1 spawn id exp6 sent <\r\n> spawn id exp6 sent <sending incremental file list\r\n> sending incremental file list spawn id exp6 sent <Scripts/rsync.log\r\n> Scripts/rsync.log spawn id exp6 sent <\r 2.29K 0% 0.00kB/s 0:00:00 > 2.29K 0% 0.00kB/s 0:00:00 spawn id exp6 sent <\r 5.26M 100% 78.42MB/s 0:00:00 (xfr#1, ir-chk=1561/3024)\r\n> 5.26M 100% 78.42MB/s 0:00:00 (xfr#1, ir-chk=1561/3024) spawn id exp6 sent <projects/JOBS/jobsearch_production/\r\n> projects/JOBS/jobsearch_production/ spawn id exp6 sent <projects/JOBS/jobsearch_production/db.sqlite3\r\n> projects/JOBS/jobsearch_production/db.sqlite3 spawn id exp6 sent <\r 5.41K 0% 0.00kB/s 0:00:00 > 5.41K 0% 0.00kB/s 0:00:00 spawn id exp6 sent <\r 7.35M 100% 73.71MB/s 0:00:00 (xfr#2, ir-chk=1025/6252)\r\n> 7.35M 100% 73.71MB/s 0:00:00 (xfr#2, ir-chk=1025/6252) spawn id exp6 sent <\r\nNumber of files: 10,529 (reg: 8,419, dir: 2,096, link: 14)> Number of files: 10,529 (reg: 8,419, dir: 2,096, link: 14)spawn id exp6 sent <\r\nNumber of created files: 0> Number of created files: 0spawn id exp6 sent <\r\nNumber of regular files transferred: 2> Number of regular files transferred: 2spawn id exp6 sent <\r\nTotal file size: 1.73G bytes> Total file size: 1.73G bytesspawn id exp6 sent <\r\nTotal transferred file size: 12.61M bytes> Total transferred file size: 12.61M bytesspawn id exp6 sent <\r\nLiteral data: 62.70K bytes> Literal data: 62.70K bytesspawn id exp6 sent <\r\nMatched data: 12.55M bytes> Matched data: 12.55M bytesspawn id exp6 sent <\r\nFile list size: 0> File list size: 0spawn id exp6 sent <\r\nFile list generation time: 0.001 seconds> File list generation time: 0.001 secondsspawn id exp6 sent <\r\nFile list transfer time: 0.000 seconds> File list transfer time: 0.000 secondsspawn id exp6 sent <\r\nTotal bytes sent: 333.42K> Total bytes sent: 333.42Kspawn id exp6 sent <\r\nTotal bytes received: 40.66K> Total bytes received: 40.66Kspawn id exp6 sent <\r\n\r\n> spawn id exp6 sent <sent 333.42K bytes received 40.66K bytes 32.53K bytes/sec\r\ntotal size is 1.73G speedup is 4,616.35> sent 333.42K bytes received 40.66K bytes 32.53K bytes/sec total size is 1.73G speedup is 4,616.35spawn id exp6 sent <\r\n> interact: received eof from spawn_id exp6 tty_set: raw = 0, echo = 1 spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/AndroidDev/ username#server:/home/username/BAK/AndroidDev/ parent: waiting for sync byte parent: telling child to go ahead parent: now unsynchronized from child spawn: returns {5069} expect: does "" (spawn_id exp7) match glob pattern "password:"? no username#server password: expect: does "username#server password: " (spawn_id exp7) match glob pattern "password:"? yes expect: set expect_out(0,string) "password:" expect: set expect_out(spawn_id) "exp7" expect: set expect_out(buffer) "username#server password:" send: sending "thepassword\r" to { exp7 } tty_raw_noecho: was raw = 0 echo = 1 spawn id exp7 sent <\r\n> spawn id exp7 sent <sending incremental file list> sending incremental file listspawn id exp7 sent <\r\n> spawn id exp7 sent <\r\n> spawn id exp7 sent <Number of files: 2,758 (reg: 2,273, dir: 485)> Number of files: 2,758 (reg: 2,273, dir: 485)spawn id exp7 sent <\r\n> spawn id exp7 sent <Number of created files: 0> Number of created files: 0spawn id exp7 sent <\r\nNumber of regular files transferred: 0\r\nTotal file size: 1.00G bytes> Number of regular files transferred: 0 Total file size: 1.00G bytesspawn id exp7 sent <\r\nTotal transferred file size: 0 bytes> Total transferred file size: 0 bytesspawn id exp7 sent <\r\nLiteral data: 0 bytes\r\nMatched data: 0 bytes> Literal data: 0 bytes Matched data: 0 bytesspawn id exp7 sent <\r\nFile list size: 0\r\nFile list generation time: 0.001 seconds> File list size: 0 File list generation time: 0.001 secondsspawn id exp7 sent <\r\nFile list transfer time: 0.000 seconds> File list transfer time: 0.000 secondsspawn id exp7 sent <\r\nTotal bytes sent: 74.25K\r\n> Total bytes sent: 74.25K spawn id exp7 sent <Total bytes received: 2.43K\r\n\r\nsent 74.25K bytes received 2.43K bytes 30.67K bytes/sec> Total bytes received: 2.43K sent 74.25K bytes received 2.43K bytes 30.67K bytes/secspawn id exp7 sent <\r\ntotal size is 1.00G speedup is 13,102.55> total size is 1.00G speedup is 13,102.55spawn id exp7 sent <\r\n> interact: received eof from spawn_id exp7 tty_set: raw = 0, echo = 1 spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/Android/ username#server:/home/username/BAK/Android/ parent: waiting for sync byte parent: telling child to go ahead parent: now unsynchronized from child spawn: returns {5074} expect: does "" (spawn_id exp8) match glob pattern "password:"? no username#server password: expect: does "username#server password: " (spawn_id exp8) match glob pattern "password:"? yes expect: set expect_out(0,string) "password:" expect: set expect_out(spawn_id) "exp8" expect: set expect_out(buffer) "username#server password:" send: sending "thepassword\r" to { exp8 } tty_raw_noecho: was raw = 0 echo = 1 spawn id exp8 sent <\r\n> spawn id exp8 sent <sending incremental file list\r\n> sending incremental file list spawn id exp8 sent <\r\nNumber of files: 3 (dir: 3)> Number of files: 3 (dir: 3)spawn id exp8 sent <\r\n> spawn id exp8 sent <Number of created files: 0> Number of created files: 0spawn id exp8 sent <\r\nNumber of regular files transferred: 0> Number of regular files transferred: 0spawn id exp8 sent <\r\nTotal file size: 0 bytes> Total file size: 0 bytesspawn id exp8 sent <\r\nTotal transferred file size: 0 bytes> Total transferred file size: 0 bytesspawn id exp8 sent <\r\n> spawn id exp8 sent <Literal data: 0 bytes\r\nMatched data: 0 bytes\r\nFile list size: 0> Literal data: 0 bytes Matched data: 0 bytes File list size: 0spawn id exp8 sent <\r\n> spawn id exp8 sent <File list generation time: 0.001 seconds\r\nFile list transfer time: 0.000 seconds\r\nTotal bytes sent: 83> File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 83spawn id exp8 sent <\r\nTotal bytes received: 21> Total bytes received: 21spawn id exp8 sent <\r\n\r\n> spawn id exp8 sent <sent 83 bytes received 21 bytes 69.33 bytes/sec\r\ntotal size is 0 speedup is 0.00> sent 83 bytes received 21 bytes 69.33 bytes/sec total size is 0 speedup is 0.00spawn id exp8 sent <\r\n> interact: received eof from spawn_id exp8 tty_set: raw = 0, echo = 1 spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/Videos/ username#server:/home/username/BAK/Videos/ parent: waiting for sync byte parent: telling child to go ahead parent: now unsynchronized from child spawn: returns {5079} expect: does "" (spawn_id exp9) match glob pattern "password:"? no username#server password: expect: does "username#server password: " (spawn_id exp9) match glob pattern "password:"? yes expect: set expect_out(0,string) "password:" expect: set expect_out(spawn_id) "exp9" expect: set expect_out(buffer) "username#server password:" send: sending "thepassword\r" to { exp9 } tty_raw_noecho: was raw = 0 echo = 1 spawn id exp9 sent <\r\n> spawn id exp9 sent <sending incremental file list\r\n> sending incremental file list spawn id exp9 sent <\r\nNumber of files: 30 (reg: 29, dir: 1)\r\nNumber of created files: 0> Number of files: 30 (reg: 29, dir: 1) Number of created files: 0spawn id exp9 sent <\r\nNumber of regular files transferred: 0> Number of regular files transferred: 0spawn id exp9 sent <\r\nTotal file size: 1.35G bytes> Total file size: 1.35G bytesspawn id exp9 sent <\r\nTotal transferred file size: 0 bytes> Total transferred file size: 0 bytesspawn id exp9 sent <\r\nLiteral data: 0 bytes> Literal data: 0 bytesspawn id exp9 sent <\r\nMatched data: 0 bytes\r\nFile list size: 0> Matched data: 0 bytes File list size: 0spawn id exp9 sent <\r\nFile list generation time: 0.001 seconds> File list generation time: 0.001 secondsspawn id exp9 sent <\r\nFile list transfer time: 0.000 seconds> File list transfer time: 0.000 secondsspawn id exp9 sent <\r\nTotal bytes sent: 587\r\n> Total bytes sent: 587 spawn id exp9 sent <Total bytes received: 11\r\n\r\nsent 587 bytes received 11 bytes 398.67 bytes/sec\r\n> Total bytes received: 11 sent 587 bytes received 11 bytes 398.67 bytes/sec spawn id exp9 sent <total size is 1.35G speedup is 2,263,533.89\r\n> total size is 1.35G speedup is 2,263,533.89 interact: received eof from spawn_id exp9 tty_set: raw = 0, echo = 1 spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/james/VirtualBox VMs/ username#server:/home/username/BAK/"VirtualBox VMs/" parent: waiting for sync byte parent: telling child to go ahead parent: now unsynchronized from child spawn: returns {5084} expect: does "" (spawn_id exp10) match glob pattern "password:"? no username#server password: expect: does "username#server password: " (spawn_id exp10) match glob pattern "password:"? yes expect: set expect_out(0,string) "password:" expect: set expect_out(spawn_id) "exp10" expect: set expect_out(buffer) "username#server password:" send: sending "thepassword\r" to { exp10 } tty_raw_noecho: was raw = 0 echo = 1 spawn id exp10 sent <\r\n> spawn id exp10 sent <sending incremental file list\r\n> sending incremental file list This is where it hangs and does not provide an error message that I can identify. I moved the remote VirtualBox directory and started the backup again, and it completed finally after running the backup for a few days. The total size of the backup was 71 GB. I ran it with expect -d which prints a very verbose output in the terminal, so that was helpful alone, because the script without -d may have looked like it was hanging only because it was not printing anything in the output. So, there was never anything wrong, I believe, other than no progress indication, which made me believe that it may not be working. Thank you for advising me about the expect -d option.
#glennjackman answered my question in a comment but did not answer the question formally. Here is the new script that works: #!/usr/bin/expect -f spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/ryan/Documents/ username#server:/home/username/BAK/Documents/ expect "password:" send "thepassword\r" expect eof sleep 1 spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/ryan/AndroidDev/ username#server:/home/username/BAK/AndroidDev/ expect "password:" send "thepassword\r" expect eof sleep 1 spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/ryan/Android/ username#server:/home/username/BAK/Android/ expect "password:" send "thepassword\r" expect eof sleep 1 spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/ryan/Videos/ username#server:/home/username/BAK/Videos/ ; expect "password:" send "thepassword\r" expect eof sleep 1 spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/ryan/VirtualBox\ VMs/ username#server:/home/username/BAK/"VirtualBox\ VMs/" ; expect "password:" send "thepassword\r" expect eof sleep 1 spawn rsync -vruEtl --progress -h --log-file=rsync.log --fuzzy --stats /home/ryan/Pictures/ username#server:/home/username/BAK/Pictures ; expect "password:" send "thepassword\r" expect eof sleep 1 spawn scp -v /home/ryan/.vimrc username#server:/home/username/BAK/ ; expect "password:" send "thepassword\r" expect eof sleep 1 spawn scp -v /home/ryan/.bashrc username#server:/home/username/BAK/ ; expect "password:" send "thepassword\r" expect eof sleep 1 spawn scp -v /home/ryan/.profile username#server:/home/username/BAK/ ; expect "password:" send "thepassword\r" expect eof sleep 1 spawn scp -v /home/ryan/.xsession* username#server:/home/username/BAK/ ; expect "password:" send "thepassword\r" expect eof sleep 1
LIST command in Pop3 server-like
I must do a pop3 server in bash and I have a problem with the LIST command. I can connect to my server, do some command but when LIST are calling my client wait long time. He probably wait for another input but I can't see what I'm doing wrong. I follow the [RFC 1939][1] : C: LIST S: +OK 2 messages (320 octets) S: 1 120 S: 2 200 S: . My minimal implementation : [...] "STAT"*) echo "+OK 1 3" ;; "LIST"*) echo "+OK 1 messages (3 octets)" echo "1 3" echo "." ;; [...] Claws Mail Log : * Account 'root#pop.kali': Connecting to POP3 server: localhost... [18:02:42] POP3< +OK [18:02:42] POP3> USER root [18:02:42] POP3< +OK USER [18:02:42] POP3> PASS ******** [18:02:42] POP3< +OK PASS [18:02:42] POP3> STAT [18:02:42] POP3< +OK 1 3 [18:02:42] POP3> UIDL [18:02:42] POP3< -ERR 'UIDL ' n'est pas une commande valide ** command not supported [18:02:42] POP3> LAST [18:02:42] POP3< -ERR 'LAST ' n'est pas une commande valide ** command not supported [18:02:42] POP3> LIST [18:02:42] POP3< +OK 1 messages (3 octets) // HERE WAIT A LONG LONG TIME // ** Session timed out. You may be able to recover by increasing the timeout value in Preferences/Other/Miscellaneous. Thank you. [1]: https://www.rfc-editor.org/rfc/rfc1939#page-6
Try making sure each line of your responses ends with CRLF, not just LF. It might be that just the commands expecting a multi-line response insist on seeing .CRLF, not just .LF. [...] "STAT"*) printf "+OK 1 3\r\n" ;; "LIST"*) printf "+OK 1 messages (3 octets)\r\n" printf "1 3\r\n" printf ".\r\n" ;; [...]