getting log from telnet session and writing relevant info to file but line gets overwritten by timestamp variable in expect script - expect

Question

Here is a sample remote session that I am automating
[sample-session]
$telnet 172.16.11.2
Trying 172.16.11.2...
Connected to 172.16.11.2.
Escape character is '^]'.
BCM96858 Broadband Router
Login: user
Password:
> voice show
Global Parameters:
------------------
BoundIfName : bronu8.2501
IP address family : IPv4
BoundIpAddr : 172.16.11.2
Voice Log Levels : general=3,cmgr=3,disp=3,sipcctk=3,bos=3,ept=3,cms=3,prov=3,lhapi=3,istw=3,dsphal=3,slicslac=3
Management Protocol : OMCI
Voice DNS Server Enable : Off
Voice DNS Server Primary : 0.0.0.0
Voice DNS Server Secondary : 0.0.0.0
Service Provider 0:
--------------------
Associated Voice Profile: 1
Locale : IND
DTMFMethod : InBand
HookFlashMethod : None
DigitMap : x+T
Log Server Addr : 0.0.0.0
Log Server Port : 0
T38 : on
V18 : on
RTPDSCPMark : 46
European flash : Off
SIP:
SIP mode : RFC3261
Domain :
Port : 5060
Transport : UDP
SIP URI for TLS calls: Off
RegExpires : 300
RegRetryInterval : 60
DSCPMark : 46
Registrar Addr : 172.16.11.201
Registrar Port : 5060
Proxy Addr : 172.16.11.201
Proxy Port : 5060
OutBoundProxy Addr : 172.16.11.201
OutBoundProxy Port : 5060
Music Server Addr : 0.0.0.0
Music Server Port : 0
Conferencing URI :
Conferencing Option : Local
Failover Enable : Off
Sip OPTIONS Enable : Off
Back-to-primary cfg : Disabled
Secondary Domain :
Secondary Registrar Addr : 0.0.0.0
Secondary Registrar Port : 5060
Secondary Proxy Addr : 0.0.0.0
Secondary Proxy Port : 5060
Secondary Outbound Proxy Addr : 0.0.0.0
Secondary Outbound Proxy Port : 5060
To Tag Matching : On
Timer B ( in ms ) : 32000
Timer F ( in ms ) : 32000
SRTP Usage Option : Disabled
Account 0:
-----------
ActivationStatus : Enabled
VoipServiceStatus : Up
CallStatus : Idle
Associated LineInst : 1
PhysEndpt : 0
Extension : 122
DisplayName :
AuthName : 122
AuthPwd : 0000
TxGain : 0 dB
RxGain : 0 dB
CALLFEATURES:
MWI : off
Caller ID number : on
Caller ID name : on
CallWaiting : on
CFWDNum :
CallFwdAll : off
CallFwdBusy : off
CallFwdNoans : off
AnonymousOutgoingCall: off
AnonymousCallRcvBlock: off
DoNotDisturb : off
CallCompOnBusy : off
SpeedDial : off
WarmLine : off
WarmLineNum :
CallBarring : off
CallBarringMode : None
CallBarringPin : 9999
CallBarringDigitMap :
NetPrivacy : on
CODECSETTINGS:
VAD : on
pTime : 20
CodecList : (0) G.711MuLaw
(1) G.711ALaw
(2) G.723.1
(3) G.726_16
(4) G.726_24
(5) G.726_32
Account 1:
-----------
ActivationStatus : Enabled
VoipServiceStatus : Up
CallStatus : Idle
Associated LineInst : 2
PhysEndpt : 1
Extension : 121
DisplayName :
AuthName : 121
AuthPwd : 0000
TxGain : 0 dB
RxGain : 0 dB
CALLFEATURES:
MWI : off
Caller ID number : on
Caller ID name : on
CallWaiting : on
CFWDNum :
CallFwdAll : off
CallFwdBusy : off
CallFwdNoans : off
AnonymousOutgoingCall: off
AnonymousCallRcvBlock: off
DoNotDisturb : off
CallCompOnBusy : off
SpeedDial : off
WarmLine : off
WarmLineNum :
CallBarring : off
CallBarringMode : None
CallBarringPin : 9999
CallBarringDigitMap :
NetPrivacy : on
CODECSETTINGS:
VAD : on
pTime : 20
CodecList : (0) G.711MuLaw
(1) G.711ALaw
(2) G.723.1
(3) G.726_16
(4) G.726_24
(5) G.726_32
> exit
Bye bye. Have a nice day!!!
Connection closed by foreign host.
[/sample-session]
My aim is to obtain lines containing ActivationStatus and VoipServiceStatus.
So I wrote this script
#!/usr/bin/expect -f
set timeout 2
set ip_addr [lindex $argv 0]
set OUTFILE "voip_status.log"
spawn telnet 172.16.11.$ip_addr
expect {
timeout { send_user "\nFailed to get login prompt for $ip_addr\n"; exit 1 }
eof { send_user "\ntelnet failure for server $ip_addr\n"; exit 2 }
"*Login:"
}
#expect "*Login:"
send -- "user\r"
expect "*Password:"
send -- "user\r"
expect ">"
send -- "voice show\r"
expect ">"
set voice_full_output $expect_out(buffer)
set voice_stat_lines [ split $voice_full_output "\n" ]
foreach line $voice_stat_lines {
if {[string match "*VoipServiceStatus*" $line]} {
set timestamp [timestamp -format %Y-%m-%d_%H:%M:%S]
set chan [open $OUTFILE a+]
#puts "ok - MATCH $line"
puts $chan "$line $timestamp"
close $chan
}
if {[string match "*ActivationStatus*" $line]} {
set chan [open $OUTFILE a+]
#puts "ok - MATCH $line"
puts $chan "$line"
close $chan
}
}
send -- "exit\r"
expect eof
Here is my output
$ cat voip_status.log
ActivationStatus : Enabled
2021-03-17_11:32:35tus : Up
ActivationStatus : Enabled
2021-03-17_11:32:35tus : Up
As you can see that time stamp has overwritten the line I had filtered out.
I wanted timestamp to appear at the end of line .
Why is this happening ?
Here I made an experimental setup which anybody can run to see this behaviour.
Contents of 'output-voice-log-for-experiment.sh' follow
[file to give input]
#!/usr/bin/perl
print "
Account 0:
-----------
ActivationStatus : Enabled
VoipServiceStatus : Up
CallStatus : Idle
Associated LineInst : 1
PhysEndpt : 0
Extension : 122
DisplayName :
AuthName : 122
AuthPwd : 0000
TxGain : 0 dB
RxGain : 0 dB
CALLFEATURES:
MWI : off
Caller ID number : on
Caller ID name : on
CallWaiting : on
CFWDNum :
CallFwdAll : off
CallFwdBusy : off
CallFwdNoans : off
AnonymousOutgoingCall: off
AnonymousCallRcvBlock: off
DoNotDisturb : off
CallCompOnBusy : off
SpeedDial : off
WarmLine : off
WarmLineNum :
CallBarring : off
CallBarringMode : None
CallBarringPin : 9999
CallBarringDigitMap :
NetPrivacy : on
CODECSETTINGS:
VAD : on
pTime : 20
CodecList : (0) G.711MuLaw
(1) G.711ALaw
(2) G.723.1
(3) G.726_16
(4) G.726_24
(5) G.726_32
Account 1:
-----------
ActivationStatus : Enabled
VoipServiceStatus : Up
CallStatus : Idle
Associated LineInst : 2
PhysEndpt : 1
Extension : 121
DisplayName :
AuthName : 121
AuthPwd : 0000
TxGain : 0 dB
RxGain : 0 dB
CALLFEATURES:
MWI : off
Caller ID number : on
Caller ID name : on
CallWaiting : on
CFWDNum :
CallFwdAll : off
CallFwdBusy : off
CallFwdNoans : off
AnonymousOutgoingCall: off
AnonymousCallRcvBlock: off
DoNotDisturb : off
CallCompOnBusy : off
SpeedDial : off
WarmLine : off
WarmLineNum :
CallBarring : off
CallBarringMode : None
CallBarringPin : 9999
CallBarringDigitMap :
NetPrivacy : on
CODECSETTINGS:
VAD : on
pTime : 20
CodecList : (0) G.711MuLaw
(1) G.711ALaw
(2) G.723.1
(3) G.726_16
(4) G.726_24
(5) G.726_32
> exit
"
[/file to give input]
Contents of 'process-experimental-echo.sh' follow
[file to process input]
#!/usr/bin/expect -f
spawn ./output-voice-log-for-experiment.sh
set timeout 20
set OUTFILE "experimental_voip_status.log"
expect ">"
set voice_full_output $expect_out(buffer)
set voice_stat_lines [ split $voice_full_output "\n" ]
foreach line $voice_stat_lines {
if {[string match "*VoipServiceStatus*" $line]} {
set timestamp [timestamp -format %Y-%m-%d_%H:%M:%S]
set chan [open $OUTFILE a+]
puts "ok - MATCH $line"
puts $chan "$line $timestamp"
close $chan
}
if {[string match "*ActivationStatus*" $line]} {
set chan [open $OUTFILE a+]
puts "ok - MATCH $line"
puts $chan "$line"
close $chan
}
}
[/file to process input]
Now simply run ./process-experimental-echo.sh and see this behaviour yourself.
Here is the output file you will get
$ cat experimental_voip_status.log
ActivationStatus : Enabled
2021-03-17_12:27:11tus : Up
ActivationStatus : Enabled
2021-03-17_12:27:11tus : Up
Thanks.

Answer 1

If you look at the output file using cat -vet it shows the non-printing control characters, tabs, and end-of-line, and you get:
ActivationStatus : Enabled^M$
VoipServiceStatus : Up^M 2021-03-17_09:02:53$
You can see ^M in the middle of a line, which is carriage-return. On a terminal this makes the cursor go back to the start of the line, and start overprinting.
This is the typical result of spawn, which runs a command through a pty, which alters each output newlines into carriage-return and newline. This is normal and desired for a pty. You can either turn it off (see stty -onlcr), or knowing that it is likely to be there, you can remove it in your script: after the foreach ... line, trim the variable $line to remove \r:
set line [string trimright $line "\r"]