expect redirect outputs giberish - bash

I have backup.sh
#!/bin/bash
expect backup.exp
and backup.exp
#!/usr/bin/expect -f
#exp_internal 1
#log_user 0
set timeout 29
puts "----------------- [exec date +%Y.%m.%d\ %H:%M:%S] ------ exp start -----"
spawn -noecho ssh andrej#10.11.22.17
expect {
timeout { send_user "\n--- failed to get expected string---\n"; exit 1 }
eof { send_user "\nSSH failure\n"; exit 1 }
" andrej $ "
}
send "sudo bash\r"
send "ls -lh\r"
send exit\r
send exit\r
interact
puts "----------------- [exec date +%Y.%m.%d\ %H:%M:%S] ------ exp ende ------"
when I run
bash backup.sh >> backup.templog 2>&1
I get
----------------- 2015.09.02 18:48:29 ------ exp start -----
Last login: Wed Sep 2 18:48:24 2015 from 10.11.22.16
^[]0;andrej#centos7c:~/andrej^G^[[?1034h^[[01;32mandrej#centos7c ^[[01;34m18:48 andrej $ ^[[00msudo bash
^[]0;root#centos7c:/home/andrej/andrej^G^[[?1034h^[[01;31mroot#centos7c ^[[01;34m18:48 andrej $ ^[[00mls -lh
total 16K
-rw-rw-r--. 1 andrej andrej 13K Jun 30 19:04 iptables.sh
drwxrwxr-x. 2 andrej andrej 6 Jun 17 22:46 ^[[0m^[[01;34mmount^[[0m
^[]0;root#centos7c:/home/andrej/andrej^G^[[01;31mroot#centos7c ^[[01;34m18:48 andrej $ ^[[00mexit
exit
^[]0;andrej#centos7c:~/andrej^G^[[01;32mandrej#centos7c ^[[01;34m18:48 andrej $ ^[[00mexit
logout
Connection to 10.11.22.17 closed.
----------------- 2015.09.02 18:48:39 ------ exp ende ------
but when I do
bash backup.sh
, I get nice output that I need :
----------------- 2015.09.02 18:52:09 ------ exp start -----
Last login: Wed Sep 2 18:48:39 2015 from 10.11.22.16
andrej#centos7c 18:52 andrej $ sudo bash
root#centos7c 18:52 andrej $ ls -lh
total 16K
-rw-rw-r--. 1 andrej andrej 13K Jun 30 19:04 iptables.sh
drwxrwxr-x. 2 andrej andrej 6 Jun 17 22:46 mount
root#centos7c 18:52 andrej $ exit
exit
andrej#centos7c 18:52 andrej $ exit
logout
Connection to 10.11.22.17 closed.
----------------- 2015.09.02 18:52:20 ------ exp ende ------
How can I get rid of extra giberish when outputing to the log file when executing the script? I am using centos 7.1 in both server and client.
If you have any sugesstion please help me out. Thank you very much.

rewriting a but to address a couple of issues:
expect has 2 built-in methods to get the current datetime: timestamp (expect-specific) and clock (from Tcl)
I turn off output (log_user 0)
when it's important, I change the prompt and use an anchored regular expression to match it
I only output the results of the ls -lh command, which I suspect is your goal.
#!/usr/bin/expect -f
#exp_internal 1
set timeout 29
proc separator {msg} {
puts [format "----------------- %s ------ %s -----" [timestamp -format "%Y.%m.%d %T"] $msg]
}
separator "exp start"
log_user 0
spawn -noecho ssh andrej#10.11.22.17
expect {
timeout { send_user "\n--- failed to get expected string---\n"; exit 1 }
eof { send_user "\nSSH failure\n"; exit 1 }
" $ "
}
send "sudo bash\r"
expect " $ "
send "PS1='$'\r"
expect -re {\$$}
send "ls -lh\r"
expect -re {(.+)\r\n\$$}
puts $expect_out(1,string)
send "exit\r"
expect " $ "
send "exit\r"
expect eof
separator "exp ende"

Related

how can find directory

when starting the tcl script a directory is created via a bash command. at the end of my script i want to read the directory name of the latest dirs. but my script does not find the newest directory but only the 2nd newest
bind pub "-|-" !aa pub:aaa
proc pub:aaa {nick host handle channel arg} {
set home "/home/user"
set bb [exec bash -c "start.sh"]
after 3000
set latest [exec bash -c "ls -td $home/jpg/*/ | head -n1"]
putnow "PRIVMSG $channel :$latest"
}
before starting it has the following folders in the directory:
drwxr-xr-x 2 user user 4096 Jun 24 18:30 aaa
drwxr-xr-x 2 user user 4096 Jun 24 18:14 bbb
after starting it has the following folders in the directory
drwxr-xr-x 2 user user 4096 Jun 24 18:30 aaa
drwxr-xr-x 2 user user 4096 Jun 24 18:14 bbb
drwxr-xr-x 2 user user 4096 Jun 24 18:35 ccc
output is :
<#testbot> aaa
it should be so
<#testbot> ccc
he finds the directory created during which the tcl script is not running
how can I display the newest, newly created directory?
regards
Instead of trying to exec out to a shell to find the most recently modified directory, I'd do it in pure tcl:
proc latest_directory {path {time mtime}} {
set dirs {}
foreach dir [glob -nocomplain -type d $path/*] {
file stat $dir s
lappend dirs $s($time) $dir
}
if {[llength $dirs] == 0} {
error "No directories found in $path"
} else {
return [lindex [lsort -integer -decreasing -stride 2 $dirs] 1]
}
}
# Then in pub:aaa
set latest [latest_directory $home/jpg]
As for why you're not getting ccc... hard to say for sure without seeing your start.sh script, but if it ends up running stuff in the background that continues after it exits, maybe it takes more than 3 seconds to create that directory?

How to prevent pexpect from echoing the password?

By default, pexpect.spawn() would not output anything. But when I specify logfile=sys.stdout it'll also echo the password (e.g. for ssh). So how can I see the real time interaction with the spawned process without the password being echoed (just like the Expect (the Tcl extension) does)?
pexepct example:
# cat expect.py
import pexpect, sys
logfile = sys.stdout if len(sys.argv) == 2 else None
ssh = pexpect.spawn('ssh foo#localhost', logfile=logfile)
ssh.delaybeforesend = 1
ssh.expect('assword:')
ssh.sendline('123456')
ssh.expect('\r\n\\$')
ssh.sendline('exit')
ssh.expect(pexpect.EOF)
ssh.wait()
# python expect.py <-- no output
# python expect.py stdout
foo#localhost's password: 123456 <-- the password is visible
Last login: Tue Mar 22 10:32:49 2016 from localhost
$ exit
exit
Connection to localhost closed.
#
Expect example:
# cat ssh.exp
spawn ssh foo#localhost
expect assword:
send "123456\r"
expect {\$}
send "exit\r"
expect eof
wait
# expect ssh.exp
spawn ssh foo#localhost
foo#localhost's password: <-- the password is invisible
Last login: Tue Mar 22 10:45:03 2016 from localhost
$ exit
Connection to localhost closed.
#
Just to make the question answered. The credit goes to Thomas K. See his comments under the question for more details.
[STEP 101] # cat foo.py
#!/usr/bin/env python3
import pexpect, sys
spawn = pexpect.spawnu if sys.version_info[0] >= 3 else pexpect.spawn
ssh = spawn('ssh -t foo#localhost bash --noprofile --norc')
ssh.logfile_read = sys.stdout
ssh.expect('assword:')
ssh.sendline('123456')
ssh.expect('bash-[.0-9]+[$#]')
ssh.sendline('exit')
ssh.expect(pexpect.EOF)
ssh.wait()
[STEP 102] #
[STEP 103] # python2 foo.py
foo#localhost's password:
bash-5.1$ exit
exit
Connection to localhost closed.
[STEP 104] #
[STEP 105] # python3 foo.py
foo#localhost's password:
bash-5.1$ exit
exit
Connection to localhost closed.
[STEP 106] #

How to decrease TCP connect() system call timeout?

In command below I enable file /dev/tcp/10.10.10.1/80 both for reading and writing and associate it with file descriptor 3:
$ time exec 3<>/dev/tcp/10.10.10.1/80
bash: connect: Operation timed out
bash: /dev/tcp/10.10.10.1/80: Operation timed out
real 1m15.151s
user 0m0.000s
sys 0m0.000s
This automatically tries to perform TCP three-way handshake. If 10.10.10.1 is not reachable as in example above, then connect system call tries to connect for 75 seconds. Is this 75 second timeout determined by bash? Or is this system default? Last but not least, is there a way to decrease this timeout value?
It's not possible in Bash without modifying the source as already mentioned, although here is the workaround by using timeout command, e.g.:
$ timeout 1 bash -c "</dev/tcp/stackoverflow.com/80" && echo Port open. || echo Port closed.
Port open.
$ timeout 1 bash -c "</dev/tcp/stackoverflow.com/81" && echo Port open. || echo Port closed.
Port closed.
Using this syntax, the timeout command will kill the process after the given time.
See: timeout --help for more options.
It is determined by TCP. It can be decreased on a per-socket basis by application code.
NB The timeout only takes effect if there is no response at all. If there is a connection refusal, the error occurs immediately.
No: there is no way of changing timeout by using /dev/tcp/
Yes, you could change default timeout for TCP connection in any programming language.
But, bash is not a programming language!
You could have a look into source code (see: Bash Homepage), you may find lib/sh/netopen.c file where you could read in _netopen4 function:
s = socket(AF_INET, (typ == 't') ? SOCK_STREAM : SOCK_DGRAM, 0);
You could read this file carefully, there are no consideration of connection timeout.
Without patching bash sources, there is no way of changing connection timeout by a bash script.
Simple HTTP client using netcat (near pure bash)
There is a little sample HTTP client written in pure bash, but using netcat:
#!/bin/bash
tmpfile=$(mktemp -p $HOME .netbash-XXXXXX)
exec 7> >(nc -w 3 -q 0 stackoverflow.com 80 >$tmpfile)
exec 6<$tmpfile
rm $tmpfile
printf >&7 "GET %s HTTP/1.0\r\nHost: stackoverflow.com\r\n\r\n" \
/questions/24317341/how-to-decrease-tcp-connect-system-call-timeout
timeout=100;
while ! read -t .001 -u 6 status ; do read -t .001 foo;done
echo STATUS: $status
[ "$status" ] && [ -z "${status//HTTP*200 OK*}" ] || exit 1
echo HEADER:
while read -u 6 -a head && [ "${head//$'\r'}" ]; do
printf "%-20s : %s\n" ${head%:} "${head[*]:1}"
done
echo TITLE:
sed '/<title>/s/<[^>]*>//gp;d' <&6
exec 7>&-
exec 6<&-
This could render:
STATUS: HTTP/1.1 200 OK
HEADER:
Cache-Control : private
Content-Type : text/html; charset=utf-8
X-Frame-Options : SAMEORIGIN
X-Request-Guid : 46d55dc9-f7fe-425f-a560-fc49d885a5e5
Content-Length : 91642
Accept-Ranges : bytes
Date : Wed, 19 Oct 2016 13:24:35 GMT
Via : 1.1 varnish
Age : 0
Connection : close
X-Served-By : cache-fra1243-FRA
X-Cache : MISS
X-Cache-Hits : 0
X-Timer : S1476883475.343528,VS0,VE100
X-DNS-Prefetch-Control : off
Set-Cookie : prov=ff1129e3-7de5-9375-58ee-5f739eb73449; domain=.stackoverflow.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly
TITLE:
bash - How to decrease TCP connect() system call timeout? - Stack Overflow
Some explanations:
We create first a temporary file (under private directory for security reason), bind and delete before using them.
$ tmpfile=$(mktemp -p $HOME .netbash-XXXXXX)
$ exec 7> >(nc -w 3 -q 0 stackoverflow.com 80 >$tmpfile)
$ exec 6<$tmpfile
$ rm $tmpfile
$ ls $tmpfile
ls: cannot access /home/user/.netbash-rKvpZW: No such file or directory
$ ls -l /proc/self/fd
lrwx------ 1 user user 64 Oct 19 15:20 0 -> /dev/pts/1
lrwx------ 1 user user 64 Oct 19 15:20 1 -> /dev/pts/1
lrwx------ 1 user user 64 Oct 19 15:20 2 -> /dev/pts/1
lr-x------ 1 user user 64 Oct 19 15:20 3 -> /proc/30237/fd
lr-x------ 1 user user 64 Oct 19 15:20 6 -> /home/user/.netbash-rKvpZW (deleted)
l-wx------ 1 user user 64 Oct 19 15:20 7 -> pipe:[2097453]
$ echo GET / HTTP/1.0$'\r\n\r' >&7
$ read -u 6 foo
$ echo $foo
HTTP/1.1 500 Domain Not Found
$ exec 7>&-
$ exec 6>&-

No response from [ -f /path/to/file ]

I am trying to get my Capistrano deploy script working, but it is not doing the symlinking as it is configured to do as shown below.
set :linked_files, %w{config/database.yml}
set :linked_dirs, %w{log tmp vendor/bundle public/system}
When it runs the related command, I get the following:
WARN [SKIPPING] No Matching Host for /usr/bin/env [ -f /path/to/shared/config/database.yml ]
If I run this command on the server, either through ssh or through logging onto the server and running the command, I get no response from the command.
user: ~
$ [ -f /path/to/shared/config/database.yml ]
user: ~
$
The file does exist in the specified location and has permissions.
user: ~
$ ll /path/to/shared/config/
total 4.0K
drwxrwxr-x 2 user group 33 Nov 30 10:58 .
drwxrwxr-x 7 user group 89 Nov 30 10:58 ..
-rwxrwxr-x 1 user group 805 Nov 30 10:58 database.yml
user: ~
Shouldn't this return a true or a false, instead of nothing? Is there a configuration I may have changed that suppresses the output? I get no response at all whether the file exists or not.
In your response to the actual question you ask, test (which is what [ is an alias for) does in fact not return output to stdout. It returns an exit code.
user: ~
$ [ -f /path/to/shared/config/database.yml ] # if the file exists
user: ~
$ echo $?
0
user: ~
$ [ -f /path/to/shared/config/database.yml ] # if the file does not exist
user: ~
$ echo $?
1
test -f /path/to/file (or [ -f /path/to/file ]) yields an exit code of 0 if the file exists or 1 if it does not. If you want to check that a file is there and echo the path to it, try:
[ -f /path/to/file ] && echo "/path/to/file"

Cronjob missing content from mail body, manually works fine

Edit: Solution found via Barmar's answer. Added full smartctl command path and it works via crontab now.
I have the below script:
#!/bin/bash
#set -x
EMAIL="admin#domain.co.uk"
FILE="/root/scripts/hddreport.txt"
HOST=`hostname`
HDD01="/dev/sda"
P=`ping -c 1 $HOST | sed '1 ! d' | awk '{print $3}'`
cd /root/scripts/
echo -en "HDD health check on the server hosting" $HOST $P > $FILE
echo -e "\n" >> $FILE
smartctl -H $HDD01 >> $FILE
# The above commands do correctly write the content to $FILE (proved by removing the rm command at the bottom and doing cat on the file after)
smartctl -H $HDD01
echo "\nEmailed you the health of the Hard Drive $HDD01\n"
mailx -s "HDD health check complete on `date`" $EMAIL < $FILE
rm $FILE
which runs fine by doing bash /root/scripts/diskhealth.sh as it shows this in my mailbox:
HDD health check on the server hosting domain.co.uk (0.0.0.0)
smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
SMART Health Status: OK
But when I let it run via crontab using any of the following syntax:
X 20 * * * /bin/bash /root/scripts/diskhealth.sh
X 20 * * * /bin/sh /root/scripts/diskhealth.sh
X 20 * * * /root/scripts/diskhealth.sh
it puts everything but the smartctl disk check:
HDD health check on the server hosting domain.co.uk (0.0.0.0)
Here's what it shows if I add extra echo lines:
This is a test
HDD health check on the server hosting domain.co.uk (0.0.0.0)
Amended script for "This is a test" below:
#!/bin/bash
#set -x
EMAIL="admin#domain.co.uk"
FILE="/root/scripts/hddreport.txt"
HOST=`hostname`
HDD01="/dev/sda"
P=`ping -c 1 $HOST | sed '1 ! d' | awk '{print $3}'`
cd /root/scripts/
echo "This is a test" > $FILE
echo -en "HDD health check on the server hosting" $HOST $P >> $FILE
echo -e "\n" >> $FILE
smartctl -H $HDD01 >> $FILE
smartctl -H $HDD01
echo "\nEmailed you the health of the Hard Drive $HDD01\n"
mailx -s "HDD health check complete on `date`" $EMAIL < $FILE
rm $FILE
Here is the /var/log/syslog output from cron:
Jun 6 20:25:01 hostname /USR/SBIN/CRON[1018112]: (root) CMD (bash /root/scripts/diskhealth.sh)
Jun 6 20:25:01 hostname postfix/pickup[1016576]: 5740356613F: uid=0 from=<root>
Jun 6 20:25:01 hostname postfix/cleanup[1018125]: 5740356613F: message-id=<20130606192501.5740356613F#hostname>
Jun 6 20:25:01 hostname postfix/qmgr[292015]: 5740356613F: from=<root#hostname>, size=465, nrcpt=1 (queue active)
Jun 6 20:25:01 hostname postfix/pickup[1016576]: 631F156613E: uid=0 from=<root>
Jun 6 20:25:01 hostname postfix/cleanup[1018125]: 631F156613E: message-id=<20130606192501.631F156613E#hostname>
Jun 6 20:25:01 hostname postfix/qmgr[292015]: 631F156613E: from=<root#hostname>, size=759, nrcpt=1 (queue active)
Jun 6 20:25:01 hostname pvemailforward[1018132]: forward mail to <root#localhost.localdomain>
Jun 6 20:25:01 hostname postfix/pickup[1016576]: B597B566148: uid=65534 from=<nobody>
Jun 6 20:25:01 hostname postfix/cleanup[1018125]: B597B566148: message-id=<20130606192501.631F156613E#hostname>
Jun 6 20:25:01 hostname postfix/local[1018131]: 631F156613E: to=<root#hostname>, orig_to=<root>, relay=local, delay=0.39, delays=0.16/0/0/0.23, dsn=2.0.0, status=sent (delivered to command: /usr/bin/pvemailforward)
Jun 6 20:25:01 hostname postfix/qmgr[292015]: 631F156613E: removed
Jun 6 20:25:01 hostname postfix/qmgr[292015]: B597B566148: from=<nobody#hostname>, size=963, nrcpt=1 (queue active)
Jun 6 20:25:01 hostname postfix/smtp[1018135]: B597B566148: to=<root#localhost.localdomain>, relay=none, delay=0.16, delays=0.12/0/0.04/0, dsn=5.4.4, status=bounced (Host or domain name not found. Name service error for name=localhost.localdomain type=A: Host not found)
Jun 6 20:25:01 hostname postfix/qmgr[292015]: B597B566148: removed
Jun 6 20:25:01 hostname postfix/cleanup[1018125]: D6570566147: message-id=<20130606192501.D6570566147#hostname>
Jun 6 20:25:01 hostname postfix/smtp[1018130]: 5740356613F: to=<admin#domain.co.uk>, relay=ASPMX.L.GOOGLE.COM[173.194.67.27]:25, delay=0.68, delays=0.12/0/0.19/0.36, dsn=2.0.0, status=sent (250 2.0.0 OK 1370546701 iy4si8635735wic.1 - gsmtp)
Jun 6 20:25:01 ds9453 postfix/qmgr[292015]: 5740356613F: removed
The email is received, just missing the smartctl output.
Cron jobs don't run your .profile. So if smartctl is in a directory you add to $PATH in your profile, it won't be found when you run via cron. Try using the full pathname to the command.

Resources