How to loop Bash array in Expect script - bash

I have an array in my bash script:
my_file=("a.txt" "b.txt" "c.txt" "d.txt" "e.txt")
Now in the same file I want to use Expect and make a looping to get some files in sftp. This is my code
/usr/bin/expect <<EOF
set timeout -1
array set param ${!my_file[#]}
spawn sftp $sftp_option $user#$host
expect "Password:"
send "$pswd\r"
expect "sftp>"
for arg in $param; do
send "mget $arg*\r"
expect "sftp>"
done
send "bye\r"
EOF
With that code I can't make a loop with that array above. And I got error like this:
wrong # args: should be "array set arrayName list"
while executing
"array set param 0 1"
Full Code:
#!/bin/bash
export PATH=$PATH:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin;
homeDir=/home/jlo_apps/daemon/jlo_ex/collection/wssh;
filePID=$homeDir/rsync_rfs_bbni.pid;
trap "rm $filePID; exit" SIGHUP SIGINT SIGTERM;
if [ -e $filePID ]; then
datenow=`date`;
echo "[ Rsync ] There are rsync processes still running...";
echo "========================= $datenow =========================";
exit 1;
else
echo $$ > $filePID;
fi
. $homeDir/get_rfs_bni.conf;
strconn="$dbuser/$dbpass#$dbhost:$dbport/$dbsid";
profile=`$sqlldr_path -s $strconn <<< "select host|| '|' ||username|| '|' ||password|| '|' ||port|| '|' ||outdir from p_epdp where bank='BBNI';" | tail -2 | head -1`;
mapfile mid < <($sqlldr_path -s $strconn <<< "select distinct(substr(mid, 0, 13)) from p_mid where kode_bank = 'BBNI';" | tail -3 | head -2);
host=$(echo $profile | cut -d"|" -f1);
user=$(echo $profile | cut -d"|" -f2);
pswd=$(echo $profile | cut -d"|" -f3);
port=$(echo $profile | cut -d"|" -f4);
outdir=$(echo $profile | cut -d"|" -f5);
/usr/bin/expect <<EOF
set timeout -1
spawn sftp $sftp_option $user#$host
expect "Password:"
send "$pswd\r"
expect "sftp>"
send "cd $outdir\r"
expect "sftp>"
send "lcd $rfshome\r"
expect "sftp>"
foreach arg {${mid[#]}} {
send "mget $arg*\r"
expect "sftp>"
}
send "bye\r"
EOF
rm $filePID;
sleep 10;
datenow=`date`;
echo "========================= $datenow =========================";
Is there any solution for this problem without separate file between bash and Expect?

The correct syntax to interpolate the array's values would be
array set param ${my_file[#]}
without a !, but this produces
array set param a.txt b.txt c.txt d.txt e.txt
However, the Expect syntax for creating an array looks like
array set param {one one.txt two two.txt}
with alternating keys and values (more like an associative array than just a list of values). But then you still can't use a shell for loop inside the Expect script; the language uses a completely different syntax (Expect is based on TCL, not shell script).
Probably you are looking for something like
/usr/bin/expect <<EOF
set timeout -1
spawn sftp $sftp_option $user#$host
expect "Password:"
send "$pswd\r"
expect "sftp>"
foreach arg {${my_file[#]}} {
send "mget $arg*\r"
expect "sftp>"
}
send "bye\r"
EOF
where I cribbed the proper TCL loop syntax from Pass bash array to expect script

It's a bit awkward looking, but you can do this:
$ my_file=("a.txt" "b.txt" "c.txt" "d.txt" "e.txt")
$ expect -f - -- "${my_file[#]}" <<'END'
foreach arg $argv {puts $arg}
END
a.txt
b.txt
c.txt
d.txt
e.txt
The -f - tells expect to use stdin for the script file.
Then -- ends the command line options, and the remaining arguments go into the argv list.
Don't forget to use a quoted heredoc, like I demonstrate. Otherwise the shell will expand the expect variables.

Related

make the bash script to be faster

I have a fairly large list of websites in "file.txt" and wanted to check if the words "Hello World!" in the site in the list using looping and curl.
i.e in "file.txt" :
blabla.com
blabla2.com
blabla3.com
then my code :
#!/bin/bash
put() {
printf "list : "
read list
run=$(cat $list)
}
put
scan_list() {
for run in $(cat $list);do
if [[ $(curl -skL ${run}) =~ "Hello World!" ]];then
printf "${run} Hello World! \n"
else
printf "${run} No Hello:( \n"
fi
done
}
scan_list
this takes a lot of time, is there a way to make the checking process faster?
Use xargs:
% tr '\12' '\0' < file.txt | \
xargs -0 -r -n 1 -t -P 3 sh -c '
if curl -skL "$1" | grep -q "Hello World!"; then
echo "$1 Hello World!"
exit
fi
echo "$1 No Hello:("
' _
Use tr to convert returns in the file.txt to nulls (\0).
Pass through xargs with -0 option to parse by nulls.
The -r option prevents the command from being ran if the input is empty. This is only available on Linux, so for macOS or *BSD you will need to check that file.txt is not empty before running.
The -n 1 permits only one file per execution.
The -t option is debugging, it prints the command before it is ran.
We allow 3 simultaneous commands in parallel with the -P 3 option.
Using sh -c with a single quoted multi-line command, we substitute $1 for the entries from the file.
The _ fills in the $0 argument, so our entries are $1.

Shell/Bash + expect script based on input list

I'm trying to run a shell script that execute an expect script based on an input list (IP Addresses), but in parallel.
Unfortunately I didn't understood how to achieve it, below my working script that execute the expect script sequentially on each IP contained in the list.
#!/bin/bash
HOST="$1"
USER="user"
PASS="password"
OUT="checkversion.csv"
rm -rf $OUT
touch $OUT
NOW=$(date +"%Y-%m-%d_%H-%M-%S")
echo "Script started at $NOW"
for i in `cat $1`
do
SCRIPT=$(expect -c "
spawn ssh -o HostKeyAlgorithms=+ssh-dss -o KexAlgorithms=+diffie-hellman-group1-sha1 -o StrictHostKeyChecking=no -c 3des-cbc -c aes128-cbc -c aes192-cbc -c aes256-cbc -c aes128-ctr -c aes192-ctr -c aes256-ctr $USER#$i
match_max 100000
expect \"*?assword:*\"
send -- \"$PASS\r\"
expect \"*>*\"
send -- \"start shell\"
send -- \"\r\"
expect \"*%*\"
send -- \"exit\"
send -- \"\r\"
expect \"*>*\"
send -- \"exit\"
send -- \"\r\"
")
VERSION=`echo "$SCRIPT" | grep JUNOS | grep -v WARNING | grep -v possible | grep -v recover`
echo "$i;$VERSION" >> $OUT
done
NOW=$(date +"%Y-%m-%d_%H-%M-%S")
echo "script finished at $NOW - note any errors above"
Could you please help me in found a method to run it in parallel and speedup the process?
Thanks
Please try changing your script as follow:
#!/bin/bash
HOST="$1"
USER="user"
PASS="password"
OUT="checkversion.csv"
rm -rf $OUT
touch $OUT
NOW=$(date +"%Y-%m-%d_%H-%M-%S")
echo "Script started at $NOW"
function runSSH {
PARAM=$1
SCRIPT=$(expect -c "
spawn ssh -o HostKeyAlgorithms=+ssh-dss -o KexAlgorithms=+diffie-hellman-group1-sha1 -o StrictHostKeyChecking=no -c 3des-cbc -c aes128-cbc -c aes192-cbc -c aes256-cbc -c aes128-ctr -c aes192-ctr -c aes256-ctr $USER#$PARAM
match_max 100000
expect \"*?assword:*\"
send -- \"$PASS\r\"
expect \"*>*\"
send -- \"start shell\"
send -- \"\r\"
expect \"*%*\"
send -- \"exit\"
send -- \"\r\"
expect \"*>*\"
send -- \"exit\"
send -- \"\r\"
")
VERSION=`echo "$SCRIPT" | grep JUNOS | grep -v WARNING | grep -v possible | grep -v recover`
echo "$PARAM;$VERSION" >> $OUT
}
for i in `cat $1`; do
# Run SSH in background
runSSH $i &
done
# Wait for background processes to finish...
wait
NOW=$(date +"%Y-%m-%d_%H-%M-%S")
echo "script finished at $NOW - note any errors above"

sftp bash script doesnt downloads files

I have a problem with my bash script.
It connects to my sftp server.
Gets the list of files to download.
Should download the files. But it doesnt do this. I can see the commands. if I write the commands manually, it works.
You can see the script and the log files here:
#!/bin/bash
LOCALDIR="/data/IMPORT/$(date +%Y%m%d)"
REMOTEDIR="/EXPORT"
FILELIST="$LOCALDIR/filelist.txt"
FILELIST2="$LOCALDIR/filelist2.txt"
SFTP="sftp -P 1234 -i /var/xxxxxx.pem -oStrictHostKeyChecking=no user#xxxxx.xxxx"
PASSPHRASE="xxxxxxxxxxxxxxxx"
mkdir -p $LOCALDIR
rm $FILELIST
rm $FILELIST2
allfilenames=()
function readFileList {
expect -c "
spawn $SFTP
expect \"assphrase\"
send \"$PASSPHRASE\r\"
expect \"sftp>\"
send \"lcd $LOCALDIR\r\"
send \"ls -l $REMOTEDIR/*\r\"
expect \"sftp>\"
send \"exit\r\"
interact " > $FILELIST
}
function getFiles {
myfilenames=("$#")
expect -c "
spawn $SFTP
expect \"assphrase\"
send \"$PASSPHRASE\r\"
expect \"sftp>\"
send \"lcd $LOCALDIR\r\"
expect \"sftp>\"
send \"cd $REMOTEDIR\r\"
expect \"sftp>\"
" >> $FILELIST2
for filepath in "${myfilenames[#]}"
do
file="$(basename -- $filepath)"
expect -c "
send \"get -P $file\r\n\"
sleep 3
expect \"sftp>\"
" >> $FILELIST2
done
expect -c "
send \"exit\r\"
" >> $FILELIST2
}
readFileList
c=0
if [[ -f "$FILELIST" ]]; then
while read line; do
filename=$(echo $line | awk '{ print $9 }')
if [[ "$filename" =~ ^$REMOTEDIR ]] ; then
allfilenames+=($filename)
fi
done < $FILELIST
fi
getFiles "${allfilenames[#]}"
filelist.txt looks like:
spawn sftp -P 1234 -i /var/xxxxxx.pem -oStrictHostKeyChecking=no user#xxxxx.xxxx
Enter passphrase for key '/var/xxxxxx.pem':
Connected to xxxxx.xxxx.
sftp> lcd /data/IMPORT/20200401
sftp> ls -l /EXPORT/*
-rw-r--r-- 0 1000472 1000472 3681 Mar 31 22:31 /EXPORT/file1.txt
-rw-r--r-- 0 1000472 1000472 14537 Mar 31 22:34 /EXPORT/file2.txt
-rw-r--r-- 0 1000472 1000472 5932 Mar 31 22:34 /EXPORT/file3.txt
sftp> exit
filelist2.txt looks like:
spawn sftp -P 1234 -i /var/xxxxxx.pem -oStrictHostKeyChecking=no user#xxxxx.xxxx
Enter passphrase for key '/var/xxxxxx.pem':
Connected to xxxxx.xxxx.
sftp> lcd /data/IMPORT/20200401
sftp> cd /EXPORT
sftp> get -P file1.txt
get -P file2.txt
get -P file3.txt
exit
expect -c 'spawn sftp ...'
expect -c 'send "get file\r" '
Here when the first expect -c completes, the SFTP connection will be closed so the second expect -c would not work. You have to use one single expect -c for one SFTP session.
It's like when you manually sftp to the server, you cannot temporarily go back to Bash and come back to sftp later.

Expect deletes ^H after spawn bash -c

I have an expect script that grabs a bunch of configs and outputs them to separate files. The files though have a bunch of ^H's though I'd like to clean out. I used the CtrlV+CtrlH to put it in the expect script, but when it runs it just does it. The sed works fine in bash, but fails in expect. How can I delete the ^H from the files?
foreach host $ip {
set output [ open "$host" w ]
set timeout 2
spawn ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 -o "StrictHostKeyChecking no" -oHostKeyAlgorithms=+ssh-dss $username#$host
expect {
eof {wait; spawn telnet $host;
expect "ame:";
send "$username\r"}
}
expect "word:"
send "$password\r"
expect "#"
send "$command\r"
sleep 2
expect {
"ore--" { send -- " "; puts $output $expect_out(buffer); exp_continue}
"#" {send -- "exit\r"}
}
puts $output $expect_out(buffer)
close $output
close
}
foreach host $ip {
spawn bash -c "sed -i 's/^H//gi; s/--More--//g' ./$host"
}
close
Here's what it looks like when it runs:
spawn bash -c sed -i 's//gi; s/--More--//g' ./192.168.50.1
spawn bash -c sed -i 's//gi; s/--More--//g' ./192.168.51.1
spawn bash -c sed -i 's//gi; s/--More--//g' ./192.168.52.1
Assuming an ASCII-based locale, this should work:
tr -d '\010' < file > file.new
\010 is octal 8, which is the ASCII equivalent of backspace.
For calling out to sed, since you don't need to interact with the process, you can use exec instead:
foreach host $ip {
exec sed -i {s/^H//gi; s/--More--//g} ./$host
}
Using Tcl's version of single quotes, which are {}

how to copy two files by SCP in expect script

please advice why
spawn scp $FILE1 $FILE2 $LOGIN#$IP:/tmp
in my expect script copy only FILE1 and not copy FILE2 ?
I try to to transfer both files by scp as
scp file1.csv file2.crt 192.8.200.1:/tmp
without expect and they transferred successfully to /tmp
so why VIA expect the only file that copied is FILE1 ??
what wrong in my syntax ?
example of my expect script
#!/usr/bin/expect -f
set FILE1 file1.csv
set FILE2 file2.crt
set multiPrompt {[#>$]}
spawn scp $FILE1 $FILE2 $LOGIN#$IP:/tmp
expect {
")?" { send "yes\r" ; exp_continue }
word: {send $PASS\r}
}
I also try this:
spawn scp "$FILE1 $FILE2" $LOGIN#$IP:/tmp
OR
spawn scp '$FILE1 $FILE2' $LOGIN#$IP:/tmp
but I get the same problem
PLEASE HELP!!!!!!!!
You can workaround it by using array and foreach in expect script, like:
#!/usr/bin/expect
set files { file1.csv file2.crt }
foreach file $files {
puts "Let's scp $file"
send "scp $file $LOGIN#$IP:/tmp"
}

Resources