I'm trying to create a script that retreives a secret from a keepass database.
The script uses Expect to get secret via the keepass cli.
Here after my script:
#!/bin/bash
set +x
entry="$1"
keepass_password="azerty"
keepass_db="canadaplatfomsecretsdb.kdbx"
keepass_entry_dir="canadaplatfomsecretsdb/k8s-enabling"
kubesecretname="artifactoryregcred"
kubenamespace="dev"
echo -e "\n"
echo "Connecting to keepass Database..."
function get_creds {
expect -d <<EOF
set timeout 10
match_max 100000000
spawn kpcli
expect "kpcli:/>"
send "open $keepass_db\n"
expect "password:"
send "$keepass_password\n"
expect ">"
send "cd $keepass_entry_dir\n"
expect "k8s-enabling>"
send "show -f $entry\n"
expect ">"
EOF
}
credentials=$(get_creds)
Here after the output form the gitlab-ci job logs
$ bash getcredsfromkeepass.sh ${entry}
Connecting to keepass Database...
expect version 5.45.4
argv[0] = expect argv[1] = -d
set argc 0
set argv0 "expect"
set argv ""
executing commands from command file
parent: waiting for sync byte
parent: telling child to go ahead
parent: now unsynchronized from child
spawn: returns {3855}
expect: does "" (spawn_id exp3) match glob pattern "kpcli:/>"? no
expect: does "No usable Term::ReadLine::* modules found.\r\nThis list was tried:\r\n * Term::ReadLine::Gnu\r\n * Term::ReadLine::Perl\r\n * Term::ReadLine::Perl5\r\nFor more information, read the documentation: perldoc kpcli\r\n" (spawn_id exp3) match glob pattern "kpcli:/>"? no
expect: read eof
expect: set expect_out(spawn_id) "exp3"
What is causing the problem and how can it be fixed?
Related
for some time I am struggling to get the exit code of a script which I am running from within expect.
It is a BASH script and the questionable part looks like this:
expect -c "
log_file $LOG
spawn su - $ora_user
expect ""
send \"source $oraenv_binary\r\"
expect \"ORACLE_SID = \[$ora_user\] \?\"
send \"$SID\r\"
expect \"The Oracle base has been set to /oracle/$SID\"
send \"$execPATHroot/$subscript $args_subscript\r\"
expect ""
send \"echo \$?\r\"
expect -re \"(\\d+)\" {
set result \$expect_out(1,string)
}
send_user \"subscript exit code: \$result\"
log_file
send \"exit\r\"
expect ""
exit [lindex \$result 3]"
sub_rc=$?
Needed to say that this is one of many tries to get the code, however, unsuccessfully. I guess that my problem lies in incorrectly escaped characters or wrong use of brackets.....
When debugging, I am getting the following:
[336] oraenv_binary=/usr/local/bin/oraenv
[338] expect -c '
log_file /var/opt/osit/oracle/log/ora_sbp_patching_root.bash.log
spawn su - oracle
expect
send "source /usr/local/bin/oraenv\r"
expect "ORACLE_SID = \[oracle\] \?"
send "H95\r"
expect "The Oracle base has been set to /oracle/H95"
send "/opt/osit/oracle/bin/ora_sbp_patching_orausr.bash -s H95 -a CHECK -p /imports/e2r2s48ifde0002/CDSAP/DB/oracle/ORA19/SBP/SBP_1915_220419_202205 -h /imports/e2r2s48ifde0002/CDSAP/DB/oracle/ORA19/SBP/SBP_1915_220419_202205/README19P_2205-70004508.HTM -u oracle\r"
expect
send "echo $?\r"
expect -re "(\d+)" {
set result $expect_out(1,string)
}
send_user "subscript exit code: $result"
log_file
send "exit\r"
expect
exit [lindex $result 3]'
.....subscript runs here OK with exit code 0 in this case
-sh-4.2$ subscript exit code: decho $?
0
-sh-4.2$ exit
logout
expected integer but got ""
while executing
"exit [lindex $result 3]"
[357] sub_rc=0
It seems to me that the regex part "(\d+)" is not OK, but perhaps, it is completely a mess... :-)
Please help.
I have read and tried these recommendations:
Is there a way to expect output without removing it from the buffer in Tcl expect?
https://wiki.tcl-lang.org/page/How+Expect+can+capture+the+exit+code+from+a+remote+command
https://www.unix.com/shell-programming-and-scripting/144812-expect-script-obtain-exit-code-remote-command.html
You have this
expect -re "(\d+)" {
set result $expect_out(1,string)
}
send_user "subscript exit code: $result"
and we can see the output is
-sh-4.2$ subscript exit code: decho $?
0
Because the regular expression "(\d+)" is in double quotes, backslash substitutions will occur, and the pattern becomes (d+) which may not match (do you get a 10 second delay at that point?) -- I suspect this is why $result is empty.
Backslashes are prevalent in regular expressions. Using braces to quote them is the way to go:
expect -re {\d+} {set result $expect_out(0,string)}
Running your expect code with expect -d (or set exp_internal 1 in the code) emits very verbose expect debug output that is extremely useful to see how your patterns are matching (or not).
Using quoted shell heredocs is (IMO) preferable to using quoted strings to encapsulate code.
Consider
expect -c "
send_user \"my home is \$env(HOME)\\n\"
"
versus
expect << 'END_CODE'
send_user "my home is $env(HOME)\n"
END_CODE
With this technique, you pass shell variable to expect through the environment:
export ora_user=oracle
expect << 'END_EXPECT'
#...
spawn su - $env(ora_user)
END_EXPECT
thanks a lot for your answer Glenn, interesting points mentioned.
Regarding the braces versus double quotes - I have changed it like that, but no effect. I have those double quotes escaped by backslash within my code - I think the effect is the same, however, definitely it looks nicer to me and evidently it is safer.
I have played with the debug mode of expect - thanks for that, I can see much more info.
I have noticed that expect holds much more stuff than I "expected" :-)
==> this is just a snippet:
.\r\n-sh-4.2$ " (spawn_id exp7) match regular expression "\d+"? (No Gate, RE only) gate=yes re=yes
expect: set expect_out(0,string) "4"
expect: set expect_out(spawn_id) "exp7"
expect: set expect_out(buffer) "\r\n-sh-4"
can't read "expect_out(1,string)": no such element in array
while executing
"set result $expect_out(1,string)"
invoked from within
"expect -re {\d+} {set result $expect_out(1,string)}"
As you can see, when I am sending the subscript to be executed I am expecting "" i.e. nothing, just the new prompt line.
However, at that point expect is full of stuff, not at all blank - I think, I need to define the prompt exactly:
-sh-4.2$
and then I need to expect it, together with echoed exit code $? and somehow separate exit code integer to get what I want...... I will keep trying.
The final code which works is as follows (much of the credit goes to Glenn for his numerous advices) :
expect -c "
log_file $LOG
spawn su - $ora_user
expect -re {\$ $}
send \"PS1='>'\r\"
expect -re {>$}
send \"source $oraenv_binary\r\"
expect {ORACLE_SID = \[$ora_user\] ? }
send \"$SID\r\"
expect \"The Oracle base has been set to /oracle/$SID\"
send \"$execPATHroot/$subscript $args_subscript\r\"
expect { (subscript) ; exp_continue }
expect -re {>$}
send \"echo \$?\r\"
expect -re {(\d+)\r\n>$} {set result \$expect_out(1,string)}
send_user \"subscript exit code:\$result\n\"
log_file
send \"exit\r\"
expect \"logout\"
exit [lindex \$result 0]"
sub_rc=$?
echo sub_rc:$sub_rc
The first thing after spawn su - $ora_user is to set the prompt by send \"PS1='>'\r\" in order to make new lines with prompt less intrusive to expect.
Then after send \"$execPATHroot/$subscript $args_subscript\r\" I have used the fact, that I have written the subscript to have every line of output populated by (subscript) keyword. So while the subscript produces the output, the expect keeps going by exp_continue.
When the $subscript ends, the prompt > appears into which the expect sends echo $? to get exit code of the $subscript.
This appears on the screen as:
>echo $?
0
>
so the code should expect the integer, return and the new line with prompt - i.e. {(\d+)\r\n>$}. At that time the expect matches the output and expect_out(1,string) is correctly populated:
send: sending "echo $?\r" to { exp7 }
Gate keeper glob pattern for '(\d+)\r\n>$' is '*
>'. Activating booster.
expect: does "" (spawn_id exp7) match regular expression "(\d+)\r\n>$"? Gate "*\r\n>"? gate=no
echo $?
0
>
expect: does "echo $?\r\n0\r\n>" (spawn_id exp7) match regular expression "(\d+)\r\n>$"? Gate "*\r\n>"? gate=yes re=yes
expect: set expect_out(0,string) "0\r\n>"
expect: set expect_out(1,string) "0"
expect: set expect_out(spawn_id) "exp7"
expect: set expect_out(buffer) "echo $?\r\n0\r\n>"
Another thing to mention is \n within send_user \"subscript exit code:\$result\n\" so to have new line next..
The last change to the code in question is:
exit [lindex \$result 0]"
I have changed the index to 0 as variable result has just one item and index 0 stands for 1st item in the list.
I've been struggling with getting the output from a remote server to a local variable or a local file.
My attempt:
#!/bin/bash
my_pass=!!psw!!
server=10.10.10.10
/usr/bin/expect << ENDOFEXPECT
exp_internal 1 ;# expect internal debugging. remove when not needed
set PROMPT ":~ ?# ?"
set timeout 30
spawn bash -c "ssh root#$server"
expect "assword:"
send "$my_pass\r"
expect -re "$PROMPT"
send -- "df -kh /\r"
expect -re "df\[^\n]+\n.+\n(.+\r\n.+)\r\n"
set command_output $expect_out(1,string)
send_user "$command_output\r"
interact
ENDOFEXPECT
echo "====================="
echo " >> $command_output"
Output:
spawn bash -c ssh root#10.10.10.10
parent: waiting for sync byte
parent: telling child to go ahead
parent: now unsynchronized from child
spawn: returns {154725}
expect: does "" (spawn_id exp4) match glob pattern "assword:"? no
Password:
expect: does "\rPassword: " (spawn_id exp4) match glob pattern "assword:"? yes
expect: set expect_out(0,string) "assword:"
expect: set expect_out(spawn_id) "exp4"
expect: set expect_out(buffer) "\rPassword:"
send: sending "!!psw!!\r" to { exp4 }
Gate keeper glob pattern for '' is ''. Not usable, disabling the performance booster.
expect: does " " (spawn_id exp4) match regular expression ""? (No Gate, RE only) gate=yes re=yes
expect: set expect_out(0,string) ""
expect: set expect_out(spawn_id) "exp4"
expect: set expect_out(buffer) ""
send: sending "df -kh /\r" to { exp4 }
Gate keeper glob pattern for 'df[^
]+
.+
(.+
.+)
' is ''. Not usable, disabling the performance booster.
expect: does " " (spawn_id exp4) match regular expression "df[^\n]+\n.+\n(.+\r\n.+)\r\n"? (No Gate, RE only) gate=yes re=no
expect: does " \r\n" (spawn_id exp4) match regular expression "df[^\n]+\n.+\n(.+\r\n.+)\r\n"? (No Gate, RE only) gate=yes re=no
Last login: Fri Dec 2 23:58:09 2022 from 10.10.10.1
Welcome to server image 2.2
expect: does " \r\nLast login: Fri Dec 2 23:58:09 2022 from 10.10.10.1\r\r\n\r\nWelcome to server image 2.2\r\n\r\n" (spawn_id exp4) match regular expression "df[^\n]+\n.+\n(.+\r\n.+)\r\n"? (No Gate, RE only) gate=yes re=no
REMY_SERVER:~ #
expect: does " \r\nLast login: Fri Dec 2 23:58:09 2022 from 10.10.10.1\r\r\n\r\nWelcome to server image 2.2\r\n\r\n\u001b[?1034h\u001b[1m\u001b[31mREMY_SERVER:~ # \u001b(B\u001b[m" (spawn_id exp4) match regular expression "df[^\n]+\n.+\n(.+\r\n.+)\r\n"? (No Gate, RE only) gate=yes re=no
expect: timed out
interact: received eof from spawn_id exp0
=====================
>>
Expected:
What I ultimately want is to get the output of df -kh into a local variable or even better, append it directly to a local file (on the local machine, not the server on which the command is executed) so that it contains something like:
$ cat ./result.txt
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 20G 18G 1,7G 92% /
Method 1: The proper way is to not use expect and use key pair access :
Step #1
Setup a SSH key pair (google it) and then copy the SSH key to the remote server. To do this I'd recommend using ssh-copy-id.
Step #2
Now with the ability to SSH to a server in place using a key, your above problem turns into this:
$ ssh root#10.10.10.10 "df -kh"
You can get fancy and use here documents (heredocs aka. here-docs) to further enhance this technique.
$ ssh root#10.10.10.10 <<EOF
> df -kh
> EOF
or put the commands in a file and pass them to ssh:
$ ssh root#10.10.10.10 < my.cmds
Method 2: Expect
See the following, expains how to use it properly and a tool to create expect scripts
https://hostadvice.com/how-to/how-to-automate-tasks-in-ssh/
First, your PROMPT regex is not matching. I see the output has some colour codes in it:
expect: does " \r\nLast login: ...REMY_SERVER:~ # \u001b(B\u001b[m" (spawn_id exp4) match regular expression ...
It's good to anchor prompt regexes, and to enclose them in braces. Try
set PROMPT { # \S*$}
Or, assuming the login shell is bash, set a new prompt that's easier to match:
send "$my_pass\r"
expect "Welcome to server"
send -- "PS1='>'\r"
set PROMPT {>$}
expect -re $PROMPT
Next, the relevant code for the question.
send -- "df -kh /\r"
expect -re "df\[^\n]+\n.+\n(.+\r\n.+)\r\n"
set command_output $expect_out(1,string)
send_user "$command_output\r"
I'd adjust your regex a touch:
set cmd "df -kh /"
send -- "$cmd\r"
expect -re "$cmd\r\n(.+)\r\n.*$PROMPT"
Then you're capturing and "echoing" the result correctly
set command_output $expect_out(1,string)
send_user "$command_output\n"
# use a newline here ......^
And to append it to a local file:
set fh [open ./results.txt a]
puts $fh $command_output
close $fh
I'm trying to write a shell function that spawns a ssh process and authentificates with a password. Then, I'd like to use the spawned process to do further stuff with expect.
Here's the function I have so far:
ssh_cmd() {
ssh $USER#$HOST 2>&1 | expect -c "
log_user 0
set timeout 5
expect password: {
send \"$PASS\n\"
sleep 2
log_user 1
}
"
}
And then I'd like to use given function in other places to interact with the ssh process like this:
ssh_cmd | expect -c "
expect '#' {
send pwd\n
send exit\n
}
expect eof
"
However, running the ssh_cmd function with -d option for expect I get the following result:
expect version 5.45.4
expect: does "" (spawn_id exp0) match glob pattern "password:"? no
ubnt#ui1's password: expect: timed out
From what I understand, the output of ssh does not get piped correctly. I know the common way to do this would be to use spawn, but that would mean the process would get killed after expect exits and I could not have a generic function that authentificates ssh sessions and keeps the process alive for further usage.
What you're designing won't work. Expect needs the process to be spawned from within the expect interpreter using the spawn command. Passing the command's stdout into expect is insufficient.
You could try this:
ssh_cmd() {
# a default bit of code if user does not provide one.
# you probably want some checking and emit an error message.
local user_code=${1:-set timeout 1; send "exit\r"; expect eof}
expect -c "
log_user 0
set timeout 5
spawn ssh -l $USER $HOST
expect password: {
send \"$PASS\n\"
sleep 2
log_user 1
}
$user_code
"
}
and then invoke it like:
ssh_cmd '
expect "#" {
send pwd\r
send exit\r
}
expect eof
'
Note that single quotes have no special meaning in expect, they are just plain characters: you probably don't want to expect the prompt to be the 3 character pattern '#'
I am trying to call a shell script and store the result in an expect variable. get_pw.sh accepts 2 args and decrypts the file using the provided md5hash. If I execute ./get_pw.sh file.test md5hash from the bash prompt it returns the password string as expected. When called from expect, the password does not get returned. The expect debug shows:
expect: does "" (spawn_id exp0) match regular expression "[^\s]"?
So it looks like the script is not returning the password string when called from expect. Relevant code:
#!/usr/bin/expect
send "./get_pw.sh file.test md5hash \r"
expect -re {[^\s]} {
set password $expect_out(0,string)
}
puts "The password is: $password"
You need to spawn a command first before you can send input and expect output from it.
To set an expect variable to the output of a command, use
set varname [exec command]
If you must do this with expect,
log_user 0
spawn -noecho get_pw.sh file hash
expect eof
set passwd [string trimright $expect_out(buffer) "\r\n"]
puts $passwd
Jens's answer looks pretty good by now ...
I have a bash+expect script which has to connect via ssh to the remote comp (and i can't use ssh keys, need password identification in here), read the file there, find specific line with the "hostname" (like "hostname aaaa1111") and store this hostname into the variable to be used after while. How can i get the value of the "hostname" parameter? I thought that line content will be in $expect_out(buffer) variable (so i can scan it and analyze), but it's not. My script is:
#!/bin/bash
----bash part----
/usr/bin/expect << ENDOFEXPECT
spawn bash -c "ssh root#$IP"
expect "password:"
send "xxxx\r"
expect ":~#"
send "cat /etc/rc.d/rc.local |grep hostname \r"
expect ":~#"
set line $expect_out(buffer)
puts "line = $line, expect_out(buffer) = $expect_out(buffer)"
...more script...
ENDOFEXPECT
When i try to see line variable, i see only this: line = , expect_out(buffer) = (buffer) What is the right way to get the line from the file into the variable?
Or is it possible to open the file on the remote computer with expect, scan the file and get what i need to the variable?
Here http://en.wikipedia.org/wiki/Expect there is an example:
# Send the prebuilt command, and then wait for another shell prompt.
send "$my_command\r"
expect "%"
# Capture the results of the command into a variable. This can be displayed,
set results $expect_out(buffer)
seems that it doesn't work in this case?
You might just want to try and do it all from expect, as expect can control bash.
The following should do what you've described. Not sure if this is exactly what you are trying to do.
#!/bin/sh
# the next line restarts using tclsh \
exec expect "$0" "$#"
spawn bash
send "ssh root#$IP\r"
expect "password:"
send "xxxx\r"
expect ":~#"
send "cat /etc/rc.d/rc.local |grep hostname \n"
expect ":~#"
set extractedOutput $expect_out(buffer)
set list [split $extractedOutput "\n"]
foreach line $list {
set re {(?x)
.*
(*)
-S.*
}
regexp $re $line total extractedValue
if {[info exists extractedValue] && [string length $extractedValue] > 1} {
set exportValue $extractedValue
break # We've got a match!
}
send "exit\r" # disconnect from the ssh session
if {[info exists exportValue] && [string length $exportValue] > 1}{
send "export VARIABLE $exportValue\r"
} else {
send_user "No exportValue was found - exiting\n"
send "exit\r"
close
exit 1
}
# now you can do more things in bash if you like