I am trying to run a local function remotely on a machine and retrieve the values of multiple variables of this function in a local function.
The first line of the heredoc section enables me to run the function remotely by defining it on the remote machine.
With the local machine named localmach and the remote machine named remotemach
#! /bin/bash
arr=()
scanHost()
{
arr+=("$(hostname)")
tmpResult=$(hostname)
}
scanHost
ssh user#remotemach "bash -s --" <<EOF
$(typeset -f scanHost)
tmpResult=$tmpResult
scanHost
EOF
echo "Local array -> ${arr[#]}"
echo "Local echo -> $tmpResult"
The snippet above returns
Local array -> localmach
Local echo -> localmach
But what I need is
Local array -> localmach remotemach
Local echo -> remotemach
In words, I need the value of the remote tmpResult AND the array arr stored locally.
Addendum :
Here I make the command hostname as an example, but in reality I am "scanning" a remote host and generate a JSON string that I store in the tmpResult variable. If I encounter problems during the scan, I append a string explaining the issue in arr. This way in my final json I can list all the problems encountered during the scan of multiple hosts.
You need to collect the result of ssh command :
#! /bin/bash
scanHost()
{
tmpResult=$(hostname)
}
scanHost
tmpResult=$(ssh user#remotemach "bash -s" <<EOF
$(typeset -f scanHost)
scanHost
echo "\$tmpResult"
EOF
)
echo "Local echo -> $tmpResult"
New Version
#! /bin/bash
arr=()
scanHost()
{
arr+=("$(hostname)")
tmpResult=$(hostname)
}
scanHost
eval "$(ssh user#remotemach "bash -s" <<EOF
$(typeset -f scanHost)
$(declare -p arr)
scanHost
declare -p arr
declare -p tmpResult
EOF
)"
echo "Local array -> ${arr[#]}"
echo "Local echo -> $tmpResult"
Use eval with caution due to side effects.
Related
I've been working through creating a script to move some files from a local machine to a remote server. As part of that process I have a function that can either be called directly or wrapped with 'declare -fp' and sent along to an ssh command. The code I have so far looks like this:
export REMOTE_HOST=myserver
export TMP=eyerep-files
doTest()
{
echo "Test moving files from $TMP with arg $1"
declare -A files=(["abc"]="123" ["xyz"]="789")
echo "Files: ${!files[#]}"
for key in "${!files[#]}"
do
echo "$key => ${files[$key]}"
done
}
moveTest()
{
echo "attempting move with wrapped function"
ssh -t "$REMOTE_HOST" "$(declare -fp doTest|envsubst); doTest ${1#Q}"
}
moveTest $2
If I run the script with something like
./myscript.sh test dev
I get the output
attempting move with wrapped function
Test moving files from eyerep-files with arg dev
Files: abc xyz
bash: line 7: => ${files[]}: bad substitution
It seems like the string expansion for the for loop is not working correctly. Is this expected behaviour? If so, is there an alternative way to loop through an array that would avoid this issue?
If you're confident that your remote account's default shell is bash, this might look like:
moveTest() {
ssh -t "$REMOTE_HOST" "$(declare -f doTest; declare -p $(compgen -e)); doTest ${1#Q}"
}
If you aren't, it might instead be:
moveTest() {
ssh -t "$REMOTE_HOST" 'exec bash -s' <<EOF
set -- ${##Q}
$(declare -f doTest; declare -p $(compgen -e))
doTest \"\$#\"
EOF
}
I managed to find an answer here: https://unix.stackexchange.com/questions/294378/replacing-only-specific-variables-with-envsubst/294400
Since I'm exporting the global variables, I can get a list of them using compgen and use that list with envsubst to specify which variables I want to replace. My finished function ended up looking like:
moveTest()
{
echo "attempting move with wrapped function"
ssh -t "$REMOTE_HOST" "$(declare -fp doTest|envsubst "$(compgen -e | awk '$0="${"$0"}"') '${1}'"); doTest ${1#Q}"
}
I have a file in which I have given all the IP addresses. The file looks like following:
[asad.javed#tarts16 ~]#cat file.txt
10.171.0.201
10.171.0.202
10.171.0.203
10.171.0.204
10.171.0.205
10.171.0.206
10.171.0.207
10.171.0.208
I have been trying to loop over the IP addresses by doing the following:
launch_sipp () {
readarray -t sipps < file.txt
for i in "${!sipps[#]}";do
ip1=(${sipps[i]})
echo $ip1
sip=(${i[#]})
echo $sip
done
But when I try to access the array I get only the last IP address which is 10.171.0.208. This is how I am trying to access in the same function launch_sipp():
local sipp=$1
echo $sipp
Ip=(${ip1[*]})
echo $Ip
Currently I have IP addresses in the same script and I have other functions that are using those IPs:
launch_tarts () {
local tart=$1
local ip=${ip[tart]}
echo " ---- Launching Tart $1 ---- "
sshpass -p "tart123" ssh -Y -X -L 5900:$ip:5901 tarts#$ip <<EOF1
export DISPLAY=:1
gnome-terminal -e "bash -c \"pwd; cd /home/tarts; pwd; ./launch_tarts.sh exec bash\""
exit
EOF1
}
kill_tarts () {
local tart=$1
local ip=${ip[tart]}
echo " ---- Killing Tart $1 ---- "
sshpass -p "tart123" ssh -tt -o StrictHostKeyChecking=no tarts#$ip <<EOF1
. ./tartsenvironfile.8.1.1.0
nohup yes | kill_tarts mcgdrv &
nohup yes | kill_tarts server &
pkill -f traf
pkill -f terminal-server
exit
EOF1
}
ip[1]=10.171.0.10
ip[2]=10.171.0.11
ip[3]=10.171.0.12
ip[4]=10.171.0.13
ip[5]=10.171.0.14
case $1 in
kill) function=kill_tarts;;
launch) function=launch_tarts;;
*) exit 1;;
esac
shift
for ((tart=1; tart<=$1; tart++)); do
($function $tart) &
ips=(${ip[tart]})
tarts+=(${tart[#]})
done
wait
How can I use different list of IPs for a function created for different purpose from a file?
How about using GNU parallel? It's an incredibly powerful wonderful-to-know very popular free linux tool, easy to install.
Firstly, here's a basic parallel tool usage ex.:
$ parallel echo {} :::: list_of_ips.txt
# The four colons function as file input syntax.†
10.171.0.202
10.171.0.201
10.171.0.203
10.171.0.204
10.171.0.205
10.171.0.206
10.171.0.207
10.171.0.208
†(Specific to parallel; see parallel usage cheatsheet here]).
But you can replace echo with just about any as complex series of commands as you can imagine / calls to other scripts. parallel loops through the input it receives and performs (in parallel) the same operation on each input.
More specific to your question, you could replace echo simply with a command call to your script
Now you would no longer need to handle any looping through ip's itself, and instead be written designed for just a single IP input. parallel will handle running the program in parallel (you can custom set the number of concurrent jobs with option -j n for any int 'n')＊ .
＊By default parallel sets the number of jobs to the number of vCPUs it automatically determines your machine has available.
$ parallel process_ip.sh :::: list_of_ips.txt
In pure Bash:
#!/bin/bash
while read ip; do
echo "$ip"
# ...
done < file.txt
Or in parallel:
#!/bin/bash
while read ip; do
(
sleep "0.$RANDOM" # random execution time
echo "$ip"
# ...
) &
done < file.txt
wait
Inside the remote server i have a condition statement.If that condition passes
status value should be set as success.
But here i am always getting Failure response while i print status variable
status='Success';
status='Success';
# !/bin/bash
declare -a server_PP=('XXXXX' 'YYYYYYY' );
declare -A results_map;
function process(){
serverList=$1[#];
servers=("${!serverList}");
status='Failure';
for serverName in "${servers[#]}"
do
ssh $serverName << EOF
if [ -f /app/Release/abc.war ]; then
echo "available - success"
status='Success';
fi
echo "***********status-inside******$status"
exit
EOF
echo "***********status-outside******$status"
results_map+=([$serverName]=$status);
done
}
process 'server_PP'
for i in "${!results_map[#]}"
do
echo "key :" $i
echo "value:" ${results_map[$i]}
done
Status variable should set as success when that condition get satisfied.
As written in pcarter's comment, the variables on both systems are independent from each other and don't get passed via ssh. Instead of setting a variable (or printing and reading the value as proposed in the comment, which is a working solution) you can use the exit code which gets passed automatically by ssh.
The following script is close to the original. For further improvements see below.
# !/bin/bash
declare -a server_PP=('XXXXX' 'YYYYYYY' );
declare -A results_map;
function process(){
serverList=$1[#];
servers=("${!serverList}");
status='Failure';
for serverName in "${servers[#]}"
do
if ssh $serverName << EOF
if [ -f /app/Release/abc.war ]; then
echo "available - success"
exit 0;
fi
echo "error"
exit 1
EOF
then
status='Success'
else
status='Failure'
fi
echo "***********status-outside******$status"
results_map+=([$serverName]=$status);
done
}
process 'server_PP'
for i in "${!results_map[#]}"
do
echo "key :" $i
echo "value:" ${results_map[$i]}
done
As you no longer need the variable assignments you can even omit the if ... and exit in the remote commands.
if ssh $serverName << EOF
[ -f /app/Release/abc.war ]
EOF
then
...
Your approach of using a heredoc as
ssh hostname <<EOF
# commands ...
EOF
has the disadvantage that you run an interactive shell on the remote system, which may print some system information or welcome message before executing your commands. You can further simplify the script (and removing the welcome message) by specifying the command or a script as command line arguments for ssh.
if ssh $serverName [ -f /app/Release/abc.war ]
then
...
If your command sequence is longer you can create a script on the remote system and run this script in the same way as ssh hostname scriptname. You could also create the script on the remote system using ssh or scp.
I want to create a function locally, echo_a in the example, and pass it with to a remote shell through ssh, here with typeset -f. The problem is that function does not have access to the local variables.
export a=1
echo_a() {
echo a: $a
}
bash <<EOF
$(typeset -f echo_a)
echo local heredoc:
echo_a
echo
echo local raw heredoc:
echo a: $a
echo
EOF
ssh localhost bash <<EOF
$(typeset -f echo_a)
echo remote heredoc:
echo_a
echo
echo remote raw heredoc:
echo a: $a
echo
EOF
Assuming the ssh connection is automatic, running the above script gives me as output:
local heredoc:
a: 1
local raw heredoc:
a: 1
remote heredoc:
a:
remote raw heredoc:
a: 1
See how the "remote heredoc" a is empty? What can I do to get 1 there?
I tested adding quotes and backslashes everywhere without success.
What am I missing? Would something else than typeset make this work?
Thanks to #Guy for the hint, it indeed is because ssh disables by default sending the environment variables. In my case, changing the server's setting was not wanted.
Hopefully we can hack around by using compgen, eval and declare.
First we identify added variables generically. Works if variables are created inside a called function too. Using compgen is neat because we don't need to export variables explicitely.
The array diff code comes from https://stackoverflow.com/a/2315459/1013628 and the compgen trick from https://stackoverflow.com/a/16337687/1013628.
# Store in env_before all variables created at this point
IFS=$'\n' read -rd '' -a env_before <<<"$(compgen -v)"
a=1
# Store in env_after all variables created at this point
IFS=$'\n' read -rd '' -a env_after <<<"$(compgen -v)"
# Store in env_added the diff betwen env_after and env_before
env_added=()
for i in "${env_after[#]}"; do
skip=
for j in "${env_before[#]}"; do
[[ $i == $j ]] && { skip=1; break; }
done
if [[ $i == "env_before" || $i == "PIPESTATUS" ]]; then
skip=1
fi
[[ -n $skip ]] || env_added+=("$i")
done
echo_a() {
echo a: $a
}
env_added holds now an array of all names of added variables between the two calls to compgen.
$ echo "${env_added[#]}"
a
I filter out also the variables env_before and PIPESTATUS as they are added automatically by bash.
Then, inside the heredocs, we add eval $(declare -p "${env_added[#]}").
declare -p VAR [VAR ...] prints, for each VAR, the variable name followed by = followed by its value:
$ a = 1
$ b = 2
$ declare -p a b
declare -- a=1
declare -- b=2
And the eval is to actually evaluate the declare lines. The rest of the code looks like:
bash <<EOF
# Eval the variables computed earlier
eval $(declare -p "${env_added[#]}")
$(typeset -f echo_a)
echo local heredoc:
echo_a
echo
echo local raw heredoc:
echo a: $a
echo
EOF
ssh rpi_301 bash <<EOF
# Eval the variables computed earlier
eval $(declare -p "${env_added[#]}")
$(typeset -f echo_a)
echo remote heredoc:
echo_a
echo
echo remote raw heredoc:
echo a: $a
echo
EOF
Finally, running the modified script gives me the wanted behavior:
local heredoc:
a: 1
local raw heredoc:
a: 1
remote heredoc:
a: 1
remote raw heredoc:
a: 1
I have a script on my local machine, but need to run it on a remote machine without copying it over there (IE, I can't sftp it over and just run it there)
I currently have the following functioning command
echo 'cd /place/to/execute' | cat - test.sh | ssh -T user#hostname
However, I also need to provide a commandline argument to test.sh.
I tried just adding it after the .sh, like I would for local execution, but that didn't work:
echo 'cd /place/to/execute' | cat - test.sh "arg" | ssh -T user#hostname
"cat: arg: No such file or directory" is the resulting error
You need to override the arguments:
echo 'set -- arg; cd /place/to/execute' | cat - test.sh | ssh -T user#hostname
The above will set the first argument to arg.
Generally:
set -- arg1 arg2 arg3
will overwrite the $1, $2, $3 in bash.
This will basically make the result of cat - test.sh a standalone script that doesn't need any arguments`.
Depends on the complexity of the script that you have. You might want to rewrite it to be able to use rpcsh functionality to remotely execute shell functions from your script.
Using https://gist.github.com/Shadowfen/2b510e51da6915adedfb saved into /usr/local/include/rpcsh.inc (for example) you could have a script
#!/bin/sh
source /usr/local/include/rpcsh.inc
MASTER_ARG=""
function ahelper() {
# used by doremotely just to show that we can
echo "master arg $1 was passed in"
}
function doremotely() {
# this executes on the remote host
ahelper $MASTER_ARG > ~/sample_rpcsh.txt
}
# main
MASTER_ARG="newvalue"
# send the function(s) and variable to the remote host and then execute it
rpcsh -u user -h host -f "ahelper doremotely" -v MASTER_ARG -r doremotely
This will give you a ~/sample_rpcsh.txt file on the remote host that contains
master arg newvalue was passed in
Copy of rpcsh.inc (in case link goes bad):
#!/bin/sh
# create an inclusion guard (to prevent multiple inclusion)
if [ ! -z "${RPCSH_GUARD+xxx}" ]; then
# already sourced
return 0
fi
RPCSH_GUARD=0
# rpcsh -- Runs a function on a remote host
# This function pushes out a given set of variables and functions to
# another host via ssh, then runs a given function with optional arguments.
# Usage:
# rpcsh -h remote_host -u remote_login -v "variable list" \
# -f "function list" -r mainfunc [-- param1 [param2]* ]
#
# The "function list" is a list of shell functions to push to the remote host
# (including the main function to run, and any functions that it calls).
#
# Use the "variable list" to send a group of variables to the remote host.
#
# Finally "mainfunc" is the name of the function (from "function list")
# to execute on the remote side. Any additional parameters specified (after
# the --)gets passed along to mainfunc.
#
# You may specify multiple -v "variable list" and -f "function list" options.
#
# Requires that you setup passwordless access to the remote system for the script
# that will be running this.
rpcsh() {
if ! args=("$(getopt -l "host:,user:,pushvars:,pushfuncs:,run:" -o "h:u:v:f:r:A" -- "$#")")
then
echo getopt failed
logger -t ngp "rpcsh: getopt failed"
exit 1
fi
sshvars=( -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null )
eval set -- "${args[#]}"
pushvars=""
pushfuncs=""
while [ -n "$1" ]
do
case $1 in
-h|--host) host=$2;
shift; shift;;
-u|--user) user=$2;
shift; shift;;
-v|--pushvars) pushvars="$pushvars $2";
shift; shift;;
-f|--pushfuncs) pushfuncs="$pushfuncs $2";
shift; shift;;
-r|--run) run=$2;
shift; shift;;
-A) sshvars=( "${sshvars[#]}" -A );
shift;;
-i) sshvars=( "${sshvars[#]}" -i $2 );
shift; shift;;
--) shift; break;;
esac
done
remote_args=( "$#" )
vars=$([ -z "$pushvars" ] || declare -p $pushvars 2>/dev/null)
ssh ${sshvars[#]} ${user}#${host} "
#set -x
$(declare -p remote_args )
$vars
$(declare -f $pushfuncs )
$run ${remote_args[#]}
"
}