Bash: how to send my own custom function through ssh - bash

Question

My goal is to do the following:
1) Check how much memory is being used by each GPU on a specific server. I accomplish this with (nvidia-smi --query-gpu=memory.free --format=csv).
2) Find the GPU with the maximum free memory. I accomplish this with my_cmd(). It works for the remote server I am currently logged into.
3) If the maximum free memory on the remote server I'm logged into is less than 1000 MiB, SSH into each other GPU server in the cluster to find the maximum free memory available. These servers are labelled according to to_check.
My current issue:
The code below works when scriptuse is given the cd command, etc.
The code below fails when scriptuse is given mycmd. It gives me the error:
bash: my_cmd: command not found.
Now, I think there's more than one problem here. First, I think I'm not providing my_cmd properly to the ssh command. Second, when I use my_cmd, I don't think I'm successfully sshing into the other servers.
Can anyone point out what is wrong and how to fix it?
The complete bash script is below.
#/bin/bash
#https://stackoverflow.com/questions/45313313/nvidia-smi-command-in-bash-vs-in-terminal-for-maximum-of-an-array/45313404#45313404
my_cmd()
{
max_idx=0
max_mem=0
idx=0
{
read _; # discard first line (header)
while read -r mem _; do # for each subsequent line, read first word into mem
if (( mem > max_mem )); then # compare against maximum mem value seen
max_mem=$mem # ...if greater, then update both that max value
max_idx=$idx # ...and our stored index value.
fi
((++idx))
done
} < <(nvidia-smi --query-gpu=memory.free --format=csv)
echo "Maximum memory seen is $max_mem, at processor $idx"
}
tocheck=('4' '5' '6' '7' '8') #The GPUs to check
it1=1
#scriptuse="my_cmd"
scriptuse= "cd ~/spatial; pwd; echo $gpuval"
while [ $it1 -lt ${#tocheck[#]} ] ; do #While we stil don't have enough free memory
echo $it1
gpuval=${tocheck[$it1]}
ssh gpu${gpuval} "${scriptuse}"
it1=$[it1+1]
done
EDIT
Thank you very much for the help, but my problem is not yet solved. I have done this:
1) Remove my_cmd from my bash script. It now looks like this:
#/bin/bash
#https://stackoverflow.com/questions/45313313/nvidia-smi-command-in-bash-vs-in-terminal-for-maximum-of-an-array/45313404#45313404
tocheck=('4' '5' '6' '7' '8') #The GPUs to check
it1=1
scriptuse= "cd ~/spatial; echo $gpuval"
while [ $it1 -lt ${#tocheck[#]} ] ; do #While we stil don't have enough free memory
echo $it1
gpuval=${tocheck[$it1]}
ssh gpu${gpuval} "${scriptuse}" /my_script.sh
it1=$[it1+1]
done
2) Create a separate bash script called my_script.sh that contains my_cmd:
#/bin/bash
#https://stackoverflow.com/questions/45313313/nvidia-smi-command-in-bash-vs-in-terminal-for-maximum-of-an-array/45313404#45313404
max_idx=0
max_mem=0
idx=0
{
read _; # discard first line (header)
while read -r mem _; do # for each subsequent line, read first word into mem
if (( mem > max_mem )); then # compare against maximum mem value seen
max_mem=$mem # ...if greater, then update both that max value
max_idx=$idx # ...and our stored index value.
fi
((++idx))
done
} < <(nvidia-smi --query-gpu=memory.free --format=csv)
echo "Maximum memory seen is $max_mem, at processor $idx"
3) Ran chmod to ensure both files can be run.
4) Ensured both files exist on all GPUs in the cluster (they have a common storage).
5) Ran ./test_run, which is the bash script from step 1.
I get the error:
./test_run.sh: line 8: cd ~/spatial; echo : No such file or directory
1
bash: /my_script.sh: No such file or directory
2
bash: /my_script.sh: No such file or directory
3
bash: /my_script.sh: No such file or directory
4
bash: /my_script.sh: No such file or directory
EDIT: The final solution
Thanks to the accepted answer below and the discussion in the comments, here's what ended up working:
1) Leave my_script as it is in the previous edit.
2) The file test_run should look like this:
#/bin/bash
tocheck=('4' '5' '6' '7' '8') #The GPUs to check
it1=1
while [ $it1 -lt ${#tocheck[#]} ] ; do #While we still don't have enough free memory
echo $it1
gpuval=${tocheck[$it1]}
ssh gpu${gpuval} ~/spatial/my_script.sh
it1=$[it1+1]
done
I think the reason this works is that all of the GPUs on the cluster have a common storage, so they all have access to /user/spatial.

Answer 1

The environment your script is running in (your shell) is totally unrelated to the environment the remote host is running in (the remote shell). If you define a function my_cmd in your shell it will not be transmitted across the wire to the remote host's shell.
Try a simpler example:
$ foo() { echo foo; }
$ foo
foo
$ ssh remote-host foo
bash: foo: command not found
This simply isn't how SSH, Bash, and Linux/POSIX are designed. Now, ssh does update some parts of the remote environment (as detailed in man ssh), but this is limited to certain environment variables, not functions.
Notably, the remote shell might not even be the same type of shell as your (e.g. yours might be Bash, but the remote shell might be Zsh), so it's not possible generally to transmit shell functions across ssh.
A much simpler and more reliable option is to create a shell script (rather than a function) that you intend to be run on the remote shell, and ensure that script exists on the remote machine. For example:
# Copy the script to the remote host's /tmp directory
scp my_cmd.sh remote-host:/tmp
# Invoke the script on the remote host
$ ssh remote-host /tmp/my_cmd.sh
Edit:
./test_run.sh: line 8: cd ~/spatial; echo : No such file or directory
Are you sure ~/spatial exists on the remote host?
bash: /my_script.sh: No such file or directory
Are you sure /my_script.sh exists on the remote host?
Again, your remote host is a wholly different environment. Just because a file or directory exists on your local machine doesn't mean it exists on the remote host unless you put it there.
Try ssh [remote-host] 'ls ~' and ssh [remote-host] 'ls /' - I bet you'll see the directory and file don't exist.