Unexpected behavior in bash scripting on a ssh function - bash

I've built a Raspberry pi cluster, with Spark and Hadoop installed, and have made a few functions in .bashrc to make communication and interaction a little easier
function otherpis {
grep "pi" /etc/hosts | awk '{print $2}' | grep -v $(hostname)
}
function clustercmd {
for pi in $(otherpis); do ssh $pi "$#"; done
$#
}
Where otherpis simply looks at the host file where i've precompiled all other raspberry pis in the cluster with their static ip addresses. I've also configured ssh with authorized keys so I don't have to enter a password everytime I ssh in.
I can call commands like
$ clustercmd date
Thu 03 Oct 2019 02:00:13 PM CDT
Thu 03 Oct 2019 02:00:11 PM CDT
Thu 03 Oct 2019 02:00:12 PM CDT
......
or
$ clustercmd sudo mkdir -p /opt/hadoop_tmp/hdfs
and it works just fine. But for some reason when I try to pass anything into the command with hadoop or spark, it says it can't find the command except for the pi that I'm invoking the command from.
$ clustercmd hadoop version | grep Hadoop
bash: hadoop: command not found
bash: hadoop: command not found
.....
Hadoop 3.2.1
But when I manually ssh into a pi and call the command, it works just fine.
$ ssh pi2
pi#pi2: $ hadoop version | grep Hadoop
Hadoop 3.2.1
I've exported all proper paths in .bashrc. I've chown of all relevant directories on each pi. No matter what I try, just the spark and hadoop commands aren't registering. Everything else is. I've even have a function that will do a file copy across the entire cluster
function clusterscp {
for pi in $(otherpis); do
cat $1 | ssh $pi "sudo tee $1" > /dev/null 2>&1
done
}
I setup hadoop and spark on the first pi, and then mass transferred all files and configurations with the above function with no problems. Any insight would help
EDIT
Adding all exported paths in .bashrc
export JAVA_HOME=$(readlink –f /usr/bin/java | sed "s:bin/java::")
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin
export HADOOP_OPTS="-XX:-PrintWarnings –Djava.net.preferIPv4Stack=true"
export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_ROOT_LOGGER="WARN,DRFA"
Note that as I stated earlier, when i'm actually SSH'd into the pi, all exported paths work fine, it is only when I try to run a clustercmd command that hadoop and spark is not found
Solved
I fixed this by moving all exports above this line in the .bashrc
# If not running interactively, don't do anything
case $- in
*i*);;
*) return;;
esac
And I added it to .profile in the home directory. This was originally suggested by mangusta, he just added a word to the file ".bash_profile", when it should be just "profile"

~/.bashrc is executed when you have already logged in and want to open a new terminal window or execute a new shell instance.
If you login into machine (locally or remotely), what actually runs is ~/.bash_profile, not ~/.bashrc.
Try including you HADOOP_HOME and SPARK_HOME folders into PATH inside of .bash_profile on all your pi_N hosts

I think you can directly debug the problem reported by bash ("command not found") by running
echo $PATH
in the ssh shells on each RPi, and comparing the result with the result of
clustercmd 'echo $PATH'
Probably this will show that mangusta's answer is correct.

Related

How to load full and same shell from SSH as from manual login? [duplicate]

I have been trying to resolve problems to be able to run openmpi on multiple nodes.
Initially I had a problem with $PATH and $LD_LIBRARY_PATH variables not being updated from .bashrc file by openmpi session, so I manually added --prefix /path/to/openmpi to resolve this issue.
Turns out that even the anaconda path variables are not being loaded as well. So ultimately I need ~/.bashrc file to be sourced from my home directory. How can I do that? Can anyone help me out please?
UPDATE 01:
I wrote a simple shell script to check the version of python
python --version
and tried to run it with openmpi on local as well as remote machine as follows:
mpirun --prefix /home/usama/.openmpi --hostfile hosts -np 4 bash script
And it returns
Python 2.7.12
Python 3.6.8 :: Anaconda, Inc.
Python 3.6.8 :: Anaconda, Inc.
Python 2.7.12
Confirming my suspicion that whatever openmpi is doing to run remote processes doesn't invoke / set proper environment variables from the ~/.bashrc file. Any help from someone who has worked with multi-node openmpi?
UPDATE 02:
A simple ssh environment grep tell me that my environment variables are not updated which might be the cause of the problem. (I have even tried to set it up in ~/.ssh/environment file)
$ ssh remote-node env | grep -i path
It seems to be loading only the /etc/environment file with only basic paths setup. How to I rectify this?
maybe you should run like this.I guess.
two ways help you!
first:
mpirun --prefix /home/usama/.openmpi --hostfile hosts -np 4 . ~/.bashrc && bash script
second:
## 1. add this line to the script
. ~/.bashrc
## 2. run command as you do
mpirun --prefix /home/usama/.openmpi --hostfile hosts -np 4 bash script

Running code without ssh in beaglebone black

I want to run some code in Beaglebone black without doing ssh when I apply power.
I have tried putting some commands to run the code in ~/.bashrc file but it only works when I login using ssh. I have tried same thing with /etc/rc.local file but didn't work even after ssh.
I have also tried #reboot my_command in crontab -e but it also requires me to login using ssh
Any suggestions??
EDIT:
root#beaglebone:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 8.6 (jessie)
Release: 8.6
Codename: jessie
root#beaglebone:~# ps aux | grep cron | grep -v grep
root 295 0.0 0.3 4428 1988 ? Ss 15:03 0:00 /usr/sbin/cron -f
Output of crontab -e: last few lines
root#beaglebone:~# crontab -e
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
##reboot /root/wiringBone-master/library/main not working
#*/5 * * * * /root/wiringBone-master/library/main works
main is the script I want to run
/etc/rc.local is a quick way. Make sure to launch into background and don't prevent the script from finishing.
Writing a proper systemd service file would be better though.
crontab -e method worked!!. My script required two overlays to execute the code which I didn't load that's why my #reboot command didn't work. I solved my problem by adding required overlays.
#reboot config-pin overlay cape-universaln
#reboot config-pin overlay BB-ADC
#reboot /root/wiringBone-master/library/main
And now my code works on reboot.
I don't know anything about beagle bone, but on a normal Linux system you'd likely do this with either an init script, or, more easily, in a cron script set to run at boot.
You'll have to check if you're environment would support either of those. Even if it doesn't have cron, it is probably running some sort of init (likely to be the thing starting SSH on boot, but YMMV).

Eval in docker-machine: terminal vs shell script

I'm trying to run a simple shell script to automate changing docker-machine environments. The problem is this, when I run the following command directly in the Mac terminal the following is outputted:
eval $(docker-machine env default)
docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
default * digitalocean Running tcp://***.**.***.***:**** v1.12.0
So basically what you would expect, however when I run the following .sh script:
#!/usr/bin/env bash
eval $(docker-machine env default)
The output is:
./run.sh
docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
default digitalocean Running tcp://***.**.***.***:**** v1.12.0
So basically, it is not setting it as active and I cannot access it.
Has anyone run into this issue before and knows how to solve it? Seems really strange to me, have got pretty much everything else running and automated apart from this facet.
Cheers, Aaron
I think you need to source your shell script
source ./myscript.sh
as the exports in the eval are being returned to the process you started to run the shell in and then being disposed of. These need to go to the parent e.g. login shell
Consider a.sh
#!/bin/bash
eval $(echo 'export a=123')
export b=234
when run in two ways
$ ./a.sh
$ echo $a
$ echo $b
$ source a.sh
$ echo $a
123
$ echo $b
234
$

Archlinux + MATE Terminal - `.bash_profile` is not being sourced

I am using Arch Linux with MATE as desktop environment. So terminal emulator is MATE Terminal. Recently I installed Jekyll with gem install jekyll. But when I ran jekyll -v it says bash: jekyll: command not found. So I tried to add path of Jekyll to PATH variable.
I ran PATH=$PATH/$HOME/.gem/ruby/2.2.0/bin and it worked perfectly. Now I can run jekyll commands. To add it permanently to PATH variable I edited the ~/.bash_profile file like following. It is not working after reboot. But
source ~/.bash_profile works perfectly.
#
# ~/.bash_profile
#
[[ -f ~/.bashrc ]] && . ~/.bashrc
export PATH="${PATH}:/home/heisenberg/.gem/ruby/2.2.0/bin"
According to ArchWiki this is the proper way to concat something permanantly to PATH. But it isn't working. Can somebody figure me out where the wrong is?
[N. B. : Adding the same line in ~/.bashrc is doing okay.]
Depending on the option it is given, bash can be run as an interactive shell or login shell. The default interactive shell mode does not read ~/.bash_profile. login shell bash do.
See:
First, some setup:
% cat ~/.bashrc
…
export BASHRC="yes"
…
% cat ~/.bash_profile
…
export BASH_PROFILE="yes"
…
Now run a regular (interactive) bash:
% bash
[galaux#magenta ~]$ echo $BASHRC
yes
[galaux#magenta ~]$ echo $BASH_PROFILE
Notice we did not get yes with this last one.
Now with a login shell:
% bash --login
[galaux#magenta ~]$ echo $BASHRC
yes
[galaux#magenta ~]$ echo $BASH_PROFILE
yes
See paragraph INVOCATION from man bash.

How to adapt bin/hdfs for executing from outside $HADOOP_HOME/bin?

I'm trying to modify the hdfs script so that it still functions although not located in $HADOOP_HOME/bin anymore, but when I execute the modified hdfs I get:
hdfs: line 110: exec: org.apache.hadoop.fs.FsShell: not found
line 110 is:
exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$#"
I've highlighted the changes I made to the script:
bin=**"$HADOOP_HOME"/bin # was** `dirname "$0"`
bin=`cd "$bin"; pwd`
./**hdfs-config.sh # was .** "$bin"/hdfs-config.sh
-
$ hadoop version
Hadoop 0.20.3-SNAPSHOT
Subversion http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append -r 1041718
Compiled by hammer on Mon Dec 6 17:38:16 CET 2010
Why don't you simply put a second copy of Hadoop on the system and let it have a different value for HADOOP_HOME?

Resources