This dockerfile has a spark path, retrieves the file name and installs the same. However, the script fails at the first echo command. It looks like the echo is being run as /bin/sh instead of /bin/sh -c.
How can I execute this echo command using /bin/sh -c? Is this the correct way to implement it, I'm planning on using the same logic for other installations such as Mongo, Node etc.
FROM ubuntu:18.04
ARG SPARK_FILE_LOCATION="http://www.us.apache.org/dist/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz"
CHAR_COUNT=`echo "${SPARK_FILE_LOCATION}" | awk -F"${DELIMITER}" '{print NF-1}'`
RUN echo $CHAR_COUNT
RUN CHAR_COUNT=`expr $CHAR_COUNT + 1`
RUN SPARK_FILE_NAME=`echo ${SPARK_FILE_LOCATION} | cut -f${CHAR_COUNT} -d"/"`
RUN Dir_name=`tar -tzf $SPARK_FILE_NAME | head -1 | cut -f1 -d"/"`
RUN echo Dir_name
/bin/sh: 1: 'echo http://www.us.apache.org/dist/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz | awk -F/ "{print NF-1}"': not found
Related
I have the below code to find out the number of instances of current script running that is running with same arg1. But looks like the script creates a subshell and executes this command which also shows up in output. What would be the better approach to find the number of instances of running script ?
$cat test.sh
#!/bin/bash
num_inst=`ps -ef | grep $0 | grep $1 | wc -l`
echo $num_inst
$ps aux | grep test.sh | grep arg1 | grep -v grep | wc -l
0
$./test.sh arg1 arg2
3
$
I am looking for a solution that matches all running instance of ./test.sh arg1 arg2 not the one with ./test.sh arg10 arg20
The reason this creates a subshell is that there's a pipeline inside the command substitution. If you run ps -ef alone in a command substitution, and then separately process the output from that, you can avoid this problem:
#!/bin/bash
all_processes=$(ps -ef)
num_inst=$(echo "$all_processes" | grep "$0" | grep -c "$1")
echo "$num_inst"
I also did a bit of cleanup on the script: double-quote all variable references to avoid weird parsing, used $() instead of backticks, and replaced grep ... | wc -l with grep -c.
You might also replace the echo "$all_processes" | ... with ... <<<"$all_processes" and maybe the two greps with a single grep -c "$0 $1":
...
num_inst=$(grep -c "$0 $1" <<<"$all_processes")
...
Modify your script like this:
#!/bin/bash
ps -ef | grep $0 | wc -l
No need to store the value in a variable, the result is printed to standard out anyway.
Now why do you get 3?
When you run a command within back ticks (fyi you should use syntax num_inst=$( COMMAND ) and not back ticks), it creates a new sub-shell to run COMMAND, then assigns the stdout text to the variable. So if you remove the use of $(), you will get your expected value of 2.
To convince yourself of that, remove the | wc -l, you will see that num_inst has 3 processes, not 2. The third one exists only for the execution of COMMAND.
I am trying to write a bash script which simply acts as an emulator. It takes input from the user and executes the command while forwarding the command along with the result onto a file. I am unable to handle inputs which have either a | or a > in them.
The only option I could find was segregating the commands based on the | into an array and run them individually. However, this does not allow > redirects.
Thanking in advance.
$cmd is a command taken as input from the user
I used the command
$cmd 2>&1 | tee -a $flname
but this does not work if there is a | or a > in $cmd
/bin/bash -c "$cmd 2>&1 | tee -a $flname" does not run/store the command either
Try this:
#!/bin/bash
read -r -p "Insert command to execute"$'\n' cmd
echo "Executing '$cmd'"
/bin/bash -c "$cmd"
# or eval "$cmd"
Example of execution:
$ ./script.sh
Insert command to execute
printf '1\n2\n3\n4\n' | grep '1\|3'
Executing 'printf '1\n2\n3\n4\n' | grep '1\|3''
1
3
Hi I want grep all aliases and run their "body"
alias | grep NAME
it's return:
alias NAME123='ping google123.com'
alias NAME321='ping google321.com'
...
I want to run:
ping google123.com
ping google321.com
but not
NAME123
NAME321
All my tries ended syntax mistake.
write_aliases_script.sh
#!/bin/bash
printf "#!/bin/bash\n\n" 1>script.sh;
alias | grep NAME |
grep -o "'\(.\+\)'" |
sed "s/'//g" 1>>script.sh;
chmod +x script.sh;
# uncomment after testing
#./script.sh;
output
$ ./write_aliases_script.sh
$ cat script.sh
#!/bin/bash
ping google123.com
ping google321.com
Using eval you can make it run but be careful to not to invoke risky aliases:
eval $(alias | awk -F "'" '/NAME/{print $2}')
This question already has answers here:
Execute a command on remote hosts via ssh from inside a bash script
(4 answers)
Closed 7 years ago.
I need a help with a bash script that connect to server as root, execute some commands and then exit from the server.
I tried this script but when login login to server performed the command not running !
#!/bin/bash
sudo ssh -o ConnectTimeout=10 $1 'exit'
if [ $? != 0 ]; then
echo "Could not connect to $1 , script stopped"
exit
fi
sudo ssh $1
echo "SRV=`cat /etc/puppet/puppet.conf | grep -i srv_domain | awk '{print $3}'`"
echo $SRV
echo "puppetMaster=`host -t srv _x-puppet._tcp.$SRV | head -1 | awk '{print $8}' | cut -f1 -d"."`"
echo $puppetMaster
'exit'
I'm surprised nobody has suggested a heredoc yet.
sudo ssh "$1" <<'EOF'
SRV=`cat /etc/puppet/puppet.conf | grep -i srv_domain | awk '{print $3}'`
echo $SRV
echo "puppetMaster=`host -t srv _x-puppet._tcp.$SRV | head -1 | awk '{print $8}' | cut -f1 -d"."`"
echo $puppetMaster
EOF
This feeds everything from the <<'EOF' until the line starting with EOF into the stdin of ssh, to be received and run by the remote shell.
The commands following ssh machine in a script are not run on the machine. They will be run on the local machine once the ssh exits.
Either specify the commands to run as an argument of ssh, or alternatively, run ssh and make it read the commands from standard input, and send the commands to it.
ssh machine ls
# or
echo ls | ssh machine
You seem to be a little confused as to what runs where.
ssh -o ConnectTimeout=10 $1 'exit'
will connect to $1, run exit, and disconnect.
ssh -o ConnectTimeout=10 $1 'echo hello world'
will print hello world on
the server and then disconnect.
ssh $1
will open up a shell on the remote. After the shell has ended, the following commands will run locally.
echo "SRV=`cat /etc/puppet/puppet.conf | grep -i srv_domain | awk '{print $3}'`"
echo $SRV
echo "puppetMaster=`host -t srv _x-puppet._tcp.$SRV | head -1 | awk '{print $8}' | cut -f1 -d"."`"
echo $puppetMaster
'exit'
What you probably want is start bash on the remote and forward to it the commands you want to give it via stdin.
echo "my commands" | ssh $1 bash
Technically, you don't need that bash -- ssh will start bash even without it (but with different rc files).
i need to run hadoop command in bash script, which go through bunch of folders on amazon S3, then write those folder names into a txt file, then do further process. but the problem is when i ran the script, seems no folder names were written to txt file. i wonder if it's the hadoop command took too long to run and the bash script didn't wait until it finished and go ahead to do further process, if so how i can make bash wait until the hadoop command finished then go do other process?
here is my code, i tried both way, neither works:
1.
listCmd="hadoop fs -ls s3n://$AWS_ACCESS_KEY:$AWS_SECRET_KEY#$S3_BUCKET/*/*/$mydate | grep s3n | awk -F' ' '{print $6}' | cut -f 4- -d / > $FILE_NAME"
echo -e "listing... $listCmd\n"
eval $listCmd
...other process ...
2.
echo -e "list the folders we want to copy into a file"
hadoop fs -ls s3n://$AWS_ACCESS_KEY:$AWS_SECRET_KEY#$S3_BUCKET/*/*/$mydate | grep s3n | awk -F' ' '{print $6}' | cut -f 4- -d / > $FILE_NAME
... other process ....
any one knows what might be wrong? and is it better to use the eval function or just use the second way to run hadoop command directly
thanks.
I would prefer to eval in this case, prettier to append the next command to this one. and I would rather break down listCmd into parts, so that you know there is nothing wrong at the grep, awk or cut level.
listCmd="hadoop fs -ls s3n://$AWS_ACCESS_KEY:$AWS_SECRET_KEY#$S3_BUCKET/*/*/$mydate > $raw_File"
gcmd="cat $raw_File | grep s3n | awk -F' ' '{print $6}' | cut -f 4- -d / > $FILE_NAME"
echo "Running $listCmd and other commands after that"
otherCmd="cat $FILE_NAME"
eval "$listCmd";
echo $? # This will print the exit status of the $listCmd
eval "$gcmd" && echo "Finished Listing" && eval "$otherCmd"
otherCmd will only be executed if $gcmd succeeds. If you have too many commands that you need to execute, then this becomes a bit ugly. If you roughly know how long it will take, you can insert a sleep command.
eval "$listCmd"
sleep 1800 # This will sleep 1800 seconds
eval "$otherCmd"