AWS_ACCESS_KEY_ID command not found when using shell to call aws cli passing environment var fail: - shell

When I use Environment Variable to call aws cli in work terminal directly it will success:
AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=xxx AWS_REGION=xxx aws elb describe-load-balancers --output json --debug
But if in shell, I first setup AWS_KEY_PAIR then execute the same command, it will returns to me AWS_ACCESS_KEY_ID=xxx: command not found
My shell script looks like this:
function test(){
AWS_KEY_PAIR="AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY} AWS_SECRET_ACCESS_KEY=${AWS_SECRET_KEY} AWS_REGION=${AWS_REGION}"
${AWS_KEY_PAIR} aws elb describe-load-balancers --output json --debug }
Could anyone tell me why it success if I run first command directly in work terminal but fail in shell script? Is the AWS_KEY_PAIR can't be setup like this? Thank you very much!

The problem lays in the way you modify the environment:
man bash : The environment for any simple command or function may be augmented temporarily by prefixing it with parameter assignments, as described above in PARAMETERS. These assignment statements affect only the environment seen by
that command.
This means that if a variable exists, you can have it temporarily modified for a single command. Eg.
$ cat test.sh
#!/usr/bin/env zsh
echo $FOO
$ FOO="hello world"
$ echo $FOO
hello world
$ FOO="hello universe" ./test.sh
hello universe
$ echo $FOO
hello world
There you see that during the execution of FOO="hello universe" ./test.sh the variable FOO is temporarily modified. This is exactly what you do during execution of your command on the command line.
In your script however, you attempt something different. You assign the assignment of your script to a variable and then try to "execute this".
$ BAR="FOO=hello"
$ echo $BAR
FOO=hello
$ $BAR
bash: FOO=hello: command not found...
As you see, it tries to execute the command FOO=hello which is not a command but actually a string you try to execute. It is similar to typing $ "FOO=hello". So you can now imagine that
$ $BAR ./test.sh
will also not execute.
There is an evil workaround here using eval, but eval is evil.
man bash : eval [arg ...]
The args are read and concatenated together into a single command. This command is then read and executed by the shell, and its exit status is returned as the value of eval. If there are no args, or only null arguments, eval returns 0.
$ BAR="FOO=hello"
$ echo $BAR
FOO=hello
$ eval $BAR
$ echo $FOO
hello
$ FOO="hello world"
$ eval $BAR ./test.sh
hello
The latter examples are exactly what you try to attempt in your function test(). You assign your variable declaration to the variable AWS_KEY_PAIR and then execute aws with the modified environment ${AWS_KEY_PAIR}, but this will not work as AWS_KEY_PAIR is nothing more then a long string. You can thus fix it by placing eval in front of it, or by typing the full key pair out as
function test(){
AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY} \
AWS_SECRET_ACCESS_KEY=${AWS_SECRET_KEY} \
AWS_REGION=${AWS_REGION} \
aws elb describe-load-balancers --output json --debug
}

Related

How do I pass multiple arguments to a shell script into `kubectl exec`?

Consider the following shell script, where POD is set to the name of a K8 pod.
kubectl exec -it $POD -c messenger -- bash -c "echo '$#'"
When I run this script with one argument, it works fine.
hq6:bot hqin$ ./Test.sh x
x
When I run it with two arguments, it blows up.
hq6:bot hqin$ ./Test.sh x y
y': -c: line 0: unexpected EOF while looking for matching `''
y': -c: line 1: syntax error: unexpected end of file
I suspect that something is wrong with how the arguments are passed.
How might I fix this so that arguments are expanded literally by my shell and then passed in as literals to the bash running in kubectl exec?
Note that removing the single quotes results in an output of x only.
Note also that I need the bash -c so I can eventually pass in file redirection: https://stackoverflow.com/a/49189635/391161.
I managed to work around this with the following solution:
kubectl exec -it $POD -c messenger -- bash -c "echo $*"
This appears to have the additional benefit that I can do internal redirects.
./Test.sh x y '> /tmp/X'
You're going to want something like this:
kubectl exec POD -c CONTAINER -- sh -c 'echo "$#"' -- "$#"
With this syntax, the command we're running inside the container is echo "$#". We then take the local value of "$#" and pass that as parameters to the remote shell, thus setting $# in the remote shell.
On my local system:
bash-5.0$ ./Test.sh hello
hello
bash-5.0$ ./Test.sh hello world
hello world

Dereference environment variable on parameter expansion in shell script

I am trying to dereference the value of environment variable using parameter expansion $#, but it doesn't seem to work.
I need to call a shell script with certain arguments. The list of arguments contain environment variables, and the environment variables are expected to be present where the shell script is to be executed. I do not know the list of commands before hand, so I am expanding those list of commands using $#. However, the script is not able to de-reference the value of environment variables.
A minimal setup which explains my problem can be done as below.
Dockerfile
FROM alpine:3.10
ENV MY_VAR=production
WORKDIR /app
COPY run.sh .
ENTRYPOINT [ "sh", "run.sh" ]
run.sh
#!/bin/sh
echo "Value of MY_VAR is" $MY_VAR
echo "Begin"
$#
echo "Done"
I can build the image using docker build . -t env-test. When I run it using docker run env-test:latest 'echo $MY_VAR', I get the below output.
Value of MY_VAR is production
Begin
$MY_VAR
Done
While the output that I am expecting is:
Value of MY_VAR is production
Begin
production
Done
SideNote: In actuality I am trying to run it using a compose file like below:
version: '3'
services:
run:
image: env-test:latest
command: echo $$MY_VAR
but it again gives me the similar result.
Expanding on the eval approach, here is a simple bash script that will use eval to evaluate a string as a sequence of bash commands:
#!/usr/bin/env bash
echo program args: $#
eval $#
but beware, eval comes with dangers:
https://medium.com/dot-debug/the-perils-of-bash-eval-cc5f9e309cae
First thing, $# it will just print the number of argument
#!/bin/sh
echo "Value of MY_VAR is" $MY_VAR
echo "Begin"
$#
echo "Done"
$# = stores all the arguments in a list of string
$* = stores all the arguments as a single string
$# = stores the number of arguments
What does $# mean in a shell script?
Second thing, When run the below command
docker run env-test:latest 'echo $MY_VAR'
It will look for $MY_VAR in host system not in the container.
To access container env you have to pass them as -e MY_VAR=test ,not as argument to docker run command which will in the ENV in host.
docker run -e MY_VAR=test env-test:latest
So the value of MY_VAR will be test not production.
To debug the argument to docker run
export MY_VAR2="value_from_host"
Now run
docker run env-test:latest "echo $MY_VAR2"
so the value will value_from_host because thes argument pick from host.
There are a more ways to skin this particular cat:
me#computer:~$ docker run -it --rm ubuntu:20.04 bash -c 'echo $HOSTNAME'
e610946f50c1
Here, we're calling on bash inside the container to process everything inside the single quotes, so variable substitution and/or expansion isn't applied by your shell, but by the shell inside the container.
Another approach is:
me#computer:~$ cat test.sh
#!/bin/bash
echo $HOSTNAME
me#computer:~$ cat test.sh | docker run -i --rm ubuntu:20.04 bash
62ba950a60fe
In this case, cat is "pushing" the contents of the script to bash the container, so it's functionally equivalent to my first example. The first method is "cleaner", however if you've got a more complex script, multi-line variables or other stuff that's difficult to put into a single command, then the second method is a better choice.
Note: The hostname is different in each example, because I'm using the --rm option which discards the container once it exits. This is great when you want to run a command in a container but don't need to keep the container afterwards.

How to do named command line arguments in Bash Scripting better way?

This is my sample Bash Script example.sh:
#!/bin/bash
# Reading arguments and mapping to respective variables
while [ $# -gt 0 ]; do
if [[ $1 == *"--"* ]]; then
v="${1/--/}"
declare $v
fi
shift
done
# Printing command line arguments through the mapped variables
echo ${arg1}
echo ${arg2}
Now if in terminal I run the bash script as follows:
$ bash ./example.sh "--arg1=value1" "--arg2=value2"
I get the correct output like:
value1
value2
Perfect! Meaning I was able to use the values passed to the arguments --arg1 and --arg2 using the variables ${arg1} and ${arg2} inside the bash script.
I am happy with this solution for now as it serves my purpose, but, anyone can suggest any better solution to use named command line arguments in bash scripts?
You can just use environment variables:
#!/bin/bash
echo "$arg1"
echo "$arg2"
No parsing needed. From the command line:
$ arg1=foo arg2=bar ./example.sh
foo
bar
There's even a shell option to let you put the assignments anywhere, not just before the command:
$ set -k
$ ./example.sh arg1=hello arg2=world
hello
world

Passing environment variables over ssh for remote expansion

I am trying to pass an environment variable to my shell script, and then echo out what I passed in. In this example $HOME is an environment variable already set.
./my_script.ksh $HOME
#! /usr/bin/ksh
my_var=${1}
echo "You sent: ${my_var}"
I want my output to be You sent: $HOME but instead it is evaluating it and is producing You Sent: /home/blah/usr
The point of this is that I plan on doing an SSH command that I need to pass $my_var too un-evaluated (aka be $HOME) because $HOME across each server is a different value and for the SSH command I want it to use the value from the server it is SSH'ing too. so was hoping to just pass it in so during the SSH it would evaluate correctly
Here is an example of what I want to have happen. Let's assume $HOME is already set on Server1 and Server2 like:
Server1 $HOME: /home/blah/usr
Server2 $HOME: /home/superblah/newusr
From Server1 execute my_script.ksh $HOME
The script is the same as above with one extra line of:
ssh my_user#Server2 "echo Server got: $my_var"
What my output currently is:
You sent: /home/blah/usr
Server got: /home/blah/usr
What I want is just that ssh command to produce:
Server got: /home/superblah/newusr
Thanks.
You can prevent resolving the variable by wrap it in single quotes
./my_script.ksh '$HOME'
I tried it like this:
[host1]$ echo $HOME
/home/user1
[host1]$ ssh user2#host2 echo '$HOME'
/home/user2
[host1]$
envsubst can be used to substitute strings of the form $HOME or $foo with the value of like-named environment variables. Thus:
#!/usr/bin/env ksh
my_var=$1
my_var_subst=$(envsubst <<<"$my_var")
echo "You sent: ${my_var}"
echo "Interpreted as: ${my_var_subst}"
...will emit:
You sent: $HOME
Interpreted as: /home/superblah/newusr
when invoked with ./myscript '$HOME'.
Note that one needs to be careful when passing values over SSH, as SSH implicitly runs sh -c "$*" with the arguments passed -- concatenating them together with whitespace and evaluating that single string as a shell command. Thus, running your command over ssh may look like:
ssh somehost $'./myscript \'$HOME\''
...or...
ssh somehost "./myscript \$HOME"
...or, to make it easier, let the shell do the escaping work for you:
remote_command=$(printf '%q ' ./myscript '$HOME')
ssh somehost "$remote_command"

Why is this bash variable empty when I just set it?

$ sudo sh -c "FOO=bar echo Result:${FOO}"
Result:
Why is the value stored in FOO not displayed?
Because bash replaces ${FOO} before calling sudo. So what sh -c actually sees is:
FOO=bar echo Result:
Besides, even if you tried
FOO=bar echo Result:${FOO}
It still won't work1. To get that right, you can do:
FOO=bar; echo Result:${FOO}
Now that that is fixed, let's get back to sh -c. To prevent bash from interpreting the string you are giving to sh -c, simply put it in ' instead of ":
sudo sh -c 'FOO=bar; echo Result:${FOO}'
1 Refer to the comments for reason.
This doesn't work, because the variable FOO is set for the following command, echo, but the ${FOO} is replaced by the enclosing shell.
If you want it to work, you must set the variable FOO for the enclosing shell and wrap the echo ... in single quotes
sudo FOO=bar sh -c 'echo Result:${FOO}'

Resources