Say I have a bash script file config.sh. It's meant to be source'd by other scripts and variables defined is used as customization of the upper-level scripts.
The problem is, if config.sh has a temporary variable and its name conflicts with upper-level scripts' variable, it breaks the upper-level one.
config.sh:
TMP1=abc
CONFIG_INPUT_DIR="$TMP1"/in
CONFIG_OUTPUT_DIR="$TMP1"/out
upper-level script:
TMP1=def
source config.sh
echo $TMP1
The last echo prints abc, not def.
Solution 1
My current solution is to append a random string to the temporary variable name to make it almost impossible to conflict. e.g:
TMP1_vFc9Uiew=abc
CONFIG_INPUT_DIR="$TMP1_vFc9Uiew"/in
CONFIG_OUTPUT_DIR="$TMP1_vFc9Uiew"/out
unset TMP1_vFc9Uiew
which is painful and makes the code hard to read, in addition not to be perfect.
Solution 2 using local keyword
After some searching, I've come to know local keyword.
But when I simply declare TMP1 as local, bash complains that config.sh: line 1: local: can only be used in a function.
So my another solution is to enclose whole config script as a function:
function config_func_rZ0Yqkpm() {
local TMP1=abc
CONFIG_INPUT_DIR="$TMP1"/in
CONFIG_OUTPUT_DIR="$TMP1"/out
}
config_func_rZ0Yqkpm
unset config_func_rZ0Yqkpm
which is better than previous solution in maintainability and readability, but there's some possibility to conflict as well as solution 1.
Question
I want to know more robust and smart solution without any possibility to conflict.
Thanks.
A trick I learned from the keychain utility is using one program to build a source-able file containing just the variables that you want to export from your program. You could modify your script to echo the variables you want to set and then source the output from your program:
$ echo $FOO
$ source <(echo FOO=bar)
$ echo $FOO
bar
$
I used echo FOO=bar to simulate the larger script; your program is probably more involved. The important part is that you must modify your program to output the variables and values you would like to set, rather than just setting them. This lets you decide which variables to expose and which ones to hold private at the cost of another shell process.
You could avoid variables and use functions in config.sh to hold your values:
get_dirname() { echo "abc"; }
CONFIG_INPUT_DIR="$(get_dirname)/in"
CONFIG_OUTPUT_DIR="$(get_dirname)/out"
unset -f get_dirname
If you're still concerned about name collision for functions, this doesn't really help you.
The "ssh-agent" method:
config.sh
#!/bin/bash
TMP=abc
printf "CONFIG_INPUT_DIR=%s/in\n" "$TMP"
printf "CONFIG_OUTPUT_DIR=%s/out\n" "$TMP"
main program:
TMP1=def
eval "$(config.sh)"
echo "$TMP1"
Related
I write a code in Bash script and the Linux does not run it correctly and printed character instead of character’s value.
Can anyone help me?
Aside from the confusion between backtick and ', you are also over-using the sub-shell syntax. You do not need to use echo $(cat $LOOP). You can just run cat $LOOP directly.
#!/bin/bash
for FILE in $(ls); do
echo "Here is ${file}:"
cat ${FILE}
echo ""
done
A couple of points of style as well:
Name your variables after the thing they represent. The loop is iterating over files in the current directory, so the name FILE is more descriptive than LOOP.
It is a good idea to get in the habit of enclosing your variable references in ${ ... } instead of just prefixing them with $. In simple scripts like this, it does not make a difference, but when your scripts get more complicated, you can get into trouble when you do not clearly delineate variable names.
I'm looking how to export from a Makefile environment variables to be exposed in the userland environment so exporting these variables from the Makefile should be accessible from the user shell.
I have tried make's export but as I understand and have tried does not export to outside of Makefile.
The idea of this is to populate Docker Compose environment variables in a elegant way and have these variables ready to use in the user shell also.
This is a fragment of what I've tried with make's export:
include docker.env
export $(shell sed -n '/=/p' docker.env)
SHELL := /bin/bash
run:
#docker-compose -f my-service.yml up -d
According with ArchWiki, each process of Bash...
Each process stores their environment in the /proc/$PID/environ file.
so once Make execute a source, export or any other command to set a new environment variable it will be applied only for that process.
As workaround I've written in the bash startup file so the variables will be in the global environment as soon as a new bash shell is loaded:
SHELL := /bin/bash
RC := ~/.bashrc
ENV := $(shell sed -n '/=/p' docker.env)
test:
#$(foreach e,$(ENV),echo $(e) >> $(RC);) \
EDIT completely reworked the answer after the OP explained in a comment that he wants the environment variables to be defined for any user shell.
If your goal is to have a set of environment variables defined for any user shell (I assume this means interactive shell), you can simply add these definitions to the shell's startup file (.bashrc for bash). From GNU make manual:
Variables in make can come from the environment in which make is run.
Every environment variable that make sees when it starts up is
transformed into a make variable with the same name and value.
However, an explicit assignment in the makefile, or with a command
argument, overrides the environment. (If the ‘-e’ flag is specified,
then values from the environment override assignments in the makefile.
See Summary of Options. But this is not recommended practice.)
Example:
$ cat .bashrc
...
export FOOBAR=foobar
export BARFOO="bar foo"
...
$ cat Makefile
all:
#printf '$$(FOOBAR)=%s\n' '$(FOOBAR)'
#printf 'FOOBAR='; printenv FOOBAR
#printf '$$(BARFOO)=%s\n' '$(BARFOO)'
#printf 'BARFOO='; printenv BARFOO
$ make
$(FOOBAR)=foobar
FOOBAR=foobar
$(BARFOO)=bar foo
BARFOO=bar foo
If you want to keep these definitions separate, you can just source the file from .bashrc:
$ cat docker.env
export FOOBAR=foobar
export BARFOO="bar foo"
$ cat .bashrc
...
source <some-path>/docker.env
...
And finally, if you don't want to add the export bash command to your file, you can parse the file in your .bashrc:
$ cat docker.env
FOOBAR=foobar
BARFOO="bar foo"
$ cat .bashrc
...
while read -r line; do
eval "export $$line"
done < <(sed -n '/=/p' <some-path>/docker.env)
...
Of course, there are some constraints for the syntax of your docker.env file (no unquoted special characters, no spaces in variable names, properly quoted values...) If your syntax is not bash-compatible it is time to ask another question about parsing this specific syntax and converting it into bash-compatible syntax.
Make cannot change the calling shell's environment without its cooperation. Of course, if you are in control, you can make the calling shell cooperate.
In broad terms, you could replace the make command with a shell alias or function which runs the real make and also sets the environment variables from the result. I will proceed to describe in more detail one way to implement this.
Whether you call this alias or function of yours make or e.g. compose is up to you really. To wrap the real make is marginally harder -- inside the function, you need to say command make, because just make would cause an infinite loop with the alias or function calling itself recursively -- so I will demonstrate this. Let's define a function (aliases suck);
make () {
# run the real make, break out on failure
command make "$#" || return
# if there is no env for us to load, we are done
test -f ./docker.env || return 0
# still here? load it
. ./docker.env
}
If you want even stricter control, maybe define a variable in the function and check inside the Makefile that the variable is set.
$(ifneq '${_composing}','function_make')
$(error Need to use the wrapper function to call make)
$(endif)
The error message is rather bewildering if you haven't read this discussion, so maybe it needs to be improved, and/or documented in a README or something. You would change the make line in the function above into
_composing='function_make' \
command make "$#" || return
The syntax var=value cmd args sets the variable var to the string value just for the duration of running the command line cmd args; it then returns to its previous state (unset, or set to its previous value).
For this particular construction, the name of the variable just needs to be reasonably unique and transparent to a curious human reader; and the value is also just a reasonably unique and reasonably transparent string which the function and the Makefile need to agree on.
Depending on what you end up storing in the environment, this could introduce complications if you need this mechanism for multiple Makefiles. Running it in directory a and then switching to a similar directory b will appear to work, but uses the a things where the poor puny human would expect the b things. (If the variables you set contain paths, relative paths fix this scenario, but complicate others.)
Extending this to a model similar to Ruby's rvm or Python's virtualenv might be worth exploring; they typically add an indicator to the shell prompt to remind you which environment is currently active, and have some (very modest) safeguards in place to warn you when your current directory and the environment disagree.
Another wart: Hardcoding make to always load docker.env is likely to produce unwelcome surprises one day. Perhaps hardcode a different file name which is specific to this hook - say, .compose_post_make_hook? It can then in turn contain something like
. ./docker.env
in this particular directory.
This question already has answers here:
Why piping input to "read" only works when fed into "while read ..." construct? [duplicate]
(4 answers)
Why does my variable set in a do loop disappear? (unix shell)
(2 answers)
Closed 6 years ago.
If I have a simple bash script set_token.sh:
#!/bin/bash
output='export AWS_ACCESS_KEY_ID="111"
export AWS_SECRET_ACCESS_KEY="222"
export AWS_SESSION_TOKEN="333"'
echo "$output" | while read line; do eval $line; done
Executed set_token.sh did not successfully set the 3 environment variables. However if I run eval on each line separately, it works.
$ eval 'export AWS_ACCESS_KEY_ID="111"'
$ eval 'export AWS_SECRET_ACCESS_KEY="222"'
$ eval 'export AWS_SESSION_TOKEN="333"'
Why is that so?
You can achieve the desired result without a loop and without eval.
source <(echo "$output")
The <() construct is a process substitution. It executes the command found inside, creates a FIFO (special first-in, first-out file), and is then transformed into an actual file path (pointing to the FIFO) which source can read from.
Of course, you could also store the actual assignments in a file rather than putting them in the output variable.
source config_file
The source command (or its more standard form .) reads commands from a file and executes them in the current shell, without launching a separate process or subshell, so variable assignments in sourced files work. Useful for config files, but of course you must be sure no one can put arbitrary commands in those files as that would be a security risk.
IMPORTANT
If you want to put declarations in a script (set_token.sh in your case), this script must be sourced (i.e. executed with source or .), not executed with bash or by calling it directly (if it is executable). Any method other than source or . will launch a child process, and there is no way for a child process to assign variables that will be visible to the parent process afterwards. Sourcing does not create a separate process, which is why assignments will work. The export keyword will make assignments visible to children process only, they cannot make assignments visible to the parent.
Not sure why you want to use eval in this case. Why not set the variables more directly like this:
export AWS_ACCESS_KEY_ID="111"
export AWS_SECRET_ACCESS_KEY="222"
export AWS_SESSION_TOKEN="333"
Your loop is running in a sub shell (because of echo "$output" | ...) and that's why your variables are not visible outside. It's not that eval is not working! Don't worry - this happens to a lot of people.
If you are insistent on using the loop and eval, you could use process substitution < <(command):
while read line; do eval $line; done < <(printf "%s\n" "$output")
printf is better than echo
see also:
Shell variables set inside while loop not visible outside of it
What is more portable? echo -e or using printf?
Fred's helpful answer contains a viable solution and good pointers (and the problem with the original approach is explained in Bash FAQ #24 - "I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates?").
That said, in your specific scenario - assuming you're willing to accept the risk of using eval - you can apply it directly to your multi-line string:
#!/bin/bash
output='export AWS_ACCESS_KEY_ID="111"
export AWS_SECRET_ACCESS_KEY="222"
export AWS_SESSION_TOKEN="333"'
# This defines all 3 AWS_* environment variables.
eval "$output"
To reiterate Fred's point: For the environment variables to take effect in the current shell, you must source the script (using builtin . or its (effective) alias source):
. ./set_token.sh
Lets say I have a shell / bash script named test.sh with:
#!/bin/bash
TESTVARIABLE=hellohelloheloo
./test2.sh
My test2.sh looks like this:
#!/bin/bash
echo ${TESTVARIABLE}
This does not work. I do not want to pass all variables as parameters since imho this is overkill.
Is there a different way?
You have basically two options:
Make the variable an environment variable (export TESTVARIABLE) before executing the 2nd script.
Source the 2nd script, i.e. . test2.sh and it will run in the same shell. This would let you share more complex variables like arrays easily, but also means that the other script could modify variables in the source shell.
UPDATE:
To use export to set an environment variable, you can either use an existing variable:
A=10
# ...
export A
This ought to work in both bash and sh. bash also allows it to be combined like so:
export A=10
This also works in my sh (which happens to be bash, you can use echo $SHELL to check). But I don't believe that that's guaranteed to work in all sh, so best to play it safe and separate them.
Any variable you export in this way will be visible in scripts you execute, for example:
a.sh:
#!/bin/sh
MESSAGE="hello"
export MESSAGE
./b.sh
b.sh:
#!/bin/sh
echo "The message is: $MESSAGE"
Then:
$ ./a.sh
The message is: hello
The fact that these are both shell scripts is also just incidental. Environment variables can be passed to any process you execute, for example if we used python instead it might look like:
a.sh:
#!/bin/sh
MESSAGE="hello"
export MESSAGE
./b.py
b.py:
#!/usr/bin/python
import os
print 'The message is:', os.environ['MESSAGE']
Sourcing:
Instead we could source like this:
a.sh:
#!/bin/sh
MESSAGE="hello"
. ./b.sh
b.sh:
#!/bin/sh
echo "The message is: $MESSAGE"
Then:
$ ./a.sh
The message is: hello
This more or less "imports" the contents of b.sh directly and executes it in the same shell. Notice that we didn't have to export the variable to access it. This implicitly shares all the variables you have, as well as allows the other script to add/delete/modify variables in the shell. Of course, in this model both your scripts should be the same language (sh or bash). To give an example how we could pass messages back and forth:
a.sh:
#!/bin/sh
MESSAGE="hello"
. ./b.sh
echo "[A] The message is: $MESSAGE"
b.sh:
#!/bin/sh
echo "[B] The message is: $MESSAGE"
MESSAGE="goodbye"
Then:
$ ./a.sh
[B] The message is: hello
[A] The message is: goodbye
This works equally well in bash. It also makes it easy to share more complex data which you could not express as an environment variable (at least without some heavy lifting on your part), like arrays or associative arrays.
Fatal Error gave a straightforward possibility: source your second script! if you're worried that this second script may alter some of your precious variables, you can always source it in a subshell:
( . ./test2.sh )
The parentheses will make the source happen in a subshell, so that the parent shell will not see the modifications test2.sh could perform.
There's another possibility that should definitely be referenced here: use set -a.
From the POSIX set reference:
-a: When this option is on, the export attribute shall be set for each variable to which an assignment is performed; see the Base Definitions volume of IEEE Std 1003.1-2001, Section 4.21, Variable Assignment. If the assignment precedes a utility name in a command, the export attribute shall not persist in the current execution environment after the utility completes, with the exception that preceding one of the special built-in utilities causes the export attribute to persist after the built-in has completed. If the assignment does not precede a utility name in the command, or if the assignment is a result of the operation of the getopts or read utilities, the export attribute shall persist until the variable is unset.
From the Bash Manual:
-a: Mark variables and function which are modified or created for export to the environment of subsequent commands.
So in your case:
set -a
TESTVARIABLE=hellohelloheloo
# ...
# Here put all the variables that will be marked for export
# and that will be available from within test2 (and all other commands).
# If test2 modifies the variables, the modifications will never be
# seen in the present script!
set +a
./test2.sh
# Here, even if test2 modifies TESTVARIABLE, you'll still have
# TESTVARIABLE=hellohelloheloo
Observe that the specs only specify that with set -a the variable is marked for export. That is:
set -a
a=b
set +a
a=c
bash -c 'echo "$a"'
will echo c and not an empty line nor b (that is, set +a doesn't unmark for export, nor does it “save” the value of the assignment only for the exported environment). This is, of course, the most natural behavior.
Conclusion: using set -a/set +a can be less tedious than exporting manually all the variables. It is superior to sourcing the second script, as it will work for any command, not only the ones written in the same shell language.
There's actually an easier way than exporting and unsetting or sourcing again (at least in bash, as long as you're ok with passing the environment variables manually):
let a.sh be
#!/bin/bash
secret="winkle my tinkle"
echo Yo, lemme tell you \"$secret\", b.sh!
Message=$secret ./b.sh
and b.sh be
#!/bin/bash
echo I heard \"$Message\", yo
Observed output is
[rob#Archie test]$ ./a.sh
Yo, lemme tell you "winkle my tinkle", b.sh!
I heard "winkle my tinkle", yo
The magic lies in the last line of a.sh, where Message, for only the duration of the invocation of ./b.sh, is set to the value of secret from a.sh.
Basically, it's a little like named parameters/arguments. More than that, though, it even works for variables like $DISPLAY, which controls which X Server an application starts in.
Remember, the length of the list of environment variables is not infinite. On my system with a relatively vanilla kernel, xargs --show-limits tells me the maximum size of the arguments buffer is 2094486 bytes. Theoretically, you're using shell scripts wrong if your data is any larger than that (pipes, anyone?)
In Bash if you export the variable within a subshell, using parentheses as shown, you avoid leaking the exported variables:
#!/bin/bash
TESTVARIABLE=hellohelloheloo
(
export TESTVARIABLE
source ./test2.sh
)
The advantage here is that after you run the script from the command line, you won't see a $TESTVARIABLE leaked into your environment:
$ ./test.sh
hellohelloheloo
$ echo $TESTVARIABLE
#empty! no leak
$
Adding to the answer of Fatal Error, There is one more way to pass the variables to another shell script.
The above suggested solution have some drawbacks:
using Export : It will cause the variable to be present out of their scope which is not a good design practice.
using Source : It may cause name collisions or accidental overwriting of a predefined variable in some other shell script file which have sourced another file.
There is another simple solution avaiable for us to use.
Considering the example posted by you,
test.sh
#!/bin/bash
TESTVARIABLE=hellohelloheloo
./test2.sh "$TESTVARIABLE"
test2.sh
#!/bin/bash
echo $1
output
hellohelloheloo
Also it is important to note that "" are necessary if we pass multiword strings.
Taking one more example
master.sh
#!/bin/bash
echo in master.sh
var1="hello world"
sh slave1.sh $var1
sh slave2.sh "$var1"
echo back to master
slave1.sh
#!/bin/bash
echo in slave1.sh
echo value :$1
slave2.sh
#!/bin/bash
echo in slave2.sh
echo value : $1
output
in master.sh
in slave1.sh
value :"hello
in slave2.sh
value :"hello world"
It happens because of the reasons aptly described in this link
Another option is using eval. This is only suitable if the strings are trusted. The first script can echo the variable assignments:
echo "VAR=myvalue"
Then:
eval $(./first.sh) ./second.sh
This approach is of particular interest when the second script you want to set environment variables for is not in bash and you also don't want to export the variables, perhaps because they are sensitive and you don't want them to persist.
Another way, which is a little bit easier for me is to use named pipes. Named pipes provided a way to synchronize and sending messages between different processes.
A.bash:
#!/bin/bash
msg="The Message"
echo $msg > A.pipe
B.bash:
#!/bin/bash
msg=`cat ./A.pipe`
echo "message from A : $msg"
Usage:
$ mkfifo A.pipe #You have to create it once
$ ./A.bash & ./B.bash # you have to run your scripts at the same time
B.bash will wait for message and as soon as A.bash sends the message, B.bash will continue its work.
I am using the sh 3.2 in Mac Os X. I have a file test.conf
config1="Configuration 1"
config2="a lot of text"
config3=...
So I only need to get the config1= and config2= parameter. How can I set a variable, that I can do this:
> echo $variable
Configuration 1
So simple, but I am not doing it work.
the sommand you are looking for is source
source test.conf
echo $config1 #echoes Configuration 1
if you need to have config1 in variable, add
varible=$config1
At a rough guess...
export `grep 'config1=' /your/config/file`
export `grep 'config2=' /your/config/file`
But remember if you put this in a shell script file, then you'll need to eval the file rather than execute it to set the variables in the current shell instance.
You could do this:
variable=`sed -n 's/^config1=//p'`
Or if you are attempting to evaluate certain parts of your file, try something like
eval `grep ^config1= test.conf`
to have config1=Configuration 1 evaluated by the current shell. (With the example you provided, this will cause a syntax error, because the value cannot contain unquoted whitespace.)
I generally recommend beginners to stay away from backticks, but this is a situation where they are a good answer.