I'm running Fastlane (a continuous build tool for iOS) in order to execute a custom shell script for decrypting a file.
This is the command.
sh "./decrypt.sh ENV['ENCRYPTION_P12']"
I cannot figured out a way to pass the environment variable to that script. Obviously, if I hardcode the pwd into the script, it works correctly.
sh "./decrypt.sh mypwd"
Any suggestions?
Expanding From Within The Immediate Shell
Assuming that sh, here, is a fastlane command that invokes a shell command with the given argument as script text:
# as a fastlane directive
sh './decrypt.sh "$ENCRYPTION_P12"'
Note that if this were being literally invoked as a command line for /bin/sh, it would need a -c argument:
# in other contexts
sh -c './decrypt.sh "$ENCRYPTION_P12"'
Note that this absolutely depends on ENCRYPTION_P12 being an environment variable -- that is, exported to the environment by the system by which it was set.
Expanding from Within The Invoked Script
That said, if it is an environment variable, you have a better option: Just use it.
That is, inside decrypt.sh, you can refer to "$ENCRYPTION_P12" without needing to set it explicitly, as the shell implicitly imports all environment variables as shell variables -- and they're passed down to child processes without any explicit actions needed.
Things to Avoid: Shell Injection Attacks
Finally, an aside: The dangerous way to do this would have been something like:
# INSECURE: DO NOT DO THIS
sh "./decrypt.sh #{ENV['ENCRYPTION_P12']}"
or
# STILL INSECURE
sh "./decrypt.sh \"#{ENV['ENCRYPTION_P12'}\""
or
# STILL INSECURE
sh "./decrypt.sh '#{ENV['ENCRYPTION_P12'}'"
...thereby substituting the value into your generated string at the Ruby level. This is dangerous, however, as that string is parsed as code -- meaning that contents of ENCRYPTION_P12 could then be leveraged in shell attacks.
For instance, consider the case (given below in bash syntax):
# this will make any of the above do Very Evil Things
ENCRYPTION_P12=$'$(rm -rf ~)\'$(rm -rf ~)\''
...for which both rms will execute if directly substituted into generated shell script (as opposed to expanded during parameter expansion -- '${foo}' -- which takes place after the expansion phases which make this dangerous have already passed).
The fastlane specific answer is https://docs.fastlane.tools/advanced/#shell-values
or, from within your Fastfile:
decrypted = sh("./decrypt" ENV[ENCRYPTION_P12])
Related
I noticed that my script was ignoring my positional arguments in old terminal tabs, but working on recently created ones, so I decided to reduce it to the following:
TAG=test
while getopts 't:' c
do
case $c in
t)
TAG=$OPTARG
;;
esac
done
echo $TAG
And running the script I have:
~ source my_script
test
~ source my_script -t "test2"
test2
~ source my_script -t "test2"
test
I thought it could be that c was an special used variable elsewhere but after changing it to other names I had the exact same problem. I also tried adding a .sh extension to the file to see it that was a problem, but nothing worked.
Am I doing something wrong ? And why does it work the first time, but not the subsequent attempts ?
I am on MacOS and I use zsh.
Thank you very much.
The problem is that you're using source to run the script (the . command does the same thing). This makes it run in your current (interactive) shell (rather than a subprocess, like scripts normally do). This means it uses the same variables as the current shell, which is necessary if you want it to change those variables, but it can also have weird effects if you're not careful.
In this case, the problem is that getopts uses the variable OPTIND to keep track of where it is in the argument list (so it doesn't process the same argument twice). The first time you run the script with -t test2, getopts processes those arguments, and leaves OPTIND set to 3 (meaning that it's already done the first two arguments, "-t" and "test2". The second time you run it with options, it sees that OPTIND is set to 3, so it thinks it's already processed both arguments and just exits the loop.
One option is to add unset OPTIND before the while getopts loop, to reset the count and make it start from the beginning each time.
But unless there's some reason for this script to run in the current shell, it'd be better to make it a standard shell script and have it run as a subprocess. To do this:
Add a "shebang" line as the first line of the script. To make the script run in bash, that'd be either #!/bin/bash or #!/usr/bin/env bash. For zsh, use #!/bin/zsh or #!/usr/bin/env zsh. Since the script runs in a separate shell process, the you can run bash scripts from zsh or zsh scripts from bash, or whatever.
Add execute permission to the script file with chmod -x my_script (or whatever the file's actual name is).
Run the script with ./my_script (note the lack of a space between . and /), or by giving the full path to the script, or by putting the script in some directory in your PATH (the directories that're automatically searched for commands) and just running my_script. Do NOT run it with the bash, sh, zsh etc commands; these override the shebang and therefore can cause confusion.
Note: adding ".sh" to the filename is not recommended; it does nothing useful, and makes the script less convenient to run since you have to type in the extension every time you run it.
Also, a couple of recommendations: there are a bunch of all-caps variable names with special meanings (like PATH and OPTIND), so unless you want one of those special meanings, it's best to use lower- or mixed-case variable names (e.g. tag instead of TAG). Also, double-quoting variable references (e.g. echo "$tag" instead of echo $tag) avoids a lot of weird parsing headaches. Run your scripts through shellcheck.net; it's good at spotting common mistakes like this.
Here's a simple test script (named "testps")
#!/bin/bash
echo PS4=$PS4
and I have set and exported PS4 like this:
export PS4='$LINENO:'
When I run it, either with ./testps or bash ./testps, the result is:
PS4=+
Looks like the value of "PS4" has been reset.
The only way I have found so far to customize PS4 is to run the script with bash -l after having added export PS4='$LINENO:' in .bashrc.
What have I missed here?
Note also that when using ksh, PS4 is initialized with its value from the environment if any.
This behavior doesn't happen with all users -- a copy of PS4 from the environment is only ignored when running as root, since shell version 4.4.
Quoting from CHANGES in the bash source:
g. Shells running as root no longer inherit PS4 from the environment, closing a security hole involving PS4 expansion performing command substitution.
This was done because environment variables are passed to setuid executables, and some setuid executables (unwisely) use system(), popen(), etc. to invoke a shell. While ld.so ignores LD_LIBRARY_PATH, LD_PRELOAD and similar when running in setuid, bash historically did not do so with environment variables that could cause arbitrary execution.
As described in https://www.openwall.com/lists/oss-security/2016/09/26/9, this issue could be exploited as follows:
env -i SHELLOPTS=xtrace PS4='$(id)' ./test
...would run id, instead of printing $(id) as part of xtrace logs, even if invocation of ./test crosses a privilege boundary.
Personally, I set PS4 inside my scripts -- it's ignored unless they're run with set -x, after all, so why not establish a meaningful value regardless?
If you want to force it to happen with fewer side effects than you get from bash -l or bash -i, set BASH_ENV to have the name of a file that can be sourced to perform your desired initialization.
Suppose I would like to set a shell in make:
SHELL:=/usr/bin/env bash
Next, suppose I have some runcom/bash file I would like to source as well. This file optionally activates a virtual environment:
if [ -d venv ]; then source venv/bin/activate fi;
However, if I write:
SHELL:=/usr/bin/env bash && source runcom/bash
This fails. However, if I deposited the venv logic into the local ~/.bashrc and write:
SHELL:=/usr/bin/env bash -l
I can get the exact functionality I need.
But, I have to deposit something that should remain downstream from the local user into the user's upstream environment -- I'd rather not.
Is there a way to get the make shell to source a file at the declaration step in the make start-up process?
This can't work:
SHELL:=/usr/bin/env bash && source runcom/bash
Because SHELL tells make how to invoke the shell; if you make the contents of SHELL be a shell script then make has to invoke the shell to interpret how to invoke the shell, which means it has to invoke the shell to invoke the shell to interpret how to invoke the shell, etc.
So, SHELL must be either a simple command or, at most, a simple set of arguments that can converted into an argv list and passed to exec(2).
So, this is really a shell question not a make question: how can you get the shell to source arbitrary stuff when it starts without changing ~/.profile or whatever?
Luckily, this is possible; see the bash man page:
BASH_ENV
If this parameter is set when bash is executing a shell script,
its value is interpreted as a filename containing commands to
initialize the shell, as in ~/.bashrc. The value of BASH_ENV is
subjected to parameter expansion, command substitution, and
arithmetic expansion before being interpreted as a filename.
PATH is not used to search for the resultant filename.
ENV Similar to BASH_ENV; used when the shell is invoked in posix
mode.
So, in your makefile you can use something like:
SHELL := /bin/bash
export BASH_ENV := runcom/bash
and that should be sufficient.
Original Title: Indirect parameter substitution breaks when the script is sourced (zsh)
zsh 5.7.1 (x86_64-apple-darwin19.0)
GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
I’m developing a shell script on a Mac and I’m trying to keep it portable between bash & zsh, so array indexing is a consideration. I know that I can set KSH_ARRAYS to get indexing to start at 0, but I decided to query the OS for the shell that’s in use and set the start index accordingly, which led to the issue described below.
It made sense (to me anyway!) to use indirect expansion, which is what led to the problem. Consider the script indirect.sh:
#! /bin/bash
declare -r ARRAY_START_BASH=0
declare -r ARRAY_START_ZSH=1
declare -r SHELL_BASH=0
declare -r SHELL_ZSH=1
# Indirect expansion is used to reference the values of the variables declared
# in this case statement e.g. ${!ARRAY_START}
case $(basename $SHELL) in
"bash" )
declare -r SHELL_ID=SHELL_BASH
declare -r ARRAY_START=ARRAY_START_BASH
;;
"zsh" )
declare -r SHELL_ID=SHELL_ZSH
declare -r ARRAY_START=ARRAY_START_ZSH
;;
* )
return 1
;;
esac
echo "Shell ID: ${!SHELL_ID} Index arrays from: ${!ARRAY_START}"
It works fine when run from the command line while in the same directory:
<my home> ~ % echo "$(./indirect.sh)"
Shell ID: 1 Index arrays from: 1
Problems arise when I source the script:
<my home> ~ % echo "$(. ~/indirect.sh)"
/Users/<me>/indirect.sh:28: bad substitution
I don’t understand why sourcing the script changes the behavior of the parameter expansion.
Is this expected behavior? If so, I’d be grateful if someone could explain it and hopefully, offer a work around.
The problem described in the original post has nothing to do with indirect expansion. The difference in behavior is a result of different shells being invoked depending on whether the script is “executed” or “sourced”. These differences reveal the basic flaw in deriving the shell from the $SHELL variable that underpins the script's design. If the shell defined in $SHELL does not match the shebang, the script will fail either when sourced or executed. An explanation follows.
Indirect expansion doesn’t offer value in the given scenario because values could just as easily be assigned directly. They’ll have to be assigned that way regardless given the different syntax used for indirect expansion between shells. In fact, other syntax differences between shells makes the entire premise for detecting the shell moot! However, putting that aside, the difference in behavior is a result of different shells being invoked based on whether the script is “executed” or “sourced”. The behavior of sourcing is well documented with numerous explanations on the web, but for context here’s how it works:
Executing a Script
Use the “./“ syntax to execute a script.
When run this way, the script executes in a sub-shell. Any changes the
script makes to it’s shell are applied to the sub-shell, not the shell
in which the script was launched, so those changes are lost when the
shell exits because the sub-shell in which it executed is destroyed as
well. For example, if the script changes the working directory, it
does so in the sub-shell. The working directory of the main shell that
launched the script is unchanged when the script terminates. If you
want to make changes to the shell in which the script was launched, it
must be sourced.
Sourcing a Script
Use the “source “ syntax to source a
script. When run this way, the script essentially becomes an argument
for the source command, which handles invoking the appropriate
execution. Some shells (e.g. ksh) use a single period “.” instead of
“source”.
When a script is executed with the “./“ syntax, the shebang at the top of the file is used to determine which shell to use. When a script is sourced, the shebang is ignored and the shell in which the script is launched is used instead. Also note that the period that appears in the “./“ command syntax used to execute a script, is not related to the period that’s occasionally used as an alias for the source command.
The script in the post uses bash in the shebang statement, so it works when executed because it’s run using bash. When it’s sourced from zsh, it encounters the incorrect indirect expansion syntax:
“${!A_VAR}"
The correct syntax is:
"${(P)A_VAR}"
However, correcting the syntax won’t help because it will then fail when executed. The shebang will invoke bash and the syntax will be wrong again. That renders indirection useless for accessing a variable designed to indicate the shell in use. More importantly, a design based on querying an environment variable for the shell is flawed due to differences in the shell that’s ultimately used depending on whether the script is executed or sourced.
To add to your answer (what I'm going to say is too long for a comment), I can not think of any application, why your script could be useful if not sourced. Actually, I came accross the need of such a script by myself in exactly one occasion:
Since I use as interactive shell not only zsh, but also sometimes bash, so I have written my .zshrc and .bashrc to set up everything (including defining variables and shell functions for interactive use). In order to safe work,
I try to put code which works under both bash and zsh into a single file (say: .commonrc), and my .zshrc and .bashrc have inside them a
source .commonrc
While many things are so different in bash and zsh, that I can't put them into .commonrc, some can, provided I do some tweaking. One reason for headache is obviously the different indexing of arrays, which you seemingly try to solve. So I have also a similar feature. However, I don't nee ca case construct for this. Instead, my .bashrc looks like this (using your naming of the variables):
...
declare -r ARRAY_START=0
source .commonrc
...
and my .zshrc looks like this:
...
declare -r ARRAY_START=1
source .commonrc
...
Since it does not happen that the .bashrc is run from a zsh and vice versa, I don't need to query what kind of shell I have.
I'm trying to access a variable declared by previous command (inside a Makefile).
Here's the Makefile:
all:
./script1.sh
./script2.sh
Here's the script declaring the variable I want to access,script1.sh:
#!/usr/bin/env bash
myVar=1234
Here's the script trying to access the variable previously defined, script2.sh:
#!/usr/bin/env bash
echo $myVar
Unfortunately when I run make, myVar isn't accessible. Is there an other way around? Thanks.
Make will run each shell command in its own shell. And when the shell exits, its environment is lost.
If you want variables from one script to be available in the next, there are constructs which will do this. For example:
all:
( . ./script1.sh; ./script2.sh )
This causes Make to launch a single shell to handle both scripts.
Note also that you will need to export the variable in order for it to be visible in the second script; unexported variables are available only to the local script, and not to subshells that it launches.
UPDATE (per Kusalananda's comment):
If you want your shell commands to populate MAKE variables instead of merely environment variables, you may have options that depend on the version of Make that you are running. For example, in BSD make and GNU make, you can use "variable assignment modifiers" including (from the BSD make man page):
!= Expand the value and pass it to the shell for execution and
assign the result to the variable. Any newlines in the result
are replaced with spaces.
Thus, with BSD make and GNU make, you could do this:
$ cat Makefile
foo!= . ./script1.sh; ./script2.sh
all:
#echo "foo=${foo}"
$
$ cat script1.sh
export test=bar
$
$ cat script2.sh
#!/usr/bin/env bash
echo "$test"
$
$ make
foo=bar
$
Note that script1.sh does not include any shebang because it's being sourced, and is therefore running in the calling shell, whatever that is. That makes the shebang line merely a comment. If you're on a system where the default shell is POSIX but not bash (like Ubuntu, Solaris, FreeBSD, etc), this should still work because POSIX shells should all understand the concept of exporting variables.
The two separate invocations of the scripts create two separate environments. The first script sets a variable in its environment and exits (the environment is lost). The second script does not have that variable in its environment, so it outputs an empty string.
You can not have environment variables pass between environments other than between the environments of a parent shell to its child shell (not the other way around). The variables passed over into the child shell are only those that the parent shell has export-ed. So, if the first script invoked the second script, the value would be outputted (if it was export-ed in the first script).
In a shell, you would source the first file to set the variables therein in the current environment (and then export them!). However, in Makefiles it's a bit trickier since there's no convenient source command.
Instead you may want to read this StackOverflow question.
EDIT in light of #ghoti's answer: #ghoti has a good solution, but I'll leave my answer in here as it explains a bit more verbosely about environment variables and what we can do and not do with them with regards to passing them between environments.