I have noticed some weird behavior when sourcing another script within my shell script. The script that I am sourcing to setup the environment in my shell script takes an optional argument, e.g.
source setup.sh version1
However in my shell script I have also have command line argument variables. For example:
./myscript.sh TEST 1
Inside myscript.sh:
#!/bin/zsh
source setup.sh
echo ROOT version setup $ROOT_SYS
...more of the script
The problem that I have noticed with my script above is that the $1 argument (TEST in this example) is used in the source setup.sh command. This causes the command to become
source setup.sh TEST
which of course fails as setup.sh does not have a version TEST.
I solved this problem by editing my script to below.
#!/bin/zsh
source setup.sh version1
echo ROOT version setup $ROOT_SYS
...more of the script
The source command now does not pick up the $1 argument.
Why/How does the source command pick up the $1 argument when I am running my shell script?
Historically, unix shells didn't allow any arguments to be passed to scripts called with the . built-in (source is an alias of . available in bash, ksh and zsh). The . built-in means “act as if this file was actually included here”.
In bash, ksh and zsh, if you pass extra arguments to the . built-in, they become positional parameters ($1 and so on) in the sourced script. If you pass zero arguments, the positional parameters of the main script remain in effect. In those shells, . behaves rather like calling a function, though not perfectly so (in particular, in bash, if you modify the positional parameters inside the sub-script, the modification is seen by the main script).
A simple way of avoiding this kind of difficulty is to only ever define functions (and perhaps variables) in the subscript. Treat it as a code library, such that merely sourcing it has no effect, and then call functions from the sub-script to actually do something.
This is because source executes the code of setup.sh as if it was in place, so when setup.sh access, say, $1, the value it has is that of the first argument of the actual script. If you want to avoid that you could either execute it:
setup.sh
or, if you need to get back some variables or values from it, change it to return the result in form of an output, something like:
ROOT_SYS=`setup.sh`
Finally, as you figured out, the source keywords also allows providing arguments to the scripts, but it bypasses current arguments if you don't provide any.
Related
I noticed that my script was ignoring my positional arguments in old terminal tabs, but working on recently created ones, so I decided to reduce it to the following:
TAG=test
while getopts 't:' c
do
case $c in
t)
TAG=$OPTARG
;;
esac
done
echo $TAG
And running the script I have:
~ source my_script
test
~ source my_script -t "test2"
test2
~ source my_script -t "test2"
test
I thought it could be that c was an special used variable elsewhere but after changing it to other names I had the exact same problem. I also tried adding a .sh extension to the file to see it that was a problem, but nothing worked.
Am I doing something wrong ? And why does it work the first time, but not the subsequent attempts ?
I am on MacOS and I use zsh.
Thank you very much.
The problem is that you're using source to run the script (the . command does the same thing). This makes it run in your current (interactive) shell (rather than a subprocess, like scripts normally do). This means it uses the same variables as the current shell, which is necessary if you want it to change those variables, but it can also have weird effects if you're not careful.
In this case, the problem is that getopts uses the variable OPTIND to keep track of where it is in the argument list (so it doesn't process the same argument twice). The first time you run the script with -t test2, getopts processes those arguments, and leaves OPTIND set to 3 (meaning that it's already done the first two arguments, "-t" and "test2". The second time you run it with options, it sees that OPTIND is set to 3, so it thinks it's already processed both arguments and just exits the loop.
One option is to add unset OPTIND before the while getopts loop, to reset the count and make it start from the beginning each time.
But unless there's some reason for this script to run in the current shell, it'd be better to make it a standard shell script and have it run as a subprocess. To do this:
Add a "shebang" line as the first line of the script. To make the script run in bash, that'd be either #!/bin/bash or #!/usr/bin/env bash. For zsh, use #!/bin/zsh or #!/usr/bin/env zsh. Since the script runs in a separate shell process, the you can run bash scripts from zsh or zsh scripts from bash, or whatever.
Add execute permission to the script file with chmod -x my_script (or whatever the file's actual name is).
Run the script with ./my_script (note the lack of a space between . and /), or by giving the full path to the script, or by putting the script in some directory in your PATH (the directories that're automatically searched for commands) and just running my_script. Do NOT run it with the bash, sh, zsh etc commands; these override the shebang and therefore can cause confusion.
Note: adding ".sh" to the filename is not recommended; it does nothing useful, and makes the script less convenient to run since you have to type in the extension every time you run it.
Also, a couple of recommendations: there are a bunch of all-caps variable names with special meanings (like PATH and OPTIND), so unless you want one of those special meanings, it's best to use lower- or mixed-case variable names (e.g. tag instead of TAG). Also, double-quoting variable references (e.g. echo "$tag" instead of echo $tag) avoids a lot of weird parsing headaches. Run your scripts through shellcheck.net; it's good at spotting common mistakes like this.
Original Title: Indirect parameter substitution breaks when the script is sourced (zsh)
zsh 5.7.1 (x86_64-apple-darwin19.0)
GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
I’m developing a shell script on a Mac and I’m trying to keep it portable between bash & zsh, so array indexing is a consideration. I know that I can set KSH_ARRAYS to get indexing to start at 0, but I decided to query the OS for the shell that’s in use and set the start index accordingly, which led to the issue described below.
It made sense (to me anyway!) to use indirect expansion, which is what led to the problem. Consider the script indirect.sh:
#! /bin/bash
declare -r ARRAY_START_BASH=0
declare -r ARRAY_START_ZSH=1
declare -r SHELL_BASH=0
declare -r SHELL_ZSH=1
# Indirect expansion is used to reference the values of the variables declared
# in this case statement e.g. ${!ARRAY_START}
case $(basename $SHELL) in
"bash" )
declare -r SHELL_ID=SHELL_BASH
declare -r ARRAY_START=ARRAY_START_BASH
;;
"zsh" )
declare -r SHELL_ID=SHELL_ZSH
declare -r ARRAY_START=ARRAY_START_ZSH
;;
* )
return 1
;;
esac
echo "Shell ID: ${!SHELL_ID} Index arrays from: ${!ARRAY_START}"
It works fine when run from the command line while in the same directory:
<my home> ~ % echo "$(./indirect.sh)"
Shell ID: 1 Index arrays from: 1
Problems arise when I source the script:
<my home> ~ % echo "$(. ~/indirect.sh)"
/Users/<me>/indirect.sh:28: bad substitution
I don’t understand why sourcing the script changes the behavior of the parameter expansion.
Is this expected behavior? If so, I’d be grateful if someone could explain it and hopefully, offer a work around.
The problem described in the original post has nothing to do with indirect expansion. The difference in behavior is a result of different shells being invoked depending on whether the script is “executed” or “sourced”. These differences reveal the basic flaw in deriving the shell from the $SHELL variable that underpins the script's design. If the shell defined in $SHELL does not match the shebang, the script will fail either when sourced or executed. An explanation follows.
Indirect expansion doesn’t offer value in the given scenario because values could just as easily be assigned directly. They’ll have to be assigned that way regardless given the different syntax used for indirect expansion between shells. In fact, other syntax differences between shells makes the entire premise for detecting the shell moot! However, putting that aside, the difference in behavior is a result of different shells being invoked based on whether the script is “executed” or “sourced”. The behavior of sourcing is well documented with numerous explanations on the web, but for context here’s how it works:
Executing a Script
Use the “./“ syntax to execute a script.
When run this way, the script executes in a sub-shell. Any changes the
script makes to it’s shell are applied to the sub-shell, not the shell
in which the script was launched, so those changes are lost when the
shell exits because the sub-shell in which it executed is destroyed as
well. For example, if the script changes the working directory, it
does so in the sub-shell. The working directory of the main shell that
launched the script is unchanged when the script terminates. If you
want to make changes to the shell in which the script was launched, it
must be sourced.
Sourcing a Script
Use the “source “ syntax to source a
script. When run this way, the script essentially becomes an argument
for the source command, which handles invoking the appropriate
execution. Some shells (e.g. ksh) use a single period “.” instead of
“source”.
When a script is executed with the “./“ syntax, the shebang at the top of the file is used to determine which shell to use. When a script is sourced, the shebang is ignored and the shell in which the script is launched is used instead. Also note that the period that appears in the “./“ command syntax used to execute a script, is not related to the period that’s occasionally used as an alias for the source command.
The script in the post uses bash in the shebang statement, so it works when executed because it’s run using bash. When it’s sourced from zsh, it encounters the incorrect indirect expansion syntax:
“${!A_VAR}"
The correct syntax is:
"${(P)A_VAR}"
However, correcting the syntax won’t help because it will then fail when executed. The shebang will invoke bash and the syntax will be wrong again. That renders indirection useless for accessing a variable designed to indicate the shell in use. More importantly, a design based on querying an environment variable for the shell is flawed due to differences in the shell that’s ultimately used depending on whether the script is executed or sourced.
To add to your answer (what I'm going to say is too long for a comment), I can not think of any application, why your script could be useful if not sourced. Actually, I came accross the need of such a script by myself in exactly one occasion:
Since I use as interactive shell not only zsh, but also sometimes bash, so I have written my .zshrc and .bashrc to set up everything (including defining variables and shell functions for interactive use). In order to safe work,
I try to put code which works under both bash and zsh into a single file (say: .commonrc), and my .zshrc and .bashrc have inside them a
source .commonrc
While many things are so different in bash and zsh, that I can't put them into .commonrc, some can, provided I do some tweaking. One reason for headache is obviously the different indexing of arrays, which you seemingly try to solve. So I have also a similar feature. However, I don't nee ca case construct for this. Instead, my .bashrc looks like this (using your naming of the variables):
...
declare -r ARRAY_START=0
source .commonrc
...
and my .zshrc looks like this:
...
declare -r ARRAY_START=1
source .commonrc
...
Since it does not happen that the .bashrc is run from a zsh and vice versa, I don't need to query what kind of shell I have.
I have a functions.sh script with a bunch of global functions that i want to use in other scripts. this functions script is written in bash (#!/bin/bash)
Those many scripts had been written over the years, so the older ones or with the #!/bin/sh (which is different from #!/bin/bash in solaris).
My question here is, when you call the functions.sh file (with . /path/to/functions.sh) from within a sh (not bash) script, is the shebang line of "functions.sh" interpreted ?
In a nutshell, can you call a bash written function script from another shell-type script (with proper shebang lines in both) ?
Thanks !
As long as you want to use the function you need to source the scripts and not execute it
source /path/to/functions.sh
or as per the POSIX standards, do
. ./path/to/functions.sh
from within the sh script, which is equivalent to including the contents of function.sh in the file at the point where the command is run.
You need to understand the difference between sourcing and executing a script.
Sourcing runs the script from the parent-shell in which the script is
invoked; all the environment variables, functions are retained until the
parent-shell is terminated (the terminal is closed, or the variables
are reset or unset),
Execute forks a new shell from the parent shell and those variables,functions
including your export variables are retained only in the sub-shell's
environment and destroyed at the end of script termination.
When you source a file, the shebang in that file is ignored (it is not on the first line since it is included in the caller script and is seen as comment).
When you include an old script with #!/bin/sh it will be handled as the shell of the caller. Most things written in /bin/sh will work in bash.
When you are running a sh or ksh script and you include (source) a bash file, all bash specific code will give problems.
I'm trying to access a variable declared by previous command (inside a Makefile).
Here's the Makefile:
all:
./script1.sh
./script2.sh
Here's the script declaring the variable I want to access,script1.sh:
#!/usr/bin/env bash
myVar=1234
Here's the script trying to access the variable previously defined, script2.sh:
#!/usr/bin/env bash
echo $myVar
Unfortunately when I run make, myVar isn't accessible. Is there an other way around? Thanks.
Make will run each shell command in its own shell. And when the shell exits, its environment is lost.
If you want variables from one script to be available in the next, there are constructs which will do this. For example:
all:
( . ./script1.sh; ./script2.sh )
This causes Make to launch a single shell to handle both scripts.
Note also that you will need to export the variable in order for it to be visible in the second script; unexported variables are available only to the local script, and not to subshells that it launches.
UPDATE (per Kusalananda's comment):
If you want your shell commands to populate MAKE variables instead of merely environment variables, you may have options that depend on the version of Make that you are running. For example, in BSD make and GNU make, you can use "variable assignment modifiers" including (from the BSD make man page):
!= Expand the value and pass it to the shell for execution and
assign the result to the variable. Any newlines in the result
are replaced with spaces.
Thus, with BSD make and GNU make, you could do this:
$ cat Makefile
foo!= . ./script1.sh; ./script2.sh
all:
#echo "foo=${foo}"
$
$ cat script1.sh
export test=bar
$
$ cat script2.sh
#!/usr/bin/env bash
echo "$test"
$
$ make
foo=bar
$
Note that script1.sh does not include any shebang because it's being sourced, and is therefore running in the calling shell, whatever that is. That makes the shebang line merely a comment. If you're on a system where the default shell is POSIX but not bash (like Ubuntu, Solaris, FreeBSD, etc), this should still work because POSIX shells should all understand the concept of exporting variables.
The two separate invocations of the scripts create two separate environments. The first script sets a variable in its environment and exits (the environment is lost). The second script does not have that variable in its environment, so it outputs an empty string.
You can not have environment variables pass between environments other than between the environments of a parent shell to its child shell (not the other way around). The variables passed over into the child shell are only those that the parent shell has export-ed. So, if the first script invoked the second script, the value would be outputted (if it was export-ed in the first script).
In a shell, you would source the first file to set the variables therein in the current environment (and then export them!). However, in Makefiles it's a bit trickier since there's no convenient source command.
Instead you may want to read this StackOverflow question.
EDIT in light of #ghoti's answer: #ghoti has a good solution, but I'll leave my answer in here as it explains a bit more verbosely about environment variables and what we can do and not do with them with regards to passing them between environments.
I am looking to execute a script but have it include another script before it executes. The problem is, the included script would be generated and the executed script would be unmodifiable. One solution I came up with, was to actually reverse the include, by having the include script as a wrapper, calling set to set the arguments for the executed script and then dotting/sourcing it. E.g.
#!/bin/bash
# Generated wrapper or include script.
: Performing some setup...
target_script=$1 ; shift
set -- "$#"
. "$target_script"
Where target_script is the script I actually want to run, importing settings from the wrapper.
However, the potential problem I face is that callers of the target script or even the target script itself may be expecting $0 to be set to the path of it's location on the file system. But because this wrapper approach overrides $0, the value of $0 may be unexpected and could produce undefined behaviour.
Is there another way to perform what is in effect, an LD_PRELOAD but in the scripted form, through bash without interfering with its runtime parameters?
I have looked at --init-file or --rcfile, but these only seem to be included for interactive shells.
Forcing interactive mode does seem to allow me to specify --rcfile:
$ bash --rcfile /tmp/x-include.sh -i /tmp/xx.sh
include_script: $0=bash, $BASH_SOURCE=/tmp/x-include.sh
target_script: $0=/tmp/xx.sh, $BASH_SOURCE=/tmp/xx.sh
Content of the x-include.sh script:
#!/bin/bash
echo "include_script: \$0=$0, \$BASH_SOURCE=$BASH_SOURCE"
Content of the xx.sh script:
#!/bin/bash
echo "target_script: \$0=$0, \$BASH_SOURCE=$BASH_SOURCE"
From the bash documentation:
When bash is started non-interactively, to run a shell script, for example, it looks for the variable BASH_ENV in
the environment, expands its value if it appears there, and uses the expanded value as the name of a file to read
and execute. Bash behaves as if the following command were executed:
if [ -n "$BASH_ENV" ]; then . "$BASH_ENV"; fi
but the value of the PATH variable is not used to search for the file name.
So that settles it then:
BASH_ENV=/tmp/x-include.sh /bin/bash /tmp/xx.sh