Is there a way to find out all the files that are sourced by bash?
Alternately, is there a single point of entry (or a first point of entry) where I can go to follow and find this information by adding a set -x at the top?
(By single point of entry, I do not mean ~/.bashrc or ~/.bash_profile because some other file higher in the source chain tells bash to load these above files in the first place).
Reviving this question because there is an automation for this:
Execute bash and carve it out of the output. -li is login interactively, -x prints out what bash is doing internally, and -c exit just tells bash to terminate immediately. Using sed to filter out the source command or the . alias.
/bin/bash -lixc exit 2>&1 | sed -n 's/^+* \(source\|\.\) //p'
There is no easy catch-all answer here, it depends on the combination of login/interactive attributes.
A login shell will source /etc/profile, and only the first one it finds among ~/.bash_profile, ~/.bash_login, and ~/.profile. You could call these independent points of entry: /etc/profile doesn't need to explicitly source one of the others, it's bash that does it.
For non-login interactive, you have /etc/bash.bashrc and ~/.bashrc, again independent.
For non-login non-interactive, the single point of entry is $BASH_ENV, if defined.
You can find the official description at the GNU Bash manual under Bash startup files.
There are several places where the process to load all startup files start.
The table in this link will make it clear:
Interactive login /etc/profile
Interactive non-login /etc/bash.bashrc
Script $BASH_ENV
Understanding login as either an Interactive login or a non-interactive shell called with the option --login. From man bash:
When bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists.
However, the most common call to an interactive shell is su - $USER which use a - as the first character of the command called (not --login).
That is the default.
Nothing prevent you from editing those files and add in /etc/profile something like:
if [ -f /etc/bash.bashrc ]; then
source /etc/bash.bashrc
fi
Which will ensure that /etc/bash.bashrc would be sourced in all cases of Interactive shells.
Care should be taken to avoid duplicating variables or actions (sourced in both files). Defining a variable and checking that it has been already set before some actions would make this process more reliable.
The starting point for scripts, from the point of view of bash, is the variable $BASH_ENV which has to be set in the environment before bash is called. That expands the search to other shells or programs that may call bash. There is no single definitive solution in this case, only what is the usual practice.
The usual practice is to not use $BASH_ENV at all, so bash would start with all the compiled-in options only.
Related
I am trying to create a simple script to zip a list of files each into its own zip file. The files are big, so I a trying to send the to background using ampersand. It works as I can see the temporary files filling up and after some time the files are created, but issuing the 'jobs' command does not list the jobs. What am I doing wrong?
#!/bin/ksh
for file in $*;do
bash -c "zip -q $file.zip $file" &
done
NATIVE CSH SOLUTION
As I said earlier, shell scripts execute in a subshell and the parent shell will not be able to list the jobs of a subshell. In order to use jobs, the jobs need to be running in the same shell.
This can be achieved by source-ing the file. Since your default shell is csh the file should contain these lines according to the csh syntax
# not a script. no need for shebang
# sourcing this file **in csh** will
# start quiet zip jobs in the background
# for all files in the working dir (*)
foreach file in (*)
zip -q "$file.zip" "$file" &
end
Keeping this file in an easily accessible location and running source /path/to/file will give you what you need.
This is the only way to do it in csh for the following reasons:
cannot be a shell script. jobs will not be possible
csh does not support shell functions
setting alias not easy due csh's foreach syntax
But also consider a few of these alternatives
A. The organisation allows for changing the login shell
Change the shell to one that allows shell functions (e.g. to bash)
chsh -s `which bash` $USER
Logout and login or simply execute bash (or your shell of choice) to start a new shell
Check you are in the right shell echo $0
Add a function to your user-level login script (~/.bashrc for bash)
# executing this command appends a bash function named `zipAll` to ~/.bashrc
# modify according to your choice of shell
cat << 'EOF' >> ~/.bashrc
zipAll() {
for file in *; do
zip -q "$file.zip" "$file" &
done
}
EOF
The function zipAll should be available from the next login onwards.
B. The organisation does not allow changing login shell
Simply execute bash (or your shell of choice) to start a new shell
Follow steps A3 to A4
Temporarily switch to a new shell with bash (or your shell of choicd) when you need this function
C. B; but you want to use bash (or other shell)
I do not know if this is a good solution. Hopefully someone will point out the ill-effects of it. Hopefully your organisation simply allows you to change the login shell
Seeing as your default shell is csh, add a line to ~/.cshrc to start bash (or your choice of shell)
echo 'bash --login' >> ~/.cshrc
Follow steps A2 to A4
Copy necessary lines from existing ~/.cshrc to ~/.bashrc (or the file corresponding to your shell)
Confusion regarding zip usage was oversight on my part. Apologies.
NB: The syntax zip -q $file $file.zip does not work with my version. But I retain it assuming that it works on OP's system
PS: The command that works with my version of zip is zip -q $file.zip file
I wrote simple command that lets me run the last N commands from terminal history. It looks like this: $> r 3 which will replay the last 3 commands.
I have the following alias in my bash profile:
alias r="history -w; runlast $1"
And then the following simple perl script for the runlast command:
#!/usr/bin/env perl
use strict;
use warnings;
my $lines = $ARGV[0] || exit;
my #last_commands = split /\n/,
`bash -ic 'set -o history; history' | tail -$lines`;
#last_commands =
grep { $_ !~ /(^r |^history |^rm )/ }
map { local $_ = $_; s/^\s+\d+\s+//; $_ }
#last_commands;
foreach my $cmd (#last_commands) {
system("$cmd");
}
This works but my bash profile has aliases and other features (e.g. color output) I want the perl script to have access to. How do I load the bash profile for perl so it runs the bash commands with my bash profile? I read somewhere that if you "source the bash profile" for perl you can get it to work. So I tried adding source ~/.bash_profile; to my r command alias but that didn't have an effect. I'm not sure if I was doing that correctly, though.
The system forks a process in which it runs a shell, which is non-login and non-interactive; so no initialization is done and you get no aliases. Also note that the shell used is /bin/sh, which is generally a link to another shell. This is often bash but not always, so run bash explicitly.
To circumvent this you need to source the file with aliases, but as bash man page says
Aliases are not expanded when the shell is not interactive, unless the expand_aliases shell option is set using shopt (see the description of shopt under SHELL BUILTIN COMMANDS below).
Thus you need shopt -s expand_aliases, as mentioned. But there is another screw: on that same physical line aliases are not yet available; so it won't work like this in a one-liner.
I'd also recommend to put aliases in .bashrc, or in a separate file that is sourced.
Solutions
Add shopt -s expand_aliases to your ~/.bashrc, and before the aliases are defined (or the file with them sourced), and run bash as a login shell
system('/bin/bash', '-cl', 'source ~/.bashrc; command');
where -l is short for --login.
In my tests the source ~/.bashrc wasn't needed; however, the man page says
When bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable.
and goes on to specify that ~/.bashrc is read when an interactive shel that is not login runs. So I added explicit sourcing.
In my tests sourcing .bashrc (with shopt added) while not running as a login shell didn't work, and I am not sure why.
This is a little heavy-handed. Also, initialization may be undesirable to run from a script.
Source ~/.bashrc and issue shopt command, and then a newline before the command
system('/bin/bash', '-c',
'source ~/.bashrc; shopt -s expand_aliases\ncommand');
Really. It works.
Finally, is this necessary? It asks for trouble, and there is probably a better design.
Other comments
The backticks (qx) is context-aware. If it's used in list context – its return assigned to an array, for example – then the command's output is returned as a list of lines. When you use it as the argument for split then it is in the scalar context though, when all output is returned in one string. Just drop split
my #last_commands = `bash -ic 'set -o history; history $lines`;
where I also use history N to get last N lines. In this case the newlines stay.
history N returns last N lines of history so there is no need to pipe to last
Regex substitution in a map can be done without changing the original
map { s/^\s+\d+\s+//r } #last_commands;
With /r modifier the s/// operator returns the new string, not changing the original. This "non-destructive substitution" has been available since v5.14
No need to explicitly use $_ in the last grep, and no need for parenthesis in regex
grep { not /^r |^history |^rm ?/ } ...
or
grep { not /^(?:r|history|rm)[ ]?/ } ...
where parens are now needed, but as it is only for grouping the ?: makes it not capture the match. I use [ ] to emphasize that that space is intended; this is not necessary.
I also added ? to make space optional since history (and r?) may have no space.
The proper solution is to have your Perl script just print the commands, and make your current interactive shell eval the string printed from your history. (I would probably get rid of Perl entirely but that's beside the point here.)
If the commands get evaluated in the current shell, you avoid many contextual problems which would be very hard or even intractable with system() or generally anything involving a new process. For example, a subprocess cannot have access to non-exported variables in the current shell. var="foo", echo "$var"; r 1 is going to be very hard to solve correctly with your current approach. Using the current interactive shell will also naturally and easily solve the problems you were having with trying to get a noninteractive subshell act like an interactive one.
Aliases suck anyway, so let's redefine r as a function:
r(){
history -w
eval $(printlast "$1")
}
... where refactoring runlast into a different script printlast is a trivial additional requirement. Or maybe just turn it into a (much simpler!) shell function:
printlast () {
history "$1" |
perl -ne 's/^\s*\d+\s+\*?//; print unless m/^(history|rm?)($|\s)'
}
With this, you can also get rid of history -w from the r definition.
Notice how we are using Perl where it is useful; but the main functionality makes sense to keep in the shell when you're dealing with the shell.
You can't source in a Bash script into a Perl script. The bash_profile has to be sourced in by the shell that executes the command. When Perl runs system, it forks a new shell each time.
You have to source in the bash_profile for each command that you run through system:
system('source ~/.bash_profile; ' + $cmd);
One more thing, system invokes a non-interactive shell. So, your Bash aliases defined in .bash_profile won't work unless you invoke:
shopt -s expand_aliases
inside that script
I am using Bash
$ echo $SHELL
/bin/bash
and starting about a year ago I stopped using Shebangs with my Bash scripts. Can
I benefit from using #!/bin/sh or #!/bin/bash?
Update: In certain situations a file is only treated as a script with the
Shebang, example
$ cat foo.sh
ls
$ cat bar.sh
#!/bin/sh
ls
$ file foo.sh bar.sh
foo.sh: ASCII text
bar.sh: POSIX shell script, ASCII text executable
On UNIX-like systems, you should always start scripts with a shebang line. The system call execve (which is responsible for starting programs) relies on an executable having either an executable header or a shebang line.
From FreeBSD's execve manual page:
The execve() system call transforms the calling process into a new
process. The new process is constructed from an ordinary file, whose
name is pointed to by path, called the new process file.
[...]
This file is
either an executable object file, or a file of data for an interpreter.
[...]
An interpreter file begins with a line of the form:
#! interpreter [arg]
When an interpreter file is execve'd, the system actually execve's the
specified interpreter. If the optional arg is specified, it becomes the
first argument to the interpreter, and the name of the originally
execve'd file becomes the second argument
Similarly from the Linux manual page:
execve() executes the program pointed to by filename. filename must be
either a binary executable, or a script starting with a line of the
form:
#! interpreter [optional-arg]
In fact, if a file doesn't have the right "magic number" in it's header, (like an ELF header or #!), execve will fail with the ENOEXEC error (again from FreeBSD's execve manpage):
[ENOEXEC] The new process file has the appropriate access
permission, but has an invalid magic number in its
header.
If the file has executable permissions, but no shebang line but does seem to be a text file, the behaviour depends on the shell that you're running in.
Most shells seem to start a new instance of themselves and feed it the file, see below.
Since there is no guarantee that the script was actually written for that shell, this can work or fail spectacularly.
From tcsh(1):
On systems which do not understand the `#!' script interpreter conven‐
tion the shell may be compiled to emulate it; see the version shell
variable. If so, the shell checks the first line of the file to see if
it is of the form `#!interpreter arg ...'. If it is, the shell starts
interpreter with the given args and feeds the file to it on standard
input.
From FreeBSD's sh(1):
If the program is not a normal executable file (i.e., if it
does not begin with the “magic number” whose ASCII representation is
“#!”, resulting in an ENOEXEC return value from execve(2)) but appears to
be a text file, the shell will run a new instance of sh to interpret it.
From bash(1):
If this execution fails because the file is not in executable format,
and the file is not a directory, it is assumed to be a shell script, a
file containing shell commands. A subshell is spawned to execute it.
You cannot always depend on the location of a non-standard program like bash. I've seen bash in /usr/bin, /usr/local/bin, /opt/fsf/bin and /opt/gnu/bin to name a few.
So it is generally a good idea to use env;
#!/usr/bin/env bash
If you want your script to be portable, use sh instead of bash.
#!/bin/sh
While standards like POSIX do not guarantee the absolute paths of standard utilities, most UNIX-like systems seem to have sh in /bin and env in /usr/bin.
Scripts should always begin with a shebang line. If a script doesn't start with this, then it may be executed by the current shell. But that means that if someone who uses your script is running a different shell than you do, the script may behave differently. Also, it means the script can't be run directly from a program (e.g. the C exec() system call, or find -exec), it has to be run from a shell.
You might be interested in an early description by Dennis M Ritchie (dmr) who invented the #! :
From uucp Thu Jan 10 01:37:58 1980
.>From dmr Thu Jan 10 04:25:49 1980 remote from research
The system has been changed so that if a file
being executed begins with the magic characters #! , the rest of the
line is understood to be the name of an interpreter for the executed
file. Previously (and in fact still) the shell did much of this job;
it automatically executed itself on a text file with executable mode
when the text file's name was typed as a command. Putting the facility
into the system gives the following benefits.
1) It makes shell scripts more like real executable files, because
they can be the subject of 'exec.'
2) If you do a 'ps' while such a command is running, its real name
appears instead of 'sh'. Likewise, accounting is done on the basis of
the real name.
3) Shell scripts can be set-user-ID.
4) It is simpler to have alternate shells available; e.g. if you like
the Berkeley csh there is no question about which shell is to
interpret a file.
5) It will allow other interpreters to fit in more smoothly.
To take advantage of this wonderful opportunity, put
#! /bin/sh
at the left margin of the first line of your shell scripts. Blanks
after ! are OK. Use a complete pathname (no search is done). At the
moment the whole line is restricted to 16 characters but this limit
will be raised.
Hope this helps
If you write bash scripts, i.e. non portable scripts containing bashisms, you should keep using the #!/bin/bash shebang just to be sure the correct interpreter is used. You should not replace the shebang by #!/bin/sh as bash will run in POSIX mode so some of your scripts might behave differently.
If you write portable scripts, i.e. scripts only using POSIX utilities and their supported options, you might keep using #!/bin/sh on your system (i.e. one where /bin/sh is a POSIX shell).
It you write stricly conforming POSIX scripts to be distributed in various platforms and you are sure they will only be launched from a POSIX conforming system, you might and probably should remove the shebang as stated in the POSIX standard:
As it stands, a strictly conforming application must not use "#!" as the first two characters of the file.
The rationale is the POSIX standard doesn't mandate /bin/sh to be the POSIX compliant shell so there is no portable way to specify its path in a shebang. In this third case, to be able to use the 'find -exec' syntax on systems unable to run a shebangless still executable script, you can simply specify the interpreter in the find command itself, eg:
find /tmp -name "*.foo" -exec sh -c 'myscript "$#"' sh {} +
Here, as sh is specified without a path, the POSIX shell will be run.
The header is useful since it specifies which shell to use when running the script. For example, #!/bin/zsh would change the shell to zsh instead of bash, where you can use different commands.
For example, this page specifies the following:
Using #!/bin/sh, the default Bourne shell in most commercial variants
of UNIX, makes the script portable to non-Linux machines, though you
sacrifice Bash-specific features ...
TL;DR: always in scripts; please not in source'd scripts
Always in your parent
FYI: POSIX compliant is #!/bin/bash, not #!/bin/sh
You want to clarify this so that nothing else overrides the interpreter your script is made for.
You don't want a user at the terminal using zsh to have trouble if your script was written for POSIX bash scripts.
You don't want to run source in your #!/bin/bash unrecognized by #!/bin/sh, someone in an sh terminal have it break the script because it is expecting the simple/POSIX . for including source'd files
You don't want e.g. zsh features - not available in other interpreters - to make their way into your bash code. So, put #!/bin/bash in all your script headers. Then, any of your zsh habits in your script will break so you know to remove them before your roll-out.
It's probably best, especially so POSIX-compliant scripts don't break in a terminal like zsh.
Not expected for included source scripts
FYI: POSIX compliant for sourcing text in a BASH script is ., not source
You can use either for sourcing, but I'll do POSIX.
Standard "shebanging" for all scripting:
parent.sh:
#!/bin/bash
echo "My script here"
. sourced.sh # child/source script, below
sourced.sh:
echo "I am a sourced child script"
But, you are allowed to do this...
sourced.sh: (optional)
#!/bin/bash
echo "I am a sourced child script"
There, the #!/bin/bash "shebang" will be ignored. The main reason I would use it is for syntax highlighting in my text editor. However, in the "proper" scripting world, it is expected that your rolled-out source'd script will not contain the shebang.
In addition to what the others said, the shebang also enables syntax highlighting in some text editors, for example vim.
$SHELL and #!/bin/bash or #!/bin/sh are different.
To start, #!/bin/sh is a symlink to /bin/bash on most Linux systems (on Ubuntu it is now /bin/dash)
But on whether to start with /bin/sh or /bin/bash:
Bash and sh are two different shells. Basically bash is sh, with more
features and better syntax. Most commands work the same, but they are
different.
Just assume if you're writing a bash script, stick with /bin/bash and not /sh because problems can arise.
$SHELL does not necessarily reflect the currently running shell.
Instead, $SHELL is the user's preferred shell, which is typically the
one set in /etc/passwd. If you start a different shell after logging
in, you can not necessarily expect $SHELL to match the current shell
anymore.
This is mine for example, but it could also be /root:/bin/dash or /root:/bin/sh depending on which shell you have input in passwd. So to avoid any problems, keep the passwd file at /bin/bash and then using $SHELL vs. #!/bin/bash wouldn't matter as much.
root#kali:~/Desktop# cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
Sources:
http://shebang.mintern.net/bourne-is-not-bash-or-read-echo-and-backslash/
https://unix.stackexchange.com/questions/43499/difference-between-echo-shell-and-which-bash
http://man.cx/sh
http://man.cx/bash
I have a hard time with interactive and non-interactive shells. I don't understand which is which.
For example, I have read that non interactive shells usually check for the BASH_ENV variable on their startup and execute whatever it points to.
So, what I did is I set the BASH_ENV to point to some script which only echoes OK. Then I typed in bash in terminal and this script echoed OK. But why? Didn't I call yet another INTERACTIVE shell by typing bash in terminal, and not the other way around? Why did it execute the bash_env? I'm on linux mint maya.
The only thing you can be certain of is what's shown in the manpage for bash (see INVOCATION) - that lists in details what startup files are run in each instance.
However, there's nothing stopping (for example) one of those startup files running other files which would normally not be run.
By way of example, if .bash_profile had the following line:
. ~/.profile
it would also run the .profile script.
In fact the manpage states:
When bash is started non-interactively, to run a shell script, for example, it looks for the variable BASH_ENV in the environment, expands its value if it appears there, and uses the expanded value as the name of a file to read and execute. Bash behaves as if the following command were executed:
if [ -n "$BASH_ENV" ]; then . "$BASH_ENV"; fi
So, if you put that exact line in your startup scripts for an interactive shell like ~/.bash_profile, you'll also source the file pointed to by BASH_ENV.
Your best bet is to examine the INVOCATION section to find out which of the files will run, and then track through them (with something like set -x at the top of the script) to see what's getting called from where.
If memory serves, Bash is only interactive if you tell it, example
bash -i
So, by you calling just bash you invoked a non-interactive Bash.
More info
-i
If the -i option is present, the shell is interactive.
Why doesn't the following work?
$ alias sayHello='/bin/echo "Hello world!"'
$ sayHello
Hello world!
$ nohup sayHello
nohup: appending output to `nohup.out'
nohup: cannot run command `sayHello': No such file or directory
(the reason I ask this question is because I've aliased my perl and python to different perl/python binaries which were optimized for my own purposes; however, nohup gives me troubles if I don't supply full path to my perl/python binaries)
Because the shell doesn't pass aliases on to child processes (except when you use $() or ``).
$ alias sayHello='/bin/echo "Hello world!"'
Now an alias is known in this shell process, which is fine but only works in this one shell process.
$ sayHello
Hello world!
Since you said "sayHello" in the same shell it worked.
$ nohup sayHello
Here, a program "nohup" is being started as a child process. Therefore, it will not receive the aliases.
Then it starts the child process "sayHello" - which isn't found.
For your specific problem, it's best to make the new "perl" and "python" look like the normal ones as much as possible. I'd suggest to set the search path.
In your ~/.bash_profile add
export PATH="/my/shiny/interpreters/bin:${PATH}"
Then re-login.
Since this is an environment variable, it will be passed to all the child processes, be they shells or not - it should now work very often.
For bash: Try doing nohup 'your_alias'. It works for me. I don't know why back quote is not shown. Put your alias within back quotes.
With bash, you can invoke a subshell interactively using the -i option. This will source your .bashrc as well as enable the expand_aliases shell option. Granted, this will only work if your alias is defined in your .bashrc which is the convention.
Bash manpage:
If the -i option is present, the shell is interactive.
expand_aliases: If set, aliases are expanded as described above under ALIASES. This option is enabled by default for interactive shells.
When an interactive shell that is not a login shell is started, bash reads and executes commands from /etc/bash.bashrc and ~/.bashrc, if these files exist.
$ nohup bash -ci 'sayHello'
If you look at the Aliases section of the Bash manual, it says
The first word of each simple command, if unquoted, is checked to see
if it has an alias.
Unfortunately, it doesn't seem like bash has anything like zsh's global aliases, which are expanded in any position.