I'm not familiar with the exec command. A bash tutorial about how to lock files throws this:
exec 200>lockfile
flock 200
...
flock -u 200
I got that it is creating a file named lockfile and assigning it an FD of 200. Then the second command locks that file. When done working with the file, the last command unlocks it.
Doing so, any other concurrent instance of the same script will stay at that second line until the first instance unlocks the file. Cool.
Now, what I don't understand is what is exec doing.
Directly from the bash command line, both options seem to work:
exec 200>lockfile
200>lockfile
But when the second option is used in the script, a "Bad file descriptor error" is raised.
Why is exec needed to avoid the error?
--- edit ---
After some more "serious research", I've found an answer here. The exec command makes the FD stay for the entire script or current shell.
So doing:
200>lockfile flock 200
Would work. But later flock -u 200 would raise a "Bad FD error".
The manual seems to mention shell replacement with given command. What does that has to do with file descriptors?
This is explained in the second sentence:
exec: exec [-cl] [-a name] file [redirection ...]
Exec FILE, replacing this shell with the specified program.
If FILE is not specified, the redirections take effect in this
shell. [...]
Essentially, doing exec 42> foo.txt from inside myscript.sh opens foo.txt for writing on FD 42 in the current process.
This is similar to running ./myscript.sh 42> foo.txt from a shell in the first place, or using open and dup2 in a C program.
Related
Bash Script:
#!/bin/bash
...
exec {LOCK} > foo.out
flock -x ${LOCK}
...
I understand that exec without an argument simply redirects all the output of the current shell to the foo.out file. Questions:
What does the first argument to exec {LOCK} mean, given that it seems to have a special significance because it is in curly braces (but not ${...}).
What is the value of ${LOCK} and where did it come from (I don't think that I defined this variable)?
This is not valid or useful bash. It will just result in two different error messages.
Instead, the intended code was this:
#!/bin/bash
...
exec {LOCK}> foo.out
flock -x ${LOCK}
...
It uses:
{name}> to open for writing and assign fd number to name
exec to apply the redirection to the current, keeping the fd open for the duration of the shell
flock to lock the assigned fd, which it will inherit from the current shell
So effectively, it creates a mutex based on the file foo.out, ensuring that only one instance is allowed to run things after the flock at a time. Any other instances will wait until the previous one is done.
Here is what I finally figured out:
exec {LOCK}> foo.out
changes stdout of the current shell to the file foo.out. The fd for the open file is set the variable ${LOCK}. Setting fd to the {LOCK} variable is a feature of bash.
flock -x ${LOCK}
is simply locking using the file descriptor.
I have a shell script which writes all output to logfile
and terminal, this part works fine, but if I execute the script
a new shell prompt only appear if I press enter. Why is that and how do I fix it?
#!/bin/bash
exec > >(tee logfile)
echo "output"
First, when I'm testing this, there always is a new shell prompt, it's just that sometimes the string output comes after it, so the prompt isn't last. Did you happen to overlook it? If so, there seems to be a race where the shell prints the prompt before the tee in the background completes.
Unfortunately, that cannot fixed by waiting in the shell for tee, see this question on unix.stackexchange. Fragile workarounds aside, the easiest way to solve this that I see is to put your whole script inside a list:
{
your-code-here
} | tee logfile
If I run the following script (suppressing the newline from the echo), I see the prompt, but not "output". The string is still written to the file.
#!/bin/bash
exec > >(tee logfile)
echo -n "output"
What I suspect is this: you have three different file descriptors trying to write to the same file (that is, the terminal): standard output of the shell, standard error of the shell, and the standard output of tee. The shell writes synchronously: first the echo to standard output, then the prompt to standard error, so the terminal is able to sequence them correctly. However, the third file descriptor is written to asynchronously by tee, so there is a race condition. I don't quite understand how my modification affects the race, but it appears to upset some balance, allowing the prompt to be written at a different time and appear on the screen. (I expect output buffering to play a part in this).
You might also try running your script after running the script command, which will log everything written to the terminal; if you wade through all the control characters in the file, you may notice the prompt in the file just prior to the output written by tee. In support of my race condition theory, I'll note that after running the script a few times, it was no longer displaying "abnormal" behavior; my shell prompt was displayed as expected after the string "output", so there is definitely some non-deterministic element to this situation.
#chepner's answer provides great background information.
Here's a workaround - works on Ubuntu 12.04 (Linux 3.2.0) and on OS X 10.9.1:
#!/bin/bash
exec > >(tee logfile)
echo "output"
# WORKAROUND - place LAST in your script.
# Execute an executable (as opposed to a builtin) that outputs *something*
# to make the prompt reappear normally.
# In this case we use the printf *executable* to output an *empty string*.
# Use of `$ec` is to ensure that the script's actual exit code is passed through.
ec=$?; $(which printf) ''; exit $ec
Alternatives:
#user2719058's answer shows a simple alternative: wrapping the entire script body in a group command ({ ... }) and piping it to tee logfile.
An external solution, as #chepner has already hinted at, is to use the script utility to create a "transcript" of your script's output in addition to displaying it:
script -qc yourScript /dev/null > logfile # Linux syntax
This, however, will also capture stderr output; if you wanted to avoid that, use:
script -qc 'yourScript 2>/dev/null' /dev/null > logfile
Note, however, that this will suppress stderr output altogether.
As others have noted, it's not that there's no prompt printed -- it's that the last of the output written by tee can come after the prompt, making the prompt no longer visible.
If you have bash 4.4 or newer, you can wait for your tee process to exit, like so:
#!/usr/bin/env bash
case $BASH_VERSION in ''|[0-3].*|4.[0-3]) echo "ERROR: Bash 4.4+ needed" >&2; exit 1;; esac
exec {orig_stdout}>&1 {orig_stderr}>&2 # make a backup of original stdout
exec > >(tee -a "_install_log"); tee_pid=$! # track PID of tee after starting it
cleanup() { # define a function we'll call during shutdown
retval=$?
exec >&$orig_stdout # Copy your original stdout back to FD 1, overwriting the pipe to tee
exec 2>&$orig_stderr # If something overwrites stderr to also go through tee, fix that too
wait "$tee_pid" # Now, wait until tee exits
exit "$retval" # and complete exit with our original exit status
}
trap cleanup EXIT # configure the function above to be called during cleanup
echo "Writing something to stdout here"
I want to execute a bash script in the current shell, not a subshell/subprocess. I believe using exec allows you to do that.
But if I run:
exec ./my_script.sh
the shell will exit and it will say "process completed". Is there a way to execute a script in the same shell somehow w/o exiting the process.
note the only thing my_script.sh does is:
export foo=bar
but if there was some error in my script, it didn't appear in the console.
As #Snakienn has said, you can / should use the "." builtin command to "source" the file containing commands; e.g.
. ./my_script.sh
What this does is to temporarily change where the shell is reading commands from. Commands and other shell input are read from the file as if they were being read from the terminal. When the shell gets to the end-of-file, it switches back to taking input from the console.
This is called "sourcing". (Not "execing".) And indeed the shell accepts source as an alternative name for the . command.
The exec command does something completely different. As the bash manual entry says:
exec [-cl] [-a name] [command [arguments]]
If command is specified, it replaces the shell. No new process is created. The arguments become the arguments to command.
The concept and terminology of exec comes from early UNIX (i.e Unix V6 or earlier) where the syscalls for running a child command were fork and exec. The procedure was:
fork the current process creating a clone of current process (the child)
in the child process exec the new command
in the parent process, wait for the child process to complete
you can try . ./my_script.sh.
The first dot stands for the current shell.
man page says:-
exec [-cl] [-a name] [command [arguments]]
If command is specified, it replaces the shell. No new process
is created. The arguments become the arguments to command. If
the -l option is supplied, the shell places a dash at the
beginning of the zeroth argument passed to command.This is
what login(1) does. The -c option causes command to be executed
with an empty environment.
if you want to execute a bash script in the current shell,you can try
bash my_script.sh or ./my_script.sh
Looking for some basic help in shell programming.
Suppose we have a command known as foobar, then what is the effect of shell invocation
exec foobar
exec 2> /var/log/foobar.log
The first exec command should only be used in a script — not at a command line terminal. It replaces the shell with the program foobar, instead of running it as a separate child process. Any commands in the script after the exec foobar will not be executed (even if the shell fails to find foobar to execute); if it is an interactive terminal session, it will report the error and continue.
exec [-cl] [-a name] [command [arguments]]
If command is supplied, it replaces the shell without creating a new process. If the -l option is supplied, the shell places a dash at the beginning of the zeroth argument passed to command. This is what the login program does. The -c option causes command to be executed with an empty environment. If -a is supplied, the shell passes name as the zeroth argument to command. If command cannot be executed for some reason, a non-interactive shell exits, unless the execfail shell option is enabled. In that case, it returns failure. An interactive shell returns failure if the file cannot be executed.
The second exec (with I/O redirection but no command) changes things so that the standard error stream goes to the file /var/log/foobar.log. Any further error messages from the shell, or from commands executed by the shell, go to the log file (unless there's another lot of I/O redirection).
If no command is specified, redirections may be used to affect the current shell environment. If there are no redirection errors, the return status is zero; otherwise the return status is non-zero.
exec foobar
will replace your shell process with foobar. I do not think you mean exec 2>/var/log/foobar.log but rather exec foobar 2>/var/log/foobar.log. This will do the same with sending 2 i.e. standard error messages to specified log file. You can read man page here.
exec(1) command is similar to exec(3) call. It replaces the code segment of calling process from that of called program. 1 and 3 signify man page sections.
I'd like to source a script (it sets up variables in my shell):
source foo.sh args
But under flock so that only one instance operates at a time (it does a lot of disk access which I'd like to ensure is serialized).
$ source flock lockfile foo.sh args
-bash: source: /usr/bin/flock: cannot execute binary file
and
$ flock lockfile source foo.sh args
flock: source: Success
don't work.
Is there some simple syntax for this I'm missing? Let's assume I can't edit foo.sh to put the locking commands inside it.
You can't source a script directly via flock because it is an external command and source is a shell builtin. You actually have two problems because of this:
flock doesn't know any command called source because it's built into bash
Even if flock could run it, the changes would not affect the state of the calling shell as you'd want with source because it's happening in a child process.
And, passing flock to source won't work because source expects a script. To do what you want, you need to lock by fd. Here's an example
#!/bin/bash
exec 9>lockfile
flock 9
echo whoopie
sleep 5
flock -u 9
Run two instances of this script in the same directory at the same time and you will see one wait for the other.