I understand that since | initiates a new process for the command(s) after the pipe, any shell command of the form cmd | cd newdir (where cmd does not change the current working directory) will leave the original process's working directory unchanged. (Not to mention that this is a bit silly since cd doesn't read input from stdin.)
However, on my machine (a CentOS 6 box, using bash, ksh, or zsh), it appears that the following command also fails to change directories:
cd newdir | cat
(Please ignore how silly it is to pipe output to cat here; I'm just trying to make a simple example.)
Why is this? Is there a way around this problem? Specifically, I'm trying to write an alias that uses popd but catches the output, discards stdout, and re-outputs stderr.
(For the curious, this is my current, non-working alias: popd 2>&1 >/dev/null | toerr && lsd. Here, toerr just catches stdin, outputs it to stderr, and returns the number of lines read/printed. lsd is a directory-name-and-contents printer that should only execute if the popd is successful. The reason I'm sending stderr to stdout, piping it, catching it, and re-outputting it on stderr is just to get it colored red using stderred, since my shell session isn't loaded with LD_PRELOAD, so Bash built-ins such as popd don't get the red-colored stderr.)
In bash, dash and ash, each command in a pipeline runs in a subshell.
In zsh, ksh, and bash with shopt -s lastpipe, all except the last command in the pipeline run in subshells.
Since cd -- as well as variables, shell options, ulimits and new file descriptors -- only affects the current process, their effects will not affect the parent shell.
Examples:
# Doesn't change directory
cd foo | cat
pwd
# Doesn't set $bar on default bash (but does on zsh and ksh)
echo foo | read bar
echo "$bar"
# Doesn't change the ulimit
ulimit -c 10000 2>&1 | grep "not permitted"
ulimit -c
The same also applies to other things that generate subshells. None of the following will change the directory:
# Command expansion creates a subshell
echo $(cd foo); pwd
# ( .. ) creates a subshell
( cd foo ); pwd
# Backgrounding a process creates a subshell
cd foo & pwd
To fix it, you have to rewrite your code to run anything that affects the environment in the main shell process.
In your particular case, you can consider using process substitution:
popd > /dev/null 2> >(toerr) && lsd
This has the additional benefit of only running lsd when popd is successful, rather than when toerr is successful like your version does.
Related
I know how to redirect stdout to a file:
exec > foo.log
echo test
this will put the 'test' into the foo.log file.
Now I want to redirect the output into the log file AND keep it on stdout
i.e. it can be done trivially from outside the script:
script | tee foo.log
but I want to do declare it within the script itself
I tried
exec | tee foo.log
but it didn't work.
#!/usr/bin/env bash
# Redirect stdout ( > ) into a named pipe ( >() ) running "tee"
exec > >(tee -i logfile.txt)
# Without this, only stdout would be captured - i.e. your
# log file would not contain any error messages.
# SEE (and upvote) the answer by Adam Spiers, which keeps STDERR
# as a separate stream - I did not want to steal from him by simply
# adding his answer to mine.
exec 2>&1
echo "foo"
echo "bar" >&2
Note that this is bash, not sh. If you invoke the script with sh myscript.sh, you will get an error along the lines of syntax error near unexpected token '>'.
If you are working with signal traps, you might want to use the tee -i option to avoid disruption of the output if a signal occurs. (Thanks to JamesThomasMoon1979 for the comment.)
Tools that change their output depending on whether they write to a pipe or a terminal (ls using colors and columnized output, for example) will detect the above construct as meaning that they output to a pipe.
There are options to enforce the colorizing / columnizing (e.g. ls -C --color=always). Note that this will result in the color codes being written to the logfile as well, making it less readable.
The accepted answer does not preserve STDERR as a separate file descriptor. That means
./script.sh >/dev/null
will not output bar to the terminal, only to the logfile, and
./script.sh 2>/dev/null
will output both foo and bar to the terminal. Clearly that's not
the behaviour a normal user is likely to expect. This can be
fixed by using two separate tee processes both appending to the same
log file:
#!/bin/bash
# See (and upvote) the comment by JamesThomasMoon1979
# explaining the use of the -i option to tee.
exec > >(tee -ia foo.log)
exec 2> >(tee -ia foo.log >&2)
echo "foo"
echo "bar" >&2
(Note that the above does not initially truncate the log file - if you want that behaviour you should add
>foo.log
to the top of the script.)
The POSIX.1-2008 specification of tee(1) requires that output is unbuffered, i.e. not even line-buffered, so in this case it is possible that STDOUT and STDERR could end up on the same line of foo.log; however that could also happen on the terminal, so the log file will be a faithful reflection of what could be seen on the terminal, if not an exact mirror of it. If you want the STDOUT lines cleanly separated from the STDERR lines, consider using two log files, possibly with date stamp prefixes on each line to allow chronological reassembly later on.
Solution for busybox, macOS bash, and non-bash shells
The accepted answer is certainly the best choice for bash. I'm working in a Busybox environment without access to bash, and it does not understand the exec > >(tee log.txt) syntax. It also does not do exec >$PIPE properly, trying to create an ordinary file with the same name as the named pipe, which fails and hangs.
Hopefully this would be useful to someone else who doesn't have bash.
Also, for anyone using a named pipe, it is safe to rm $PIPE, because that unlinks the pipe from the VFS, but the processes that use it still maintain a reference count on it until they are finished.
Note the use of $* is not necessarily safe.
#!/bin/sh
if [ "$SELF_LOGGING" != "1" ]
then
# The parent process will enter this branch and set up logging
# Create a named piped for logging the child's output
PIPE=tmp.fifo
mkfifo $PIPE
# Launch the child process with stdout redirected to the named pipe
SELF_LOGGING=1 sh $0 $* >$PIPE &
# Save PID of child process
PID=$!
# Launch tee in a separate process
tee logfile <$PIPE &
# Unlink $PIPE because the parent process no longer needs it
rm $PIPE
# Wait for child process, which is running the rest of this script
wait $PID
# Return the error code from the child process
exit $?
fi
# The rest of the script goes here
Inside your script file, put all of the commands within parentheses, like this:
(
echo start
ls -l
echo end
) | tee foo.log
Easy way to make a bash script log to syslog. The script output is available both through /var/log/syslog and through stderr. syslog will add useful metadata, including timestamps.
Add this line at the top:
exec &> >(logger -t myscript -s)
Alternatively, send the log to a separate file:
exec &> >(ts |tee -a /tmp/myscript.output >&2 )
This requires moreutils (for the ts command, which adds timestamps).
Using the accepted answer my script kept returning exceptionally early (right after 'exec > >(tee ...)') leaving the rest of my script running in the background. As I couldn't get that solution to work my way I found another solution/work around to the problem:
# Logging setup
logfile=mylogfile
mkfifo ${logfile}.pipe
tee < ${logfile}.pipe $logfile &
exec &> ${logfile}.pipe
rm ${logfile}.pipe
# Rest of my script
This makes output from script go from the process, through the pipe into the sub background process of 'tee' that logs everything to disc and to original stdout of the script.
Note that 'exec &>' redirects both stdout and stderr, we could redirect them separately if we like, or change to 'exec >' if we just want stdout.
Even thou the pipe is removed from the file system in the beginning of the script it will continue to function until the processes finishes. We just can't reference it using the file name after the rm-line.
Bash 4 has a coproc command which establishes a named pipe to a command and allows you to communicate through it.
Can't say I'm comfortable with any of the solutions based on exec. I prefer to use tee directly, so I make the script call itself with tee when requested:
# my script:
check_tee_output()
{
# copy (append) stdout and stderr to log file if TEE is unset or true
if [[ -z $TEE || "$TEE" == true ]]; then
echo '-------------------------------------------' >> log.txt
echo '***' $(date) $0 $# >> log.txt
TEE=false $0 $# 2>&1 | tee --append log.txt
exit $?
fi
}
check_tee_output $#
rest of my script
This allows you to do this:
your_script.sh args # tee
TEE=true your_script.sh args # tee
TEE=false your_script.sh args # don't tee
export TEE=false
your_script.sh args # tee
You can customize this, e.g. make tee=false the default instead, make TEE hold the log file instead, etc. I guess this solution is similar to jbarlow's, but simpler, maybe mine has limitations that I have not come across yet.
Neither of these is a perfect solution, but here are a couple things you could try:
exec >foo.log
tail -f foo.log &
# rest of your script
or
PIPE=tmp.fifo
mkfifo $PIPE
exec >$PIPE
tee foo.log <$PIPE &
# rest of your script
rm $PIPE
The second one would leave a pipe file sitting around if something goes wrong with your script, which may or may not be a problem (i.e. maybe you could rm it in the parent shell afterwards).
I'm trying to write a bash script where every command is passed through a function that evaluates the command using this line:
eval $1 2>&1 >>~/max.log | tee --append ~/max.log
An example of a case where it does not work is when trying to evaluate a cd command:
eval cd /usr/local/src 2>&1 >>~/max.log | tee --append ~/max.log
The part the causes the issue is the | tee --append ~/max.log part. Any idea why I'm experiencing issues?
From the bash(1) man page:
Each command in a pipeline is executed as a separate process (i.e., in a subshell).
Therefore, cd can not change the working directory of the current shell when used in a pipeline. To work around this restriction, the usual approach would be to group cd with other commands and redirect the output of the group command:
{
cd /usr/local/src
command1
command2
} | tee --append ~/max.log
Without breaking your existing design, you could instead handle cd specially in your filter function:
# eval all commands (will catch any cd output, but will not change directory):
eval $1 2>&1 >>~/max.log | tee --append ~/max.log
# if command starts with "cd ", execute it once more, but in the current shell:
[[ "$1" == cd\ * ]] && $1
Depending on your situation, this may not be enough: You may have to handle other commands that modify the environment or shell variables like set, history, ulimit, read, pushd, and popd as well. In that case it would probably be a good idea to re-think the program's design.
I have a Bash script that runs a program with parameters. That program outputs some status (doing this, doing that...). There isn't any option for this program to be quiet. How can I prevent the script from displaying anything?
I am looking for something like Windows' "echo off".
The following sends standard output to the null device (bit bucket).
scriptname >/dev/null
And if you also want error messages to be sent there, use one of (the first may not work in all shells):
scriptname &>/dev/null
scriptname >/dev/null 2>&1
scriptname >/dev/null 2>/dev/null
And, if you want to record the messages, but not see them, replace /dev/null with an actual file, such as:
scriptname &>scriptname.out
For completeness, under Windows cmd.exe (where "nul" is the equivalent of "/dev/null"), it is:
scriptname >nul 2>nul
Something like
script > /dev/null 2>&1
This will prevent standard output and error output, redirecting them both to /dev/null.
An alternative that may fit in some situations is to assign the result of a command to a variable:
$ DUMMY=$( grep root /etc/passwd 2>&1 )
$ echo $?
0
$ DUMMY=$( grep r00t /etc/passwd 2>&1 )
$ echo $?
1
Since Bash and other POSIX commandline interpreters does not consider variable assignments as a command, the present command's return code is respected.
Note: assignement with the typeset or declare keyword is considered as a command, so the evaluated return code in case is the assignement itself and not the command executed in the sub-shell:
$ declare DUMMY=$( grep r00t /etc/passwd 2>&1 )
$ echo $?
0
Try
: $(yourcommand)
: is short for "do nothing".
$() is just your command.
Like andynormancx' post, use this (if you're working in an Unix environment):
scriptname > /dev/null
Or you can use this (if you're working in a Windows environment):
scriptname > nul
This is another option
scriptname |& :
Take a look at this example from The Linux Documentation Project:
3.6 Sample: stderr and stdout 2 file
This will place every output of a program to a file. This is suitable sometimes for cron entries, if you want a command to pass in absolute silence.
rm -f $(find / -name core) &> /dev/null
That said, you can use this simple redirection:
/path/to/command &>/dev/null
In your script you can add the following to the lines that you know are going to give an output:
some_code 2>>/dev/null
Or else you can also try
some_code >>/dev/null
I want an advanced shell or command line in Unix which has the following features:
output to err and out are in different colours.
I should be able to highlight (or find) keywords in the output of the executing command.
indicator in the OS task-bar/title as a command is running or completed.
I am looking at an advanced shell that boosts productivity. Is there any alternative?
Re: output to err and out are in different colours ... can be done in Bash.
# colourize stderr in current shell
# note: use sed in line-buffering mode
(
exec 2> >(sed -l -e $'s/.*/\033[31m&\033[m/')
ls -ld / xxxxx
)
# colourize stderr & stdout in current shell
(
exec 1> >(sed -l -e $'s/.*/\033[32m&\033[m/') 2> >(sed -l -e $'s/.*/\033[31m&\033[m/')
ls -ld / xxxxx
)
That isn't a trivial proposition.
There are shells that work with the terminal to echo the currently executing command in the title bar, such as bash on MacOS X.
The commands are autonomous and do not, in general, colour-code their output. So, to get colour-coded output, the shell will have to capture the error outputs of the commands it runs, and arrange to display that information appropriately colour-coded.
Searching the output requires the terminal program to keep the output it displays in a searchable form, and some program (probably the terminal program or possibly the shell) will have to respond to searching operations.
Emacs allows you to run interactive shells such as bash or zsh.
rc works similarly when run in Plan9 -- I'm not sure about its Unix ports.
In a C program I can write argv[0] and the new name shows up in a ps listing.
How can I do this in bash?
You can do it when running a new program via exec -a <newname>.
Just for the record, even though it does not exactly answer the original poster's question, this is something trivial to do with zsh:
ARGV0=emacs nethack
I've had a chance to go through the source for bash and it does not look like there is any support for writing to argv[0].
I'm assuming you've got a shell script that you wish to execute such that the script process itself has a new argv[0]. For example (I've only tested this in bash, so i'm using that, but this may work elsewhere).
#!/bin/bash
echo "process $$ here, first arg was $1"
ps -p $$
The output will be something like this:
$ ./script arg1
process 70637 here, first arg was arg1
PID TTY TIME CMD
70637 ttys003 0:00.00 /bin/bash ./script arg1
So ps shows the shell, /bin/bash in this case. Now try your interactive shell's exec -a, but in a subshell so you don't blow away the interactive shell:
$ (exec -a MyScript ./script arg1)
process 70936 here, first arg was arg1
PID TTY TIME CMD
70936 ttys008 0:00.00 /bin/bash /path/to/script arg1
Woops, still showing /bin/bash. what happened? The exec -a probably did set argv[0], but then a new instance of bash started because the operating system read #!/bin/bash at the top of your script. Ok, what if we perform the exec'ing inside the script somehow? First, we need some way of detecting whether this is the "first" execution of the script, or the second, execed instance, otherwise the second instance will exec again, and on and on in an infinite loop. Next, we need the executable to not be a file with a #!/bin/bash line at the top, to prevent the OS from changing our desired argv[0]. Here's my attempt:
$ cat ./script
#!/bin/bash
__second_instance="__second_instance_$$"
[[ -z ${!__second_instance} ]] && {
declare -x "__second_instance_$$=true"
exec -a MyScript "$SHELL" "$0" "$#"
}
echo "process $$ here, first arg was $1"
ps -p $$
Thanks to this answer, I first test for the environment variable __second_instance_$$, based on the PID (which does not change through exec) so that it won't collide with other scripts using this technique. If it's empty, I assume this is the first instance, and I export that environment variable, then exec. But, importantly, I do not exec this script, but I exec the shell binary directly, with this script ($0) as an argument, passing along all the other arguments as well ($#). The environment variable is a bit of a hack.
Now the output is this:
$ ./script arg1
process 71143 here, first arg was arg1
PID TTY TIME CMD
71143 ttys008 0:00.01 MyScript ./script arg1
That's almost there. The argv[0] is MyScript like I want, but there's that extra arg ./script in there which is a consequence of executing the shell directly (rather than via the OS's #! processing). Unfortunately, I don't know how to get any better than this.
Update for Bash 5.0
Looks like Bash 5.0 adds support for writing to special variable BASH_ARGV0, so this should become far simpler to accomplish.
(see release announcement)
( exec -a foo bash -c 'echo $0' )
ps and others inspect two things, none of which is argv0: /proc/PID/comm (for the "process name") and /proc/PID/cmdline (for the command-line). Assigning to argv0 will not change what ps shows in the CMD column, but it will change what the process usually sees as its own name (in output messages, for example).
To change the CMD column, write to /proc/PID/comm:
echo -n mynewname >/proc/$$/comm; ps
You cannot write to or modify /proc/PID/cmdline in any way.
Process can set their own "title" by writing to the memory area in which argv & envp are located (note that this is different than setting BASH_ARGV0). This has the side effect of changing /proc/PID/cmdline as well, which is what some daemons do in order to prettify (hide?) their command lines. libbsd's setproctitle() does exactly that, but you cannot do that in Bash without support of external tools.
I will just add that this must be possible at runtime, at least in some environments. Assigning $0 in perl on linux does change what shows up in ps. I do not know how that is implemented, however. If I can find out, i'll update this.
edit:
Based on how perl does it, it is non-trivial. I doubt there is any bask built in way at runtime but don't know for sure. You can see how perl does sets the process name at runtime.
Copy the bash executable to a different name.
You can do this in the script itself...
cp /bin/bash ./new-name
PATH=$PATH:.
exec new-name $0
If you are trying to pretend you are not a shell script you can rename the script itself to something cool or even " " (a single space) so
exec new-name " "
Will execute bash your script and appears in the ps list as just new-name.
OK so calling a script " " is a very bad idea :)
Basically, to change the name
bash script
rename bash and rename the script.
If you are worried, as Mr McDoom. apparently is, about copying a binary to a new name (which is entirely safe) you could also create a symlink
ln -s /bin/bash ./MyFunkyName
./MyFunkyName
This way, the symlink is what appears in the ps list. (again use PATH=$PATH:. if you dont want the ./)