This answer, "How to profile a bash shell script?", seems to nearly perfectly cover what I'm trying to accomplish here. I currently have some zsh scripts that modify the prompt, however I think some updates to oh-my-zsh have evoked some issues that I need to hunt down. The sluggishness from time to time is unbearable.
To this end, how would you adapt the prompt sections in this example answer to work with zsh vs bash?
Presently I have modified /etc/zshenv such that it has the initial suggested code from the example:
PS4='+ $(date "+%s.%N")\011 '
exec 3>&2 2>/tmp/bashstart.$$.log
set -x
And my ~/.zshrc has the following appended to it's tail:
set +x
exec 2>&3 3>&-
Of course these are not valid for ZSH shell customization. My prompt rendering code utilizes oh-my-zsh customizations. I could prepend the appropriate code to the prompt I suppose or I'm open to other suggestions.
Calling date for each command will fork and exec, which adds overhead which may interfere with your measurements.
Instead, you could use
PS4=$'+ %D{%s.%6.}\011 '
to log timestamps with lower overhead (up to millisecond precision).
For some notes on processing the resulting logs, see http://blog.xebia.com/profiling-zsh-shell-scripts/
You may need to do
setopt prompt_subst
if it's not already.
Also, in order to interpret the octal escape for tab, use $'':
PS4=$'+ $(date "+%s.%N")\011 '
You may also find some of these escapes to be useful:
%? The return status of the last command executed just before the prompt.
%_ The status of the parser, i.e. the shell constructs (like `if' and `for') that have been started on the command
line. If given an integer number that many strings will be printed; zero or negative or no integer means print as
many as there are. This is most useful in prompts PS2 for continuation lines and PS4 for debugging with the
XTRACE option; in the latter case it will also work non-interactively.
%i The line number currently being executed in the script, sourced file, or shell function given by %N. This is most
useful for debugging as part of $PS4.
%I The line number currently being executed in the file %x. This is similar to %i, but the line number is always a
line number in the file where the code was defined, even if the code is a shell function.
%L The current value of $SHLVL.
%N The name of the script, sourced file, or shell function that zsh is currently executing, whichever was started
most recently. If there is none, this is equivalent to the parameter $0. An integer may follow the `%' to spec‐
ify a number of trailing path components to show; zero means the full path. A negative integer specifies leading
components.
%x The name of the file containing the source code currently being executed. This behaves as %N except that function
and eval command names are not shown, instead the file where they were defined.
Related
I noticed that my script was ignoring my positional arguments in old terminal tabs, but working on recently created ones, so I decided to reduce it to the following:
TAG=test
while getopts 't:' c
do
case $c in
t)
TAG=$OPTARG
;;
esac
done
echo $TAG
And running the script I have:
~ source my_script
test
~ source my_script -t "test2"
test2
~ source my_script -t "test2"
test
I thought it could be that c was an special used variable elsewhere but after changing it to other names I had the exact same problem. I also tried adding a .sh extension to the file to see it that was a problem, but nothing worked.
Am I doing something wrong ? And why does it work the first time, but not the subsequent attempts ?
I am on MacOS and I use zsh.
Thank you very much.
The problem is that you're using source to run the script (the . command does the same thing). This makes it run in your current (interactive) shell (rather than a subprocess, like scripts normally do). This means it uses the same variables as the current shell, which is necessary if you want it to change those variables, but it can also have weird effects if you're not careful.
In this case, the problem is that getopts uses the variable OPTIND to keep track of where it is in the argument list (so it doesn't process the same argument twice). The first time you run the script with -t test2, getopts processes those arguments, and leaves OPTIND set to 3 (meaning that it's already done the first two arguments, "-t" and "test2". The second time you run it with options, it sees that OPTIND is set to 3, so it thinks it's already processed both arguments and just exits the loop.
One option is to add unset OPTIND before the while getopts loop, to reset the count and make it start from the beginning each time.
But unless there's some reason for this script to run in the current shell, it'd be better to make it a standard shell script and have it run as a subprocess. To do this:
Add a "shebang" line as the first line of the script. To make the script run in bash, that'd be either #!/bin/bash or #!/usr/bin/env bash. For zsh, use #!/bin/zsh or #!/usr/bin/env zsh. Since the script runs in a separate shell process, the you can run bash scripts from zsh or zsh scripts from bash, or whatever.
Add execute permission to the script file with chmod -x my_script (or whatever the file's actual name is).
Run the script with ./my_script (note the lack of a space between . and /), or by giving the full path to the script, or by putting the script in some directory in your PATH (the directories that're automatically searched for commands) and just running my_script. Do NOT run it with the bash, sh, zsh etc commands; these override the shebang and therefore can cause confusion.
Note: adding ".sh" to the filename is not recommended; it does nothing useful, and makes the script less convenient to run since you have to type in the extension every time you run it.
Also, a couple of recommendations: there are a bunch of all-caps variable names with special meanings (like PATH and OPTIND), so unless you want one of those special meanings, it's best to use lower- or mixed-case variable names (e.g. tag instead of TAG). Also, double-quoting variable references (e.g. echo "$tag" instead of echo $tag) avoids a lot of weird parsing headaches. Run your scripts through shellcheck.net; it's good at spotting common mistakes like this.
Consider the following script:
#!/bin/bash
CMD="echo hello world > /tmp/hello.out"
${CMD}
The output for this is:
hello world > /tmp/hello.out
How can I modify CMD so that the output gets redirected to hello.out?
For my use case, it is not feasible to either do this:
${CMD} > /tmp/hello.out
or to add this at the top of the script:
exec > /tmp/hello.out
No, there is no way to make a redirection happen from a variable.
Why?
The first thing the shell does with a command line is:
Each line that the shell reads from the standard input or a script is called a pipeline; it contains one or more commands separated by zero or
more pipe characters (|). For each pipeline it reads, the shell breaks it up into commands, sets up the I/O for the pipeline, then does the following for each command (Figure 7-1):
From: Learning the bash Shell Unix Shell Programming . Chapter Preview / Figure . Pdf
That means that even before starting with the first word of a command line, the redirections are set up.
The "Parameter Expansion" happens quite a lot latter (in step 6 of the Figure).
There is no way to set up redirections after a variable is expanded.
Unless ...
The "command line is reprocessed" using eval.
eval "$CMD"
But this comes with a lot of danger.
The command line is changed by the first processing in the 12 steps detailed in the book (quotes are removed, variables expanded, words split, etc.).
It is usually quite difficult to estimate all the changes and consequences before the line is actually processed.
And then, it is processed again.
You can use eval to instruct the shell to reinterpret the variable content as a shell command:
eval $CMD
To set the scene, I'm editing my /.bash_profile, and wanted the current directory along with files in said directory within my PS1. While it works, I'm getting a strange error with the spacing:
(~) ( 12)
I don't understand why this is, as — to my knowledge — the code shouldn't be adding that whitespace.
lines=$(ls | wc -l)
PS1='\n\e[2m(\w) ($lines)\e[22m\n> '
I have no idea as to why this is happening, and someone with some Bash wizardry would be handy.
PS1 is a little bit different than most variables, at least in how it is used. Its value isn't simply printed to standard error to display the prompt, as is obvious if you compare your prompt to the output of echo "$PS1".
The evaluation rules aren't identical to normal evaluation rules, as you can see here that the leading whitespace in the value of $lines isn't removed from the unquoted expansion of $lines when the prompt is finally displayed.
One way to work around this is to replace the plain parameter expansion with a shell command that does undergo normal evaluation, then capture the output of that command for use in the prompt:
PS1='\n\e[2m(\w) ($(echo $lines))\e[22m\n> '
Better yet, don't parse the output of ls: create an array and output its length:
PS1='\n\e[2m(\w) ($(f=(*); echo ${#f[#]}))\e[22m\n> '
(Also, unless you are resetting the value of lines prior to each prompt, say in PROMPT_COMMAND, you are always going to show the file count of the directory you were in when lines was set, probably your home directory. Getting the file count in the prompt itself gets you the file count of the current directory, whatever that might be.)
I use ZSH for my terminal shell, and whilst I've written several functions to automate specific tasks, I've never really attempted anything that requires the functionality I'm after at the moment.
I've recently re-written a blog using Jekyll and I want to automate the production of blog posts and finally the uploading of the newly produced files to my server using something like scp.
I'm slightly confused about the variable bindings/usage in ZSH; for example:
DATE= date +'20%y-%m-%d'
echo $DATE
correctly outputs 2011-08-23 as I'd expect.
But when I try:
DATE= date +'20%y-%m-%d'
FILE= "~/path/to/_posts/$DATE-$1.markdown"
echo $FILE
It outputs:
2011-08-23
blog.sh: line 4: ~/path/to/_posts/-.markdown: No such file or directory
And when run with what I'd be wanting the blog title to be (ignoring the fact the string needs to be manipulated to make it more url friendly and that the route path/to doesn't exist)
i.e. blog "blog title", outputs:
2011-08-23
blog.sh: line 4: ~/path/to/_posts/-blog title.markdown: No such file or directory
Why is $DATE printing above the call to print $FILE rather than the string being included in $FILE?
Two things are going wrong here.
Firstly, your first snippet is not doing what I think you think it is. Try removing the second line, the echo. It still prints the date, right? Because this:
DATE= date +'20%y-%m-%d'
Is not a variable assignment - it's an invocation of date with an auxiliary environment variable (the general syntax is VAR_NAME=VAR_VALUE COMMAND). You mean this:
DATE=$(date +'20%y-%m-%d')
Your second snippet will still fail, but differently. Again, you're using the invoke-with-environment syntax instead of assignment. You mean:
# note the lack of a space after the equals sign
FILE="~/path/to/_posts/$DATE-$1.markdown"
I think that should do the trick.
Disclaimer
While I know bash very well, I only started using zsh recently; there may be zshisms at work here that I'm not aware of.
Learn about what a shell calls 'expansion'. There are several kinds, performed in a particular order:
The order of word expansion is as follows:
tilde expansion
parameter expansion
command substitution
arithmetic expansion
pathname expansion, unless set -f is in effect
quote removal, always performed last
Note that tilde expansion is only performed when the tilde is not quoted; viz.:
$ FILE="~/.zshrc"
$ echo $FILE
~/.zshrc
$ FILE=~./zshrc
$ echo $FILE
/home/user42/.zshrc
And there must be no spaces around the = in variable assignments.
Since you asked in a comment where to learn shell programming, there are several options:
Read the shell's manual page man zsh
Read the specification of the POSIX shell, http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html, especially if you want to run your scripts on different operating systems (and you will find yourself in that situation one fine day!)
Read books about shell programming.
Hang out in the usenet newsgroup comp.unix.shell where a lot of shell wizards answer questions
While taking a look at this awesome thread I noticed that some examples use
PS1="Blah Blah Blah"
and some use
PROMPT_COMMAND="Blah Blah Blah"
(and some use both) when setting the prompt in a Bash shell. What is the difference between the two? A Stack Overflow search and even a bit of broader Google searching aren't getting me results, so even a link to the right place to look for the answer would be appreciated.
PROMPT_COMMAND can contain ordinary Bash statements whereas the PS1 variable can also contain the special characters, such as '\h' for hostname, in the variable.
For example, here is my Bash prompt that uses both PROMPT_COMMAND and PS1. The Bash code in PROMPT_COMMAND works out what Git branch you might be in and displays that at the prompt, along with the exit status of the last run process, hostname and basename of the pwd.
The variable RET stores the return value of the last executed program. This is convenient to see if there was an error and the error code of the last program I ran in the terminal. Note the outer ' surrounding the entire PROMPT_COMMAND expression. It includes PS1 so that this variable is reevaluated each time the PROMPT_COMMAND variable is evaluated.
PROMPT_COMMAND='RET=$?;\
BRANCH="";\
ERRMSG="";\
if [[ $RET != 0 ]]; then\
ERRMSG=" $RET";\
fi;\
if git branch &>/dev/null; then\
BRANCH=$(git branch 2>/dev/null | grep \* | cut -d " " -f 2);\
fi;
PS1="$GREEN\u#\h $BLUE\W $CYAN$BRANCH$RED$ERRMSG \$ $LIGHT_GRAY";'
Example output looks like this in a non-Git directory:
sashan#dhcp-au-122 Documents $ false
sashan#dhcp-au-122 Documents 1 $
And in a Git directory you see the branch name:
sashan#dhcp-au-122 rework mybranch $
Update
After reading the comments and Bob's answer, I think that writing it as he describes is better. It's more maintainable than what I originally wrote above, where the PS1 variable is set inside the PROMPT_COMMAND, which itself is a super complicated string that is evaluated at runtime by Bash.
It works, but it's more complicated than it needs to be. To be fair, I wrote that PROMPT_COMMAND for myself about 10 years ago and it worked and didn't think too much about it.
For those curious as to how I've amended my things, I've basically put the code for the PROMPT_COMMAND in a separate file (as Bob described) and then echo the string that I intend to be PS1:
GREEN="\[\033[0;32m\]"
CYAN="\[\033[0;36m\]"
RED="\[\033[0;31m\]"
PURPLE="\[\033[0;35m\]"
BROWN="\[\033[0;33m\]"
LIGHT_GRAY="\[\033[0;37m\]"
LIGHT_BLUE="\[\033[1;34m\]"
LIGHT_GREEN="\[\033[1;32m\]"
LIGHT_CYAN="\[\033[1;36m\]"
LIGHT_RED="\[\033[1;31m\]"
LIGHT_PURPLE="\[\033[1;35m\]"
YELLOW="\[\033[1;33m\]"
WHITE="\[\033[1;37m\]"
RESTORE="\[\033[0m\]" #0m restores to the terminal's default colour
if [ -z $SCHROOT_CHROOT_NAME ]; then
SCHROOT_CHROOT_NAME=" "
fi
BRANCH=""
ERRMSG=""
RET=$1
if [[ $RET != 0 ]]; then
ERRMSG=" $RET"
fi
if which git &>/dev/null; then
BRANCH=$(git branch 2>/dev/null | grep \* | cut -d " " -f 2)
else
BRANCH="(git not installed)"
fi
echo "${GREEN}\u#\h${SCHROOT_CHROOT_NAME}${BLUE}\w \
${CYAN}${BRANCH}${RED}${ERRMSG} \$ $RESTORE"
And in my .bashrc file:
function prompt_command {
RET=$?
export PS1=$(~/.bash_prompt_command $RET)
}
PROMPT_DIRTRIM=3
export PROMPT_COMMAND=prompt_command
From the GNU Bash documentation page (Bash Reference Manual):
PROMPT_COMMAND
If set, the value is interpreted as a command to execute before
the printing of each primary prompt ($PS1).
I never used it, but I could have used this back when I only had sh.
The difference is that PS1 is the actual prompt string used, and PROMPT_COMMAND is a command that is executed just before the prompt. If you want the simplest, most flexible way of building a prompt, try this:
Put this in your .bashrc file:
function prompt_command {
export PS1=$(~/bin/bash_prompt)
}
export PROMPT_COMMAND=prompt_command
Then write a script (Bash, Perl, or Ruby: your choice), and place it in file ~/bin/bash_prompt.
The script can use any information it likes to construct a prompt. This is much simpler, IMO, because you don't have to learn the somewhat baroque substitution language that was developed just for the PS1 variable.
You might think that you could do the same by simply setting PROMPT_COMMAND directly to ~/bin/bash_prompt, and setting PS1 to the empty string.
This at first appears to work, but you soon discover that the readline code expects PS1 to be set to the actual prompt, and when you scroll backwards in history, things get messed up as a result.
This workaround causes PS1 to always reflect the latest prompt (since the function sets the actual PS1 variable used by the invoking instance of the shell), and this makes readline and command history work fine.
From man bash:
PROMPT_COMMAND
If set, the value is executed as a command prior to issuing each primary prompt.
PS1
The value of this parameter is expanded (see PROMPTING below) and used as the primary prompt string. The default value is ''\s-\v\$ ''.
If you simply want to set the prompt string, using PS1 alone is enough:
PS1='user \u on host \h$ '
If you want to do something else just before printing the prompt, use PROMPT_COMMAND. For example, if you want to sync cached writes to disk, you can write:
PROMPT_COMMAND='sync'
Yeah, so to try to really nail this down:
PROMPT_COMMAND is a handy Bash convenience variable/function, but there is, strictly speaking, nothing that cannot also be done using PS1 alone, correct?
I mean, if one wants to set another variable with scope outside the prompt: depending on the shell, that variable would probably need to be declared first outside $PS1 or (worst case) one might have to get fancy with something waiting on a FIFO prior to calling $PS1 (and armed again at the end of $PS1); the \u \h might cause some trouble, particularly if you're using some fancy regex; but otherwise: one can accomplish anything PROMPT_COMMAND can by using command substitution within $PS1 (and, maybe in corner cases, explicit subshells)?
Right?
The difference is that
if you output an incomplete line from PROMPT_COMMAND, it will screw your Bash prompt
PS1 substitutes \H and friends
PROMPT_COMMAND runs its contents, and PS1 uses its contents as the prompt.
PS1 does variable expansion and command substitution at each prompt. There isn't any need to use PROMPT_COMMAND to assign a value to PS1 or to run arbitrary code. You can easily do export PS1='$(uuidgen) $RANDOM' once in file .bash_profile. Just use single quotes.
I spent so much time on this I just wanted to share what worked for me. I looked at a lot of the SO posts about PROMPT_COMMAND and PS1 and tried many combinations of single quotes, double quotes, calling functions... I could not get the prompt to update each time without printing control characters or the literal expanded but not processed prompt string, or without just setting PS1 in PROMPT_COMMAND as we are advised not to do. My problem was setting variables (colors) that contained control characters; these had to be hard-coded after the variable name in PS1. PROMPT_COMMAND is set to a function that sets variables and they are used (escaped) in a double-quoted PS1 string. This is for a powerline-style prompt that changes colors with each command.
icon1=#unicode powerline char like
#these: https://github.com/ryanoasis/powerline-extra-symbols#glyphs
icon2=#same
#array of ANSI colors. 2 for rgb mode then the rgb values
#then 'm' without '\]' control character. these are from
#the solarized theme https://ethanschoonover.com/solarized/
declare -a colors=(
"2;220;50;47m"
"2;203;75;22m"
"2;181;137;0m"
"2;133;153;0m"
"2;42;161;152m"
"2;38;139;210m"
"2;108;113;196m"
"2;211;54;130m"
"2;0;43;54m"
"2;7;54;66m"
"2;88;110;117m"
"2;101;123;131m"
"2;131;148;150m"
"2;147;161;161m"
)
#outside of vars set in PROMPT_COMMAND it's ok to have control chars
LEN=${#colors[#]}
BG="\[\e[48;"#set bg color
FG="\[\e[38;"#set fg color
TRANSP="1m\]"#transparency
BASE2="2;238;232;213m\]"#fg (text) color
myfunc(){
RAND=$(($RANDOM % $LEN))
COLOR1=${colors[$RAND]}
COLOR2=${colors[($RAND + 1) % $LEN]}
COLOR3=${colors[($RAND + 2) % $LEN]}
}
PROMPT_COMMAND=myfunc
#note double quotes and escaped COLOR vars followed by hard-coded '\]' control chars
PS1="$BG$TRANSP$FG\$COLOR1\]$icon1$BG\$COLOR1\]$FG$TRANSP$BG\$COLOR1\]$FG$BASE2
[username hard-coded in unicode] $BG\$COLOR2\]$FG\$COLOR1\]$icon2$BG\$COLOR2\]$FG$BASE2
\w $BG\$COLOR3\]$FG\$COLOR2\]$icon2$BG\$COLOR3\]$FG$BASE2 [more unicode]
\[\e[0m\]$FG\$COLOR3\]$icon2\[\e[0m\] "
That ought to get you going!