I often see key=value bash-script to pass variables to a bash script. Here is an example:
$ echo $0
-bash
$ cat foo.sh
#!/usr/bin/env bash
echo "key1: $key1"
$ key1=value1 ./foo.sh
key1: value1
I have checked Bash Reference Manual. But I can't a description related to this usage.
Section 3.7.4 "Environment"
The environment for any simple command or function may be augmented temporarily by prefixing it with parameter assignments, as described in Shell Parameters. These assignment statements affect only the environment seen by that command.
Apparently the online version of the documentation fails to describe the syntax of a simple command completely. It reads:
3.2.1 Simple Commands
A simple command is the kind of command encountered most often. It’s just a sequence of words separated by blanks, terminated by one of the shell’s control operators (see Definitions). The first word generally specifies a command to be executed, with the rest of the words being that command’s arguments.
There is some information about the variable assignments that may precede a simple command on the section Simple Commands Expansion:
3.7.1 Simple Command Expansion
When a simple command is executed, the shell performs the following expansions, assignments, and redirections, from left to right.
The words that the parser has marked as variable assignments (those preceding the command name) and redirections are saved for later processing.
...
The text after the = in each variable assignment undergoes tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal before being assigned to the variable.
The manual page bundled with the shell seems to be better at this. It says (the emphasis is mine):
Simple Commands
A simple command is a sequence of optional variable assignments followed by blank-separated words and redirections, and terminated by a control operator.
You can always read the documentation of bash (and of any other CLI program installed on your computer) using man.
Type man bash on your terminal and use the usual keys (<spacebar>, b, /, q etc.) to navigate in the document. Behind the scenes, man uses your default pager program (which is, most probably, less) to put the information on screen.
Related
I have a simple bash script that does something like the following:
#!/bin/bash
a=$(curl -s http://metadata/endpoint/internal)
echo "$a - bar"
(this is just a simplification). Note the use of the two $ signs to execute a command and resolve a variable.
Using Terraform, I want to write this file to a GCP instance during startup. Per Terraform's instructions, I'm attempting to avoid using the File Provisioner. Using the metadata_startup_script field in google_compute_instance2, I am including the script so that I can write it to a particular location.
E.g.
metadata_startup_script = <<-EOF
#!/bin/bash -xe
sudo tee /etc/myservice/serv.conf > /dev/null <<EOI
${file("${path.module}/scripts/simple_bash.sh")}
EOI
EOF
Terraform is interpolating the $ in the subscript somewhere (either in the loading into the metadata_startup_script, or in the writing out into the script to disk.
So, depending on what I use to try to escape the interpolation, it still fails to write. For example, I have tried (in the subscript):
echo "\$a - bar"
echo "${“$”}a - bar"
echo "$$a - bar"
According to the terraform docs, I'm supposed to use $$, but when I do it in the above, I get:
echo "1397a - bar"
All which fail to replicate the original script.
I’m just looking for the exact bash script, as written, to be written to disk.
My goal would be to do the above without extra escape sequences (as detailed here - Escaping dollar sign in Terraform) so that i can continue to run the original script (for debugging purposes).
I would also prefer not to build a packer image with the original script in it.
Thanks!
I don't think it's Terraform eating your variable interpolations here, because Terraform only understands ${ (a dollar sign followed by a brace) as starting an interpolation, whereas your example contains only $a.
However, you do seem to be embedded one bash script inside another, so it seems plausible to me that the outer bash is resolving your $a before the inner bash gets a chance to look at it. If so, you can use the literal variant of bash heredoc syntax, as described in answers to How to cat <> a file containing code?, so that the outer bash will take the content as literal and leave it to the inner bash to evaluate.
metadata_startup_script = <<-EOF
#!/bin/bash -xe
sudo tee /etc/myservice/serv.conf > /dev/null <<'EOI'
${file("${path.module}/scripts/simple_bash.sh")}
EOI
EOF
Notice that I wrote <<'EOI' instead of <<EOI, following the guidance from that other question in combination with the main Bash documentation on "here documents" (bold emphasis mine):
This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen. All of the lines read up to that point are then used as the standard input (or file descriptor n if n is specified) for a command.
The format of here-documents is:
[n]<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic expansion, or filename expansion is performed on word. If any part of word is quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion, the character sequence \newline is ignored, and \ must be used to quote the characters \, $, and `.
If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing delimiter. This allows here-documents within shell scripts to be indented in a natural fashion.
If your machine image is configured to run cloud-init at startup -- this is often but not always what is responsible for executing metadata_startup_script -- you may be able to achieve a similar effect without so much Bash scripting indirection by using Cloud Config YAML instead of a shell script directly.
For example, if your intent is only to write the content of that file into the designated location in the filesystem, you could potentially follow the Writing out arbitrary files example:
metadata_startup_script = <<-EOF
#cloud-config
${yamlencode({
write_files = [
{
encoding = "b64"
content = filebase64("${path.module}/scripts/simple_bash.sh")
path = "/etc/myservice/serv.conf"
owner = "root:root"
permissions = "0644"
},
]
}}
EOF
Cloud-init evaluates its modules at various points in the startup lifecycle. The Write Files module used here is specified to run once on the first boot of an instance, which matches how Cloud-init would typically treat a naked shell script too.
I do not think your issue is related to TF interpolation. I think you have problems because of normal bash interpolation, as its bash which is going to try to resolve $ in your /etc/myservice/serv.conf while writing its content.
The regular solution is to use 'EOI', not EOI:
metadata_startup_script = <<-EOF
#!/bin/bash -xe
sudo tee /etc/myservice/serv.conf > /dev/null <<'EOI'
${file("${path.module}/scripts/simple_bash.sh")}
EOI
EOF
So I generally create job files with a list of commands in it. Then I execute it like so
cat jobFile | while read a; do $a; done
Which always works in bash. However, I've just started working in Mac which apparently uses zsh. And this command fails with "no such file" etc. I've tested the job file by running few lines from it manually, so it should be fine.
I've found questions on zsh read inbut they tend to be reading in from variables e.g. $a=('a' 'b' 'c') or echo $a
Thank you for your answers!
In bash, unquoted parameter expansions always undergo word-splitting, so if a="foo bar", then $a expands to two words, foo and bar. As a command, this means running the command foo with an argument bar.
In zsh, parameter expansions to not undergo word-splitting by default, which means the same expansion $a would produce a single word foo bar, treated as the name of the command to execute.
In either case, relying on parameter expansion to "parse" a shell command is fragile; in addition to word-splitting, the expansion is subject to pathname expansion (globbing), and you are limited to simple commands and their arguments. No pipes, lists (&&, ||), or redirections allowed, as everything will be treated as the command name and a sequence of arguments.
What you want in both shells is to simply treat your job file as a shell script, which can be executed in the current shell using the . command:
. jobFile
Why are you executing it in such a cumbersome way? Assuming jobFile is a file holding a sequence of bash commands, you can simply run it as
bash jobFile
If it contains a sequence of zsh commands, you can likewise run it as
zsh jobFile
If you follow this approach, I would however reflect in the name of the job file, what shell it is intended for, i.e.
bash jobFile.bash
zsh jobFile.zsh
and, if you write a job file so that it is supposed to be compatible with either shell, I would name it jobFile.sh.
When using bash, there are self-referential variables that don't seem to be available in zsh. For example, $? gets the exit status of the most recent foreground process, $_ gets the last parameter of the previous command, etc.
Are there equivalents to these in zsh? I ask with reference to this question for bash.
bash calls these the special parameters (to distinguish them from ordinary variables and the positional parameters). zsh implements essentially the same set, but documents them along with other parameters set by the shell, though it does also tag each one as special.
See man zshparam:
PARAMETERS SET BY THE SHELL
In the parameter lists that follow, the mark `<S>' indicates that the
parameter is special. `<Z>' indicates that the parameter does not
exist when the shell initializes in sh or ksh emulation mode.
The following parameters are automatically set by the shell:
! <S> The process ID of the last command started in the background
with &, or put into the background with the bg builtin.
# <S> The number of positional parameters in decimal. Note that some
confusion may occur with the syntax $#param which substitutes
the length of param. Use ${#} to resolve ambiguities. In par-
ticular, the sequence `$#-...' in an arithmetic expression is
interpreted as the length of the parameter -, q.v.
ARGC <S> <Z>
Same as #.
$ <S> The process ID of this shell. Note that this indicates the
original shell started by invoking zsh; all processes forked
from the shells without executing a new program, such as sub-
shells started by (...), substitute the same value.
[... etc ...]
I saw an expression a=($(cat)) which I am not able to understand from it's working mechanism perspective.
Functionally it takes input from the standard input and assigns it to variable a (which forms an array).
My understanding is , when shell executes the inner parenthesese it executes the cat command which brings the standard input, and when you type a few lines on the standard input and press CTRL+D it returns the lines to the outer parenthesese which then assign the lines to an array a.
My question is why this expression gives error when I remove the $ and write it as a=((cat)).
It is because $(..) is a command substitution syntax to run commands on. The cat in your example run in a sub-shell under this construct. Without it the command cat and ( are interpreted literally which the shell does not like
From the bash(1) - Linux man page
Command Substitution
Command substitution allows the output of a command to replace the command name. There are two forms:
$(command) (or) command
Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command, with any trailing newlines deleted.
The arithmetic operator in bash is $((..)) which is not the syntax you are using in your example
I was trying to run a bash shell script in ESXi shell.
Think ESXi is posix compatible, posix compiler threw much syntax errors.
please let me know if anyone has the answer
The manual lists the changed behavior when running in POSIX mode. You should adapt your script accordingly.
The following list is what’s changed when POSIX mode is in effect:
When a command in the hash table no longer exists, Bash will re-search $PATH to find the new location. This is also available with ‘shopt -s checkhash’.
The message printed by the job control code and builtins when a job exits with a non-zero status is ‘Done(status)’.
The message printed by the job control code and builtins when a job is stopped is ‘Stopped(signame)’, where signame is, for example, SIGTSTP.
The bg builtin uses the required format to describe each job placed in the background, which does not include an indication of whether the job is the current or previous job.
Reserved words appearing in a context where reserved words are recognized do not undergo alias expansion.
The POSIX PS1 and PS2 expansions of ‘!’ to the history number and ‘!!’ to ‘!’ are enabled, and parameter expansion is performed on the values of PS1 and PS2 regardless of the setting of the promptvars option.
The POSIX startup files are executed ($ENV) rather than the normal Bash files.
Tilde expansion is only performed on assignments preceding a command name, rather than on all assignment statements on the line.
The default history file is ~/.sh_history (this is the default value of $HISTFILE).
The output of ‘kill -l’ prints all the signal names on a single line, separated by spaces, without the ‘SIG’ prefix.
The kill builtin does not accept signal names with a ‘SIG’ prefix.
Non-interactive shells exit if filename in . filename is not found.
Non-interactive shells exit if a syntax error in an arithmetic expansion results in an invalid expression.
Non-interactive shells exit if there is a syntax error in a script read with the . or source builtins, or in a string processed by the eval builtin.
Redirection operators do not perform filename expansion on the word in the redirection unless the shell is interactive.
Redirection operators do not perform word splitting on the word in the redirection.
Function names must be valid shell names. That is, they may not contain characters other than letters, digits, and underscores, and may not start with a digit. Declaring a function with an invalid name causes a fatal syntax error in non-interactive shells.
POSIX special builtins are found before shell functions during command lookup.
The time reserved word may be used by itself as a command. When used in this way, it displays timing statistics for the shell and its completed children. The TIMEFORMAT variable controls the format of the timing information.
When parsing and expanding a ${...} expansion that appears within double quotes, single quotes are no longer special and cannot be used to quote a closing brace or other special character, unless the operator is one of those defined to perform pattern removal. In this case, they do not have to appear as matched pairs.
The parser does not recognize time as a reserved word if the next token begins with a ‘-’.
If a POSIX special builtin returns an error status, a non-interactive shell exits. The fatal errors are those listed in the POSIX standard, and include things like passing incorrect options, redirection errors, variable assignment errors for assignments preceding the command name, and so on.
A non-interactive shell exits with an error status if a variable assignment error occurs when no command name follows the assignment statements. A variable assignment error occurs, for example, when trying to assign a value to a readonly variable.
A non-interactive shell exists with an error status if a variable assignment error occurs in an assignment statement preceding a special builtin, but not with any other simple command.
A non-interactive shell exits with an error status if the iteration variable in a for statement or the selection variable in a select statement is a readonly variable.
Process substitution is not available.
Assignment statements preceding POSIX special builtins persist in the shell environment after the builtin completes.
Assignment statements preceding shell function calls persist in the shell environment after the function returns, as if a POSIX special builtin command had been executed.
The export and readonly builtin commands display their output in the format required by POSIX.
The trap builtin displays signal names without the leading SIG.
The trap builtin doesn’t check the first argument for a possible signal specification and revert the signal handling to the original disposition if it is, unless that argument consists solely of digits and is a valid signal number. If users want to reset the handler for a given signal to the original disposition, they should use ‘-’ as the first argument.
The . and source builtins do not search the current directory for the filename argument if it is not found by searching PATH.
Subshells spawned to execute command substitutions inherit the value of the -e option from the parent shell. When not in POSIX mode, Bash clears the -e option in such subshells.
Alias expansion is always enabled, even in non-interactive shells.
When the alias builtin displays alias definitions, it does not display them with a leading ‘alias ’ unless the -p option is supplied.
When the set builtin is invoked without options, it does not display shell function names and definitions.
When the set builtin is invoked without options, it displays variable values without quotes, unless they contain shell metacharacters, even if the result contains nonprinting characters.
When the cd builtin is invoked in logical mode, and the pathname constructed from $PWD and the directory name supplied as an argument does not refer to an existing directory, cd will fail instead of falling back to physical mode.
The pwd builtin verifies that the value it prints is the same as the current directory, even if it is not asked to check the file system with the -P option.
When listing the history, the fc builtin does not include an indication of whether or not a history entry has been modified.
The default editor used by fc is ed.
The type and command builtins will not report a non-executable file as having been found, though the shell will attempt to execute such a file if it is the only so-named file found in $PATH.
The vi editing mode will invoke the vi editor directly when the ‘v’ command is run, instead of checking $VISUAL and $EDITOR.
When the xpg_echo option is enabled, Bash does not attempt to interpret any arguments to echo as options. Each argument is displayed, after escape characters are converted.
The ulimit builtin uses a block size of 512 bytes for the -c and -f options.
The arrival of SIGCHLD when a trap is set on SIGCHLD does not interrupt the wait builtin and cause it to return immediately. The trap command is run once for each child that exits.