I would like to have command in Makefile that allows me to change service's target in docker-compose.yaml respectively to target given command parameter. I have something like this:
example:
#[[ "$${target:-dev}" == #(dev|test|release) ]] || exit 1
yq -y -i '.services."$(service)".build.target = "$(target)"' docker-compose.yaml
This almost do the job, but I have this parameter to be optional. It should fallback to "dev" when not provided. In second line there is validation which check if given target has allowed value. If target is not specified in command it should fallback to "dev" and it work. But when I try similar syntax it doesn't work in third line. I tried different combination like
yq -y -i '.services."$(service)".build.target = "$(target:-dev)"' docker-compose.yaml
yq -y -i '.services."$(service)".build.target = "${target:-dev}"' docker-compose.yaml
yq -y -i '.services."$(service)".build.target = "${{target:-dev}}"' docker-compose.yaml
yq -y -i '.services."$(service)".build.target = "$(${target:-dev})"' docker-compose.yaml
// etc
But in most cases I got empty string instead of "dev", what's wrong?
The values $(target) is a make variable, not a shell variable. Make variables can't be manipulated using shell constructs like ${foo:-bar}.
If you want to do something like that you have to use make constructs. If you're willing to restrict yourself to GNU make, you can use:
$(or $(target),dev)
If you want to be completely portable you can convert your make variable into a shell variable, then use shell constructs, like this:
target='$(target)'; yq -u -i ".services.\"...\".build.target = \"$${target:-dev}\"" ...
By the way I should point out: make always invokes /bin/sh when running recipes, unless you specifically request that it use a different shell by setting SHELL. This line in your recipe:
#[[ "$${target:-dev}" == #(dev|test|release) ]] || exit 1
is using a bunch of bash-specific features which are definitely NOT portable to the POSIX shell. If your system's /bin/sh just happens to be bash (for example, you're on a Red Hat type system) this will work. But if you try to run this on a system where /bin/sh is a strict POSIX shell, like dash (for example, if you're on a Debian type system including Ubuntu) it will fail.
If your makefile requires recipes to run in bash, not POSIX sh, you need to add:
SHELL := /bin/bash
or similar to it.
Related
I'm facing some problems to pass some environment parameters to docker run in a relatively generic way.
Our first iteration was to load a .env file into the environment via these lines:
set -o allexport;
. "${PROJECT_DIR}/.env";
set +o allexport;
And then manually typing the --env VARNAME=$VARNAME as options for the docker run command. But this can be quite annoying when you have dozens of variables.
Then we tried to just pass the file, with --env-file .env, and it seems to work, but it doesn't, because it does not play well with quotes around the variable values.
Here is where I started doing crazy/ugly things. The basic idea was to do something like:
set_docker_parameters()
{
grep -v '^$' "${PROJECT_DIR}/.env" | while IFS= read -r LINE; do
printf " -e %s" "${LINE}"
done
}
docker run $(set_docker_parameters) --rm image:label command
Where the parsed lines are like VARIABLE="value", VARIABLE='value', or VARIABLE=value. Blank lines are discarded by the piped grep.
But docker run complains all the time about not being called properly. When I expand the result of set_docker_parameters I get what I expected, and when I copy its result and replace $(set_docker_parameters), then docker run works as expected too, flawless.
Any idea on what I'm doing wrong here?
Thank you very much!
P.S.: I'm trying to make my script 100% POSIX-compatible, so I'll prefer any solution that does not rely on Bash-specific features.
Based on the comments of #jordanm I devised the following solution:
docker_run_wrapper()
{
# That's not ideal, but in any case it's not directly related to the question.
cmd=$1
set --; # Unset all positional arguments ($# will be emptied)
# We don't have arrays (we want to be POSIX compatible), so we'll
# use $# as a sort of substitute, appending new values to it.
grep -v '^$' "${PROJECT_DIR}/.env" | while IFS= read -r LINE; do
set -- "$#" "--env";
set -- "$#" "${LINE}";
done
# We use $# in a clearly non-standard way, just to expand the values
# coming from the .env file.
docker run "$#" "image:label" /bin/sh -c "${cmd}";
}
Then again, this is not the code I wrote for my particular use case, but a simplification that shows the basic idea. If you can rely on having Bash, then it could be much cleaner, by not overloading $# and using arrays.
I have two aliases:
alias ls="ls -G"
alias la="ls -aFhlT"
I know that after you type your alias, but before you execute, you can type Meta-Control-e (probably Alt-Control-e, but possibly Esc-Control-e) to expand what you've typed.
So, if I expand my alias la using this method I get:
ls -aFhlT
However, what I really want is to see:
ls -G -aFhlT
Is there any way to achieve this besides typing Meta-Control-e a second time?
--OR--
Is there any way to confirm that my execution of la actually executed ls -G -aFhlT (other than knowing how nested aliases work and trusting that it did what I think it did)?
I'm trying to do this on macOS, but a general bash solution will also be accepted.
This question rides the fine line between using an alias and using a function. When aliases get even slightly complicated, it is generally better to write a function instead. That being said, I did find a solution for this question that allows for expanding aliases as desired.
I wrote a bash function for this:
xtrace() {
local eval_cmd
printf -v eval_cmd '%q ' "${#}"
{ set -x
eval "${eval_cmd}"
} 2>&1 | grep '^++'
return "${PIPESTATUS[0]}"
}
The -v flag of printf will store the output of printf in the specified variable.
The printf format string %q will print the associated argument ($# in this case) shell-quoted, reusable as input. This eliminates the dangers associated with passing arbitrary code/commands to eval.
I then use a command group { ... } so I can control the functionality of set -x, which tells bash to print a trace of all executed commands. For my purposes, I do not care about any output except for the fully expanded command, so I redirect stderr and grep for the output line that starts with "++". This will be the line that shows the fully expanded command.
Finally, I return the value of PIPESTATUS[0], which contains the return code of the last command executed in the command group (i.e. the eval command).
Thus, we will get something like the following:
$ xtrace la; echo $?
++ ls -G -aFhlT
0
Much thanks to #CharlesDuffy for the set -x recommendation as well as the input sanitation for eval.
Trying to compare the process list before and after running a bash script of tests. Having trouble, since ps returns 1, and I'm not sure how to compare the before and after when I have them.
Ideally, it would look something like this. Forgive the crude pseudo-code:
run-tests:
ps -x
export before=$?
# run tests and scripts
ps -x
export after=$?
# compare before and after
Suggests and advice appreciated.
I'm assuming you want to count the number of running processes before and after (your question wasn't overly clear on that). If so, you can pipe ps into wc:
export before=`ps --no-headers | wc -l`
-- EDIT ---
I reread the question, and it may be that you're looking for the actual processes that differ. If that's the case, then, you can capture the output in variables and compare those:
target:
# before=$$(ps --no-headers); \
run test; \
after=$$(ps --no-headers); \
echo "differing processes:"; \
comm -3 <(echo "$before") <(echo "$after")
A few quick notes on this: I concatenated all the lines using \'s as you mentioned you used makefiles, and the scope of a variable is only the recipe line in which it's defined. By concatenating the lines, the variables have a scope of the whole recipe.
I used double $$ as your original post suggested a makefile, and a makefile $$ will expand to a single $ in the bash code to be run.
Doing var=$(command) in bash assigns var the output of command
I used the <() convention which is specific to bash. This lets you treat the output of a command as file, without having to actually create a file. Notice that I put quotes around the variables -- this is required, otherwise the bash will ignore newlines when expanding the variable.
I'm using the following command really often (the real path is longer):
>>/root/error.log 2>&1 >>/root/out.log
but now I'm trying to shorten it like that:
log="$(>>/root/error.log 2>&1 >>/root/out.log)"
But when I use the variable in another file, it doesnt work at all:
apt-get update -y ${log}
Do you have any idea, how to get this working?
Edit:
At the moment the command looks like this:
apt-get update -y >>/root/error.log 2>&1 >>/root/out.log
or
apt-get install openssl >>/root/error.log 2>&1 >>/root/out.log
... and is working fine. But i want a shorter version for it like:
apt-get update -y ${log}
or
apt-get install openssl ${log}
for example
What you are trying to encapsulate is NOT a command. It is the redirection portion of a command you want to execute.
This will not work. You can have file names from variables in redirections, but not the >, >> or &1 parts.
The simplest solution is to create a function.
execute_and_log()
{
"$#" >>/root/out.log 2>>/root/error.log
}
execute_and_log any_command with args
Note that "$#" (the double quotes are very important here) expands to a list of all positional arguments in the current context (which could be no argument at all, which would expand to nothing), each argument being protected from further word splitting, as if called as "$1" "$2" ...
Please note that eval can be used to execute a command after performing expansions on it, but while it can be used properly, it is tricky and should be seen as a last resort, as there often are simpler and less error-prone solutions.
Another possibility is this :
main_log="long/path/to/file1"
err_log="long/path/to/file2"
apt-get update -y >>"$main_log" 2>>"$err_log"
As part of my makefile I need to download and build ZLib. However I want to ensure that when I download ZLib, it is correct by comparing the sha256 of the downloaded .tar.gz against the known correct sha256 value. This need to work on multiple platforms.
I have so far something like the following, however the value of ZLIB_SHA256_ACTUAL always seems to be blank when I compare it with ZLIB_SHA256, so my makefile always exits with an error because the checksums are not the same. I am newish to Makefiles, can someone tell me what I am doing wrong please?
ZLIB_VER = 1.2.11
ZLIB_SHA256 = c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1
SHA256_CMD = sha256sum
ifeq ($(PLATFORM), OS_MACOSX)
SHA256_CMD = openssl sha256 -r
endif
ifeq ($(PLATFORM), OS_SOLARIS)
SHA256_CMD = digest -a sha256
endif
libz.a:
-rm -rf zlib-$(ZLIB_VER)
curl -O -L http://zlib.net/zlib-$(ZLIB_VER).tar.gz
ZLIB_SHA256_ACTUAL = $(SHA256_CMD) zlib-$(ZLIB_VER).tar.gz
ifneq ($(ZLIB_SHA256), $(ZLIB_SHA256_ACTUAL))
$(error zlib-$(ZLIB_VER).tar.gz checksum mismatch, expected="$(ZLIB_SHA256)" actual="$(ZLIB_SHA256_ACTUAL)")
endif
tar xvzf zlib-$(ZLIB_VER).tar.gz
cd zlib-$(ZLIB_VER) && CFLAGS='-fPIC' ./configure --static && make
cp zlib-$(ZLIB_VER)/libz.a .
A makefile consists of two different programming languages in one file. Most of the file uses makefile syntax, that make understands and parses. But the recipes of the rules use shell syntax, which make doesn't try to interpret: it just passes the contents of the recipe to the shell to interpret.
The recipe is the part of the makefile indented with a TAB character, after a target definition. So in your example above, the target definition is libz.a: and all the lines after that which are indented with a TAB, are recipe lines. They are passed to the shell, not run by make.
The recipe is a single block of lines; you cannot intersperse recipe lines with makefile lines. Once make sees the first non-recipe line, that's the end of the recipe and make starts treating the remaining lines as if they were makefile lines.
Let's look at your rule:
libz.a:
-rm -rf zlib-$(ZLIB_VER)
curl -O -L http://zlib.net/zlib-$(ZLIB_VER).tar.gz
OK, this is fine: you've created a target libz.a and provided two command lines, which are valid shell commands, in your recipe.
ZLIB_SHA256_ACTUAL = $(SHA256_CMD) zlib-$(ZLIB_VER).tar.gz
OK, now you have problems; this is a make variable assignment, not a shell command, but since you've indented it with a TAB make will not interpret it: make will just pass it to the shell. That's not a valid shell command (in the shell, variable assignments cannot have spaces around the equal sign); this is trying to run a program named literally ZLIB_SHA256_ACTUAL and pass it the arguments = and the expansion of the SHA256_CMD variable. Even if this was recognized as a make assignment it wouldn't do what you want since it would just set the value of the variable to the string openssl sha256 -r zlib-1.2.11.tar.gz: you want to run that command and set the variable to the output.
Then the next lines:
ifneq ($(ZLIB_SHA256), $(ZLIB_SHA256_ACTUAL))
$(error zlib-$(ZLIB_VER).tar.gz checksum mismatch, expected="$(ZLIB_SHA256)" actual="$(ZLIB_SHA256_ACTUAL)")
endif
Again, this is wrong because these are make commands but you've put them into a recipe which means they'll be passed to the shell, but the shell doesn't know anything about them.
However, they never get the chance to be passed to the shell because the one thing make does with a recipe before it sends it off to the shell is expand all make variables and functions. So, when make expands this it runs the error function and that immediately fails and make never has a chance to try to run the recipe.
This is the tricky part of make. Maybe I've just confused you with all of the above stuff.
The short, simple answer is: you have to use shell commands to perform operations in a recipe. You cannot use make commands (like ifeq etc.), and if you want to set variables in a recipe they have to be shell variables, not make variables.
So, you want something like this, which uses shell syntax not make syntax for the variable assignment and test.
EDIT Note your SHA generation command doesn't print just the SHA it also prints the name of the file, so you can't compare them as strings: they'll never be the same. You need to do something fancier; there are many ways to go about it. Here I decided to use case to do the comparison:
libz.a:
-rm -rf zlib-$(ZLIB_VER)
curl -O -L http://zlib.net/zlib-$(ZLIB_VER).tar.gz
ZLIB_SHA256_ACTUAL=`$(SHA256_CMD) zlib-$(ZLIB_VER).tar.gz`; \
case "$$ZLIB_SHA256_ACTUAL " in \
($(ZLIB_SHA256)\ *) : ok ;; \
(*) echo zlib-$(ZLIB_VER).tar.gz checksum mismatch, expected=\"$(ZLIB_SHA256)\" actual=\"$$ZLIB_SHA256_ACTUAL\"; \
exit 1 ;; \
esac
tar xvzf zlib-$(ZLIB_VER).tar.gz
cd zlib-$(ZLIB_VER) && CFLAGS='-fPIC' ./configure --static && $(MAKE)
cp zlib-$(ZLIB_VER)/libz.a .
Note that each logical line in the recipe is passed to a new instance of the shell, so if you want to set a shell variable and test its value you have to combine physical lines into one logical line with the backslash/newline syntax.
Also, when running a sub-make in a recipe you should always use the variable $(MAKE) and never use just make.