In a Makefile I'm trying to do the following:
Here I'm getting a couple things I need:
S3_BUCKET_NAME_STATE=$(shell terraform output -json | jq '.S3_Bucket.value')
DYNAMODB_LOCK=$(shell terraform output -json | jq '.dynamo_db_lock.value')
And with the result of that (i.e.: the content in the variables) I want to do some replacements in a file, like this:
sed 's/{{ BUCKET_NAME }}/$$(S3_BUCKET_NAME_STATE)/g;' variables.tf.template > variables.tf)
But it keeps on not doing it right. I've tried using ${} instead but i can't make it work...
Help is appreciated!
You didn't mention whether these are make variables or shell variables, so I'm going to assume they're make variables.
Here's a working example of how to do this. It has 3 files: test.json, variables.tf.template, and Makefile.
Makefile:
S3_BUCKET_NAME_STATE := $(shell cat test.json | jq -r '.S3_Bucket.value')
all:
sed 's/{{ BUCKET_NAME }}/$(S3_BUCKET_NAME_STATE)/g;' variables.tf.template > variables.tf
test.json:
{"S3_Bucket": {"value": "Foo"}}
variables.tf.template:
some stuff {{ BUCKET_NAME }}
I made the following changes:
Using := instead of =. The equals operator is the lazy substitution operator, which expands the macro at the time that it's used rather than at the time of defintion.
Using -r with jq. This gets the raw string value.
Using $(var) instead of $$(var). Make interprets a double dollar sign as an escaped dollar sign.
I tested this on GNU Make 3.81 and 4.3.
Related
Following curl usage is not liking braces inside braces. I tried escaping but it didnt help ... any ideas?
echo "$(curl -s -u ${USERNAME}:${PASSWORD} GET ${hostName}/${path}.json| jq -r -c '[.field1,.field2] | \"(.[0]) ,(.[1])"')"
Result:
jq: error: syntax error, unexpected INVALID_CHARACTER (Unix shell
quoting issues?) at <top-level>, line 1:
error near (.[0]`)`
To debug cases like this, it's best to break things down into the basic components first. In your case, I'd guess you intended the jq filter to be:
[.field1,.field2] | "\(.[0]), \(.[1])"
To test:
jq -r -c '[.field1,.field2] | "\(.[0]), \(.[1])"'
{"field1": 1, "field2": 2}
1, 2
Once you have that part right, the rest is easy. Assuming you are using a decent shell, you could probably make life simple for yourself by using $() rather than "$()", or avoiding command substitution altogether.
From the development and testing point of view, it might make sense to put the jq program that you know to be correct into a file, say program.jq, and then you can in sequence:
(a) verify it in stand-alone mode, using jq -r -c -f program.jq
(b) plug the jq invocation into your pipeline to verify that there isn't another problem elsewhere.
If using program.jq isn't ultimately satisfactory, you might want to consider setting a shell variable to the jq program, e.g.
JQ='[.field1,.field2] | "\(.[0]), \(.[1])"'
echo $(jq -r -c "$JQ")
Of course the last line above is just there for testing. Maybe you don't even need $() at all?
Trying to compare the process list before and after running a bash script of tests. Having trouble, since ps returns 1, and I'm not sure how to compare the before and after when I have them.
Ideally, it would look something like this. Forgive the crude pseudo-code:
run-tests:
ps -x
export before=$?
# run tests and scripts
ps -x
export after=$?
# compare before and after
Suggests and advice appreciated.
I'm assuming you want to count the number of running processes before and after (your question wasn't overly clear on that). If so, you can pipe ps into wc:
export before=`ps --no-headers | wc -l`
-- EDIT ---
I reread the question, and it may be that you're looking for the actual processes that differ. If that's the case, then, you can capture the output in variables and compare those:
target:
# before=$$(ps --no-headers); \
run test; \
after=$$(ps --no-headers); \
echo "differing processes:"; \
comm -3 <(echo "$before") <(echo "$after")
A few quick notes on this: I concatenated all the lines using \'s as you mentioned you used makefiles, and the scope of a variable is only the recipe line in which it's defined. By concatenating the lines, the variables have a scope of the whole recipe.
I used double $$ as your original post suggested a makefile, and a makefile $$ will expand to a single $ in the bash code to be run.
Doing var=$(command) in bash assigns var the output of command
I used the <() convention which is specific to bash. This lets you treat the output of a command as file, without having to actually create a file. Notice that I put quotes around the variables -- this is required, otherwise the bash will ignore newlines when expanding the variable.
I've been learning make and am struggling to figure something out. I have some rules with this general structure.
FILE = "myfile.txt"
test :
YOUR = $(subst my,your,$(FILE));\
cat $(FILE) $(YOUR)
I would expect the end result to be running the command:
cat myfile.txt yourfile.txt
Instead I get the following...
YOUR = "yourfile.txt";\
cat "myfile.txt"
/bin/sh: YOUR: command not found
make: *** [test] Error 1
If instead of using the subst function, I just do YOUR="yourfile" in the makefile, everything looks fine. Any suggestions or have I missed something pretty fundamental? I should add that I'm using tabs and not spaces to start the lines for the commands within the rule.
FILE = "myfile.txt"
test :
$(eval YOUR = $(subst my,your,$(FILE)))
cp $(FILE) $(YOUR)
You have to use the eval function in the recipe (Define make variable at rule execution time)
You need to distinguish between what make executes and what the shell executes. Your line with YOUR = starts with a tab and is part of the actions of a rule, so it is executed by the shell, which can't find a program YOUR to execute with some arguments.
Place the expansion outside the rule:
YOUR = $(subst my,your,$(FILE))
test:
cat $(FILE) $(YOUR)
Note that shell assignments require no space around the equals sign, and use ${} rather than $() to reference variables: YOUR=${FILE/my/your} in Bash (and if written in a make rule, you'd need $$ in place of $ so that the shell sees a single dollar sign and make does not try the variable expansion that it doesn't understand). The shell uses $() to execute the command contained within, and the result is often captured in a variable: YOUR=$(echo "${FILE}" | sed 's/my/your/').
If you only need the variable in the shell recipe and not in the make context then you don't need to bother playing with eval (which are hoisted) and can just assign to shell variables instead.
For example:
FILE = "myfile.txt"
test :
YOUR='$(subst my,your,$(FILE))';\
cat $(FILE) "$${YOUR}"
I am learning how to write Makefile. I found the .VARIABLES variable which holds all the variables valid in Makefile.
I can check the variables' name with command like this:
test:
#echo "${.VARIABLES}" | tr ' ' '\n'
But I don't know how to should the values of them. Can anyone teach me to to do that?
If you are using GNU Make, then you can use a crafty combination of foreach and .VARIABLES like this:
test:
$(foreach var,$(.VARIABLES),$(info $(var) = $($(var))))
I have a variable containing list of files separated with string _NEWLINE_. I need to output that variable into a file so that each file is in a separate line. The trick is that it needs to works on FreeBSD and Solaris.
This is what I am trying now:
echo "lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf" | sed 's|_NEWLINE_|\'$'\n|g'
This works on FreeBSD and in shell on Solaris. But when run in GNUmakefile on Solaris I am getting this (notice $ at the end of each line):
lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf$
lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src$
lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf$
If I remove \'$' from sed then it works on Solaris but doesn't on FreeBSD. Maybe there is a way of telling which version to use depending on which system the makefile is executed?
EDIT:
Thanks to the solution proposed by bobbogo I created an exemplary makefile that provides the desired outcome and seems to be working on both FreeBSD and Solaris:
one-line := lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/\
priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/comm\
unity.conf: lib/alarms/priv/snmp_conf/community.conf
many-lines := { echo '$(subst _NEWLINE_,' && echo ',${one-line})'; }
.PHONY: all
all:
$(shell $(many-lines) > test.txt)
If this is GNU make, then do it all in make.
one-line := lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf
define \n
endef
many-lines := $(subst _NEWLINE_,${\n},${one-line})
Now ${many-lines} has just what you want. Annoyingly, it's quite hard to use in shell lines. If you do this:
tgt:
echo '${many-lines}'
make will invoke a separate shell for each line. The first shell invocation will get an un-paired ' and exit with an error.
.ONESHELL:
tgt:
echo '${many-lines}'
will work in an invasive sort of way. The proper fix is to ensure each line of ${many-lines} has valid sh syntax. Some mouthfull like:
echolines = $(subst ${\n},'${\n}echo ',echo '${many-lines}')
.PHONY: aa
aa:
$(call echolines,${many-lines})
Sheesh.
Tried many different solutions, including defining \n as mentioned in this answer: Add a newline in Makefile 'foreach' loop
The real problem is inconsistent implementation of the echo command across platforms, and the fact that by default make invokes shell commands using sh, which itself is quite inflexible.
I found a better way thanks to this answer: "echo -e" when called directly and when called via a shell-script
The better way is to use printf instead of echo
Construct the string with \n instead of _NEWLINE_ to separate parts that go into separate lines in the output file:
some_string = lib/alarms-1.2/priv/snmp_conf/target_params.conf: lib/alarms/priv/snmp_conf/target_params.conf\nlib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf\n
and then in the makefile print it simply as this:
#printf "$(some_string)" >> $(some_file)
Works on both, FreeBSD and Solaris.
Disclaimer: I have no experience with Solaris or FreeBSD... here goes anyway.
In make, you can use $(patsubst pattern,replacement,text) to substitute a pattern.
try this...
FILENAMES := "lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf"
.PHONY: all
all:
#echo $(patsubst _NEWLINE_,${\n},$(FILENAMES))
As an alternative, I think your first approach will work, if you just double the $ to "escape" it:
sed 's|_NEWLINE_|\'$$'\n|g'