I have a list of objects in a Makefile variable called OBJECTS which is too big for the command buffer. Therefore I'm using the following method to create a file listing the objects (to pass to ar):
objects.lst:
$(foreach OBJ,$(OBJECTS),$(shell echo "$(OBJ)">>$#))
While this works it is extremely slow (on Cygwin at least) and I don't like relying on shell commands and redirection.
Additionlly foreach is not intended for this purpose - it is evaluated before any commands are run which means I can't for example rm -f objects.lst before appending.
Is there a better way? I don't want to use incremental archiving as that causes problems with multiple jobs.
The only thing I can think of is parsing the Makefile with a separate script to read the object list or storing the object list in a separate file. Both solutions have their own problems though.
Try something like:
OBJECTS:=a b c d
objects.lst:
echo > $# <<EOF $(OBJECTS)
i.e. make use of the <<EOF functionality that is built into the shell. It does not have any max-length limitations.
In the following example I also replaced echo with a simple Perl script to split the arguments onto new lines but this is the jist of it..
objects.lst:
echo $(wordlist 1,99,$(OBJECTS))>$#
echo $(wordlist 100,199,$(OBJECTS))>>$#
echo $(wordlist 200,299,$(OBJECTS))>>$#
echo $(wordlist 300,399,$(OBJECTS))>>$#
...
How about something like this:
OBJECTS_AM=$(filter a% b% c% d% e% f% g% h% i% j% k% l% m%,$(OBJECTS))
OBJECTS_NZ=$(filter-out a% b% c% d% e% f% g% h% i% j% k% l% m%,$(OBJECTS))
objects.lst:
$(shell echo "$(OBJECTS_AM)">$#)
$(shell echo "$(OBJECTS_NZ)">>$#)
You might need to split it one or two more times, but it's not that bad, especially as the distribution of file names doesn't change all that often.
Here's a patch to gnu make that lets you directly write a variable into a file.
It creates a new 'writefile' function, similar to the existing 'info' function, except it takes a filename argument and writes to the file:
https://savannah.gnu.org/bugs/?35384
Related
I have a Makefile, trying to loop over a series of strings in a recipe and make them lower case.
My goal: I have a series of commands I would like to run on different files with the appropriate suffix.
# files in directory: test_file1.txt test_file2.txt test_file3.txt
MYLIST = file1 file2 file3
recipe:
for name in $(MYLIST) ; do \
$(eval FILENAME=`echo $($name) | tr A-Z a-z`) \
echo "Final name : test_${FILENAME}.txt" ; \
done
My problem, FILENAME always resolves to blank:
File name: test_.txt
I hope to see:
File name: test_file1.txt
File name: test_file2.txt
File name: test_file3.txt
You cannot mix make functions and shell commands like this. Make works like this: when it decides that your target is out of date and it wants to run the recipe, first it will expand the entire recipe string. Second it sends that expanded string to the shell to run.
So in your case, the $(eval ...) (which is a make operation) is expanded one time, then the resulting string is passed to the shell to run. The shell runs the for loop and all that stuff.
You have to use shell variables here to store values obtained by running your shell for loop. You cannot use make variables or make functions.
In general if you ever think about using $(shell ...) or $(eval ...) inside a recipe, you are probably going down the wrong road.
I have a script that I call with an application, I can't run it from command line. I derive the directory where the script is called and in the next variable go up 1 level where my files are stored. From there I have 3 variables with the full path and file names (with wildcard), which I will refer to as "masks".
I need to find and "do something with" (copy/write their names to a new file, whatever else) to each of these masks. The do something part isn't my obstacle as I've done this fine when I'm working with a single mask, but I would like to do it cleanly in a single loop instead of duplicating loop and just referencing each mask separately if possible.
Assume in my $FILESFOLDER directory below that I have 2 existing files, aaa0.csv & bbb0.csv, but no file matching the ccc*.csv mask.
#!/bin/bash
SCRIPTFOLDER=${0%/*}
FILESFOLDER="$(dirname "$SCRIPTFOLDER")"
ARCHIVEFOLDER="$FILESFOLDER"/archive
LOGFILE="$SCRIPTFOLDER"/log.txt
FILES1="$FILESFOLDER"/"aaa*.csv"
FILES2="$FILESFOLDER"/"bbb*.csv"
FILES3="$FILESFOLDER"/"ccc*.csv"
ALLFILES="$FILES1
$FILES2
$FILES3"
#here as an example I would like to do a loop through $ALLFILES and copy anything that matches to $ARCHIVEFOLDER.
for f in $ALLFILES; do
cp -v "$f" "$ARCHIVEFOLDER" > "$LOGFILE"
done
echo "$ALLFILES" >> "$LOGFILE"
The thing that really spins my head is when I run something like this (I haven't done it with the copy command in place) that log file at the end shows:
filesfolder/aaa0.csv filesfolder/bbb0.csv filesfolder/ccc*.csv
Where I would expect echoing $ALLFILES just to show me the masks
filesfolder/aaa*.csv filesfolder/bbb*.csv filesfolder/ccc*.csv
In my "do something" area, I need to be able to use whatever method to find the files by their full path/name with the wildcard if at all possible. Sometimes my network is down for maintenance and I don't want to risk failing a change directory. I rarely work in linux (primarily SQL background) so feel free to poke holes in everything I've done wrong. Thanks in advance!
Here's a light refactoring with significantly fewer distracting variables.
#!/bin/bash
script=${0%/*}
folder="$(dirname "$script")"
archive="$folder"/archive
log="$folder"/log.txt # you would certainly want this in the folder, not $script/log.txt
shopt -s nullglob
all=()
for prefix in aaa bbb ccc; do
cp -v "$folder/$prefix"*.csv "$archive" >>"$log" # append, don't overwrite
all+=("$folder/$prefix"*.csv)
done
echo "${all[#]}" >> "$log"
The change in the loop to append the output or cp -v instead of overwrite is a bug fix; otherwise the log would only contain the output from the last loop iteration.
I would probably prefer to have the files echoed from inside the loop as well, one per line, instead of collect them all on one humongous line. Then you can remove the array all and instead simply
printf '%s\n' "$folder/$prefix"*.csv >>"$log"
shopt -s nullglob is a Bash extension (so won't work with sh) which says to discard any wildcard which doesn't match any files (the default behavior is to leave globs unexpanded if they don't match anything). If you want a different solution, perhaps see Test whether a glob has any matches in Bash
You should use lower case for your private variables so I changed that, too. Notice also how the script variable doesn't actually contain a folder name (or "directory" as we adults prefer to call it); fixing that uncovered a bug in your attempt.
If your wildcards are more complex, you might want to create an array for each pattern.
tmpspaces=(/tmp/*\ *)
homequest=($HOME/*\?*)
for file in "${tmpspaces[#]}" "${homequest[#]}"; do
: stuff with "$file", with proper quoting
done
The only robust way to handle file names which could contain shell metacharacters is to use an array variable; using string variables for file names is notoriously brittle.
Perhaps see also https://mywiki.wooledge.org/BashFAQ/020
If I have a list of things such as this:
nodes = A B C
How would I generate dynamic targets that have dynamic prerequisites. For example (this isn't working but might help explain what I want).
# node.csr is a file that already exists, like a template
# this would create outputs like node-A-csr.json, node-B-csr.json
# I am basically guessing at the syntax here
node-%-csr.json: node-csr.json
sed 's/$${node}/$*' node-csr.json > $#
# this would create outputs like node-A-key.pem node-A.pem and would require node-A-csr.json
node-%-key.pem node-%.pem: node-%-csr.json
# some command that generates node-NAME-key.pem and node-NAME-csr.pem
$(nodes): node-%-key.pem node-%.pem
Id basically like to be able to run make all and have those targets run for everything in my list.
I am very new to Makefiles and I just dont see how something like this would work and the documentation and syntax of Make is extremely confusing to me.
I am willing to use any tool to do this but seems like Make is very standard.
You can use Make's Substitution References to generate the 'all' target. This will be enough to start processing of all rules.
Notice minor change: the 'node-csr.json' should have the token NODE where the actual node name is to be inserted
# First rule is the default
default: all
nodes = A B C
# use node-csr as a template, replacing NODE with current ID: A, B, C
node-%-csr.json: node-csr.json
sed 's/__NODE__/$*/' node-csr.json > $#
# this would create node-A-key.pem node-A.pem from node-A-csr.json
# Put real commands here
node-%-key.pem node-%.pem: node-%-csr.json
ls -l node-$*-csr.json > node-$*-key.csr
ls -l node-$*-csr.json > node-$*.pem
# all will be node-A-key.pem, node-B-key.pem, ... node-C.pem
all: $(nodes:%=node-%-key.pem) $(nodes:%=node-%.pem)
echo "done"
# Using pathsubst: all: $(patsubst %,node-%-key.pem,$(nodes)) $(pathsubst
Please pay attention to tab/spaces, some version are sensitive. You will have to put back tabs before all commands (sed, ls, ...)
Substitution References man: https://www.gnu.org/software/make/manual/html_node/Substitution-Refs.html
pathsubst function: https://www.gnu.org/software/make/manual/html_node/Text-Functions.html
I have a text file called OPTIONS.txt storing all flags of Makefile:
arg1=foo arg2="-foo -bar"
I want to pass all flags in this file to make. However,
make `cat OPTIONS.txt`
fails with make: invalid option -- 'a'. It seems that shell interprets it as:
make arg1=foo arg2="-foo -bar"
^argv[1] ^argv[2] ^argv[3]
Is there any way to make it interpreted as:
make arg1=foo arg2="-foo -bar"
^argv[1] ^--------argv[2]
Since you control the options file, store the options one per line:
arg1=foo
arg2="-foo -bar"
Then in the shell, you'll read the file into an array, one element per line:
readarray -t opts < OPTIONS.txt
Now you can invoke make and keep the options whole:
make "${opts[#]}"
If you want the shell to interpret quotes after backtick expansion you need to use eval, like this:
eval make `cat OPTIONS.txt`
however just be aware that this evaluates everything, so if you have quoted content outside of the backticks you'll get the same issue:
eval make `cat OPTIONS.txt` arg4="one two"
will give an error. You'd have to double-quote the arg4, something like this:
eval make `cat OPTIONS.txt` arg4='"one two"'
In general it's tricky to do stuff like this from the command line, outside of scripts.
ETA
The real problem here is that we don't have a set of requirements. Why do you want to put these into a file, and what kind of things are you adding; are they only makefile variable assignments, or are there other make options here as well such as -k or similar?
IF the OP controls (can change) the format of the file AND the file contains content only used by make AND the OP doesn't care about the variables being command line assignments vs. regular assignments AND there are only variable assignments and not other options, then they can just (a) put each variable assignment on its own line, (b) remove all quotes, and (c) use include OPTIONS.txt from inside the makefile to "import" them.
I'm using Makefiles with "make" for a lot of things like starting / stopping / configuring services I've written. Sometimes I'd like to read an input from the user. The only ways I know are either make the user pass his input with NAME=VALUE when executing make, or by putting a command like read -p "setting X: " var ; echo $$var into the Makefile.
NAME=VALUE has the disadvantage that the user must manually set it and I can't "ask" him to enter a value. read has the disadvantage that the read value can not (or I don't know how) be saved in a variable and so it can't be used multiple times.
Is there a way to read user input into a variable during executing a specific makefile target? (I don't want to put FILE ?= 'read -p "value: " var ; echo $$var' in the header because the value is only needed for one target, and when I put that line in the target itself, I get the error "/bin/bash: FILE: Command not found. ".
I use intermediate files for this purpose.
INPUT = dialog --inputbox 80 10 10
all: case1 case2
case1: read-input
echo $(shell cat read-input) in case 1
case2: read-input
echo $(shell cat read-input) in case 2
.INTERMEDIATE: read-input
read-input:
$(INPUT) 2>$#