I want in my makefile to retrieve all textfiles (at the moment i just got one) from a certain subfolder
and call in a loop a specific python script with each text file as an input parameter.
This is the code i currently have:
run_analysis:
#echo "Get text files"
txt_files=$(wildcard ./input/*.txt)
#echo "Current text files are:"
#echo $(txt_files)
for txt_file in $(txt_files); do \
#echo "Iteration" \
#echo $(txt_file ) \
python ./scripts/my_test_script.py $(txt_file ) ; \
done
It seems the wildcard results are not stored in the variable.
My output looks the following:
Get text files
txt_files=./input/test_text_1.txt
Current text files are:
for txt_file in ; do \
#echo "Iteration" \
#echo \
python ./scripts/my_test_script.py ; \
done
Each line in a Makefile recipe is executed in a separate shell instance by default.
Saving the files in a variable doesn't appear to serve any useful purpose anyway. Just inline the wildcard.
run_analysis:
for txt_file in ./input/*.txt; do \
python ./scripts/my_test_script.py "$$txt_file"; \
done
(Notice also how txt_file is a shell variable, not a Make variable.)
Better yet, change your Python script so it accepts a list of input files.
run_analysis:
python ./scripts/my_test_script.py ./input/*.txt
Maybe add incessant chatter with logging.debug() inside the Python script if you want to see exactly what it's doing. Unlike hard-coded echo, logging can easily be turned off once you are confident that your code works.
Related
I have a Makefile, trying to loop over a series of strings in a recipe and make them lower case.
My goal: I have a series of commands I would like to run on different files with the appropriate suffix.
# files in directory: test_file1.txt test_file2.txt test_file3.txt
MYLIST = file1 file2 file3
recipe:
for name in $(MYLIST) ; do \
$(eval FILENAME=`echo $($name) | tr A-Z a-z`) \
echo "Final name : test_${FILENAME}.txt" ; \
done
My problem, FILENAME always resolves to blank:
File name: test_.txt
I hope to see:
File name: test_file1.txt
File name: test_file2.txt
File name: test_file3.txt
You cannot mix make functions and shell commands like this. Make works like this: when it decides that your target is out of date and it wants to run the recipe, first it will expand the entire recipe string. Second it sends that expanded string to the shell to run.
So in your case, the $(eval ...) (which is a make operation) is expanded one time, then the resulting string is passed to the shell to run. The shell runs the for loop and all that stuff.
You have to use shell variables here to store values obtained by running your shell for loop. You cannot use make variables or make functions.
In general if you ever think about using $(shell ...) or $(eval ...) inside a recipe, you are probably going down the wrong road.
I have a make target, that first calls a CAE tool which generates reports. When this is done, the target calls a python script that shall take the content of the CAE reports (or more specific some grep'ed lines of the reports) as argument.
A minimum example is
target1:
date > ./bar.txt
echo $(shell cat ./bar.txt)
Problem is, that make expands the $(shell cat ./bar.txt) before the first command has been called and bar.txt has been updated. So for this minimum example, the echo prints the content of bar.txt before the update (the date from the previous target run).
(
I know that I simply could write this example in another way without variables and the shell function call, this is just for the sake of showing the problem where I call a tool that takes an argument from a shell call. So actually I want to do sth like this:
target1:
cae_tool_call
report_eval.py -text "$(shell cat $(generated_report) | grep 'foo')"
where cae_tool_call generates the generated_report. And this -text "argument" does not resolve the argument without an explicit call of the shell function.
)
I already tried with actual shell variables (instead of make variables), double escapes, immediate vs deferred variables but have no working solution yet. Any ideas?
#######################################
Edit to show some unexpected behavior:
I have this python script argument_example.py
import argparse
def main():
parser = argparse.ArgumentParser()
parser.add_argument("-r", "--reporttext", help="text string", required=True)
args=parser.parse_args()
if args.reporttext:
print(args.reporttext)
main()
It just prints the text given with argument -r.
And I have these two make targets:
####################################
#this does not work
REPORTNAME := ./bar.txt
variable_report:
date > $REPORTNAME
python3 ./argument_example.py --reporttext "`(cat $REPORTNAME)`"
####################################
#this works
static_report:
date > ./bar.txt
python3 ./argument_example.py --reporttext "`(cat ./bar.txt)`"
When calling variable_report, the python scripts prints the outdated bar.txt content. When calling static_report, the python script prints the updated content.
make recipes are already shell scripts. Never use the shell make function inside a recipe. In your first simple example use:
target1:
date > bar.txt
cat bar.txt
In your other example use:
generated_report := name-of-generated-report
target1:
cae_tool_call
report_eval.py -text "`cat $(generated_report) | grep 'foo'`"
Or even better:
generated_report := name-of-generated-report
target1:
cae_tool_call
report_eval.py -text "`grep 'foo' $(generated_report)`"
Referencing a previous answer, but still having difficulties converting my folder of .md files:
for i in src/*.md; do perl markdown/Markdown.pl --html4tags $i > output/${i%.*}.html; done;
Unfortunately (for my test file "index.md")it's throwing the error:
line 11: output/src/index.html: No such file or directory
I'm not sure how to get it to direct output to just "output/index.html".
Any thoughts? (I'm not interested in using another soluton like pandoc, just trying to do this in bash)
The expansion of src/*.md will yield elements that all start with src/. You can remove the path of a file, yield only the filename sans directory, with dirname.
Since you're using the ${variable%match} replacement pattern to replace .md with .html, it would probably be easiest to create a new variable, here $j, to hold the results of basename.
for i in src/*.md; do j="$(basename $i)"; perl markdown/Markdown.pl --html4tags $i > output/${j%.*}.html; done;
The error message means that the directory output/src, relative to the working directory in which the command is executed, does not exist. You can do a
mkdir -p output/src; for i in ....
You can avoid the for loop, take advantage of modern multi-core CPUs, test what its going to do in advance without actually doing anything and get everything done in parallel with GNU Parallel like this:
parallel --dry-run perl markdown/Markdown.pl --html4tags {} \> output/{/.}.html ::: src/*md
Sample Output
perl markdown/Markdown.pl --html4tags src/a.md > output/a.html
If that looks correct, run it again but without the --dry-run to do it for real.
I've put some common utility scripts into common.sh, which I want to use in my RPM specfile during %pre. common.sh is located in the root of the RPM package.
What I was planning to do is simply call something like source common.sh, but how can I access common.sh from the RPM during %pre?
I was able to solve this using RPM macros, the following way:
Before doing rpmbuild I have put common.spec into the SPECS folder.
common.spec
%define mymacro() (echo -n "My arg is %1 " ; sleep %1 ; echo done.)
I've added %include SPECS/common.spec as the first line of my actual spec file.
Usage example
%pre
%mymacro 5
My arg is 5 done.
Multi-line macros
Pretty fragile syntactically imo, but you can put line breaks into your macros using \. That will tell the RPM builder that it should continue parsing the macro. Given the previous macro as an example:
%define mymacro() (echo -n "My arg is %1 " ; \
sleep %1 ; \
echo done.)
This way the code will be still parsed back into a single line, hence the ; on the first and second line.
I have a list of objects in a Makefile variable called OBJECTS which is too big for the command buffer. Therefore I'm using the following method to create a file listing the objects (to pass to ar):
objects.lst:
$(foreach OBJ,$(OBJECTS),$(shell echo "$(OBJ)">>$#))
While this works it is extremely slow (on Cygwin at least) and I don't like relying on shell commands and redirection.
Additionlly foreach is not intended for this purpose - it is evaluated before any commands are run which means I can't for example rm -f objects.lst before appending.
Is there a better way? I don't want to use incremental archiving as that causes problems with multiple jobs.
The only thing I can think of is parsing the Makefile with a separate script to read the object list or storing the object list in a separate file. Both solutions have their own problems though.
Try something like:
OBJECTS:=a b c d
objects.lst:
echo > $# <<EOF $(OBJECTS)
i.e. make use of the <<EOF functionality that is built into the shell. It does not have any max-length limitations.
In the following example I also replaced echo with a simple Perl script to split the arguments onto new lines but this is the jist of it..
objects.lst:
echo $(wordlist 1,99,$(OBJECTS))>$#
echo $(wordlist 100,199,$(OBJECTS))>>$#
echo $(wordlist 200,299,$(OBJECTS))>>$#
echo $(wordlist 300,399,$(OBJECTS))>>$#
...
How about something like this:
OBJECTS_AM=$(filter a% b% c% d% e% f% g% h% i% j% k% l% m%,$(OBJECTS))
OBJECTS_NZ=$(filter-out a% b% c% d% e% f% g% h% i% j% k% l% m%,$(OBJECTS))
objects.lst:
$(shell echo "$(OBJECTS_AM)">$#)
$(shell echo "$(OBJECTS_NZ)">>$#)
You might need to split it one or two more times, but it's not that bad, especially as the distribution of file names doesn't change all that often.
Here's a patch to gnu make that lets you directly write a variable into a file.
It creates a new 'writefile' function, similar to the existing 'info' function, except it takes a filename argument and writes to the file:
https://savannah.gnu.org/bugs/?35384