Considering a first script (language does not matter, it could be replaced by whatever you like, the idea is what it does: replacing, stopping,..)
my_script.py
for line in sys.stdin:
for line in file:
if line:
if 'plop' in line:
raise Exception("'plop' found, aborting target")
line = re.sub(r'not possible yet', r'nice' , line):
I want to use this script for some of the targets in this simplified makefile:
not_usefull:
#echo "this is not possible yet and will never be"
something_nice:
#echo "this is not possible yet"
something_very_nice:
#echo "this is not possible yet again"
something_bad:
#echo "this is not possible yet plop"
# The following part is not working
# and I don't like to call make in the recipe
# but you get the idea
ifeq ($(MAKING_IMPOSSIBLE_NICE),yes)
something_%:
#$(MAKE) $# | my_script.py
endif
use and expected outputs:
#text replaced
make something_bad MAKING_IMPOSSIBLE_NICE=no
> "this is not possible yet plop"
make something_nice MAKING_IMPOSSIBLE_NICE=no
> "this is not possible yet"
make something_nice MAKING_IMPOSSIBLE_NICE=yes
> "this is nice"
make something_very_nice MAKING_IMPOSSIBLE_NICE=yes
> "this is nice again"
#no effect here because the target does not start by 'something_'
make not_usefull MAKING_IMPOSSIBLE_NICE=no
> "this is not possible yet and will never be"
make not_usefull MAKING_IMPOSSIBLE_NICE=yes
> "this is not possible yet and will never be"
#target aborted because target ouput contains forbidden 'plop'
make something_very_nice MAKING_IMPOSSIBLE_NICE=no
> "this is not possible yet again"
make something_bad MAKING_IMPOSSIBLE_NICE=yes
> "'plop' found, aborting target"
I would like to call a parsing/trimming script (my_script.py) after some specific targets, so the output could be formatted (add colors, replace 'helo' by 'Hello'), and action could be triggered (like aborting target if a specific string appears).
Note that I would like the solution to be working on current and new makefile targets.
I have no control on how many target there will be as they are not created by me.
I have no control on the target output and it could change anytime.
How could I do this?
Related
My knowledge of Make is small. I have been told that everything you put after make (that does not contain "-") is a target.
Well a building process I have is failing.
First there is a line
make path/to/configuration_file
configuration_file is not a target. It is a autogenerated configuration file buried inside the directory structure ("path/to") that is of the form
#
# Boot Configuration
#
#
# DRAM Component
#
CONFIG_DRAM_TYPE_LPDDR4=y
# CONFIG_DRAM_TYPE_DDR4 is not set
CONFIG_DDR_SIZE=0x80000000
#
# Boot Device
#
# CONFIG_ENABLE_EMMC_BOOT is not set
# CONFIG_ENABLE_NAND_BOOT is not set
CONFIG_ENABLE_SPINAND_BOOT=y
# CONFIG_ENABLE_SPINOR_BOOT is not set
CONFIG_EMMC_ACCESS_8BIT=y
# CONFIG_EMMC_ACCESS_4BIT is not set
# CONFIG_EMMC_ACCESS_1BIT is not set
so I cannot understand how this is a target. For reference, when I run make there is a Makefile but this Makefile does not reference this file.
Still this line is going well.
The path where it fails says
make diags
and I have verified there is no "diags" target.
I will print here the error file that can give us more info of what is happening
GEN cortex_a/output/Makefile
Init diag test "orc_scheduler" ...
remoteconfig: Failed to generate configure in cortex_a/soc/visio/tests/orc_scheduler!
Makefile:11 recipe for target 'orc_scheduler-init' failed
make[10]: *** [orc_scheduler-init] Error 25
At least what I would like to know is how to interpret this error message. I don't know what the "11" or the "10" or the "25" refers to.
make is fundamentally a tool for automatically running commands in the right order so you don't have to type them in yourself. So all the commands make runs are commands that you could just type into your shell prompt. And all the errors that those commands generate are the same ones that you would see if you typed the command yourself. So, looking at make to try to understand those errors is looking in the wrong place: you have to look at the documentation for whatever command was invoked.
A "target" is just a file that make knows how to build. The fact that when you typed make <somefile> is didn't give you an error that it doesn't know how to build <somefile>, means that <somefile> is a target as far as your makefiles are concerned.
The error message Makefile:11: simply refers to the filename Makefile, line 11, which is where the command that make ran, that failed, can be found. But this likely won't help you solve the problem of why the command failed (unless the problem is you invoked it with the wrong arguments and you need to adjust the makefile to specify different arguments).
The command that failed generated the message:
remoteconfig: Failed to generate configure in cortex_a/soc/visio/tests/orc_scheduler!
I don't know what that means, but it's not related to make. You'll need to find out what this remoteconfig command is, what it does, and why it failed. It's unfortunate that it doesn't show any better error message as to why it failed to "generate configure", but again there's nothing make can do about that.
If you want to learn more about make you can look at the GNU make manual (note, GNU make is only one implementation of make; there are others and they are fundamentally the same but different in details).
I'm running a metagenomics pipeline in Snakemake. I am running MetaSPAdes for my assemblies, but it's not uncommon that MetaSPAdes will often fail for particular samples. If MetaSPAdes fails, I want to run MEGAHIT on only the samples that it failed for. Is there any way to create this sort of rule dependancy in Snakemake?
For example:
generate a particular file if a rule fails (in this case, assembly with MetaSPAdes). I suppose this would mean that the output of the MetaSPAdes rule needs to be either the contigs, or a "this failed" output file. This would help Snakemake recognize not to re-run this rule.
create a list of samples that the rule failed for, and
run a different rule only on this list of samples with failed MetaSPAdes assemblies (in this case, run MEGAHIT instead on those samples).
Has anyone figured out an elegant way to do something like this?
I'm not familiar with the programs you mention but I think you don't need separate rules for what you need. You can write a single rule that tries to run metaspades first and if it fails try megahit. For example:
rule assembly:
input:
'{sample}.in',
output:
'{sample}.out',
run:
import subprocess
p = subprocess.Popen("MetaSPAdes {input} > {output}", shell= True, stderr= subprocess.PIPE, stdout= subprocess.PIPE)
stdout, stderr= p.communicate()
if p.returncode != 0:
shell("megahit {input} > {output}")
stdout, stderr= p.communicate() captures the stderr, stdout and return code of the process. You can analyse stderr and/or the returncode to decide what to do next. You probably need something more than the above but hopefully the idea is about right.
I am trying to test if certain files, called up in a list of textfiles, are in a certain directory. Every once in a while (and I am quite certain I use the same statements every time) I get an error, complaining that the echo command cannot be found.
The textfiles I have in my directory /audio/playlists/ are named according to their date on which they are supposed to be used: 20130715.txt for example for today:
me#computer:/some/dir# ls /audio/playlists/
20130715.txt 20130802.txt 20130820.txt 20130907.txt 20130925.txt
20130716.txt 20130803.txt 20130821.txt 20130908.txt 20130926.txt
(...)
me#computer:/some/dir# cat /audio/playlists/20130715.txt
#A Comment line goes here
00:00:00 141-751.mp3
00:03:35 141-704.mp3
00:06:42 140-417.mp3
00:10:46 139-808.mp3
00:15:13 136-126.mp3
00:20:26 071-007.mp3
(...)
23:42:22 136-088.mp3
23:46:15 128-466.mp3
23:50:15 129-592.mp3
23:54:29 129-397.mp3
So much for the facts. The following statement, which lets me test if all files called upon in all of the textfiles in the given directory are actually a file in the directory /audio/mp3/, produces an error:
me#computer:/some/dir# for i in $(cat /audio/playlists/*.txt|cut -c 10-16|sort|uniq); do [ -f "/audio/mp3s/$i.mp3" ] || echo $i; done
echo: command not found
me#computer:/some/dir#
I would guess bash wants to complain about the "A Comment"-line (actually " line ") not being a file, but why would that cause echo not to be found? Again, mostly this works, but every so often I get this error. Any help is greatly appreciated.
That space before echo isn't U+0020, it's U+00A0. And indeed, the command " echo" doesn't exist.
is there any way to force the execution of task in Rake, even if the prerequisites are already met?
I am looking for the equivalent of the --always-make option for GNU/make (http://www.gnu.org/software/make/manual/make.html#Options-Summary )
Example Rakefile:
file "myfile.txt" do
system "touch myfile.txt"
puts "myfile.txt created"
end
How would the --always-make option work:
# executing the rule for the first time creates a file:
$: rake myfile.txt
myfile.txt created
# executing the rule a second time returns no output
# because myfile.txt already exists and is up to date
$: rake myfile.txt
# if the --always-make option is on,
# the file is remade even if the prerequisites are met
$: rake myfile.txt --always-make
myfile.txt created
I am running Rake version 0.9.2.2, but I can't find any option in the --help and man pages.
If I undersand you correctly, you can manually execute the task using Rake::Task.
task "foo" do
puts "Doing something in foo"
end
task "bar" => "foo" do
puts "Doing something in bar"
Rake::Task["foo"].execute
end
When you run rake bar, you'll see:
Doing something in foo
Doing something in bar
Doing something in foo
If you use Rake::Task, it will be executed without checking any pre-requisites. Let me know if this doesn't help you.
I barely know enough makefile stuff to hack away at other people's makefiles, and sometimes not even that much. My current employer has a fairly sophisticated make environment, and I added a couple targets to the makefile, mostly to self-document all the switches I needed. (There might be a better way to do this...if there is, please let me know!!)
Anyway, the first target works as expected. The second target is identical to the first, except that the BENCH assignment has a "2" at the end. Otherwise, its the same exact call. At some point during execution, the BENCH variable gets used to create a directory. The first target creates the directory as expected. The second target creates the directory using the default BENCH name instead of the one passed to it. Somehow it lost the variable setting at the command line, even though almost identical syntax worked for the first target.
Why does it lose my variable settings when it calls itself? Is there some make subtlety that I'm missing? Is this even a valid thing to do?
comp_gate_oct04_ff:
make clean all BENCH=GATE_OCT04_FF SIMULATOR=NC SIM_64BIT=1 CORNER=FF NETLIST=oct04 | tee all_gate_oct04_ff.log
#tail all_gate_oct04_ff.log | mail -s "Compile for GATE_OCT04_FF is complete" $(SMS_ADDR) $(USER)
comp_gate_oct04_ff2:
make clean all BENCH=GATE_OCT04_FF2 SIMULATOR=NC SIM_64BIT=1 CORNER=FF NETLIST=oct04 | tee all_gate_oct04_ff2.log
#tail all_gate_oct04_ff2.log | mail -s "Compile for GATE_OCT04_FF2 is complete" $(SMS_ADDR) $(USER)
I can't see why you're getting this error, and I can't reproduce it. Try something simpler:
comp_gate_oct04_ff:
make foo BENCH=GATE_OCT04_FF
comp_gate_oct04_ff2:
make foo BENCH=GATE_OCT04_FF2
foo:
#echo BENCH is $(BENCH)
Tell us how this behaves and we'll go from there.
EDIT:
All right, we've cut the problem in half. Now try this:
comp_gate_oct04_ff:
make clean
make all BENCH=GATE_OCT04_FF SIMULATOR=NC SIM_64BIT=1 CORNER=FF NETLIST=oct04
comp_gate_oct04_ff2:
make clean
make all BENCH=GATE_OCT04_FF2 SIMULATOR=NC SIM_64BIT=1 CORNER=FF NETLIST=oct04