Let's say I have a target who needs to compile some files. That target has another target as a prerequisite, one that obtains the files.
Let's say this:
task :obtain do
# obtain files from somewhere
end
task :compile => :obtain do
# do compilation
end
Let's say that the :obtain target doesn't always places the files in the same folder. How would I pass :compile the path that :obtain found? Environment variables?
Using ENV['something'] is in my opinion preferable, because if you do it this way (as opposed to $global or #instance variables) you can treat those as task arguments, and use the sub task from commandline easily.
On the other hand if you keep your code in separate classes / modules / methods, you will not even have to deal with those sorts of hacks + your code will be more testable.
One way would be to store it in a global variable:
task :obtain do
$obtained_dir = "/tmp/obtained"
end
task :compile => :obtain do
puts "compiling files in #{$obtained_dir}"
end
Instance variables (i.e. #obtained_dir) should also work.
Another way would be to pull the "obtain" code into a method, as follows:
task :obtain do
obtain_files
end
task :compile do
obtained_dir = obtain_files
puts "compiling files in #{obtained_dir}"
end
def obtain_files
#obtain files from somewhere
"/tmp/obtained_files"
end
Related
AFAIK, Rake comes with two types of tasks: Rake::Task which runs unconditionally, and Rake::FileTask which runs only if the file it is named after doesn't exist, or is older that one of its prerequisites.
Is there a conventional way to customize the logic that decides if a task needs to run? For example, if I wanted to not only verify the existence of a file, but also test its contents somehow.
I can see the method Rake::Task#needed? handles this, and overriding that in a subclass does indeed work. But is there a more idiomatic way to do this? Something that would be more suitable to include directly in a Rakefile?
I'm imagining something like this:
need :process do
# Check if file is already processed
end
task :process do
# Process file in-place
end
which would skip the task if all of its need blocks return true.
Is there a conventional way to customize the logic that decides if a
task needs to run?
Yes; a way to do it is:
declare the "needed tasks" as task_1, and let it exist if needed operations aren't proceeded
declare the "secondary tasks" as task_2 with prerequisite: task_1
so your rake file will be like:
# check if needed is done
def needed_done?
return false # TODO: edit checking logic
end
desc "do prerequisite stuff"
task :do_needed do
p "do needed stuff"
unless needed_done?
p "needed stuff wasn't done ^^'"
exit 1
end
end
desc "process other stuff, if prerequisite is meet"
task :process => [:do_needed] do
p "process other stuff"
end
now when you ran the task process with:
rake process
do_needed will automatically ran first, if needed_done? then process will run, else you'll exit without running it
I'm using rake to help compile Coffeescript for a Chrome extension I'm writing.
My Rakefile looks like this:
COFFEE = FileList['src/*.coffee']
JS = COFFEE.ext 'js'
directory 'extension'
rule '.js' => ['.coffee', 'extension'] do |t|
`coffee -c -o extension #{t.source}`
end
desc "Build the extension in the 'extension' directory"
task :build => ['extension', JS] do
cp File.join('src', 'manifest.json'), 'extension'
end
When I only have one .coffee file in my src directory, there's no problem. But as soon as I have more than one .coffee files it errors:
$ rake build
> rake aborted!
> Don't know how to build task 'src/app.js src/background.js'
>
> Tasks: TOP => build
> (See full trace by running task with --trace)
Is it possible to specify a FileList as a dependency? How else would I tell rake that I want all my Coffeescript files compiled durring the build task?
Rake’s dependency list is an Array of task names. When you use a FileList as one of its elements, you nest arrays – effectively, this:
task :build => ['extension', ['src/app.js', 'src/background.js']] do
Rake just uses the String representation of all passed dependency Array elements, which is why it complains about being unable to build a 'src/app.js src/background.js' task (note how this is one string).
Splatting your FileList (or flattening the dependency Array) will solve the issue, i.e.:
task :build => ['extension', *JS] do
or
task :build => ['extension', JS].flatten do
Try this:
files = Dir.entries('path/to/scripts').select { |f| f.include? '.coffee' }
files.each do |file_path|
`coffee -c -o extension #{file_path}`
end
So far in my search it looks like the only way to accomplish what I want is to either have a task which loops through my FileList and compiles each one explicitly (like in the answer from #nicooga). Or, I can loop through everything in the FileList and add it as a dependency to the build task.
I don't like either of these because rake has FileLists for getting groups of files, rules for defining how to handle kinds of files, and a nice syntax for defining dependencies, but apparently no way to combine all three of those together.
So, my solution for now is to go with the second option, adding each file as a dependency. The shortest way I've found to do this is to concat the FileList onto the dependency array. So now my build task looks like this:
task :build => ['extension'].concat(JS) do
cp File.join('src', 'manifest.json'), 'extension'
end
And thanks to the comment by #kopischke, this can even be shortened to ['extension' *JS] using the splat operator.
Late to the game, but a FileList is lazy, and that is useful, especially if you have lots of file matching.
Think of C or C++ where you potentially can have many that require dependencies.
Most of the answers here require the FileList to be expanded/evaluated. Since Rake is Ruby and task is where the last argument is basically a Hash, you will be evaluating the array.
One of the answers suggests ['extension', JS].flatten. This approach will search the directory and collect all the .js files. This approach is fine when you have a task that will always get executed/invoked. However, you don't want to evaluate the FileList and invoke its search if the task is will not get executed/invoked.
The best way to use a FileList is the following
fl = FileList['extension', "src/*.coffee"] do |c|
c.ext('js')
end
file "somefile" => fl do
#some stuff
end
Then if "somefile" doesn't exist, or the "somefile" task is not even invoked, the read of your Rakefile doesn't invoke the FileList expansion, saving a bunch of time.
I've got a helper class that scans my entire project directory and collects a list of source files and the corresponding (target) object files. The dependencies on the compile task is defined after scanning the source directory as shown below.
CLEAN.include(FileList[obj_dir + '**/*.o'])
CLOBBER.include(FileList[exe_dir + '**/*.exe'])
$proj = DirectoryParser.new(src_dir)
$proj.source_files.each do |source_file|
file source_file.obj_file do
sh "gcc -c ..."
end
end
$proj.obj_files.each do |obj_file|
task :compile => obj_file
end
task :compile do
end
Since $proj is global, the DirectoryParser.new() is invoked when any of the tasks are called including clean and clobber. This makes the clean and clobber tasks slow and that is not desirable.
To get around the problem I moved all the generation of File dependencies into the default task. This makes my clean and clobber tasks fast, however, I can't call my compile or link tasks independently now.
CLEAN.include(FileList[obj_dir + '**/*.o'])
CLOBBER.include(FileList[exe_dir + '**/*.exe'])
task :compile => $proj.source_files do # Throws error!
end
task :default => do
$proj = DirectoryParser.new(src_dir)
$proj.source_files.each do |source_file|
file source_file.obj_file do
sh "gcc -c ..."
end
end
$proj.obj_files.each do |obj_file|
task :compile => obj_file
end
... compile
... link
... execute
end
How do I get around this problem? I am sure someone has previously encountered a similar problem. I'd appreciate any help.
You could try a two step approach.
Create a new task generate_dependencies.
This task builds a (static) rake file with your dependencies and actions.
This generated rakefile can be loaded in your rake file.
Some sample code (untested):
GENERATED = 'generated_dependencies.rb'
task :generate_dependencies do
$proj = DirectoryParser.new(src_dir)
File.open(GENERATED, 'w') do |f|
$proj.source_files.each do |source_file|
f << <<-code
file #{source_file.obj_file} do
sh "gcc -c " #etc.
end
code
end
$proj.obj_files.each do |obj_file|
f << "task :compile => #{obj_file}"
end
#~ ... compile
#~ ... link
#~ ... execute
end
end
require GENERATED
Now you have two steps:
create an empty 'generated_dependencies.rb' (so you get no error when you call the script the first time)
call rake generate_dependencies
Check the generated file - if it's not good, change the generator ;)
call rake compile or rake link (or rake if you want to use the default task) ... - the dependencies are defined in the generated file.
When something changes (new files), continue from step 2.
If the structure stays the same (no new files, only code changes) you need only step 4.
I managed to get around this problem elegantly by using the Singleton design pattern and moving away from using Rake file/task dependencies completely. DirectoryParser is now a singleton class (by mixing in Ruby's built-in 'singleton' library)
CLEAN.include(FileList[obj_dir + '**/*.o'])
CLOBBER.include(FileList[exe_dir + '**/*.exe'])
task :compile do
$proj = DirectoryParser.instance
$proj.source_files.each do |source_file|
sh "gcc -c ..." unless uptodate?(obj_file, source_file)
end
end
task :link do
$proj = DirectoryParser.instance
...
end
Now my clean/clobber tasks are fast and I can still call compile/link tasks independently.
I'm building a specialized pipeline, and basically, every step in the pipeline involves taking one file as input and creating a different file as output. Not all files are in the same directory, all output files are of a different format, and because I'm using several different programs, different actions have to be taken to appease the different programs.
This has led to some complicated file management in my code, and the more I try to organize the file directories, the more ugly it's getting. Just about every class involves some sort of code like the following:
#fileName = File.basename(file)
#dataPath = "#{$path}/../data/"
MzmlToOther.new("mgf", "#{#dataPath}/spectra/#{#fileName}.mzML", 1, false).convert
system("wine readw.exe --mzXML #{#file}.raw #{$path}../data/spectra/#{File.basename(#file + ".raw", ".raw")}.mzXML 2>/dev/null")
fileName = "#{$path}../data/" + parts[0] + parts[1][6..parts[1].length-1].chomp(".pep.xml")
Is there some sort of design pattern, or ruby gem, or something to clean this up? I like writing clean code, so this is really starting to bother me.
You could use a Makefile.
Make is essential a DSL designed for handling converting one type of file to another type via running an external program. As an added bonus, it will handle only performing the steps necessary to incrementally update your output if some set of source files change.
If you really want to use Ruby, try a rakefile. Rake will do this, and it's still Ruby.
You can make this as sophisticated as you want but this basic script will match a file suffix to a method which you can then call with the file path.
# a conversion method can be used for each file type if you want to
# make the code more readable or if you need to rearrange filenames.
def htm_convert file
"HTML #{file}"
end
# file suffix as key, lambda as value, the last uses an external method
routines = {
:log => lambda {|file| puts "LOG #{file}"},
:rb => lambda {|file| puts "RUBY #{file}"},
:haml => lambda {|file| puts "HAML #{file}"},
:htm => lambda {|file| puts htm_convert(file) }
}
# this loops recursively through the directory and sub folders
Dir['**/*.*'].each do |f|
suffix = f.split(".")[-1]
if routine = routines[suffix.to_sym]
routine.call(f)
else
puts "UNPROCESSED -- #{f}"
end
end
I have a Rakefile that compiles the project in two ways, according to the global variable $build_type, which can be :debug or :release (the results go in separate directories):
task :build => [:some_other_tasks] do
end
I wish to create a task that compiles the project with both configurations in turn, something like this:
task :build_all do
[ :debug, :release ].each do |t|
$build_type = t
# call task :build with all the tasks it depends on (?)
end
end
Is there a way to call a task as if it were a method? Or how can I achieve anything similar?
If you need the task to behave as a method, how about using an actual method?
task :build => [:some_other_tasks] do
build
end
task :build_all do
[:debug, :release].each { |t| build t }
end
def build(type = :debug)
# ...
end
If you'd rather stick to rake's idioms, here are your possibilities, compiled from past answers:
This always executes the task, but it doesn't execute its dependencies:
Rake::Task["build"].execute
This one executes the dependencies, but it only executes the task if
it has not already been invoked:
Rake::Task["build"].invoke
This first resets the task's already_invoked state, allowing the task to
then be executed again, dependencies and all:
Rake::Task["build"].reenable
Rake::Task["build"].invoke
Note that dependencies already invoked are not automatically re-executed unless they are re-enabled. In Rake >= 10.3.2, you can use the following to re-enable those as well:
Rake::Task["build"].all_prerequisite_tasks.each(&:reenable)
for example:
Rake::Task["db:migrate"].invoke
task :build_all do
[ :debug, :release ].each do |t|
$build_type = t
Rake::Task["build"].reenable
Rake::Task["build"].invoke
end
end
That should sort you out, just needed the same thing myself.
task :invoke_another_task do
# some code
Rake::Task["another:task"].invoke
end
task :build_all do
[ :debug, :release ].each do |t|
$build_type = t
Rake::Task["build"].execute
end
end
If you want each task to run regardless of any failures, you can do something like:
task :build_all do
[:debug, :release].each do |t|
ts = 0
begin
Rake::Task["build"].invoke(t)
rescue
ts = 1
next
ensure
Rake::Task["build"].reenable # If you need to reenable
end
return ts # Return exit code 1 if any failed, 0 if all success
end
end
I would suggest not to create general debug and release tasks if the project is really something that gets compiled and so results in files. You should go with file-tasks which is quite doable in your example, as you state, that your output goes into different directories.
Say your project just compiles a test.c file to out/debug/test.out and out/release/test.out with gcc you could setup your project like this:
WAYS = ['debug', 'release']
FLAGS = {}
FLAGS['debug'] = '-g'
FLAGS['release'] = '-O'
def out_dir(way)
File.join('out', way)
end
def out_file(way)
File.join(out_dir(way), 'test.out')
end
WAYS.each do |way|
desc "create output directory for #{way}"
directory out_dir(way)
desc "build in the #{way}-way"
file out_file(way) => [out_dir(way), 'test.c'] do |t|
sh "gcc #{FLAGS[way]} -c test.c -o #{t.name}"
end
end
desc 'build all ways'
task :all => WAYS.map{|way|out_file(way)}
task :default => [:all]
This setup can be used like:
rake all # (builds debug and release)
rake debug # (builds only debug)
rake release # (builds only release)
This does a little more as asked for, but shows my points:
output directories are created, as necessary.
the files are only recompiled if needed (this example is only correct for the simplest of test.c files).
you have all tasks readily at hand if you want to trigger the release build or the debug build.
this example includes a way to also define small differences between debug and release-builds.
no need to reenable a build-task that is parametrized with a global variable, because now the different builds have different tasks. the codereuse of the build-task is done by reusing the code to define the build-tasks. see how the loop does not execute the same task twice, but instead created tasks, that can later be triggered (either by the all-task or be choosing one of them on the rake commandline).