How do I run multiple rake tasks at once? - ruby

Question

I'm using a Rake task that runs multiple scraping scripts and exports category data for 35 different cities of a site to 35 different CSV files.
The problem I'm having is that when I run the master Rake task from the root directory of the folder, it creates a new file in the parent directory "resultsForCity.csv" instead of seeing the current CSV file within that given subfolder and adding the data to it. To get around it, I thought I should make my master Rake task (within the parent directory) run slave Rake tasks that then run the scraping scripts, but that didn't work either.
However, if I cd into one of the city folders and run the scraper or Rake task from there, it adds the data to the corresponding CSV file located within that subfolder. Am I not clearly defining dependencies or something else?
Things I've tried:
I've tried requiring each individual rakefile within my master rake task.
Tried iterating over all files and loading the rake tasks and received a stack too deep error.
Tried searching on Stackoverflow for 7 days now.
Here's my Rake task code:
require "rake"
task default: %w[getData]
task :getData do
Rake::FileList.new("**/*.rb*").each do |file|
ruby file
end
end
And here's my scraper code:
require "nokogiri"
require "open-uri"
require "csv"
url = "http:// example.com/atlanta"
doc = Nokogiri::HTML(open(url))
CSV.open("resultsForAtlanta.csv", "wb") do |csv|
doc.css(".tile-title").each do |item|
csv << [item.text.tr("[()]+0-9", ""), item.text.tr("^0-9$", "")]
end
doc.css(".tile-subcategory").each do |tile|
csv << [tile.text.tr("[()]+0-9", ""), tile.text.tr("^0-9$", "")]
end
end
Any help would be more than greatly appreciated.

Answer 1

What if you let your scraper script take an output filename and use the directory structure to help you build the output filenames.
Assuming you have a directory tree something like
Atlanta/scraper.rb
LosAngeles/scraper.rb
...
where scraper.rb is your scraping script, you should be able to write the task somewhat like this:
task :getData do
Rake::FileList.new("**/scraper.rb").each do |scraper_script|
dir = File.dirname(file)
city = File.basename(dir)
csv_file = File.join(dir, "resultsFor#{city}.csv")
ruby [scraper_script, csv_file].join(" ")
end
end
and then your Ruby script could just grab the filename off the command line like this:
CSV.open(ARGV[1], "wb") do |csv|
...
end