It's first time I'm using rake, and I figure a problem with file dependencies.
To make a test, in Rakefile.rb I put this code:
task :ffile do
f1 = "config.yaml"
f2 = "Rakefile.rb"
if File.file? f1 then puts "## test file task on #{f1}" end
if File.file? f2 then puts "## test file task on #{f2}" end
file "#{f1}" => "#{f2}" do
puts "lol"
end
file "#{f2}" => "#{f1}" do
puts "lul"
end
file "#{f1}" do
puts "lil"
end
file "#{f2}" do
puts "lal"
end
end
I'm on Windows 10, and when run
rake ffile
the result is
Starting rake operations...
## test file task on config.yaml
## test file task on Rakefile.rb
that is file method do nothing in all four cases. I tried also to remove the quote (i.e. f1 instead "#{f1}" and so on in all file) but obtain the same result.
Clearly every time I save Rakefile.rb while testing, so I'm sure that should be trigger one of the file methos.
There's syntax error? It's troubleshoot with Windows 10?
Thanks
The file method is actually a DSL to define a file task https://github.com/ruby/rake/blob/b63f7d33d72782f646ef95da10300d273c9b8006/lib/rake/dsl_definition.rb#L75, which has to be invoked manually.
Rake::Task[f1].invoke
Rake::Task[f2].invoke
Since your dsl definitions have circular dependency: TOP => config.yaml => Rakefile.rb => config.yaml, adding the above code will raise an error.
But I think you could get the idea how to invoke the file tasks.
Thanks larrylv, I've understood!
I have two errors:
circular, as larrylv said
and I don't invoke the file task! (larrylv said)
So I report a simple example that works
f1 = "config.yaml"
f2 = "Rakefile.rb"
task :ffile => ["#{f1}"]
file "#{f1}" => ["#{f2}"] do
puts "lul"
end
With this, the command rake ffile works as aspected.
Thanks!
Related
I'm writing a script that Creek and an .xlsx file and uses it to update the prices and weights of products in a database. The .xlsx file is located on an AWS server, so Creek copies the file down and stores it in a Tempfile while it is in use.
The issue is, at some point the Tempfile seems to be prematurely deleted, and since Creek continues to call on it whenever it iterates through a sheet, the script fails. Interestingly, my coworker's environment runs the script fine, though I haven't found a difference between what we're running.
Here is the script I've written:
require 'creek'
class PricingUpdateWorker
include Sidekiq::Worker
def perform(filename)
# This points to the file in the root bucket
file = bucket.files.get(filename)
# Make public temporarily to open in Creek
file.public = true
file.save
creek_sheets = Creek::Book.new(file.public_url, remote: true).sheets
# Close file to public
file.public = false
file.save
creek_sheets.each_with_index do |sheet, sheet_index|
p "---------- #{sheet.name} ----------"
sheet.simple_rows.each_with_index do |row, index|
next if index == 0
product = Product.find_by_id(row['A'].to_i)
if product
if row['D']&.match(/N\/A/) || row['E']&.match(/N\/A/)
product.delete
p '*** deleted ***'
else
product.price = row['D']&.to_f&.round(2)
product.weight = row['E']&.to_f
product.request_for_quote = false
product.save
p 'product updated'
end
else
p "#{row['A']} | product not found ***"
end
end
end
end
private
def connection
#connection ||= Fog::Storage.new(
provider: 'AWS',
aws_access_key_id: ENV['AWS_ACCESS_KEY_ID'],
aws_secret_access_key: ENV['AWS_SECRET_ACCESS_KEY']
)
end
def bucket
# Grab the file from the bucket
#bucket ||= connection.directories.get 'my-aws-bucket'
end
end
And the logs:
"---------- Sheet 1 ----------"
"product updated"
"product updated"
... I've cut out a bunch more of these...
"product updated"
"product updated"
"---------- Sheet 2 ----------"
rails aborted!
Errno::ENOENT: No such file or directory # rb_sysopen - /var/folders/9m/mfcnhxmn1bqbm6h91rx_rd8m0000gn/T/file20190920-19247-c6x4zw
"/var/folders/9m/mfcnhxmn1bqbm6h91rx_rd8m0000gn/T/file20190920-19247-c6x4zw" is the temporary file, and as you can see, it's been collected already, even though I'm still using it, and I believe it is still in scope. Any ideas what could be causing this? It's especially odd that my coworker can run this just fine.
In case it's helpful, here is a little code from Creek:
def initialize path, options = {}
check_file_extension = options.fetch(:check_file_extension, true)
if check_file_extension
extension = File.extname(options[:original_filename] || path).downcase
raise 'Not a valid file format.' unless (['.xlsx', '.xlsm'].include? extension)
end
if options[:remote]
zipfile = Tempfile.new("file")
zipfile.binmode
zipfile.write(HTTP.get(path).to_s)
# I added the line below this one, and it fixes the problem by preventing the file from being marked for garbage collection, though I shouldn't need to take steps like that.
# ObjectSpace.undefine_finalizer(zipfile)
zipfile.close
path = zipfile.path
end
#files = Zip::File.open(path)
#shared_strings = SharedStrings.new(self)
end
EDIT: Someone wanted to know exactly how I was running my code, so here it is.
I run the following rake task by executing bundle exec rails client:pricing_update[client_updated_prices.xlsx] in the command line.
namespace :client do
desc 'Imports the initial database structure & base data from uploaded .xlsx file'
task :pricing_update, [:filename] => :environment do |t, args|
PricingUpdateWorker.new.perform(args[:filename])
end
end
I should also mention that I'm running Rails, so the Gemfile.lock keeps the gem versions consistent between me and my coworker. My fog version is 2.0.0 and my rubyzip version is 1.2.2.
Finally it seems that the bug is not in Creek gem at all but rather in the rubyzip gem having trouble with xlsx files as noted in this issue It seems to depend on how the source of the file was generated. I created a simple 2 page spreadsheet in Google sheets and it works fine, but a random xlsx file may not.
require 'creek'
def test_creek(url)
Creek::Book.new(url, remote: true).sheets.each_with_index do |sheet, index|
p "----------Name: #{sheet.name} Index: #{index} ----------"
sheet.simple_rows.each_with_index do |row, i|
puts "#{row} index: #{i}"
end
end
end
test_creek 'https://tc-sandbox.s3.amazonaws.com/creek-test.xlsx'
# works fine should output
"----------Name: Sheet1 Index: 0 ----------"
{"A"=>"foo ", "B"=>"sheet", "C"=>"one"} index: 0
"----------Name: Sheet2 Index: 1 ----------"
{"A"=>"bar", "B"=>"sheet", "C"=>"2.0"} index: 0
test_creek 'http://dev-builds.libreoffice.org/tmp/test.xlsx'
# raises error
When there is a syntax error in an i18n locale YAML file, Psych::SyntaxError is raised. When this exception is encountered during Rails bootup (for example, when production is restarted), Rails crashes.
Is there any way to capture this exception and somehow recover from it without having Rails crash altogether?
Is there any way to check locale files for syntax errors before commit or deploy in an automated way?
I'm not sure if there's a way to recover from this error, but I've created a rake task that ensures that a given YAML file is syntactically valid (run via a pre-commit git hook for any changed YAML files):
namespace :yaml do
desc "Check YAML syntax for errors."
task :check_syntax do
require 'YAML'
require 'colorize'
puts "Checking YAML files..."
filenames = (ENV['FILENAMES'].split(',') || []).push(ENV['FILENAME']).uniq.compact
fails = 0
filenames.each do |file|
print "#{file}... "
begin
YAML.load_file(file)
rescue Psych::SyntaxError => e
fails += 1
print "Failed! ".red
print "[#{e.message.sub(/\A.*: /, '')}]"
puts
next
end
print "Success!".green
puts
end
puts
exit fails > 0 ? 1 : 0
end
end
I am working on an HTML5/JavaScript app with Ruby packaging to be used on multiple platforms including a dash display unit on a car. The code base is the same across platforms except for a single wrapper file that houses all of the API-specific calls for a single platform: i.e. "sdk.web.js" or "sdk.car.js".
I'm hoping to modify the current PackageTask to allow an input for a target platform. Depending on the target, I want the corresponding wrapper file to be renamed to "sdk.js" and included in the package, disregarding the rest to keep the zip archive as small as possible.
From terminal, I want to say something like:
rake target "web"
which would put together a package including "sdk.web.js" renamed "sdk.js". Anyone know if this is possible, and how I would modify my existing RakeFile (below) in order to accomplish this?
require 'fileutils'
require 'rake/packagetask'
VERSION = ""
task :default => [:package]
def parse_version(filename)
f = File.open(filename, "rb")
contents = f.read
m = contents.match('var version = "([0-9.]+)";')
if m
return m[1]
else
return nil
end
end
desc "Package up the app into a zip file"
Rake::PackageTask.new("myApp") do |p|
p.version = parse_version("js/application.js")
p.need_zip = true
p.package_files = FileList["*", "**/*"]
p.package_files.exclude(".git", "pkg/*", "Rakefile", ".rvmrc", "Gemfile", "Gemfile.lock", ".project", ".settings", ".gitignore", "data/store/*", "docs", "docs/*")
end
In general, you can pass arguments to rake tasks. See for instance this page.
However, you don't have access to the package task from PackageTask. A solution would be to define your own packaging task which would add the right js script and then invoke package manually. For instance (untested):
Rake::PackageTask.new("myApp") do |p|
[snip]
p.package_files << "sdk.js"
end
task 'custom_packaging', :sdk do |t, args|
# Copy the right file to sdk.js
FileUtils.cp_f "sdk.#{args[:sdk]}.js", "sdk.js"
Rake::Task["package"].invoke
end
I have been playing around with Rake and Albacore, to see if I can replace our existing MSBuild script that deploys software with something that isn't XML. I have a task that will change the debug value inside a web.config to false. The task takes the directory of the web.config as an argument, but I can't quite figure out the syntax needed to supply this argument in the default task.
require 'albacore'
require 'nokogiri'
deployment_path = 'c:/test-mars-deploy'
task :default => [ :build, :publish, :update_web_config['c:/test-mars-deploy'] ]
task :update_web_config, :deploy_path do |t, args|
deployment_path = #{args[:deploy_path]}
web_config_path = File.join deployment_path, 'Web.config'
File.open(web_config_path, 'r+') do |f|
doc = Nokogiri::XML(f)
puts 'finding attribute'
attribute = doc.xpath('/configuration/system.web/compilation')
attribute.attr('debug', 'false')
puts attribute.to_xml
end
File.delete(web_config_path)
File.new(web_config_path, 'w') do |f|
f.write(doc.to_s)
end
end
The task dependency notation doesn't support passing arguments. It only takes names or symbols referring to task names.
task :default => [ :build, :publish, :update_web_config['c:/test-mars-deploy'] ]
You'd need to do something like this.
task :default => [ :build, :publish ] do
Rake::Task[:update_web_config].invoke 'c:/test-mars-deploy'
end
Remember, though, invoke will only work once per task, even with different arguments. It's the real dependency chain invoke. But, it will call all dependent tasks. You can use execute if you need multiple executions, but that won't call dependent tasks.
Rake::Task[:update_web_config].invoke 'c:/test-mars-deploy'
Rake::Task[:update_web_config].execute 'c:/test-mars-deploy2'
Rake::Task[:update_web_config].execute 'c:/test-mars-deploy3'
In general, I don't recommend either of these approaches. Calling invoke or execute seems to me to indicate a poorly structured task. You simply don't have this problem if you don't prematurely parameterize.
web_config = 'c:/test-mars-deploy/Web.config'
task :update_web_config do
File.open(web_config, 'r+') do |file|
# ...
end
end
If you must parameterize, provide an array or FileList and generate the tasks per item.
web_configs = FileList['c:/test-*/Web.config']
web_configs.each do |config|
task config do
File.open(config, 'r+') do |file|
# ...
end
end
end
task :update_all_web_configs => web_configs
Better yet, I published a config update task that does all of this mess for you! Provide a FileList to update and a hash of xpath queries => replacements.
appconfig :update_web_configs do |x|
x.files = FileList['c:/test-*/Web.config']
x.replacements = {
"/configuration/system.web/compilation/#debug" => 'False' }
end
I think you might have to use the old style parameter passing, eg:
nicholas#hal:/tmp$ cat Rakefile
task :default => :all
deploy_path = ENV['deploy_path'] || "c:/some_path"
task :all do |t, args|
puts deploy_path.inspect
end
And invoke with:
nicholas#hal:/tmp$ rake
(in /tmp)
"c:/some_path"
Or, to override the path:
nicholas#hal:/tmp$ rake deploy_path=c:/other_path
(in /tmp)
"c:/other_path"
basically, you name your args as extra symbols after the name of the task. an args param will get passed into the block that responds to the name of your args, and you can invoke the task passing the args in square brackets ([])
ree-1.8.7-2010.02#rails3 matt#Zion:~/setup$ cat lib/tasks/blah.rake
task :blah, :n do |t, args|
puts args.n
end
ree-1.8.7-2010.02#rails3 matt#Zion:~/setup$ rake blah[20]
(in /home/matt/setup)
20
The task dependency notation does in fact support passing arguments. For example, say "version" is your argument:
task :default, [:version] => [:build]
task :build, :version do |t,args|
version = args[:version]
puts version ? "version is #{version}" : "no version passed"
end
Then you can call it like so:
$ rake
no version passed
or
$ rake default[3.2.1]
version is 3.2.1
or
$ rake build[3.2.1]
version is 3.2.1
However, I have not found a way to avoid specifying the task name (default or build) while passing in arguments. Would love to hear if anyone knows of a way.
I'm struggling with getting rubyzip to append directories to a zipoutputstream. (I want the output stream so I can send it from a rails controller). My code follows this example:
http://info.michael-simons.eu/2008/01/21/using-rubyzip-to-create-zip-files-on-the-fly/
When modified to include directories in the list of files to add I get the following error:
Any help would be greatly appreciated.
UPDATE
After trying a number of solutions I had best success with zipruby which has a clean api and good examples: http://zipruby.rubyforge.org/.
Zip::ZipFile.open(path, Zip::ZipFile::CREATE) do |zip|
songs.each do |song|
zip.add "record/#{song.title.parameterize}.mp3", song.file.to_file.path
end
end
OOOOOuuuhh...you DEFINITELY want ZIPPY. It's a Rails plugin that abstracts a lot of the complexity in rubyzip, and lets you create what you're talking about, including directories (from what I recall).
Here you go:
http://github.com/toretore/zippy
And direct from the zippy site:
Example controller:
def show
#gallery = Gallery.find(params[:id])
respond_to do |format|
format.html
format.zip
end
end
Example view:
zip['description.txt'] = #gallery.description
#gallery.photos.each do |photo|
zip["photo_#{photo.id}.png"] = File.open(photo.url)
end
edit: Amending per user comment:
Hmm...the whole objective of using Zippy is to make it a whole lot easier to use ruby zip.
Ya might want to take a second (or first) look...
Here's how to make a directory with directories:
some_var = Zippy.open('awsum.zip') do |zip|
%w{dir_a dir_b dir_c diri}.each do |dir|
zip["bin/#{dir}/"]
end
end
...
send_file some_var, :file_name => ...
Zippy will work for this. There may be a more cool way to do this but since there are essentially no docs, here's what I came up with for recursively copying directories with Zippy in a Rakefile. This Rakefile is used in a Rails environment so I put gem requirements in my Gemfile:
#Gemfile
source 'http://rubygems.org'
gem 'rails'
gem 'zippy'
And this is the Rakefile
#Rakefile
def add_file( zippyfile, dst_dir, f )
zippyfile["#{dst_dir}/#{f}"] = File.open(f)
end
def add_dir( zippyfile, dst_dir, d )
glob = "#{d}/**/*"
FileList.new( glob ).each { |f|
if (File.file?(f))
add_file zippyfile, dst_dir, f
end
}
end
task :myzip do
Zippy.create 'my.zip' do |z|
add_dir z, 'my', 'app'
add_dir z, 'my', 'config'
#...
add_file z, 'my', 'config.ru'
add_file z, 'my', 'Gemfile'
#...
end
end
Now I can use it like this:
C:\> cd my
C:\my> rake myzip
and it will produce my.zip which contains an inner directory called 'my' with copies of selected files and directories.
I was able to get directories working with the same ZipOutputStream used in the original article.
All I had to do was add the directory when calling zos.put_next_entry.
For example:
require 'zip/zip'
require 'zip/zipfilesystem'
t = Tempfile.new("some-weird-temp-file-basename-#{request.remote_ip}")
# Give the path of the temp file to the zip outputstream, it won't try to open it as an archive.
Zip::ZipOutputStream.open(t.path) do |zos|
some_file_list.each do |file|
# Create a new entry with some arbitrary name
zos.put_next_entry("myfolder/some-funny-name.jpg") # Added myfolder/
# Add the contents of the file, don't read the stuff linewise if its binary, instead use direct IO
zos.print IO.read(file.path)
end
end
# End of the block automatically closes the file.
# Send it using the right mime type, with a download window and some nice file name.
send_file t.path, :type => 'application/zip', :disposition => 'attachment', :filename => "some-brilliant-file-name.zip"
# The temp file will be deleted some time...
t.close
I just changed zos.put_next_entry('some-funny-name.jpg') to zos.put_next_entry('myfolder/some-funny-name.jpg'), and the resulting zipfile had a nested folder called myfolder that contained the files.