I'm writing a script that Creek and an .xlsx file and uses it to update the prices and weights of products in a database. The .xlsx file is located on an AWS server, so Creek copies the file down and stores it in a Tempfile while it is in use.
The issue is, at some point the Tempfile seems to be prematurely deleted, and since Creek continues to call on it whenever it iterates through a sheet, the script fails. Interestingly, my coworker's environment runs the script fine, though I haven't found a difference between what we're running.
Here is the script I've written:
require 'creek'
class PricingUpdateWorker
include Sidekiq::Worker
def perform(filename)
# This points to the file in the root bucket
file = bucket.files.get(filename)
# Make public temporarily to open in Creek
file.public = true
file.save
creek_sheets = Creek::Book.new(file.public_url, remote: true).sheets
# Close file to public
file.public = false
file.save
creek_sheets.each_with_index do |sheet, sheet_index|
p "---------- #{sheet.name} ----------"
sheet.simple_rows.each_with_index do |row, index|
next if index == 0
product = Product.find_by_id(row['A'].to_i)
if product
if row['D']&.match(/N\/A/) || row['E']&.match(/N\/A/)
product.delete
p '*** deleted ***'
else
product.price = row['D']&.to_f&.round(2)
product.weight = row['E']&.to_f
product.request_for_quote = false
product.save
p 'product updated'
end
else
p "#{row['A']} | product not found ***"
end
end
end
end
private
def connection
#connection ||= Fog::Storage.new(
provider: 'AWS',
aws_access_key_id: ENV['AWS_ACCESS_KEY_ID'],
aws_secret_access_key: ENV['AWS_SECRET_ACCESS_KEY']
)
end
def bucket
# Grab the file from the bucket
#bucket ||= connection.directories.get 'my-aws-bucket'
end
end
And the logs:
"---------- Sheet 1 ----------"
"product updated"
"product updated"
... I've cut out a bunch more of these...
"product updated"
"product updated"
"---------- Sheet 2 ----------"
rails aborted!
Errno::ENOENT: No such file or directory # rb_sysopen - /var/folders/9m/mfcnhxmn1bqbm6h91rx_rd8m0000gn/T/file20190920-19247-c6x4zw
"/var/folders/9m/mfcnhxmn1bqbm6h91rx_rd8m0000gn/T/file20190920-19247-c6x4zw" is the temporary file, and as you can see, it's been collected already, even though I'm still using it, and I believe it is still in scope. Any ideas what could be causing this? It's especially odd that my coworker can run this just fine.
In case it's helpful, here is a little code from Creek:
def initialize path, options = {}
check_file_extension = options.fetch(:check_file_extension, true)
if check_file_extension
extension = File.extname(options[:original_filename] || path).downcase
raise 'Not a valid file format.' unless (['.xlsx', '.xlsm'].include? extension)
end
if options[:remote]
zipfile = Tempfile.new("file")
zipfile.binmode
zipfile.write(HTTP.get(path).to_s)
# I added the line below this one, and it fixes the problem by preventing the file from being marked for garbage collection, though I shouldn't need to take steps like that.
# ObjectSpace.undefine_finalizer(zipfile)
zipfile.close
path = zipfile.path
end
#files = Zip::File.open(path)
#shared_strings = SharedStrings.new(self)
end
EDIT: Someone wanted to know exactly how I was running my code, so here it is.
I run the following rake task by executing bundle exec rails client:pricing_update[client_updated_prices.xlsx] in the command line.
namespace :client do
desc 'Imports the initial database structure & base data from uploaded .xlsx file'
task :pricing_update, [:filename] => :environment do |t, args|
PricingUpdateWorker.new.perform(args[:filename])
end
end
I should also mention that I'm running Rails, so the Gemfile.lock keeps the gem versions consistent between me and my coworker. My fog version is 2.0.0 and my rubyzip version is 1.2.2.
Finally it seems that the bug is not in Creek gem at all but rather in the rubyzip gem having trouble with xlsx files as noted in this issue It seems to depend on how the source of the file was generated. I created a simple 2 page spreadsheet in Google sheets and it works fine, but a random xlsx file may not.
require 'creek'
def test_creek(url)
Creek::Book.new(url, remote: true).sheets.each_with_index do |sheet, index|
p "----------Name: #{sheet.name} Index: #{index} ----------"
sheet.simple_rows.each_with_index do |row, i|
puts "#{row} index: #{i}"
end
end
end
test_creek 'https://tc-sandbox.s3.amazonaws.com/creek-test.xlsx'
# works fine should output
"----------Name: Sheet1 Index: 0 ----------"
{"A"=>"foo ", "B"=>"sheet", "C"=>"one"} index: 0
"----------Name: Sheet2 Index: 1 ----------"
{"A"=>"bar", "B"=>"sheet", "C"=>"2.0"} index: 0
test_creek 'http://dev-builds.libreoffice.org/tmp/test.xlsx'
# raises error
When calling the vsim command, how can I make ModelSim automatically use the top-level VHDL entity (or entities)? I'm writing a generic script for running VHDL simulations.
Currently I'm doing the following to compile and simulate:
vcom design.vhd testbench.vhd
vsim -c -do "onElabError resume; run -all; exit" MY_TB
How can I make it so that ModelSim automatically simulates MY_TB without explicitly specifying it.
Automatically using the top-level module(s) is possible with Verilog:
vlog -writetoplevels my.tops design.v testbench.v
vsim -c -do "onElabError resume; run -all; exit" -f my.tops
My way was to create a script in Ruby. It's very easy to find the entity/architecture names and then invoke Modelsim. The outline of the solution is:
Open the VHDL file:
file_contents = File.read(file_name)
Use a regular expression to find the entity and architecture names:
match = file_contents.match(/entity (.*) is/)
entity = match && match[1]
match = file_contents.match(/architecture (.*) of/)
architecture = match && match[1]
Invoke Modelsim in command-line mode:
vsim -do {run -all; quit} -lib /temp/work -c entity_name(architecture_name)
I've included my entire script below for reference. I think its highlights are:
vhdl -m → "make" project by compiling all VHDL files in the directory
vhdl -f text → find text in VHDL files
vhdl -s filename → simulate filename
# vhdl.rb: Swiss army knife for VHDL-related tasks.
# vhdl --compile file_name # Compile file with ModelSim
# vhdl --simulate file_name # Simulate file with ModelSim
# vhdl --find text # Find text in VHDL source files
# vhdl --make dir # Compile all files in dir
require 'optparse'
require 'pathname'
begin
gem 'colorize' # gem install colorize
require 'colorize'
rescue Gem::LoadError
puts "Gem 'colorize' is not installed. You can install it by typing:\ngem install colorize"
end
WORK = Pathname.new("/temp/work")
def ensure_library_work_exists
if WORK.exist?
puts "Found library work at #{WORK.expand_path}"
else
puts "Creating library work at #{WORK.expand_path}..."
system('mkdir \temp\work')
system('vlib /temp/work')
end
end
def compile(file_name)
ensure_library_work_exists
puts "Compiling #{file_name}..."
puts cmd_line = "vcom -2008 -work #{WORK} #{file_name}"
system(cmd_line)
end
def simulate(file_name)
ensure_library_work_exists
puts "Simulating #{file_name}..."
compile(file_name)
entity, architecture = find_entity_architecture(file_name)
if entity && architecture
cmd_line = "vsim -lib #{WORK} -c #{$entity || entity}(#{$architecture || architecture}) -do \"run -all; quit\""
system(cmd_line)
end
end
def find_entity_architecture(file_name)
file_contents = File.read(file_name)
match = file_contents.match(/entity (.*) is/)
entity = match && match[1]
match = file_contents.match(/architecture (.*) of/)
architecture = match && match[1]
[ entity, architecture ]
end
def find_text(text)
puts cmd_line = "findstr /sni /C:\"#{text}\" *.vhd *.vhdl"
system(cmd_line)
end
$before_context = 1
$after_context = 3
def find_filenames(path)
filenames = Dir["#{path}/**/*.vhd"]
puts "Found #{filenames.count} VHDL files"
filenames
end
def grep_text(text)
filenames = find_filenames(".")
filenames.each do |filename|
output = `grep #{text} #{filename} --recursive --line-number --before-context=#{$before_context} --after-context=#{$after_context}`
last_output_was_blank = true
if output.empty?
print ".".green
last_output_was_blank = true
else
puts() if last_output_was_blank
puts filename.green, output
last_output_was_blank = false
end
end
end
def compile_vhdl_file(file_name)
vcom_output = `vcom -2008 -work #{WORK} #{file_name}`
if vcom_output.match(/\*\* Error/)
error_message = vcom_output[/(\*\* Error.*$)/m]
raise error_message
end
puts vcom_output
end
def make(path)
ensure_library_work_exists
filenames = Dir["#{path}/**/*.vhd"]
puts "Found #{filenames.count} VHDL files:"
puts filenames.join(" ").green
pass_number = 0
failed_filenames = []
begin
pending_files_count = filenames.count - failed_filenames.count
puts ">>> Starting pass #{pass_number} -- pending #{pending_files_count}, failed #{failed_filenames.count}".yellow
previously_failed_filenames = failed_filenames.dup
failed_filenames.clear
filenames.each do |filename|
begin
compile_vhdl_file(filename)
rescue
puts "#{$!}".red
failed_filenames << filename
end
end
pass_number += 1
end while failed_filenames.any? and (failed_filenames != previously_failed_filenames)
if failed_filenames.any?
puts "*** Error: Failed to compile:".red
puts failed_filenames.join("\n").red
exit
end
end
OptionParser.new do |o|
o.on('-e', '--ent entity_name') { |entity| $entity = entity }
o.on('-a', '--arch architecture_name') { |architecture| $architecture = architecture }
o.on('-c', '--compile file') { |file| compile(file) }
o.on('-s', '--simulate file') { |file| simulate(file) }
o.on('-f', '--find text') { |text| find_text(text) }
o.on('-b', '--before-context before_context') { |lines_count| $before_context = lines_count }
o.on('-t', '--after-context after_context') { |lines_count| $after_context = lines_count }
o.on('-g', '--grep text') { |text| grep_text(text) }
o.on('-m', '--make [dir]') { |dir| make(dir || '.') }
o.parse!
end
Trying to write a script that will search through a directory and sub-directories for specific files. I would like to do know how a certain directory or directories come up to call a method.
this is what I have tried and failed:
def display_directory(path)
list = Dir[path+'/*']
return if list.length == 0
list.each do |f|
if File.directory? f #is it a directory?
if File.directory?('config')
puts "this is the config folder"
end
printf "%-50s %s\n", f, "is a directory:".upcase.rjust(25)
else
printf "%-50s %s\n", f, "is not a directory:".upcase.rjust(25)
end
end
end
start = File.join("**")
puts "Processing directory\n\n".upcase.center(30)
display_directory start
this is what I want to happen.
app
app/controllers
app/helpers
app/mailers
app/models
app/models/bugzilla
app/models/security
app/views
app/views/auth
app/views/calendar
app/views/layouts
app/views/step
app/views/step_mailer
app/views/suggestion
app/views/suggestion_mailer
app/views/task
app/views/user
bin
--------------------------------------
config <----------(call method foo)
config/environments
config/initializers
config/locales
--------------------------------------
db
db/bugzilla
db/migrate
db/security
lib
lib/tasks
log
public
public/images
public/javascripts
public/stylesheets
script
script/performance
script/process
--------------------------
test <---------(call method foobar)
test/fixtures
test/fixtures/mailer
test/functional
test/integration
test/performance
test/unit
--------------------------
vendor
vendor/plugins
Instead
if File.directory?('config')
Try
if f.path.include?('config')
but this will work for every directory that have config on the name. You can put a larger substring to make a better match.
Also, it is very idiomatic in ruby use do..end for multiline blocks and {..} for single line.
I figured out a way. this works pretty well. I've added a method to show all the files in mentioned directory when reached.
def special_dir(path)
puts "------------------------------------"
sp_path = Dir.glob(File.join(path,"*","**"))
sp_path.each do |cf|
puts "\t" + cf
end
end
def walk(path)
list = Dir[path+'/*'].reject{ |r| r['doc'] || r['tmp']}
list.each do |x|
path = File.join(path, x)
if File.directory?(x)
if x =~ /config/ or x =~ /test/
special_dir(x)
else
puts "#{x}"
walk(path)
end
else
#puts x
end
end
end
start = File.join("**")
walk start
I have the following Rakefile:
task :test_commas, :arg1 do |t, args|
puts args[:arg1]
end
And want to call it with a single string argument containing commas. Here's what I get:
%rake 'test_commas[foo, bar]'
foo
%rake 'test_commas["foo, bar"]'
"foo
%rake "test_commas['foo, bar']"
'foo
%rake "test_commas['foo,bar']"
'foo
%rake "test_commas[foo\,bar]"
foo\
I'm currently using the workaround proposed in this pull request to rake, but is there a way to accomplish this without patching rake?
Changing rake is quite a dirty fix. Try using OptionParser instead. The syntax is following
$ rake mytask -- -s 'some,comma,sepparated,string'
the -- is necessary to skip the rake way of parsing the arguments
and here's the ruby code:
task :mytask do
options = {}
optparse = OptionParser.new do |opts|
opts.on('-s', '--string ARG', 'desc of my argument') do |str|
options[:str] = str
end
opts.on('-h', '--help', 'Display this screen') do
puts opts
exit
end
end
begin
optparse.parse!
mandatory = [:str]
missing = mandatory.select{ |param| options[param].nil? }
if not missing.empty?
puts "Missing options: #{missing.join(', ')}"
puts optparse
exit
end
rescue OptionParser::InvalidOption, OptionParser::MissingArgument
puts $!.to_s
puts optparse
exit
end
puts "Performing task with options: #{options.inspect}"
# #TODO add task's code
end
I'm not sure it's possible. Looking at lib/rake/application.rb, the method for parsing the task string is:
def parse_task_string(string)
if string =~ /^([^\[]+)(\[(.*)\])$/
name = $1
args = $3.split(/\s*,\s*/)
else
name = string
args = []
end
[name, args]
end
It appears that the arguments string is split by commas, so you cannot have an argument that contains a comma, at least not in the current rake-0.9.2.
Eugen already answered, why it doesn't work.
But perhaps the following workaround may help you:
task :test_commas, :arg1, :arg2 do |t, args|
arg = args.to_hash.values.join(',')
puts "Argument is #{arg.inspect}"
end
It takes two arguments, but joins them to get the 'real' one.
If you have more then one comma, you need more arguments.
I did some deeper research and found one (or two) solution.
I don't think it's a perfect solution, but it seems it works.
require 'rake'
module Rake
class Application
#usage:
# rake test_commas[1\,2\,3]
def parse_task_string_masked_commas(string)
if string =~ /^([^\[]+)(\[(.*)\])$/
name = $1
args = $3.split(/\s*(?<!\\),\s*/).map{|x|x.gsub(/\\,/,',')}
else
name = string
args = []
end
[name, args]
end
#Usage:
# rake test_commas[\"1,2\",3]
#" and ' must be masked
def parse_task_string_combined(string)
if string =~ /^([^\[]+)(\[(.*)\])$/
name = $1
args = $3.split(/(['"].+?["'])?,(["'].+?["'])?/).map{|ts| ts.gsub(/\A["']|["']\Z/,'') }
args.shift if args.first.empty?
else
name = string
args = []
end
[name, args]
end
#~ alias :parse_task_string :parse_task_string_masked_commas
alias :parse_task_string :parse_task_string_combined
end
end
desc 'Test comma separated arguments'
task :test_commas, :arg1 do |t, args|
puts '---'
puts "Argument is #{args.inspect}"
end
The version parse_task_string_masked_commasallows calls with masked commas:
rake test_commas[1\,2\,3]
The version parse_task_string_combined allows:
rake test_commas[\"1,2,3\"]
At least under windows, the " (or ') must be masked. If not, they are already deleted until the string reached Rake::Aplication (probably shell substitution)
Have you tried escaping the , with a \?
Rake task variables are not very convenient generally, instead use environment variables:
task :my_task do
ENV["MY_ARGUMENT"].split(",").each_with_index do |arg, i|
puts "Argument #{i}: #{arg}"
end
end
Then you can invoke with
MY_ARGUMENT=foo,bar rake my_task
Or if you prefer
rake my_task MY_ARGUMENT=foo,bar
# rake -v 10.5
rake test_commas\['foo\,bar'\]
works like a charm.
For rake version 12.3.2,
rake test_commas["foo\,bar"]
works well
Another simple workaround is to use a different delimiter in your arguments.
It's pretty simple to swap to a pipe | character (or another) instead of your comma separated args. Rake parses your arguments correctly in this case and allows you to split the first for an array.
desc 'Test pipe separated arguments'
task :test_pipes, :arg1, :arg2 do |t, args|
puts ":arg1 is: #{ args[:arg1] }"
puts ":arg2 still works: #{ args[:arg2] }"
puts "Split :arg1 is: #{ args[:arg1].split('|') }"
end
Call it with:
rake test_pipes["foo|bar",baz]
We can convert each string to symbol,
As,
task :create do
ARGV.each { |a| task a.to_sym do ; end }
puts ARGV[1]
puts ARGV[2]
puts ARGV[3]
puts ARGV[4]
end
end
run as,
rake create '1,s' '2' 'ww ww' 'w,w'
I am adding tons of file to my archive it looks like this:
print "Starting ..."
Zip::ZipFile.open(myarchive, 'w') do |zipfile|
my_tons_of_files.each do |file|
print "Adding #{file.filename} to archive ... \r"
# ...
end
print "\n Saving archive"
# !!! -> waiting about 10-15 minutes
# but I want to show the percentage of completed job
end
After all files are added to my archive it starts to compress them all (about 10-15 minutes).
How can I indicate what is actually going on with rubyzip gem (actually I want to show percentage like current_file_number/total_files_count).
You can override Zip::ZipFile.commit:
require 'zip'
require 'zip/zipfilesystem'
module Zip
class ZipFile
def commit
return if ! commit_required?
on_success_replace(name) {
|tmpFile|
ZipOutputStream.open(tmpFile) {
|zos|
total_files = #entrySet.length
current_files = 1
#entrySet.each do |e|
puts "Current file: #{current_files}, Total files: #{total_files}"
current_files += 1
e.write_to_zip_output_stream(zos)
end
zos.comment = comment
}
true
}
initialize(name)
end
end
end
print "Starting ..."
Zip::ZipFile.open(myarchive, 'w') do |zipfile|