Context
I am auto-requiring all files in a directory structure via
# base.rb
dir = File.dirname(__FILE__)
path = File.join(dir, '**', '*.rb')
Dir[path].each { |file| require File.expand_path(file, __FILE__) }
and am calling this snippet through a require statement in a separate file, api.rb.
Problem
This code snippet includes itself (base.rb) as well as api.rb.
Question
Is there a 'clean' way to do this type of auto-requiring while dynamically avoiding including the file that has called the auto-require'er (i.e. api.rb)?
Remember that when you require a file identified by a certain path more than once each subsequent call to require will return false and the file won't be reevaluated. As a result if your base.rb, which requires everything else, is itself required, further attepts to require it should not lead to a reevaluation.
Let's demonstrate it using an example. Create a lib directory with 3 files inside.
# lib/a.rb
require 'base'
puts :a
# lib/b.rb
require 'base'
puts :b
# lib/base.rb
$counter ||= 0
puts "Evaluated base.rb #{$counter += 1} times"
dir = File.dirname(__FILE__)
path = File.join(dir, '**', '*.rb')
Dir[path].each { |file| require File.expand_path file }
Execute lib/base.rb directly. base.rb will be evaluated twice: firstly, when it's executed directly; secondly, when it's required by a.rb. Notice, that it is not evaluated when it's required from b.rb.
$ ruby -I lib lib/base.rb
Evaluated base.rb 1 times
Evaluated base.rb 2 times
a
b
Compare with requireing it. Now base.rb is evaluated once only, because attempts to require it in a.rb and b.rb were preceded by having the file required using the command line -r switch.
$ ruby -I lib -r base -e 'puts :ok'
Evaluated base.rb 1 times
a
b
ok
Kernel.caller
returns the current execution stack as an array of strings. You can avoid
calling require on filenames which you find in that array. Multiple files with
the same basename would trip this up. I don't see a way to get a more precise
list of ancestor files.
$ head *.rb
==> A.rb <==
require 'base'
puts :A
==> B.rb <==
require 'base'
puts :B
==> CA.rb <==
require 'base'
puts :CA
==> base.rb <==
dir = File.dirname(__FILE__)
path = File.join(dir, '**', '*.rb')
required = caller.map { |frame| /^(.+):\d+:in `require'$/.match(frame) and File.basename $1 }.compact
Dir[path].each { |file| required.include?(File.basename file) or require File.expand_path file }
$ ruby A.rb
B
CA
A
$ ruby B.rb
A
CA
B
$ ruby CA.rb
A
B
CA
$ ruby base.rb
B
CA
A
$
Related
I've just started using OptionParser for Ruby and I wanted to use flags that would use more than just one argument.
For instance, I would like to be able to run:
script --move src dst
Note how src and dst are not separated using a coma.
My initial idea was:
opts.on("-m src dst", "--move src dst ", "move file from SRCto DST") do |src|
# do something
end
But this is not working. I assume that this is not the right approach. But how could this be done?
The example under the "Complete Example" section of the OptionParser details how a list of items can be accepted.
Here is a sample program based on that example. The third parameter Array in opts.on indicates that input src, dst should be used to create an array. To run this sample, you need to do gem install trollop.
# test.rb
require 'optparse'
options = {}
OptionParser.new do |opt|
opt.on("-m src, dst", "--move src, dst", Array, "Move from src to dst") do |list|
options[:src] = list[0]
options[:dst] = list[1]
end
end.parse!
puts options # It's a hash of parsed options
Sample run:
> ruby test.rb -m from,to
{:src=>"src", :dst=>"dst"}
>ruby test.rb -h
Usage: test [options]
-m, --move src, dst Move from src to dst
The above script forces one to separate the options using comma.
As indicated by "Really Cheap Command-Line Option Parsing in Ruby", there seems to be a gem, trollop, that can be quite easy to use for command-line parsing.
A sample program based on Trollop is given below, which allows usage of spaces for specifying options with multiple values
# test.rb
require "trollop"
opts = Trollop::options do
banner "Command line parsing using Trollop"
opt :move, "--move src dst', Move from src to dst", :short => "-m", :long => "--move", :type => :strings
end
# An array of option values
p opts.move
Sample run:
>ruby test.rb -m hello world
["hello", "world"]
>ruby test.rb -h
Command line parsing using Trollop
-m, --move=<s+> '--move src dst', Move from src to dst
-h, --help Show this message
There is a subtle difference between the help output between the two approaches. Trollop produces help text where --move=<s+> does not indicate clearly that it needs accepts two values, so I had to repeat the command syntax description.
OptionParser doesn't support that; It could be patched to do so, but I'm not sure it's worth the trouble.
Consider this code:
require 'optparse'
options = {}
OptionParser.new do |opt|
opt.on('-m', '--move') { |o| options[:move] = o }
end.parse!
from_name, to_name = ARGV
puts "Should move: #{ options.key?(:move) }"
puts "From: #{ from_name }"
puts "To: #{ to_name }"
Saving it and running it with various combinations of the parameters returns:
> ruby test.rb --move from to
Should move: true
From: from
To: to
> ruby test.rb from to
Should move: false
From:
To:
If the code is supposed to move files by default then don't bother with the --move flag, simply use:
test.rb from to
and consider removing the OptionParser block entirely.
If the code is supposed to normally copy with the option to move, then --move becomes more sensible to act as a flag that moving is desired.
ruby test.rb --move from to
I'd have code that tests for options[:move] and run the code to move instead of copy at that point.
In either case, the filenames shouldn't be tied to the flag, they should be supplied separately and retrieved from ARGV after OptionParser has finished parsing the command-line and removing entries it's handled.
I'm following "The Bastards Book of Ruby" and I am trying to build a webscraper using nokogiri but about a quarter of the way into it when I attempt to run the code it throws the error:
Crawler.rb:6:in `mkdir': No such file or directory # dir_s_mkdir - data-hold/nobel (Errno::ENOENT)
from Crawler.rb:6:in `<main>'
My code is as follows:
require 'rubygems'
require 'nokogiri'
require 'open-uri'
DATA_DIR = "data-hold/nobel"
Dir.mkdir(DATA_DIR) unless File.exists?(DATA_DIR)
BASE_WIKIPEDIA_URL = "http://en.wikipedia.org"
LIST_URL = "#{BASE_WIKIPEDIA_URL}/wiki/List_of_Nobel_laureates"
HEADERS_HASH = {"User-Agent" => "Ruby/#{RUBY_VERSION}"}
page = Nokogiri::HTML(open(LIST_URL))
rows = page.css('div.mw-content-ltr table.wikitable tr')
rows[1..-2].each do |row|
hrefs = row.css("td a").map{ |a|
a['href'] if a['href'] =~ /^\/wiki\//
}.compact.uniq
hrefs.each do |href|
remote_url = BASE_WIKIPEDIA_URL + href
local_fname = "#{DATA_DIR}/#{File.basename(href)}.html"
unless File.exists?(local_fname)
puts "Fetching #{remote_url}..."
begin
wiki_content = open(remote_url, HEADERS_HASH).read
rescue Exception=>e
puts "Error: #{e}"
sleep 5
else
File.open(local_fname, 'w'){|file| file.write(wiki_content)}
puts "\t...Success, saved to #{local_fname}"
ensure
sleep 1.0 + rand
end # done: begin/rescue
end # done: unless File.exists?
end # done: hrefs.each
end # done: rows.each
I have literally no idea why it is not creating a new directory to store the data in. I know I must be missing something extremely simple...
My best guess is that not only does "data-hold/nobel" not exist, "data-hold/" does not exist either. Since mkdir does not recursively create all parent directories of the directory you want to create, an error is thrown.
To fix this, you could use FileUtils.mkdir_p, which does create all parent directories.
Be sure to include fileutils before using mkdir_p.
Bundled the requirements into a method with the proper debug message. Works as expected.
$:~/rubyterminals/file_tansfer$ cat mkdir_mthod.rb
#!/usr/bin/env ruby
require 'fileutils'
def run
my_dir="/home/rubyterminals/file_tansfer/new_dir"
create_a_directory(my_dir)
end
def create_a_directory(dir_name)
if dir_name
# dir_name was specified, ensure it is created and writable.
unless File.exist?(dir_name)
begin
FileUtils.mkdir_p(dir_name)
puts "just made the following dir #{dir_name}"
rescue Errno::EACCES => e
abort "Failed to create #{dir_name}: #{e.message}"
end
end
end
end
run
tested it :
-SVE1411EGXB:~/rubyterminals/file_tansfer$ ./mkdir_mthod.rb
just made the following dir /home/rubyterminals/file_tansfer/new_dir
Hope this help.
2 small questions to create the effect I'm looking for.
How do I check if a file exists within a directory with the extension of .zip?
If it does exist I need to make a folder with the same name as the .zip without the .zip extension for the folder.
Then I need to extract the files into the folder.
Secondly, what do I do if there are more than one .zip files in the folder?
I'm doing something like this and trying to put it into ruby
`mkdir fileNameisRandom`
`unzip fileNameisRandom.zip -d fileNameisRandom`
On a similar post I found something like
Dir.entries("#{Dir.pwd}").select {|f| File.file? f}
which I know checks all files within a directory and makes sure they are a file.
The problem is I don't know how to make sure that it is only an extension of .zip
Also, I found the Glob function which checks the extension of a filename from: http://ruby-doc.org/core-1.9.3/Dir.html
How do I ensure the file exists in that case, and if it doesn't I can print out an error then.
From the comment I now have
if Dir['*.zip'].first == nil #check to see if any exist
puts "A .zip file was not found"
elsif Dir['*.zip'].select {|f| File.file? f} then #ensure each of them are a file
#use a foreach loop to go through each one
Dir['*.zip'].select.each do |file|
puts "#{file}"
end ## end for each loop
end
Here's a way of doing this with less branching:
# prepare the data
zips= Dir['*.zip'].select{ |f| File.file? }
# check if data is sane
if zips.empty?
puts "No zips"
exit 0 # or return
end
# process data
zips.each do |z|
end
This pattern is easier to follow for fellow programmers.
You can also do it using a ruby gem called rubyzip
Gemfile:
source 'https://rubygems.org'
gem 'rubyzip'
run bundle
unzip.rb:
require 'zip'
zips= Dir['*.zip'].select{ |f| File.file? }
if zips.empty?
puts "No zips"
exit 0 # or return
end
zips.each do |zip|
Zip::File.open(zip) do |files|
files.each do |file|
# write file somewhere
# see here https://github.com/rubyzip/rubyzip
end
end
end
I finally pieced together different information from tutorials and used #rogerdpack and his comment for help.
require 'rubygems/package'
#require 'zlib'
require 'fileutils'
#move to the unprocessed directory to unpack the files
#if a .tgz file exists
#take all .tgz files
#make a folder with the same name
#put all contained folders from .tgz file inside of similarly named folder
#Dir.chdir("awaitingApproval/")
if Dir['*.zip'].first == nil #check to see if any exist, I use .first because Dir[] returns an array
puts "A .zip file was not found"
elsif Dir['*.zip'].select {|f| File.file? f} then #ensure each of them are a file
#use a foreach loop to go through each one
Dir['*.zip'].select.each do |file|
puts "" #newlie for each file
puts "#{file}" #print out file name
#next line based on `mkdir fileNameisRandom`
`mkdir #{Dir.pwd}/awaitingValidation/#{ File.basename(file, File.extname(file)) }`
#next line based on `unzip fileNameisRandom.zip -d fileNameisRandom`
placement = "awaitingValidation/" + File.basename(file, File.extname(file))
puts "#{placement}"
`sudo unzip #{file} -d #{placement}`
puts "Unzip complete"
end ## end for each loop
end
When using Tempfile Ruby is creating a file with a thread-safe and inter-process-safe name. I only need a file name in that way.
I was wondering if there is a more straight forward approach way than:
t = Tempfile.new(['fleischwurst', '.png'])
temp_path = t.path
t.close
t.unlink
Dir::Tmpname.create
You could use Dir::Tmpname.create. It figures out what temporary directory to use (unless you pass it a directory). It's a little ugly to use given that it expects a block:
require 'tmpdir'
# => true
Dir::Tmpname.create(['prefix-', '.ext']) {}
# => "/tmp/prefix-20190827-1-87n9iu.ext"
Dir::Tmpname.create(['prefix-', '.ext'], '/my/custom/directory') {}
# => "/my/custom/directory/prefix-20190827-1-11x2u0h.ext"
The block is there for code to test if the file exists and raise an Errno::EEXIST so that a new name can be generated with incrementing value appended on the end.
The Rails Solution
The solution implemented by Ruby on Rails is short and similar to the solution originally implemented in Ruby:
require 'tmpdir'
# => true
File.join(Dir.tmpdir, "YOUR_PREFIX-#{Time.now.strftime("%Y%m%d")}-#{$$}-#{rand(0x100000000).to_s(36)}-YOUR_SUFFIX")
=> "/tmp/YOUR_PREFIX-20190827-1-wyouwg-YOUR_SUFFIX"
File.join(Dir.tmpdir, "YOUR_PREFIX-#{Time.now.strftime("%Y%m%d")}-#{$$}-#{rand(0x100000000).to_s(36)}-YOUR_SUFFIX")
=> "/tmp/YOUR_PREFIX-20190827-1-140far-YOUR_SUFFIX"
Dir::Tmpname.make_tmpname (Ruby 2.5.0 and earlier)
Dir::Tmpname.make_tmpname was removed in Ruby 2.5.0. Prior to Ruby 2.4.4 it could accept a directory path as a prefix, but as of Ruby 2.4.4, directory separators are removed.
Digging in tempfile.rb you'll notice that Tempfile includes Dir::Tmpname. Inside you'll find make_tmpname which does what you ask for.
require 'tmpdir'
# => true
File.join(Dir.tmpdir, Dir::Tmpname.make_tmpname("prefix-", nil))
# => "/tmp/prefix-20190827-1-dfhvld"
File.join(Dir.tmpdir, Dir::Tmpname.make_tmpname(["prefix-", ".ext"], nil))
# => "/tmp/prefix-20190827-1-19zjck1.ext"
File.join(Dir.tmpdir, Dir::Tmpname.make_tmpname(["prefix-", ".ext"], "suffix"))
# => "/tmp/prefix-20190827-1-f5ipo7-suffix.ext"
Since Dir::Tmpname.make_tmpname was removed in Ruby 2.5.0, this one falls back to using SecureRandom:
require "tmpdir"
def generate_temp_filename(ext=".png")
filename = begin
Dir::Tmpname.make_tmpname(["x", ext], nil)
rescue NoMethodError
require "securerandom"
"#{SecureRandom.urlsafe_base64}#{ext}"
end
File.join(Dir.tmpdir, filename)
end
Since you only need the filename, what about using the SecureRandom for that:
require 'securerandom'
filename = "#{SecureRandom.hex(6)}.png" #=> "0f04dd94addf.png"
You can also use SecureRandom.alphanumeric
I found the Dir:Tmpname solution did not work for me. When evaluating this:
Dir::Tmpname.make_tmpname "/tmp/blob", nil
Under MRI Ruby 1.9.3p194 I get:
uninitialized constant Dir::Tmpname (NameError)
Under JRuby 1.7.5 (1.9.3p393) I get:
NameError: uninitialized constant Dir::Tmpname
You might try something like this:
def temp_name(file_name='', ext='', dir=nil)
id = Thread.current.hash * Time.now.to_i % 2**32
name = "%s%d.%s" % [file_name, id, ext]
dir ? File.join(dir, name) : name
end
I'm struggling with getting rubyzip to append directories to a zipoutputstream. (I want the output stream so I can send it from a rails controller). My code follows this example:
http://info.michael-simons.eu/2008/01/21/using-rubyzip-to-create-zip-files-on-the-fly/
When modified to include directories in the list of files to add I get the following error:
Any help would be greatly appreciated.
UPDATE
After trying a number of solutions I had best success with zipruby which has a clean api and good examples: http://zipruby.rubyforge.org/.
Zip::ZipFile.open(path, Zip::ZipFile::CREATE) do |zip|
songs.each do |song|
zip.add "record/#{song.title.parameterize}.mp3", song.file.to_file.path
end
end
OOOOOuuuhh...you DEFINITELY want ZIPPY. It's a Rails plugin that abstracts a lot of the complexity in rubyzip, and lets you create what you're talking about, including directories (from what I recall).
Here you go:
http://github.com/toretore/zippy
And direct from the zippy site:
Example controller:
def show
#gallery = Gallery.find(params[:id])
respond_to do |format|
format.html
format.zip
end
end
Example view:
zip['description.txt'] = #gallery.description
#gallery.photos.each do |photo|
zip["photo_#{photo.id}.png"] = File.open(photo.url)
end
edit: Amending per user comment:
Hmm...the whole objective of using Zippy is to make it a whole lot easier to use ruby zip.
Ya might want to take a second (or first) look...
Here's how to make a directory with directories:
some_var = Zippy.open('awsum.zip') do |zip|
%w{dir_a dir_b dir_c diri}.each do |dir|
zip["bin/#{dir}/"]
end
end
...
send_file some_var, :file_name => ...
Zippy will work for this. There may be a more cool way to do this but since there are essentially no docs, here's what I came up with for recursively copying directories with Zippy in a Rakefile. This Rakefile is used in a Rails environment so I put gem requirements in my Gemfile:
#Gemfile
source 'http://rubygems.org'
gem 'rails'
gem 'zippy'
And this is the Rakefile
#Rakefile
def add_file( zippyfile, dst_dir, f )
zippyfile["#{dst_dir}/#{f}"] = File.open(f)
end
def add_dir( zippyfile, dst_dir, d )
glob = "#{d}/**/*"
FileList.new( glob ).each { |f|
if (File.file?(f))
add_file zippyfile, dst_dir, f
end
}
end
task :myzip do
Zippy.create 'my.zip' do |z|
add_dir z, 'my', 'app'
add_dir z, 'my', 'config'
#...
add_file z, 'my', 'config.ru'
add_file z, 'my', 'Gemfile'
#...
end
end
Now I can use it like this:
C:\> cd my
C:\my> rake myzip
and it will produce my.zip which contains an inner directory called 'my' with copies of selected files and directories.
I was able to get directories working with the same ZipOutputStream used in the original article.
All I had to do was add the directory when calling zos.put_next_entry.
For example:
require 'zip/zip'
require 'zip/zipfilesystem'
t = Tempfile.new("some-weird-temp-file-basename-#{request.remote_ip}")
# Give the path of the temp file to the zip outputstream, it won't try to open it as an archive.
Zip::ZipOutputStream.open(t.path) do |zos|
some_file_list.each do |file|
# Create a new entry with some arbitrary name
zos.put_next_entry("myfolder/some-funny-name.jpg") # Added myfolder/
# Add the contents of the file, don't read the stuff linewise if its binary, instead use direct IO
zos.print IO.read(file.path)
end
end
# End of the block automatically closes the file.
# Send it using the right mime type, with a download window and some nice file name.
send_file t.path, :type => 'application/zip', :disposition => 'attachment', :filename => "some-brilliant-file-name.zip"
# The temp file will be deleted some time...
t.close
I just changed zos.put_next_entry('some-funny-name.jpg') to zos.put_next_entry('myfolder/some-funny-name.jpg'), and the resulting zipfile had a nested folder called myfolder that contained the files.