I have series of zip files under #workingdir, and am trying to unzip the files that match #Regexp, and print the lines from them.
require 'zip/zip'
#workingdir = '/my/dir/structure/*.zip'
#Regexp = '/yup:maybe.*nope/i'
Dir.glob(#workingdir) do |zips|
Zip::ZipFile.open(zips) do |file|
file.each do |search|
tempFile = file.read(search)
tempFile.each do |line|
if (line =~ #Regexp ) then
p line
end
end
end
end
end
Below is the error message from IRB:
NoMethodError: undefined method `each' for #<String:0x0000000168bf40>
from (irb):70:in `block (3 levels) in irb_binding'
from /var/lib/gems/1.9.1/gems/rubyzip2-2.0.2/lib/zip/zip.rb:1122:in `each'
from /var/lib/gems/1.9.1/gems/rubyzip2-2.0.2/lib/zip/zip.rb:1122:in `each'
from /var/lib/gems/1.9.1/gems/rubyzip2-2.0.2/lib/zip/zip.rb:1265:in `each'
from (irb):68:in `block (2 levels) in irb_binding'
from /var/lib/gems/1.9.1/gems/rubyzip2-2.0.2/lib/zip/zip.rb:1381:in `open'
from (irb):67:in `block in irb_binding'
from (irb):66:in `glob'
from (irb):66
from /usr/bin/irb:12:in `<main>'
I tried tempFile.grep, and received the same error, except that grep was an undefined method. I believe I need to define a class.
Turns out my code had two problems. 1) My regular expression was being processed as a string (I should not have used the quotes). 2) Seeing as it runs fine otherwise on Ruby 1.8.7, I suspect the is a difference in how 1.8.7 and 1.9.1 process the 'each' method. If anyone has additional insights, I'm more than happy to hear them. The code below works fine on 1.8.7:
require 'zip/zip'
#workingdir = '/my/dir/structure/*.zip'
#Regexp = /regexp/i
Dir.glob(#workingdir) do |zips|
Zip::ZipFile.open(zips) do |file|
file.each do |search|
tempFile = file.read(search)
tempFile.each do |line|
if (line =~ #Regexp) then
puts zips + ': ' + line.chomp
end
end
end
end
end
Thanks again everyone!
Related
here's my code:
> !#usr/bin/ruby
require 'fileutils'
Dir.chdir "/home/john/Documents"
if (Dir.exist?("Photoshoot") === false) then
Dir.mkdir "Photoshoot"
puts "Directory: 'Photoshoot' created"
end
Dir.chdir "/run/user/1000/gvfs"
camdirs = Dir.glob('*')
numcams = camdirs.length
camnum = 0
campath = []
while camnum < numcams do
campath.push("/run/user/1000/gvfs/#{camdirs[camnum]}/DCIM")
puts campath[camnum]
camnum += 1
end
campath.each do |path|
Dir.chdir (path)
foldnum = 0
foldir = Dir.glob('*')
puts foldir
Dir.entries("#{path}/#{foldir[foldnum]}").each do |filename|
filetype = File.extname(filename)
if filetype == ".JPG"
FileUtils.mv("#{path}/#{foldir[foldnum]}/#{filename}", "/home/john/Documents/Photoshoot")
end
foldnum += 1
end
end
puts "#{numcams} cameras detected"
I'm just trying to go into some cameras I have connected and extract all the images into a file but its giving me this error. One of the things that's messing me up is that the images are stored in sub-folders under DCIM. When I just use .entries it gives me the folders the images are in as well as the images.
/usr/lib/ruby/2.3.0/fileutils.rb:1387:in `copy': unknown file type: /run/user/1000/gvfs/gphoto2:host=%5Busb%3A002%2C021%5D/DCIM//IMG_0092.JPG (RuntimeError)
from /usr/lib/ruby/2.3.0/fileutils.rb:472:in `block in copy_entry'
from /usr/lib/ruby/2.3.0/fileutils.rb:1498:in `wrap_traverse'
from /usr/lib/ruby/2.3.0/fileutils.rb:469:in `copy_entry'
from /usr/lib/ruby/2.3.0/fileutils.rb:530:in `rescue in block in mv'
from /usr/lib/ruby/2.3.0/fileutils.rb:527:in `block in mv'
from /usr/lib/ruby/2.3.0/fileutils.rb:1571:in `block in fu_each_src_dest'
from /usr/lib/ruby/2.3.0/fileutils.rb:1585:in `fu_each_src_dest0'
from /usr/lib/ruby/2.3.0/fileutils.rb:1569:in `fu_each_src_dest'
from /usr/lib/ruby/2.3.0/fileutils.rb:517:in `mv'
from /home/john/Desktop/TestExtract.rb:34:in `block (2 levels) in <main>'
from /home/john/Desktop/TestExtract.rb:31:in `each'
from /home/john/Desktop/TestExtract.rb:31:in `block in <main>'
from /home/john/Desktop/TestExtract.rb:26:in `each'
from /home/john/Desktop/TestExtract.rb:26:in `<main>'
/run/user/1000/gvfs/gphoto2:host=%5Busb%3A002%2C022%5D/DCIM
/run/user/1000/gvfs/gphoto2:host=%5Busb%3A002%2C021%5D/DCIM
/run/user/1000/gvfs/gphoto2:host=%5Busb%3A002%2C020%5D/DCIM
104___03
105___04
106___05
102___01
[Finished in 0.1s with exit code 1]
[shell_cmd: ruby "/home/john/Desktop/TestExtract.rb"]
[dir: /home/john/Desktop]
[path: /home/john/bin:/home/john/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin]
Any advice? I can't figure out what's wrong.
The reason the path to your files looks strange is because your camera storage has been mounted using FUSE. If you look very closely, you'll see that it is looking for:
/run/user/1000/gvfs/gphoto2:host=%5Busb%3A002%2C021%5D/DCIM//IMG_0092.JPG
You have two forward slashes before the final filename. Try correcting this on line 34 of your app.
If the problem still manifests then it is possible that the user running the operation in Ruby does not have permission to that filesystem or the manner in which the paths are constructed by FUSE is not compatible with Ruby FileUtils.
You can try to run:
cat /run/user/1000/gvfs/gphoto2:host=%5Busb%3A002%2C021%5D/DCIM/IMG_0092.JPG
as the same user that is running the Ruby process to ensure you have read permission to the filesystem.
When I am using Kiba ELT, I have followed the tutorials in YouTube as well as the tutorials provided by the owner. Yet, I am getting this error:
bitlasoft#Bitlasoft-TS-22:~/test01$ bundle exec kiba movies.etl
{
"title: Blade Runner" => "title: Minority Report"
}
/home/bitlasoft/.rvm/gems/ruby-2.2.2/gems/themoviedb-1.0.1/lib/themoviedb/api.rb:37:in `set_response': Tmdb::InvalidApiKeyError (Tmdb::InvalidApiKeyError)
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/gems/themoviedb-1.0.1/lib/themoviedb/search.rb:75:in `fetch_response'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/gems/themoviedb-1.0.1/lib/themoviedb/search.rb:60:in `fetch'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/gems/themoviedb-1.0.1/lib/themoviedb/resource.rb:41:in `search'
from /home/bitlasoft/test01/common.rb:30:in `process'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/gems/kiba-0.6.1/lib/kiba/runner.rb:35:in `block (3 levels) in process_rows'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/gems/kiba-0.6.1/lib/kiba/runner.rb:34:in `each'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/gems/kiba-0.6.1/lib/kiba/runner.rb:34:in `block (2 levels) in process_rows'
from /home/bitlasoft/test01/common.rb:12:in `block in each'
from /home/bitlasoft/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/csv.rb:1739:in `each'
from /home/bitlasoft/test01/common.rb:11:in `each'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/gems/kiba-0.6.1/lib/kiba/runner.rb:33:in `block in process_rows'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/gems/kiba-0.6.1/lib/kiba/runner.rb:32:in `each'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/gems/kiba-0.6.1/lib/kiba/runner.rb:32:in `process_rows'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/gems/kiba-0.6.1/lib/kiba/runner.rb:13:in `run'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/gems/kiba-0.6.1/lib/kiba/cli.rb:13:in `run'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/gems/kiba-0.6.1/bin/kiba:5:in `<top (required)>'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/bin/kiba:23:in `load'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/bin/kiba:23:in `<main>'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:15:in `eval'
from /home/bitlasoft/.rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:15:in `<main>'
bitlasoft#Bitlasoft-TS-22:~/test01$
Here is my movies.etl and common.rb config:
require_relative 'common'
api_key = IO.read('.themoviedb')
source CSVSource, filename: 'movies.csv'
limit ENV['LIMIT']
show_me!
transform MovieDBlookup,
api_key: api_key,
title_field: 'title'
show_me!
require 'csv'
require 'awesome_print'
class CSVSource
def initialize(filename:)
#filename = filename
end
def each
csv = CSV.open(#filename, headers: true)
csv.each do |row|
yield(row.to_hash)
end
csv.close
end
end
require 'themoviedb'
class MovieDBlookup
def initialize(api_key:, title_field:)
#title_field = title_field
Tmdb::Api.key(api_key)
end
def process(row)
movie = Tmdb::Movie.find(row[#title_field]).first
row[:vote_average] = movie.vote_average
row[:vote_count] = movie.vote_count
row
end
end
def show_me!
transform do |row|
ap row
row
end
end
def limit(x)
x = Integer(x || -1)
return if x == -1
transform do |row|
#counter ||= 0
#counter += 1
abort("stopping....")if #counter >= x
row
end
end
(Kiba owner here) - the error you get isn't Kiba specific ; it looks like the API key you provided for themoviedb is invalid. Did you sign-up for the movie db here, and is the API key provided copy-pasted into the file you are loading (.themoviedb)?
One problem I could think of would be that you'd have a line ending character in that file (carriage return / line feed), in which case maybe just calling api_key = IO.read('.themoviedb').strip could help.
Again, this isn't Kiba specific, but hope this helps!
I have a directory with 100+ zip files and I need to read the files inside the zip files to do some data processing, without unzipping the archive.
Is there a Ruby library to read the contents of files in zip archives, without unzipping the file?
Using rubyzip gives an error:
require 'zip'
Zip::File.open('my_zip.zip') do |zip_file|
# Handle entries one by one
zip_file.each do |entry|
# Extract to file/directory/symlink
puts "Extracting #{entry.name}"
entry.extract('here')
# Read into memory
content = entry.get_input_stream.read
end
end
Gives this error:
test.rb:12:in `block (2 levels) in <main>': undefined method `read' for Zip::NullInputStream:Module (NoMethodError)
from .gem/ruby/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:42:in `call'
from .gem/ruby/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:42:in `block in each'
from .gem/ruby/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:41:in `each'
from .gem/ruby/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:41:in `each'
from .gem/ruby/gems/rubyzip-1.1.6/lib/zip/central_directory.rb:182:in `each'
from test.rb:6:in `block in <main>'
from .gem/ruby/gems/rubyzip-1.1.6/lib/zip/file.rb:99:in `open'
from test.rb:4:in `<main>'
The Zip::NullInputStream is returned if the entry is a directory and not a file, could that be the case?
Here's a more robust variation of the code:
#!/usr/bin/env ruby
require 'rubygems'
require 'zip'
Zip::File.open('my_zip.zip') do |zip_file|
# Handle entries one by one
zip_file.each do |entry|
if entry.directory?
puts "#{entry.name} is a folder!"
elsif entry.symlink?
puts "#{entry.name} is a symlink!"
elsif entry.file?
puts "#{entry.name} is a regular file!"
# Read into memory
entry.get_input_stream { |io| content = io.read }
# Output
puts content
else
puts "#{entry.name} is something unknown, oops!"
end
end
end
I came across the same issue and checking for if entry.file?, before entry.get_input_stream.read, resolved the issue.
require 'zip'
Zip::File.open('my_zip.zip') do |zip_file|
# Handle entries one by one
zip_file.each do |entry|
# Extract to file/directory/symlink
puts "Extracting #{entry.name}"
entry.extract('here')
# Read into memory
if entry.file?
content = entry.get_input_stream.read
end
end
end
Upgrade steps were performed as mentioned in http://guides.rubyonrails.org/3_2_release_notes.html
Right now I'm on Ruby 1.8.7 and Rails 3.1.x (using system ruby). I installed RVM with ruby 1.8.7 and added to rails 3.2 and then I get the following error:
[app]$ rails console
Faraday: you may want to install system_timer for reliable timeouts
$HOME/src/qbol/tapp/config/environment.rb:16:in `add': undefined method `>' for nil:NilClass (NoMethodError)
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/tagged_logging.rb:55:in `add'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/tagged_logging.rb:61:in `info'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activerecord-3.2.17/lib/active_record/railtie.rb:86
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/lazy_load_hooks.rb:36:in `instance_eval'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/lazy_load_hooks.rb:36:in `execute_hook'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/lazy_load_hooks.rb:26:in `on_load'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/lazy_load_hooks.rb:25:in `each'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/lazy_load_hooks.rb:25:in `on_load'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activerecord-3.2.17/lib/active_record/railtie.rb:80
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/initializable.rb:30:in `instance_exec'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/initializable.rb:30:in `run'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/initializable.rb:55:in `run_initializers'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/initializable.rb:54:in `each'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/initializable.rb:54:in `run_initializers'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/application.rb:136:in `initialize!'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/railtie/configurable.rb:30:in `send'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/railtie/configurable.rb:30:in `method_missing'
from $HOME/src/app/config/environment.rb:48
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:251:in `require'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:251:in `require'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:236:in `load_dependency'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:251:in `require'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/application.rb:103:in `require_environment!'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/commands.rb:40
from script/rails:6:in `require'
from script/rails:6
This is on my development box. Any idea why this is happening?
EDIT: Below is the environment.rb. The error is happening on the last line App::Application.initialize!
# Load the rails application
require File.expand_path('../application', __FILE__)
module ActiveSupport
class BufferedLogger
def self.current_user
Thread.current[:user]
end
def self.current_user=(user)
Thread.current[:user] = user
end
def add(severity, message = nil, progname = nil, &block)
return if #level > severity
message = (message || (block && block.call) || progname).to_s
level = {
0 => "DEBUG",
1 => "INFO ",
2 => "WARN ",
3 => "ERROR",
4 => "FATAL"
}[severity] || "UNKNOWN"
user=BufferedLogger.current_user
if(!user.nil?)
idstr = "uid:#{user.id}"
if !user.current_app_user.nil?
idstr.concat(", acid: #{user.current_app_user.account_id}")
end
else
idstr=""
end
message = "[%s: %s #{idstr}] %s" %
["#{level} pid: #{$$}", Time.now.strftime("%y-%m-%d %H:%M:%S"), message]
message = "#{message}\n" unless message[-1] == ?\n
buffer << message
auto_flush
message
end
end
end
# Initialize the rails application
App::Application.initialize!
The error is happening inside ActiveSupport::BufferedLogger#add method call and you appear to be monkey-patching this class. The internals of ActiveSupport::BufferedLogger have likely changing between the versions of Rails you're using.
Try removing all of the ActiveSupport::BufferedLogger code from your environment.rb to work past this error and get your app running. Then, if you still need the monkeypatch (don't know why), you'd have to rewrite it on top of the newer version of the class.
I can start a pry session of a command line app like this
pry -r ./todo.rb
However, if I want to call the list function
pry -r ./todo.rb list
I'm getting an error message.
Without pry, I call the list function
ruby todo.rb list
This is the error message
/Users/michaeljohnmitchell/.rvm/gems/ruby-1.9.2-p290#global/gems/pry-0.9.10/lib/pry/repl_file_loader.rb:16:in `initialize': No such file: /Users/michaeljohnmitchell/Sites/todo/bin/list (RuntimeError)
from /Users/michaeljohnmitchell/.rvm/gems/ruby-1.9.2-p290#global/gems/pry-0.9.10/lib/pry/pry_class.rb:161:in `new'
from /Users/michaeljohnmitchell/.rvm/gems/ruby-1.9.2-p290#global/gems/pry-0.9.10/lib/pry/pry_class.rb:161:in `load_file_through_repl'
from /Users/michaeljohnmitchell/.rvm/gems/ruby-1.9.2-p290#global/gems/pry-0.9.10/lib/pry/cli.rb:162:in `block in <top (required)>'
from /Users/michaeljohnmitchell/.rvm/gems/ruby-1.9.2-p290#global/gems/pry-0.9.10/lib/pry/cli.rb:65:in `call'
from /Users/michaeljohnmitchell/.rvm/gems/ruby-1.9.2-p290#global/gems/pry-0.9.10/lib/pry/cli.rb:65:in `block in parse_options'
from /Users/michaeljohnmitchell/.rvm/gems/ruby-1.9.2-p290#global/gems/pry-0.9.10/lib/pry/cli.rb:65:in `each'
from /Users/michaeljohnmitchell/.rvm/gems/ruby-1.9.2-p290#global/gems/pry-0.9.10/lib/pry/cli.rb:65:in `parse_options'
from /Users/michaeljohnmitchell/.rvm/gems/ruby-1.9.2-p290#global/gems/pry-0.9.10/bin/pry:16:in `<top (required)>'
from /Users/michaeljohnmitchell/.rvm/gems/ruby-1.9.2-p290#global/bin/pry:19:in `load'
from /Users/michaeljohnmitchell/.rvm/gems/ruby-1.9.2-p290#global/bin/pry:19:in `<main>'
Source Code
TODO_FILE = 'todo.txt'
def read_todo(line)
line.chomp.split(/,/)
end
def write_todo(file,name,created=Time.now,completed='')
file.puts("#{name},#{created},#{completed}")
end
command = ARGV.shift
case command
when 'new'
new_task = ARGV.shift
File.open(TODO_FILE,'a') do |file|
write_todo(file,new_task)
puts "Task added."
end
when 'list'
File.open(TODO_FILE,'r') do |file|
counter = 1
file.readlines.each do |line|
name,created,completed = read_todo(line)
printf("%3d - %s\n",counter,name)
printf(" Created : %s\n",created)
unless completed.nil?
printf(" Completed : %s\n",completed)
end
counter += 1
end
end
when 'done'
task_number = ARGV.shift.to_i
binding.pry
File.open(TODO_FILE,'r') do |file|
File.open("#{TODO_FILE}.new",'w') do |new_file|
counter = 1
file.readlines.each do |line|
name,created,completed = read_todo(line)
if task_number == counter
write_todo(new_file,name,created,Time.now)
puts "Task #{counter} completed"
else
write_todo(new_file,name,created,completed)
end
counter += 1
end
end
end
`mv #{TODO_FILE}.new #{TODO_FILE}`
end
Update
when I try
pry -r ./todo.rb -e list
I'm getting the following error
NameError: undefined local variable or method `list' for main:Object
From pry --help:
-e, --exec A line of code to execute in context before the session starts
So, if your list method is defined on main (if you don't know, it probably is), then you can do this:
pry -r ./todo.rb -e list
Update
Pry doesn't let you pass in arguments for scripts it loads (or at least it isn't documented). But all is not lost, you can call pry from your script. Just drop this at wherever you want to inspect:
require 'pry'; binding.pry
This will spawn a pry session that has access to all the local variables and methods.
I think you can use:
ruby -rpry ./todo.rb -e list