Improve speed of the file search in Ruby - ruby

Given a directory with about 100 000 small files (each files is about 1kB).
I need to get list of these files and iterate over it in order to find files with the same name but different case (the files are on Linux ext4 FS).
Currently, I use some code like this:
def similar_files_in_folder(file_path, folder, exclude_folders = false)
files = Dir.glob(file_path, File::FNM_CASEFOLD)
files_set = files.select{|f| f.start_with?(folder)}
return files_set unless exclude_folders
files_set.reject{|entry| File.directory? entry}
end
dir_entries = Dir.entries(#directory) - ['.', '..']
dir_entries.map do |file_name|
similar_files_in_folder(file_name, #directory)
end
The issue with this approach is that the snippet takes a lot!!! of time to finish.
It is about some hours on my system.
Is there another way to achieve the same goal but much faster in Ruby?
Limitation: I can't load the file list in memory and then just compare the names in down case, because in the #directory new files are appear.
So, I need to scan the #directory on each iteration.
Thanks for any hint.

If I understand your code correctly, this already returns an array of all those 100k filenames:
dir_entries = Dir.entries(#directory) - ['.', '..']
#=> ["foo.txt", "bar.txt", "BAR.txt", ...]
I would group this array by the lowercase filename:
dir_entries.group_by(&:downcase)
#=> {"foo.txt"=>["foo.txt"], "bar.txt"=>["bar.txt", "BAR.txt"], ... }
And select the ones with more than 1 occurrences:
dir_entries.group_by(&:downcase).select { |k, v| v.size > 1 }
#=> {"bar.txt"=>["bar.txt", "BAR.txt"], ...}

What I meant by my comment was that you could search for a string as you traverse the filesystem, instead of first building up a huge array of all possible files and only then searching. I wrote something similar to a linux find <path> | grep --color -i <pattern> , except highlighting the pattern only in basename:
require 'find'
#find files whose basename matches a pattern (and output results to console)
def find_similar(s, opts={})
#by default, path is '.', case insensitive, no bash terminal coloring
opts[:verbose] ||= false
opts[:path] ||= '.'
opts[:insensitive]=true if opts[:insensitive].nil?
opts[:color]||=false
boldred = "\e[1m\e[31m\\1\e[0m" #contains an escaped \1 for regex
puts "searching for \"#{s}\" in \"#{opts[:path]}\", insensitive=#{opts[:insensitive]}..." if opts[:verbose]
reg = opts[:insensitive] ? /(#{s})/i : /(#{s})/
dir,base = '',''
Find.find(opts[:path]) {|path|
dir,base = File.dirname(path), File.basename(path)
if base =~ reg
if opts[:color]
puts "#{dir}/#{base.gsub(reg, boldred)}"
else
puts path
end
end
}
end
time = Time.now
#find_similar('LOg', :color=>true) #similar to find . | grep --color -i LOg
find_similar('pYt', :path=>'c:/bin/sublime3/', :color=>true, :verbose=>true)
puts "search took #{Time.now-time}sec"
example output (cygwin), but also works if run from cmd.exe

Related

Recursively rename folders based on contents in Ruby

I have a collection of folders (within a folder) that all need to be renamed based on their contents.
Specifically, I'd like to rename "/working_directory/my_folder/my_file.extension" to /working_directory/my_file/my_file.extension"
There are a few other files within /my_folder/. How might I recursively do this using ruby?
I'm new to ruby and programming, I have so tried to just extract the file names, but have not have much luck. The attempt at itterating through the folders. This will cycle through /working_directory/ every time Find.find is called. The intent is to search /working_directory/my_folder/ only for the file with the .fls extension.
require 'find'
Path = "/working_directory/"
Dir.foreach(Path) do |file|
puts file
new_dir = Path+file
puts new_dir
Find.find(new_dir) do |i| # this is intended to by /working_directory/my_folder/
fls_file << i if i =~ /.*\.fls$/
puts fls_file
end
end
Assuming, the my_file is to be chosen by extension, one might do:
Dir["/working_directory/**/*"].select do |dir_or_file|
File.directory? dir_or_file # select only directories, recursively
end.inject({}) do |memo, dir|
new_name = Dir["#{dir}/*.extension"].to_a
unless new_name.size == 1 # check if the folder contains only one proper file
puts "Multiple/No choices; can not rename dir [#{dir}] ⇒ skipping..."
next memo # skip if no condition met
end
my_file = new_name.first[/[^\/]+(?=\.extension\z)/] # get my_name
memo[dir] = dir.gsub /[^\/]+(?=\/#{myfile}\.extension\z)/, my_file
memo
end.each do |old, neu|
# dry run to make sure everything is OK
puts "Gonna rename #{old} to #{neu}"
# uncomment the lines below as you are certain the code works properly
# neu_folder = neu[/(.*?)([^\/]+\z)/, 1]
# FileUtils.mkdir neu_folder unless File.exist? neu_folder
# FileUtils.mv old, neu # rename
end
The rename is done after the main processing for the sake of previous iterator consistency, probably in this case it might be done in the previous loop, instead of injecting old: neu pairs into hash and iterating it later.
We are heavily using string parsing with regexps here.
my_file = new_name.first[/[^\/]+(?=\.extension\z)/] # get my_name
this line gets a new folder name by parsing a tail of the string, containing no slashes and trailing with '.extension\z' (see positive lookahead.)
memo[dir] = dir.gsub /[^\/]+(?=\/#{myfile}\.extension\z)/, my_file
This line assigns a new element on an accumulator hash, substituting the old folder name with the new one.

In Ruby, how would I find all .csv files in a folder and print out path to the files that contain the word "meh"?

Title pretty much says it all, looking to search for all .csv files and puts out a list of all files with the word meh in the name. Assume there are a few.
EDIT:
This method is significantly more direct and efficient:
d = Dir.new('.')
d.entries.select do |e|
/^.+\.csv$/.match(e) && IO.readlines(e).grep(/meh/).length > 0
end
This should do it assuming you want to search the current directory
d = Dir.new('.')
# This will find all files whose path ends in .csv
csvs = d.entries.select {|e| /^.+\.csv$/.match(e)}
# This will find all .csv files that contain one or more instance
# of the pattern /meh/
mehs = csvs.select do |e|
f = File.open(e)
[*f.each_line].grep(/meh/).length > 0
end

Ruby class is outputting files as well as directories

Why does below class output directory's as well as filenames on line "print "\n"+f" ?
I just want to output the files but directories are also being outputted.
class Sort
require 'find'
directoryToSort = "c:\\test"
total_size = 0
Find.find(directoryToSort) do |path|
if FileTest.directory?(path)
if File.basename(path)[0] == ?.
Find.prune # Don't look any further into this directory.
else
Dir.foreach(path) do
|f|
# do whatever you want with f, which is a filename within the
# given directory (not fully-qualified)
if !FileTest.directory? f
print "\n"+f
end
end
next
end
else
end
end
end
It says right there in a comment:
# do whatever you want with f, which is a filename within the
# given directory (not fully-qualified)
key being "not fully-qualified" part. you need to do something like:
if !FileTest.directory? (path + File::SEPARATOR + f)
Consider using the Ruby standard File.directory? method instead.
you need File.directory?( filename ) to check if it's a filename
you probably want to do something along these lines....
this is a helper method for doing recursive directory descend and executing a block depending
on if the filename matches a certain Regular Expressions.. a bit overkill for you, but maybe this helps.
# recursiveDirectoryDescend
# do action for files matching regexp
#
# (not very elegant solution, but just for illustration purposes. Pulled from some very old code.)
def recursive_dir_descend(dir,regexp,action)
olddir = Dir.pwd
dirp = Dir.open(dir)
Dir.chdir(dir)
pwd = Dir.pwd
for file in dirp
file.chomp
next if file =~ /^\.\.?$/ # ON UNIX, ignore '.' and '..' directories
filename = "#{pwd}/#{file}"
if File.directory?(filename) # CHECK IF DIRECTORY
recursive_dir_descend(filename,regexp,action)
else
if file =~ regexp
eval action # execute action on filename
end
end
end
Dir.chdir(olddir)
end

Open a file case-insensitively in Ruby under Linux

Is there a way to open a file case-insensitively in Ruby under Linux? For example, given the string foo.txt, can I open the file FOO.txt?
One possible way would be reading all the filenames in the directory and manually search the list for the required file, but I'm looking for a more direct method.
One approach would be to write a little method to build a case insensitive glob for a given filename:
def ci_glob(filename)
glob = ''
filename.each_char do |c|
glob += c.downcase != c.upcase ? "[#{c.downcase}#{c.upcase}]" : c
end
glob
end
irb(main):024:0> ci_glob('foo.txt')
=> "[fF][oO][oO].[tT][xX][tT]"
and then you can do:
filename = Dir.glob(ci_glob('foo.txt')).first
Alternatively, you can write the directory search you suggested quite concisely. e.g.
filename = Dir.glob('*').find { |f| f.downcase == 'foo.txt' }
Prior to Ruby 3.1 it was possible to use the FNM_CASEFOLD option to make glob case insensitive e.g.
filename = Dir.glob('foo.txt', File::FNM_CASEFOLD).first
if filename
# use filename here
else
# no matching file
end
The documentation suggested FNM_CASEFOLD couldn't be used with glob but it did actually work in older Ruby versions. However, as mentioned by lildude in the comments, the behaviour has now been brought inline with the documentation and so this approach shouldn't be used.
You can use Dir.glob with the FNM_CASEFOLD flag to get a list of all filenames that match the given name except for case. You can then just use first on the resulting array to get any result back or use min_by to get the one that matches the case of the orignial most closely.
def find_file(f)
Dir.glob(f, File::FNM_CASEFOLD).min_by do |f2|
f.chars.zip(f2.chars).count {|c1,c2| c1 != c2}
end
end
system "touch foo.bar"
system "touch Foo.Bar"
Dir.glob("FOO.BAR", File::FNM_CASEFOLD) #=> ["foo.bar", "Foo.Bar"]
find_file("FOO.BAR") #=> ["Foo.Bar"]

One-liner to recursively list directories in Ruby?

What is the fastest, most optimized, one-liner way to get an array of the directories (excluding files) in Ruby?
How about including files?
Dir.glob("**/*/") # for directories
Dir.glob("**/*") # for all files
Instead of Dir.glob(foo) you can also write Dir[foo] (however Dir.glob can also take a block, in which case it will yield each path instead of creating an array).
Ruby Glob Docs
I believe none of the solutions here deal with hidden directories (e.g. '.test'):
require 'find'
Find.find('.') { |e| puts e if File.directory?(e) }
For list of directories try
Dir['**/']
List of files is harder, because in Unix directory is also a file, so you need to test for type or remove entries from returned list which is parent of other entries.
Dir['**/*'].reject {|fn| File.directory?(fn) }
And for list of all files and directories simply
Dir['**/*']
As noted in other answers here, you can use Dir.glob. Keep in mind that folders can have lots of strange characters in them, and glob arguments are patterns, so some characters have special meanings. As such, it's unsafe to do something like the following:
Dir.glob("#{folder}/**/*")
Instead do:
Dir.chdir(folder) { Dir.glob("**/*").map {|path| File.expand_path(path) } }
Fast one liner
Only directories
`find -type d`.split("\n")
Directories and normal files
`find -type d -or -type f`.split("\n")`
Pure beautiful ruby
require "pathname"
def rec_path(path, file= false)
puts path
path.children.collect do |child|
if file and child.file?
child
elsif child.directory?
rec_path(child, file) + [child]
end
end.select { |x| x }.flatten(1)
end
# only directories
rec_path(Pathname.new(dir), false)
# directories and normal files
rec_path(Pathname.new(dir), true)
In PHP or other languages to get the content of a directory and all its subdirectories, you have to write some lines of code, but in Ruby it takes 2 lines:
require 'find'
Find.find('./') do |f| p f end
this will print the content of the current directory and all its subdirectories.
Or shorter, You can use the ’**’ notation :
p Dir['**/*.*']
How many lines will you write in PHP or in Java to get the same result?
Here's an example that combines dynamic discovery of a Rails project directory with Dir.glob:
dir = Dir.glob(Rails.root.join('app', 'assets', 'stylesheets', '*'))
Dir.open(Dir.pwd).map { |h| (File.file?(h) ? "#{h} - file" : "#{h} - folder") if h[0] != '.' }
dots return nil, use compact
Although not a one line solution, I think this is the best way to do it using ruby calls.
First delete all the files recursively
Second delete all the empty directories
Dir.glob("./logs/**/*").each { |file| File.delete(file) if File.file? file }
Dir.glob("./logs/**/*/").each { |directory| Dir.delete(directory) }

Resources