Script to append files - ruby

I am trying to write a script to do the following:
There are two directories A and B. In directory A, there are files called "today" and "today1". In directory B, there are three files called "today", "today1" and "otherfile".
I want to loop over the files in directory A and append the files that have similar names in directory B to the files in Directory A.
I wrote the method below to handle this but I am not sure if this is on track or if there is a more straightforward way to handle such a case?
Please note I am running the script from directory B.
def append_data_to_daily_files
directory = "B"
Dir.entries('B').each do |file|
fileName = file
next if file == '.' or file == '..'
File.open(File.join(directory, file), 'a') {|file|
Dir.entries('.').each do |item|
next if !(item.match(/fileName/))
File.open(item, "r")
file<<item
item.close
end
#file.puts "hello"
file.close
}
end
end

In my opinion, your append_data_to_daily_files() method is trying to do too many things -- which makes it difficult to reason about. Break down the logic into very small steps, and write a simple method for each step. Here's a start along that path.
require 'set'
def dir_entries(dir)
Dir.chdir(dir) {
return Dir.glob('*').to_set
}
end
def append_file_content(target, source)
File.open(target, 'a') { |fh|
fh.write(IO.read(source))
}
end
def append_common_files(target_dir, source_dir)
ts = dir_entries(target_dir)
ss = dir_entries(source_dir)
common_files = ts.intersection(ss)
common_files.each do |file_name|
t = File.join(target_dir, file_name)
s = File.join(source_dir, file_name)
append_file_content(t, s)
end
end
# Run script like this:
# ruby my_script.rb A B
append_common_files(*ARGV)
By using a Set, you can easily figure out the common files. By using glob you can avoid the hassle of filtering out the dot-directories. By designing the code to take its directory names from the command line (rather than hard-coding the names in the script), you end up with a potentially re-usable tool.

My solution....
def append_old_logs_to_daily_files
directory = "B"
#For each file in the folder "B"
Dir.entries('B').each do |file|
fileName = file
#skip dot directories
next if file == '.' or file == '..'
#Open each file
File.open(File.join(directory, file), 'a') {|file|
#Get each log file from the current directory in turn
Dir.entries('.').each do |item|
next if item == '.' or item == '..'
#that matches the day we are looking for
next if !(item.match(fileName))
#Read the log file
logFilesToBeCopied = File.open(item, "r")
contents = logFilesToBeCopied.read
file<<contents
end
file.close
}
end
end

Related

How do I open each file in a directory with Ruby?

I need to open each file inside a directory. My attempt at this looks like:
Dir.foreach('path/to/directory') do |filename|
next if filename == '.' || filename == '..'
puts "working on #{filename}"
# this is where it crashes
file = File.open(filename, 'r')
#some code
file.close
# more code
end
My code keeps crashing at File.open(filename, 'r'). I'm not sure what filename should be.
The filename should include the path to the file when the file is not in the same directory than the Ruby file itself:
path = 'path/to/directory'
Dir.foreach(path) do |filename|
next if filename == '.' || filename == '..'
puts "working on #{filename}"
file = File.open("#{path}/#{filename}", 'r')
#some code
file.close
# more code
end
I recommend using Find.find.
While we can use various methods from the Dir class, it will look and retrieve the list of files before returning, which can be costly if we're recursively searching multiple directories or have a huge number of files embedded in the directories.
Instead, Find.find will walk the directories, returning both the directories and files as each is found. A simple check lets us decide which we want to continue processing or whether we want to skip it. The documentation has this example which should be easy to understand:
The Find module supports the top-down traversal of a set of file paths.
For example, to total the size of all files under your home directory, ignoring anything in a “dot” directory (e.g. $HOME/.ssh):
require 'find'
total_size = 0
Find.find(ENV["HOME"]) do |path|
if FileTest.directory?(path)
if File.basename(path)[0] == ?.
Find.prune # Don't look any further into this directory.
else
next
end
else
total_size += FileTest.size(path)
end
end
I'd go for Dir.glob or File.find. But not Dir.foreach as it returns . and .. which you don't want.
Dir.glob('something/*').each do |filename|
next if File.directory?(filename)
do_something_with_the_file(filename)
end

How do I copy .htaccess files using Rake?

I am in the process of creating some build scripts, using Rake, that will be used as part of the overall process of deploying our web services to the cloud via Docker containers. In order to accomplish this we combine resources from several repos using Rake to "assemble" the directory/file layout. This all work well save for one item, .htaccess files.
Here is the copy function that I've created:
require 'fileutils'
EXT_ALLOWED = ["html", "css", "js", "svg", "otf", "eot", "ttf", "woff", "jpeg", "map", "ico", "map", "png", "db", "php", "conf"]
def copy_to(dest, src, trim="")
files = FileList.new()
EXT_ALLOWED.each {|ext| files.include "#{src}/**/*.#{ext}"}
files.each do |file|
dir = File.dirname(file)
filename = File.basename(file)
trimming = "/shared/" + trim + "(.*)"
path = dir.match(trimming)
if path == nil || dest == path[1] + '/'
bin = dest
else
bin = File.join(dest, path[1] + '/')
end
puts "copying #{file} to #{bin}"
FileUtils.mkdir_p(bin)
FileUtils.cp file, bin
end
end
The usage for this would be:
desc 'copies from shared/admin to the base server directory'
task :admin do
# Copy admin over
dest = 'www-server/'
src = '../shared/admin'
trim = "admin/"
copy_to dest, src, trim
end
The trim variable is there to make sure files are copied to the appropriate directories. In this case files in admin are copied directly to www-server without an admin subdirectory.
I, naively, tried adding "htaccess" to the EXT_ALLOWED array, but that failed.
I have also followed some items online, but most have to do with Octopress which does not solve the problem.
The .htaccess file is in ../shared/admin and needs to end up in www-server/, can I make that happen within this function? Or do I need to write something specifically for file names beginning with dots?
In this case, looking for a quick and dirty (yes...I feel dirty doing it this way!) option, I wrote a function which specifically looks for the .htaccess file in a particular directory:
def copy_htaccess(src, dest)
files = Dir.glob("#{src}/.*")
files.each do |file|
filename = File.basename(file)
if filename == ".htaccess"
puts "copying #{file} to #{dest}"
FileUtils.mkdir_p(dest)
FileUtils.cp file, dest
end
end
end
With the usage being performed this way:
desc 'copies the .htaccess file from one root to the web root'
task :htaccess do
src = '../shared/admin'
dest = 'www-server/'
copy_htaccess src, dest
end
Here I am able to use Dir.glob() to list all files starting with a ., then test for the .htaccess file and perform the copying.
I will be looking into ways to modifying the single copy function to make this cleaner, if possible. Perhaps this can be done by globbing the directory and adding the files starting with . to the files array.
EDIT: Rather than creating an additional function I found that I could just push the .htaccess file's information onto the end of the files array in the original copying function, after first checking if it exists in the source directory:
if File.file?("#{src}/.htaccess")
files.push("#{src}/.htaccess")
end
Making the whole function as shown below:
def copy_to(dest, src, trim="")
files = FileList.new()
EXT_ALLOWED.each {|ext| files.include "#{src}/**/*.#{ext}"}
if File.file?("#{src}/.htaccess")
files.push("#{src}/.htaccess")
end
files.each do |file|
dir = File.dirname(file)
filename = File.basename(file)
trimming = "/shared/" + trim + "(.*)"
path = dir.match(trimming)
if path == nil || dest == path[1] + '/'
bin = dest
else
bin = File.join(dest, path[1] + '/')
end
puts "copying #{file} to #{bin}"
FileUtils.mkdir_p(bin)
FileUtils.cp file, bin
end
end
Note that I am using .file? to test for an actual file where .exists? can return a directories truthiness. In the end you can use either method depending on your situation.

Directory walk call method when directory is reached

Trying to write a script that will search through a directory and sub-directories for specific files. I would like to do know how a certain directory or directories come up to call a method.
this is what I have tried and failed:
def display_directory(path)
list = Dir[path+'/*']
return if list.length == 0
list.each do |f|
if File.directory? f #is it a directory?
if File.directory?('config')
puts "this is the config folder"
end
printf "%-50s %s\n", f, "is a directory:".upcase.rjust(25)
else
printf "%-50s %s\n", f, "is not a directory:".upcase.rjust(25)
end
end
end
start = File.join("**")
puts "Processing directory\n\n".upcase.center(30)
display_directory start
this is what I want to happen.
app
app/controllers
app/helpers
app/mailers
app/models
app/models/bugzilla
app/models/security
app/views
app/views/auth
app/views/calendar
app/views/layouts
app/views/step
app/views/step_mailer
app/views/suggestion
app/views/suggestion_mailer
app/views/task
app/views/user
bin
--------------------------------------
config <----------(call method foo)
config/environments
config/initializers
config/locales
--------------------------------------
db
db/bugzilla
db/migrate
db/security
lib
lib/tasks
log
public
public/images
public/javascripts
public/stylesheets
script
script/performance
script/process
--------------------------
test <---------(call method foobar)
test/fixtures
test/fixtures/mailer
test/functional
test/integration
test/performance
test/unit
--------------------------
vendor
vendor/plugins
Instead
if File.directory?('config')
Try
if f.path.include?('config')
but this will work for every directory that have config on the name. You can put a larger substring to make a better match.
Also, it is very idiomatic in ruby use do..end for multiline blocks and {..} for single line.
I figured out a way. this works pretty well. I've added a method to show all the files in mentioned directory when reached.
def special_dir(path)
puts "------------------------------------"
sp_path = Dir.glob(File.join(path,"*","**"))
sp_path.each do |cf|
puts "\t" + cf
end
end
def walk(path)
list = Dir[path+'/*'].reject{ |r| r['doc'] || r['tmp']}
list.each do |x|
path = File.join(path, x)
if File.directory?(x)
if x =~ /config/ or x =~ /test/
special_dir(x)
else
puts "#{x}"
walk(path)
end
else
#puts x
end
end
end
start = File.join("**")
walk start

In Ruby- Parsing Directory and reading first row of the file

Below is the piece of code that is supposed read the directory and for each file entry prints the first row of the file. The issue is x is not visible so file is not being parsed.
Dir.foreach("C:/fileload/src") do |file_name|
x = file_name
puts x
f = File.open("C:/fileload/src/" +x)
f.readlines[1..1].each do |line|
puts line
end
end
Why are you assigning x to file_name? You can use file_name directly. And if you are only reading the first line of the file, why not try this?
#!/usr/bin/ruby
dir = "C:/fileload/src"
Dir.foreach(dir) do |file_name|
full = File.join(dir, file_name)
if File.file?(full)
f = File.open(full)
puts f.first
f.close
end
end
You should use File.join to safely combine paths in Ruby. I also checked that you are opening a file using the File.file? method.
You have no visibility issue with x. You should be using File::join or Pathname#+ to build your file paths. You should exclude non-files from consideration. You're selecting the second line, not the first with [1..1]. Here's a cleaner correct replacement for your sample code.
dir = "C:/fileload/src"
Dir.foreach(dir).
map { |fn| File.join(dir,fn) }.
select { |fn| File.file?(fn) }.
each { |fn| puts File.readlines(fn).first }

How to open and read files line-by-line from a directory?

I am trying to read file lines from a directory containing about 200 text files, however, I can't get Ruby to read them line-by-line. I did it before, using one text file, not reading them from a directory.
I can get the file names as strings, but I am struggling to open them and read each line.
Here are some of the methods I've tried.
Method 1:
def readdirectory
#filearray = []
Dir.foreach('mydirectory') do |i|
# puts i.class
#filearray.push(i)
#filearray.each do |s|
# #words =IO.readlines('s')
puts s
end#do
# puts #words
end#do
end#readdirectory
Method 2:
def tryread
Dir.foreach('mydir'){
|x| IO.readlines(x)
}
end#tryread
Method 3:
def tryread
Dir.foreach('mydir') do |s|
File.readlines(s).each do |line|
sentence =line.split
end#inner do
end #do
end#tryread
With every attempt to open the string passed by the loop function, I keep getting the error:
Permission denied - . (Errno::EACCES)
sudo ruby reader.rb or whatever your filename is.
Since permissions are process based you can not read files with elevated permissions if the process reading does not have them.
Only solutions are either to run the script with more permissions or call another process which is already running with higher permissions to read for you.
Thanks for all replies,I did a bit of trial and error and got it to work.This is the syntax I used
Dir.entries('lemmatised').each do |s|
if !File.directory?(s)
file = File.open("pathname/#{s}", 'r')
file.each_line do |line|
count+=1
#words<<line.split(/[^a-zA-Z]/)
end # inner do
puts #words
end #if
end #do
Try this one,
#it'll hold the lines
f = []
#here test directory contains all the files,
#write the path as per the your computer,
#mine's as you can see, below
#fetch filenames and keep in sorted order
a = Dir.entries("c:/Users/lordsangram/desktop/test")
#read the files, line by line
Dir.chdir("c:/Users/lordsangram/desktop/test")
#beginning for i = 1, to ignore first two elements of array a,
#which has no associated file names
2.upto(a.length-1) do |i|
File.readlines("#{a[i]}").each do |line|
f.push(line)
end
end
f.each do |l|
puts l
end
#the Tin Man -> you need to avoid processing "." and ".." which are listed in Dir.foreach and give the permission denied error. A simple if should fix all your apporoaches.
Dir.foreach(ARGV[0]) do |f|
if f != "." and f != ".."
# code to process file
# example
# File.open(ARGV[0] + "\\" + f) do |file|
# end
end
end

Resources