I think this is a pretty simple question, but I'm stuck.
I'm trying to write a script to look in the first directory, then check a second directory to see if there are matching files. If the files do match, I need to copy the files from the second directory to a third directory, only if they do not already exist in the third directory.
Here's a copy of what I have so far. The code still overwrites the files in the third directory even if they already exist. What am I doing wrong?
Thanks!
#!/usr/bin/ruby
require 'FileUtils'
library_path_1 = ARGV[0]
library_path_2 = ARGV[1]
library_path_3 = ARGV[2]
dir_1 = Dir.glob(library_path_1 + "/**/*").select{ |x| File.file? x }
dir_2 = Dir.glob(library_path_2 + "/**/*").select{ |x| File.file? x }
destination = Dir.glob(library_path_3 + "/**/*").select{ |x| File.file? x }
dir_1.each do |filename|
dir_2.each do |path|
destination.each do |existing_file|
existing_file = File.basename(existing_file)
if path.include?(File.basename(filename))
FileUtils.cp(path, library_path_3) unless File.exists?(existing_file)
end
end
end
end
bn = dir1.map { |fn| File.basename fn } - destination.map { |fn| File.basename fn }
dir2.each { |fn| FileUtils.cp(fn, library_path_3) if bn.include? File.basename(fn) }
dir_1.each do |file|
filename = File.basename(file)
if exists_in_dir2?(filename) && exists_in_dir3?(filename) == false
copy_to_destination(filename)
end
end
def copy_to_destination(filename)
File.copy(filename, "#{library_path_3}/{filename}")
end
def exists_in_dir2?(filename)
dir_2.each do |path|
path.include?("#{filename}")
end
end
def exists_in_dir3?(filename)
destination.each do |existing_file|
File.exist?("#{filename}")
end
end
It seems you are validating files by its just file names, but you have to check them by full name, like this:
existing_file = File.basename(existing_file)
if path.include?(File.basename(filename))
FileUtils.cp(path, library_path_3) unless File.exist?(File.join(library_path_3, existing_file))
end
Here's the final script that is working based on #Cary Swoveland's answer. Thanks all!
require 'FileUtils'
library_path_1 = ARGV[0]
library_path_2 = ARGV[1]
library_path_3 = ARGV[2]
# check the managed media folder for files, look in original capture scratch for all files in managed media.
# if files in managed media exist in original capture scratch, copy them to the destination folder.
managed_media_folder = Dir.glob(library_path_1 + "/**/*").select{ |x| File.file? x }
original_capture_scratch = Dir.glob(library_path_2 + "/**/*").select{ |x| File.file? x }
destination = Dir.glob(library_path_3 + "/**/*").select{ |x| File.file? x }
bn = managed_media_folder.map { |fn| File.basename fn } - destination.map { |fn| File.basename fn }
original_capture_scratch.each do |fn|
if bn.include? File.basename(fn)
puts fn
FileUtils.cp(fn, library_path_3)
end
end
I have a directory structure with sub-directories:
../../../../../MY_PROJECT/TEST_A/cats/
../../../../../MY_PROJECT/TEST_B/dogs/
../../../../../MY_PROJECT/TEST_A/tigers/
../../../../../MY_PROJECT/TEST_A/elephants/
each of which has a file that ends with ".sln":
../../../../../MY_PROJECT/TEST_A/cats/cats.sln
../../../../../MY_PROJECT/TEST_B/dogs/dogs.sln
...
These files contain information specific to their directory. I would like to do the following:
Create a file "myfile.txt" within each sub-directory, and write some strings to them:
../../../../../MY_PROJECT/TEST_A/cats/myfile.txt
../../../../../MY_PROJECT/TEST_B/dogs/myfile.txt
../../../../../MY_PROJECT/TEST_A/tigers/myfile.txt
../../../../../MY_PROJECT/TEST_A/elephants/myfile.txt
Copy a specific string in the ".sln" files to the myfile.txt of certain directories using the following method:
def parse_sln_files
sln_files = Dir["../../../../../MY_PROJECT/TEST_*/**/*.sln"]
sln_files.each do |file_name|
File.open(file_name) do |f|
f.each_line { |line|
if line =~ /C Source files ="..\\/ #"
path = line.scan(/".*.c"/)
puts path
end
}
end
end
end
I would like to do something like this:
def create_myfile
Dir['../../../../../MY_PROJECT/TEST_*/*/'].each do |dir|
File.new File.join(dir, 'myfile.txt'), 'w+'
Dir['../../../../../TEST/TEST_*/*/myfile.txt'].each do |path|
File.open(path,'w+') do |f|
f.puts "some text...."
f.puts "some text..."
f.puts # here I would like to return the result of parse_sln_files
end
end
end
end
Any suggestions on how to express this?
It seems like you want to read list of C file names from a Visual C++ Solution file, and store in a separate file in the same directory. You may have to merge the two loops that you have shown in your code, and do something like this:
def parse_sln_and_store_source_files
sln_files = Dir["../../../../../MY_PROJECT/TEST_*/**/*.sln"]
sln_files.each do |file_name|
#### Lets collect source file names in this array
source_file_names = []
File.open(file_name) do |f|
f.each_line { |line|
if line =~ /C Source files ="..\\/ #"
path = line.scan(/".*.c"/)
############ Add path to array ############
source_file_names << path
end
}
end
#### lets create `myfile.txt` in same dir as that of .sln
test_file = File.expand_path(File.dirname(file_name)) + "/myfile.txt"
File.open(test_file,'w+') do |f|
f.puts "some text...."
f.puts "some text..."
##### Iterate over source file names & write to file
source_file_names.each { |n| f.puts n }
end
end
end
This can be done bit more elegantly with few more refactoring. Also note that this is not tested code, hopefully, you get the gist of what I am suggesting.
I am trying to write a script to do the following:
There are two directories A and B. In directory A, there are files called "today" and "today1". In directory B, there are three files called "today", "today1" and "otherfile".
I want to loop over the files in directory A and append the files that have similar names in directory B to the files in Directory A.
I wrote the method below to handle this but I am not sure if this is on track or if there is a more straightforward way to handle such a case?
Please note I am running the script from directory B.
def append_data_to_daily_files
directory = "B"
Dir.entries('B').each do |file|
fileName = file
next if file == '.' or file == '..'
File.open(File.join(directory, file), 'a') {|file|
Dir.entries('.').each do |item|
next if !(item.match(/fileName/))
File.open(item, "r")
file<<item
item.close
end
#file.puts "hello"
file.close
}
end
end
In my opinion, your append_data_to_daily_files() method is trying to do too many things -- which makes it difficult to reason about. Break down the logic into very small steps, and write a simple method for each step. Here's a start along that path.
require 'set'
def dir_entries(dir)
Dir.chdir(dir) {
return Dir.glob('*').to_set
}
end
def append_file_content(target, source)
File.open(target, 'a') { |fh|
fh.write(IO.read(source))
}
end
def append_common_files(target_dir, source_dir)
ts = dir_entries(target_dir)
ss = dir_entries(source_dir)
common_files = ts.intersection(ss)
common_files.each do |file_name|
t = File.join(target_dir, file_name)
s = File.join(source_dir, file_name)
append_file_content(t, s)
end
end
# Run script like this:
# ruby my_script.rb A B
append_common_files(*ARGV)
By using a Set, you can easily figure out the common files. By using glob you can avoid the hassle of filtering out the dot-directories. By designing the code to take its directory names from the command line (rather than hard-coding the names in the script), you end up with a potentially re-usable tool.
My solution....
def append_old_logs_to_daily_files
directory = "B"
#For each file in the folder "B"
Dir.entries('B').each do |file|
fileName = file
#skip dot directories
next if file == '.' or file == '..'
#Open each file
File.open(File.join(directory, file), 'a') {|file|
#Get each log file from the current directory in turn
Dir.entries('.').each do |item|
next if item == '.' or item == '..'
#that matches the day we are looking for
next if !(item.match(fileName))
#Read the log file
logFilesToBeCopied = File.open(item, "r")
contents = logFilesToBeCopied.read
file<<contents
end
file.close
}
end
end
I have to following code in ruby:
<%
files = Dir.glob('/**/*')
files.each do |file|
puts file
end
%>
It outputs (for example):
/dirA/file1.txt
/dirA/file2.txt
/dirB/file1.txt
/file1.txt
/file2.txt
/subdirA/file1.txt
I want it to output it like this:
/file1.txt
/file2.txt
/dirA/file1.txt
/dirA/file2.txt
/dirB/file1.txt
/subdirA/file1.txt
Basically, I'd like to have the files displayed before the directories. Is there a sorting command I can use?
I believe this should work for you:
files = Dir.glob('**/*')
files = files.map { |file| [file.count("/"), file] }
files = files.sort.map { |file| file[1] }
files.each do |file|
puts file
end
Change "/" to ?/ if you're on Ruby 1.8.
Or, as a one-liner: :)
Dir.glob('**/*').map { |file| [file.count("/"), file] }.sort.map { |file| file[1] }.each { |file| puts file }
d,f = Dir.glob('*').partition{|d|test(?d,d)}
d.sort.each{|x|puts x}
f.sort.each{|y|puts y}
I want to get all file names from a folder using Ruby.
You also have the shortcut option of
Dir["/path/to/search/*"]
and if you want to find all Ruby files in any folder or sub-folder:
Dir["/path/to/search/**/*.rb"]
Dir.entries(folder)
example:
Dir.entries(".")
Source: http://ruby-doc.org/core/classes/Dir.html#method-c-entries
The following snippets exactly shows the name of the files inside a directory, skipping subdirectories and ".", ".." dotted folders:
Dir.entries("your/folder").select { |f| File.file? File.join("your/folder", f) }
To get all files (strictly files only) recursively:
Dir.glob('path/**/*').select { |e| File.file? e }
Or anything that's not a directory (File.file? would reject non-regular files):
Dir.glob('path/**/*').reject { |e| File.directory? e }
Alternative Solution
Using Find#find over a pattern-based lookup method like Dir.glob is actually better. See this answer to "One-liner to Recursively List Directories in Ruby?".
This works for me:
If you don't want hidden files[1], use Dir[]:
# With a relative path, Dir[] will return relative paths
# as `[ './myfile', ... ]`
#
Dir[ './*' ].select{ |f| File.file? f }
# Want just the filename?
# as: [ 'myfile', ... ]
#
Dir[ '../*' ].select{ |f| File.file? f }.map{ |f| File.basename f }
# Turn them into absolute paths?
# [ '/path/to/myfile', ... ]
#
Dir[ '../*' ].select{ |f| File.file? f }.map{ |f| File.absolute_path f }
# With an absolute path, Dir[] will return absolute paths:
# as: [ '/home/../home/test/myfile', ... ]
#
Dir[ '/home/../home/test/*' ].select{ |f| File.file? f }
# Need the paths to be canonical?
# as: [ '/home/test/myfile', ... ]
#
Dir[ '/home/../home/test/*' ].select{ |f| File.file? f }.map{ |f| File.expand_path f }
Now, Dir.entries will return hidden files, and you don't need the wildcard asterix (you can just pass the variable with the directory name), but it will return the basename directly, so the File.xxx functions won't work.
# In the current working dir:
#
Dir.entries( '.' ).select{ |f| File.file? f }
# In another directory, relative or otherwise, you need to transform the path
# so it is either absolute, or relative to the current working dir to call File.xxx functions:
#
home = "/home/test"
Dir.entries( home ).select{ |f| File.file? File.join( home, f ) }
[1] .dotfile on unix, I don't know about Windows
In Ruby 2.5 you can now use Dir.children. It gets filenames as an array except for "." and ".."
Example:
Dir.children("testdir") #=> ["config.h", "main.rb"]
http://ruby-doc.org/core-2.5.0/Dir.html#method-c-children
Personally, I found this the most useful for looping over files in a folder, forward looking safety:
Dir['/etc/path/*'].each do |file_name|
next if File.directory? file_name
end
This is a solution to find files in a directory:
files = Dir["/work/myfolder/**/*.txt"]
files.each do |file_name|
if !File.directory? file_name
puts file_name
File.open(file_name) do |file|
file.each_line do |line|
if line =~ /banco1/
puts "Found: #{line}"
end
end
end
end
end
this code returns only filenames with their extension (without a global path)
Dir.children("/path/to/search/")
=> [file_1.rb, file_2.html, file_3.js]
While getting all the file names in a directory, this snippet can be used to reject both directories [., ..] and hidden files which start with a .
files = Dir.entries("your/folder").reject {|f| File.directory?(f) || f[0].include?('.')}
This is what works for me:
Dir.entries(dir).select { |f| File.file?(File.join(dir, f)) }
Dir.entries returns an array of strings. Then, we have to provide a full path of the file to File.file?, unless dir is equal to our current working directory. That's why this File.join().
Dir.new('/home/user/foldername').each { |file| puts file }
You may also want to use Rake::FileList (provided you have rake dependency):
FileList.new('lib/*') do |file|
p file
end
According to the API:
FileLists are lazy. When given a list of glob patterns for possible
files to be included in the file list, instead of searching the file
structures to find the files, a FileList holds the pattern for latter
use.
https://docs.ruby-lang.org/en/2.1.0/Rake/FileList.html
One simple way could be:
dir = './' # desired directory
files = Dir.glob(File.join(dir, '**', '*')).select{|file| File.file?(file)}
files.each do |f|
puts f
end
def get_path_content(dir)
queue = Queue.new
result = []
queue << dir
until queue.empty?
current = queue.pop
Dir.entries(current).each { |file|
full_name = File.join(current, file)
if not (File.directory? full_name)
result << full_name
elsif file != '.' and file != '..'
queue << full_name
end
}
end
result
end
returns file's relative paths from directory and all subdirectories
If you want get an array of filenames including symlinks, use
Dir.new('/path/to/dir').entries.reject { |f| File.directory? f }
or even
Dir.new('/path/to/dir').reject { |f| File.directory? f }
and if you want to go without symlinks, use
Dir.new('/path/to/dir').select { |f| File.file? f }
As shown in other answers, use Dir.glob('/path/to/dir/**/*') instead of Dir.new('/path/to/dir') if you want to get all the files recursively.
In addition to the suggestions in this thread, I wanted to mention that if you need to return dot files as well (.gitignore, etc), with Dir.glob you would need to include a flag as so:
Dir.glob("/path/to/dir/*", File::FNM_DOTMATCH)
By default, Dir.entries includes dot files, as well as current a parent directories.
For anyone interested, I was curious how the answers here compared to each other in execution time, here was the results against deeply nested hierarchy. The first three results are non-recursive:
user system total real
Dir[*]: (34900 files stepped over 100 iterations)
0.110729 0.139060 0.249789 ( 0.249961)
Dir.glob(*): (34900 files stepped over 100 iterations)
0.112104 0.142498 0.254602 ( 0.254902)
Dir.entries(): (35600 files stepped over 100 iterations)
0.142441 0.149306 0.291747 ( 0.291998)
Dir[**/*]: (2211600 files stepped over 100 iterations)
9.399860 15.802976 25.202836 ( 25.250166)
Dir.glob(**/*): (2211600 files stepped over 100 iterations)
9.335318 15.657782 24.993100 ( 25.006243)
Dir.entries() recursive walk: (2705500 files stepped over 100 iterations)
14.653018 18.602017 33.255035 ( 33.268056)
Dir.glob(**/*, File::FNM_DOTMATCH): (2705500 files stepped over 100 iterations)
12.178823 19.577409 31.756232 ( 31.767093)
These were generated with the following benchmarking script:
require 'benchmark'
base_dir = "/path/to/dir/"
n = 100
Benchmark.bm do |x|
x.report("Dir[*]:") do
i = 0
n.times do
i = i + Dir["#{base_dir}*"].select {|f| !File.directory? f}.length
end
puts " (#{i} files stepped over #{n} iterations)"
end
x.report("Dir.glob(*):") do
i = 0
n.times do
i = i + Dir.glob("#{base_dir}/*").select {|f| !File.directory? f}.length
end
puts " (#{i} files stepped over #{n} iterations)"
end
x.report("Dir.entries():") do
i = 0
n.times do
i = i + Dir.entries(base_dir).select {|f| !File.directory? File.join(base_dir, f)}.length
end
puts " (#{i} files stepped over #{n} iterations)"
end
x.report("Dir[**/*]:") do
i = 0
n.times do
i = i + Dir["#{base_dir}**/*"].select {|f| !File.directory? f}.length
end
puts " (#{i} files stepped over #{n} iterations)"
end
x.report("Dir.glob(**/*):") do
i = 0
n.times do
i = i + Dir.glob("#{base_dir}**/*").select {|f| !File.directory? f}.length
end
puts " (#{i} files stepped over #{n} iterations)"
end
x.report("Dir.entries() recursive walk:") do
i = 0
n.times do
def walk_dir(dir, result)
Dir.entries(dir).each do |file|
next if file == ".." || file == "."
path = File.join(dir, file)
if Dir.exist?(path)
walk_dir(path, result)
else
result << file
end
end
end
result = Array.new
walk_dir(base_dir, result)
i = i + result.length
end
puts " (#{i} files stepped over #{n} iterations)"
end
x.report("Dir.glob(**/*, File::FNM_DOTMATCH):") do
i = 0
n.times do
i = i + Dir.glob("#{base_dir}**/*", File::FNM_DOTMATCH).select {|f| !File.directory? f}.length
end
puts " (#{i} files stepped over #{n} iterations)"
end
end
The differences in file counts are due to Dir.entries including hidden files by default. Dir.entries ended up taking a bit longer in this case due to needing to rebuild the absolute path of the file to determine if a file was a directory, but even without that it was still taking consistently longer than the other options in the recursive case. This was all using ruby 2.5.1 on OSX.
When loading all names of files in the operating directory you can use
Dir.glob("*)
This will return all files within the context that the application is running in (Note for Rails this is the top level directory of the application)
You can do additional matching and recursive searching found here https://ruby-doc.org/core-2.7.1/Dir.html#method-c-glob
if you create directories with spaces:
mkdir "a b"
touch "a b/c"
You don't need to escape the directory names, it will do it automatically:
p Dir["a b/*"] # => ["a b/c"]