I have files (with different extensions) that are added every 10 minutes on a windows share (A) and want to copy them to linux server (B) and do some operations on them with a script.
Using ruby and FilesUtils How can i create a script that will copy only the last added files (or have a watcher that will copy the files to folder B whenever they are added to my folder A)
update this is what i have so far
require 'fileutils'
require 'time'
class Copier
def initialize(from,to)
puts "copying files... puts #{Time.now} \n"
my_files = Dir["#{from}/*.*"].sort_by { |a| File.stat(a).mtime }
my_files.each do |filename|
name = File.basename(filename)
orig = "#{filename}"
dest = "#{to}/#{name}"
FileUtils.cp(orig, dest)
puts "cp file : from #{orig} => to #{dest}"
end
end
end
Copier.new("/mnt/windows_share", "linux_folder")
But it copies all the files each time it is called...
This is what I ended up doing to get the files modified in the last 10 minutes and then copy them from the a windows share folder to the linux folder:
require 'fileutils'
require 'time'
class Copier
def initialize(from,to)
puts "copying files... puts #{Time.now} \n"
my_files = Dir["#{from}/*.*"].select { |fname| File.mtime(fname) > (Time.now - (60*10)) })
my_files.each do |filename|
name = File.basename(filename)
orig = "#{filename}"
dest = "#{to}/#{name}"
FileUtils.cp(orig, dest)
puts "cp file : from #{orig} => to #{dest}"
end
end
end
Copier.new("/mnt/windows_share", "linux_folder")
Related
I'm adapting the RubyZip recursive zipping example (found here) to work with write_buffer instead of open and am coming across a host of issues. I'm doing this because the zip archive I'm producing has word documents in it and I'm getting errors on opening those word documents. Therefore, I'm trying the work-around that RubyZip suggests, which is using write_buffer instead of open (example found here).
The problem is, I'm getting errors because I'm using an absolute path, but I'm not sure how to get around that. I'm getting the error "#//', name must not start with />"
Second, I'm not sure what to do to mitigate the issue with word documents. When I used my original code, which worked and created an actual zip file, any word document in that zip file had the following error upon opening: "Word found unreadable content in Do you want to recover the contents of this document? If you trust the source of this document, click Yes." The unreadable content error is the reason why I went down the road of attempting to use write_buffer.
Any help would be appreciated.
Here is the code that I'm currently using:
require 'zip'
require 'zip/zipfilesystem'
module AdvisoryBoard
class ZipService
def initialize(input_dir, output_file)
#input_dir = input_dir
#output_file = output_file
end
# Zip the input directory.
def write
entries = Dir.entries(#input_dir) - %w[. ..]
path = ""
buffer = Zip::ZipOutputStream.write_buffer do |zipfile|
entries.each do |e|
zipfile_path = path == '' ? e : File.join(path, e)
disk_file_path = File.join(#input_dir, zipfile_path)
#file = nil
#data = nil
if !File.directory?(disk_file_path)
#file = File.open(disk_file_path, "r+b")
#data = #file.read
unless [#output_file, #input_dir].include?(e)
zipfile.put_next_entry(e)
zipfile.write #data
end
#file.close
end
end
zipfile.put_next_entry(#output_file)
zipfile.put_next_entry(#input_dir)
end
File.open(#output_file, "wb") { |f| f.write(buffer.string) }
end
end
end
I was able to get word documents to open without any warnings or corruption! Here's what I ended up doing:
require 'nokogiri'
require 'zip'
require 'zip/zipfilesystem'
class ZipService
# Initialize with the directory to zip and the location of the output archive.
def initialize(input_dir, output_file)
#input_dir = input_dir
#output_file = output_file
end
# Zip the input directory.
def write
entries = Dir.entries(#input_dir) - %w[. ..]
::Zip::File.open(#output_file, ::Zip::File::CREATE) do |zipfile|
write_entries entries, '', zipfile
end
end
private
# A helper method to make the recursion work.
def write_entries(entries, path, zipfile)
entries.each do |e|
zipfile_path = path == '' ? e : File.join(path, e)
disk_file_path = File.join(#input_dir, zipfile_path)
if File.directory? disk_file_path
recursively_deflate_directory(disk_file_path, zipfile, zipfile_path)
else
put_into_archive(disk_file_path, zipfile, zipfile_path, e)
end
end
end
def recursively_deflate_directory(disk_file_path, zipfile, zipfile_path)
zipfile.mkdir zipfile_path
subdir = Dir.entries(disk_file_path) - %w[. ..]
write_entries subdir, zipfile_path, zipfile
end
def put_into_archive(disk_file_path, zipfile, zipfile_path, entry)
if File.extname(zipfile_path) == ".docx"
Zip::File.open(disk_file_path) do |zip|
doc = zip.read("word/document.xml")
xml = Nokogiri::XML.parse(doc)
zip.get_output_stream("word/document.xml") {|f| f.write(xml.to_s)}
end
zipfile.add(zipfile_path, disk_file_path)
else
zipfile.add(zipfile_path, disk_file_path)
end
end
end
I want to extract one single content type file from a ZIP package into a directory that doesn't yet exist. My code so far:
require 'zip'
Dir.mkdir 'new_folder'
#I create the folder
def unzip_file (file_path, destination)
Zip::File.open(file_path) { |zip_file|
zip_file.glob('*.xml'){ |f| #I want to extract .XML files only
f_path = File.join(Preprocess, f.name)
FileUtils.mkdir_p(File.dirname(f_path))
puts "Extract file to %s" % f_path
zip_file.extract(f, f_path)
}
}
end
The folder gets successfully created, but no extraction is done at any directory. I suspect that there is something wrong within the working directory. Any help?
I believe you forgot to call your unzip method to begin with...
Nevertheless, this is how I would do this:
require 'zip'
def unzip_file (file_path, destination)
Zip::File.open(file_path) do |zip_file|
zip_file.each do |f| #I want to extract .XML files only
next unless File.extname(f.name) == '.xml'
FileUtils.mkdir_p(destination)
f_path = File.join(destination, File.basename(f.name))
puts "Extract file to %s" % f_path
zip_file.extract(f, f_path)
end
end
end
zip_file = 'random.zip' # change this to zip file's name (full path or even relative path to zip file)
out_dir = 'new_folder' # change this to the name of the output folder
unzip_file(zip_file, out_dir) # this runs the above method, supplying the zip_file and the output directory
EDIT
Adding an additional method called unzip_files that call unzip_file on all zipped files in a directory.
require 'zip'
def unzip_file (file_path, destination)
Zip::File.open(file_path) do |zip_file|
zip_file.each do |f| #I want to extract .XML files only
next unless File.extname(f.name) == '.xml'
FileUtils.mkdir_p(destination)
f_path = File.join(destination, File.basename(f.name))
puts "Extract file to %s" % f_path
zip_file.extract(f, f_path)
end
end
end
def unzip_files(directory, destination)
FileUtils.mkdir_p(destination)
zipped_files = File.join(directory, '*.zip')
Dir.glob(zipped_files).each do |zip_file|
file_name = File.basename(zip_file, '.zip') # this is the zipped file name
out_dir = File.join(destination, file_name)
unzip_file(zip_file, out_dir)
end
end
zipped_files_dir = 'zips' # this is the folder containing all the zip files
output_dir = 'output_dir' # this is the main output directory
unzip_files(zipped_files_dir, output_dir)
Trying to write a script that will search through a directory and sub-directories for specific files. I would like to do know how a certain directory or directories come up to call a method.
this is what I have tried and failed:
def display_directory(path)
list = Dir[path+'/*']
return if list.length == 0
list.each do |f|
if File.directory? f #is it a directory?
if File.directory?('config')
puts "this is the config folder"
end
printf "%-50s %s\n", f, "is a directory:".upcase.rjust(25)
else
printf "%-50s %s\n", f, "is not a directory:".upcase.rjust(25)
end
end
end
start = File.join("**")
puts "Processing directory\n\n".upcase.center(30)
display_directory start
this is what I want to happen.
app
app/controllers
app/helpers
app/mailers
app/models
app/models/bugzilla
app/models/security
app/views
app/views/auth
app/views/calendar
app/views/layouts
app/views/step
app/views/step_mailer
app/views/suggestion
app/views/suggestion_mailer
app/views/task
app/views/user
bin
--------------------------------------
config <----------(call method foo)
config/environments
config/initializers
config/locales
--------------------------------------
db
db/bugzilla
db/migrate
db/security
lib
lib/tasks
log
public
public/images
public/javascripts
public/stylesheets
script
script/performance
script/process
--------------------------
test <---------(call method foobar)
test/fixtures
test/fixtures/mailer
test/functional
test/integration
test/performance
test/unit
--------------------------
vendor
vendor/plugins
Instead
if File.directory?('config')
Try
if f.path.include?('config')
but this will work for every directory that have config on the name. You can put a larger substring to make a better match.
Also, it is very idiomatic in ruby use do..end for multiline blocks and {..} for single line.
I figured out a way. this works pretty well. I've added a method to show all the files in mentioned directory when reached.
def special_dir(path)
puts "------------------------------------"
sp_path = Dir.glob(File.join(path,"*","**"))
sp_path.each do |cf|
puts "\t" + cf
end
end
def walk(path)
list = Dir[path+'/*'].reject{ |r| r['doc'] || r['tmp']}
list.each do |x|
path = File.join(path, x)
if File.directory?(x)
if x =~ /config/ or x =~ /test/
special_dir(x)
else
puts "#{x}"
walk(path)
end
else
#puts x
end
end
end
start = File.join("**")
walk start
This is the code I'm trying.
require 'zlib'
Dir.glob('*.*').each do |file|
Zlib::GzipWriter.open('output.gz') do |gz|
gz.mtime = File.mtime(file)
gz.orig_name = File.basename(file)
gz.write IO.binread(file)
end
end
I've tried different variations of this. There doesn't seem to be a howto for "multiple files" online. I keep ending up with the first file name in the ouput.gz, and I think it may have the content of the last file from the directory (not sure). But that's besides the point. I just want to put each file as separate entities in a compressed file. The more cross platform compatible it is the better.
This answer is taken from http://old.thoughtsincomputation.com/posts/tar-and-a-few-feathers-in-ruby who took it from the RubyGems library.
require 'rubygems'
require 'rubygems/package'
require 'zlib'
require 'fileutils'
module Util
module Tar
# Creates a tar file in memory recursively
# from the given path.
#
# Returns a StringIO whose underlying String
# is the contents of the tar file.
def tar(path)
tarfile = StringIO.new("")
Gem::Package::TarWriter.new(tarfile) do |tar|
Dir[File.join(path, "**/*")].each do |file|
mode = File.stat(file).mode
relative_file = file.sub /^#{Regexp::escape path}\/?/, ''
if File.directory?(file)
tar.mkdir relative_file, mode
else
tar.add_file relative_file, mode do |tf|
File.open(file, "rb") { |f| tf.write f.read }
end
end
end
end
tarfile.rewind
tarfile
end
# gzips the underlying string in the given StringIO,
# returning a new StringIO representing the
# compressed file.
def gzip(tarfile)
gz = StringIO.new("")
z = Zlib::GzipWriter.new(gz)
z.write tarfile.string
z.close # this is necessary!
# z was closed to write the gzip footer, so
# now we need a new StringIO
StringIO.new gz.string
end
# un-gzips the given IO, returning the
# decompressed version as a StringIO
def ungzip(tarfile)
z = Zlib::GzipReader.new(tarfile)
unzipped = StringIO.new(z.read)
z.close
unzipped
end
# untars the given IO into the specified
# directory
def untar(io, destination)
Gem::Package::TarReader.new io do |tar|
tar.each do |tarfile|
destination_file = File.join destination, tarfile.full_name
if tarfile.directory?
FileUtils.mkdir_p destination_file
else
destination_directory = File.dirname(destination_file)
FileUtils.mkdir_p destination_directory unless File.directory?(destination_directory)
File.open destination_file, "wb" do |f|
f.print tarfile.read
end
end
end
end
end
end
end
### Usage Example: ###
#
# include Util::Tar
#
# io = tar("./Desktop") # io is a TAR of files
# gz = gzip(io) # gz is a TGZ
#
# io = ungzip(gz) # io is a TAR
# untar(io, "./untarred") # files are untarred
#
First off, that will keep overwriting output.gz, leaving it containing only the last file compressed.
Second, the gzip format does not hold multiple files. It only holds one. You need to use the .tar.gz or .zip format. .zip is more "cross platform compatible". Take a look at rubyzip.
I am adding tons of file to my archive it looks like this:
print "Starting ..."
Zip::ZipFile.open(myarchive, 'w') do |zipfile|
my_tons_of_files.each do |file|
print "Adding #{file.filename} to archive ... \r"
# ...
end
print "\n Saving archive"
# !!! -> waiting about 10-15 minutes
# but I want to show the percentage of completed job
end
After all files are added to my archive it starts to compress them all (about 10-15 minutes).
How can I indicate what is actually going on with rubyzip gem (actually I want to show percentage like current_file_number/total_files_count).
You can override Zip::ZipFile.commit:
require 'zip'
require 'zip/zipfilesystem'
module Zip
class ZipFile
def commit
return if ! commit_required?
on_success_replace(name) {
|tmpFile|
ZipOutputStream.open(tmpFile) {
|zos|
total_files = #entrySet.length
current_files = 1
#entrySet.each do |e|
puts "Current file: #{current_files}, Total files: #{total_files}"
current_files += 1
e.write_to_zip_output_stream(zos)
end
zos.comment = comment
}
true
}
initialize(name)
end
end
end
print "Starting ..."
Zip::ZipFile.open(myarchive, 'w') do |zipfile|