The code in this question works, but only with a single directory. I can also make it output a file archive as well. But not both a file and a directory, or two directories. I am hoping to make it work with a list of paths, including directories and files that are all placed in the same archive. If I try to add more than one path, the tarfile becomes corrupted. I thought I could continue adding files/data to archive as long as the TarWriter object is open.
QUESTION: In addition to how I can make the above example work with multiple paths (in linked post), can someone please help explain how files and directories are added into an archive? I have looked at the directory structure/format, but I can't seem to understand why this wouldn't work with more than one directory/file.
You can add multiple paths to Dir object
Dir[File.join(path1, '**/*'), File.join(path2, '**/*')]
After which the code would be something like this:
BLOCKSIZE_TO_READ = 1024 * 1000
def create_tarball(path)
tar_filename = Pathname.new(path).realpath.to_path + '.tar'
File.open(tar_filename, 'wb') do |tarfile|
Gem::Package::TarWriter.new(tarfile) do |tar|
Dir[File.join(path1, '**/*'), File.join(path2, '**/*')].each do |file|
mode = File.stat(file).mode
relative_file = file.sub(/^#{ Regexp.escape(path) }\/?/, '')
if File.directory?(file)
tar.mkdir(relative_file, mode)
else
tar.add_file(relative_file, mode) do |tf|
File.open(file, 'rb') do |f|
while buffer = f.read(BLOCKSIZE_TO_READ)
tf.write buffer
end
end
end
end
end
end
end
tar_filename
end
Related
I want to remove the following characters from several files in a folder. What I have so far is this:
str.delete! '!##$%^&*()
which I think will work to remove the characters. What do I need to do to make it run through all the files in the folder?
You clarified your question, stating you want to remove certain characters from the contents of files in a directory. I created a straight forward way to traverse a directory (and optionally, subdirectories) and remove specified characters from the file contents. I used String#delete like you started with. If you want to remove more advanced patterns you might want to change it to String#gsub with regular expressions.
The example below will traverse a tmp directory (and all subdirectories) relative to the current working directory and remove all occurrences of !, $, and # inside the files found. You can of course also pass the absolute path, e.g., C:/some/dir. Notice I do not filter on files, I assume it's all text files in there. You can of course add a file extension check if you wish.
def replace_in_files(dir, chars, subdirs=true)
Dir[dir + '/*'].each do |file|
if File.directory?(file) # Traverse inner directories if subdirs == true
replace_in_files(file, chars, subdirs) if subdirs
else # Replace file contents
replaced = File.read(file).delete(chars)
File.write(file, replaced)
end
end
end
replace_in_files('tmp', '!$#')
I think this might work, although I'm a little shaky on the Dir class in Ruby.
Dir.foreach('/path/to/dir') do |file|
file.delete '!##$%^&*()
end
There's a more general version of your question here: Iterate through every file in one directory
Hopefully a more thorough answer will be forthcoming but maybe this'll get you where you need.
Dir.foreach('filepath') do |f|
next if Dir.exists?(f)
file = File.new("filepath/#{f}",'r+')
text = file.read.delete("'!##$%^&*()")
file.rewind
file.write(text)
file.close
end
The reason you can't do
file.write(file.read.delete("'!##$%^&*()"))
is that file.read leaves the "cursor" at the end of the text. Instead of writing over the file, you would be appending to the file, which isn't what you want.
You could also add a method to the File class that would move the cursor to the beginning of the file.
class File
def newRead
data = self.read
self.rewind
data
end
end
Dir.foreach('filepath') do |f|
next if Dir.exists?(f)
file = File.new("filepath/#{f}",'r+')
file.write(file.newRead.delete("'!##$%^&*()"))
file.close
end
I am taking screenshots using selenium for my cucumber test. I want one of my steps to place a screenshot file in a folder with a folder name generated using input from the step + time stamp.
Here is what I have accomplished so far:
Then /^screen shots are placed in the folder "(.*)"$/ do |folder_name|
time = Time.now.strftime("%Y%m%d%H%M%S")
source ="screen_shots"
destination = "screen_shots\_#{folder_name}_#{time}"
if !Dir.exists? destination
Dir.new destination
end
Dir.glob(File.join(source, '*')).each do |file|
if File.exists? (file)
File.move file, File.join(destination, File.basename(file))
end
end
end
If the directory does not exist, I want to create it. Then I want to place all screenshots into the new directory.
The folder is to be created in the same directory as the screenshots and then all screenshot files are to be moved into the folder. I am still learning ruby, and my attempts to put this together are not working out at all:
Desktop > cucumber_project_folder > screenshots_folder > shot1.png, shot2.png
In short, I want to create a new directory in screenshots and move shot1.png and shot2.png into it. How can I do so?
Based on the answer given this is the solution (for cucumber)
Then /^screen shots are placed in the folder "(.*)" contained in "(.*)"$/ do |folder_name, source_path|
date_time = Time.now.strftime('%m-%d-%Y %H:%M:%S')
source = Pathname.new(source_path)
destination = source + "#{folder_name}_#{date_time}"
destination.mkdir unless destination.exist?
files = source.children.find_all { |f| f.file? and f.fnmatch?('*.png') }
FileUtils.move(files, destination)
end
The source path is indicated in the step so different users do not have to modify the definition.
I'm not sure what's going on with your first line of code
Then /^screen shots are placed in the folder "(.*)"$/ do |folder_name|
as it's not Ruby code, but I've made it work with a notional line from a file.
The Pathname class allows things like destination.exist? instead of File.exist?(destination). It also lets you build composite paths with + and provides the children method.
The FileUtils module provides the move facility.
Note that Ruby allows forward slashes to be used in Windows paths, and it is usually easier to use them instead of having to escape backslashes everywhere.
I've also added a hyphen between the date and the time in the directory name, as otherwise it's pretty much unreadable.
require 'pathname'
require 'fileutils'
source = Pathname.new('C:/my/source')
line = 'screen shots are placed in the folder "screenshots"'
/^screen shots are placed in the folder "(.*)"$/.match(line) do |m|
folder_name = m[1]
date_time = Time.now.strftime('%Y%m%d-%H%M%S')
destination = source + "#{folder_name}_#{date_time}"
destination.mkdir unless destination.exist?
jpgs = source.children.find_all { |f| f.file? and f.fnmatch?('*.jpg') }
FileUtils.move(jpgs, destination)
end
There is a zipfile. It can either have 10 files or one folder. This folder will have the 10 files. Now, if the zipfile has a folder, then i have to move all the files one directory above i.e.
zipfile.zip has folder. folder has 10 files. normally, if i unzip, i get zipfile/folder/10files. Now, I have to get like zipfile/10files. ie. move all the files one directory above.
How to do this? Please help.
If you don't mind using Linux unzip and really aren't worried about subdirectories:
def unzip(file)
to = File.join(File.dirname(file), File.basename(file, ".*"))
Dir.mkdir(to) unless File.exists?(to)
`unzip -j #{file} -d #{to}`
end
# unzip('yourfile.zip')
This method creates a new directory in the same directory as the zip file with the same name as the zipfile (minus extension). It then extracts (using unzip) the zip file into that directory, ignoring all paths (the -j flag tells unzip to junk paths).
EDIT
Per your comment, here is a way to do it without system calls:
def unzip(file)
Zip::ZipFile.open(file) do |zipfile|
to = File.join(File.dirname(file), File.basename(file, ".*"))
FileUtils.mkdir(to) unless File.exists? to
zipfile.each do |f|
if f.file? # Don't extract directories
fpath = File.join(to, File.basename(f.name))
zipfile.extract(f, fpath) unless File.exists?(fpath)
end
end
end
end
I'm trying to crawl FTP and pull down all the files recursively.
Up until now I was trying to pull down a directory with
ftp.list.each do |entry|
if entry.split(/\s+/)[0][0, 1] == "d"
out[:dirs] << entry.split.last unless black_dirs.include? entry.split.last
else
out[:files] << entry.split.last unless black_files.include? entry.split.last
end
But turns out, if you split the list up until last space, filenames and directories with spaces are fetched wrong.
Need a little help on the logic here.
You can avoid recursion if you list all files at once
files = ftp.nlst('**/*.*')
Directories are not included in the list but the full ftp path is still available in the name.
EDIT
I'm assuming that each file name contains a dot and directory names don't. Thanks for mentioning #Niklas B.
There are a huge variety of FTP servers around.
We have clients who use some obscure proprietary, Windows-based servers and the file listing returned by them look completely different from Linux versions.
So what I ended up doing is for each file/directory entry I try changing directory into it and if this doesn't work - consider it a file :)
The following method is "bullet proof":
# Checks if the give file_name is actually a file.
def is_ftp_file?(ftp, file_name)
ftp.chdir(file_name)
ftp.chdir('..')
false
rescue
true
end
file_names = ftp.nlst.select {|fname| is_ftp_file?(ftp, fname)}
Works like a charm, but please note: if the FTP directory has tons of files in it - this method takes a while to traverse all of them.
You can also use a regular expression. I put one together. Please verify if it works for you as well as I don't know it your dir listing look different. You have to use Ruby 1.9 btw.
reg = /^(?<type>.{1})(?<mode>\S+)\s+(?<number>\d+)\s+(?<owner>\S+)\s+(?<group>\S+)\s+(?<size>\d+)\s+(?<mod_time>.{12})\s+(?<path>.+)$/
match = entry.match(reg)
You are able to access the elements by name then
match[:type] contains a 'd' if it's a directory, a space if it's a file.
All the other elements are there as well. Most importantly match[:path].
Assuming that the FTP server returns Unix-like file listings, the following code works. At least for me.
regex = /^d[r|w|x|-]+\s+[0-9]\s+\S+\s+\S+\s+\d+\s+\w+\s+\d+\s+[\d|:]+\s(.+)/
ftp.ls.each do |line|
if dir = line.match(regex)
puts dir[1]
end
end
dir[1] contains the name of the directory (given that the inspected line actually represents a directory).
As #Alex pointed out, using patterns in filenames for this is hardly reliable. Directories CAN have dots in their names (.ssh for example), and listings can be very different on different servers.
His method works, but as he himself points out, takes too long.
I prefer using the .size method from Net::FTP.
It returns the size of a file, or throws an error if the file is a directory.
def item_is_file? (item)
ftp = Net::FTP.new(host, username, password)
begin
if ftp.size(item).is_a? Numeric
true
end
rescue Net::FTPPermError
return false
end
end
I'll add my solution to the mix...
Using ftp.nlst('**/*.*') did not work for me... server doesn't seem to support that ** syntax.
The chdir trick with a rescue seems expensive and hackish.
Assuming that all files have at least one char, a single period, and then an extension, I did a simple recursion.
def list_all_files(ftp, folder)
entries = ftp.nlst(folder)
file_regex = /.+\.{1}.*/
files = entries.select{|e| e.match(file_regex)}
subfolders = entries.reject{|e| e.match(file_regex)}
subfolders.each do |subfolder|
files += list_all_files(ftp, subfolder)
end
files
end
nlst seems to return the full path to whatever it finds non-recursively... so each time you get a listing, separate the files from the folders, and then process any folder you find recrsively. Collect all the file results.
To call, you can pass a starting folder
files = list_all_files(ftp, "my_starting_folder/my_sub_folder")
files = list_all_files(ftp, ".")
files = list_all_files(ftp, "")
files = list_all_files(ftp, nil)
I'm trying to use RubyZip to package up some files. At the moment I have a method which happily zips on particular directory and sub-directories.
def zip_directory(zipfile)
Dir["#{#directory_to_zip}/**/**"].reject{|f| reject_file(f)}.each do |file_path|
file_name = file_path.sub(#directory_to_zip+'/','');
zipfile.add(file_name, file_path)
end
end
However, I want to include a file from a completely different folder. I have a the following method to solve this:
def zip_additional(zipfile)
additional_files.reject{|f| reject_file(f)}.each do |file_path|
file_name = file_path.split('\\').last
zipfile.add(file_name, file_path)
end
end
While the file is added, it also copies the directory structure instead of placing the file at the root of the folder. This is really annoying and makes it more difficult to work with.
How can I get around this?
Thanks
Ben
there is setting to include (or exclude) the full path for zip libraries, check that setting
Turns out it was because the filename had the pull path in. My split didn't work as the path used a / instead of a . With the path removed from the filename it just worked.