I need to get the modification time of a group of files on a server. I know how to get this on a local computer, but File.mtime doesn't work via FTP. How would I convert this code to work on a server?
files_sorted_by_time = Dir['*'].select { |f|
((Time.now - File.mtime(f)).to_i / 604800) < 7
}
You want Net::FTP#mtime.
Example from documentation:
Net::FTP.open('ftp.netlab.co.jp') do |ftp|
ftp.login
files = ftp.chdir('pub/lang/ruby/contrib')
files = ftp.list('n*')
ftp.getbinaryfile('nif.rb-0.91.gz', 'nif.gz', 1024)
ftp.mtime('file.pdf')
end
You can use #mtime with #nlst to filter through the list of remote files.
Net::FTP.open('ftp.netlab.co.jp') do |ftp|
ftp.login
ftp.nlst do |file|
if ftp.mtime(file) # ...
end
end
Related
This is my IMAP code:
imap = Net::IMAP.new('outlook.office365.com', 993, true)
imap.login('USERNAME', 'PASSWORD')
imap.examine("FOLDER1")
imap.search(["SINCE", "17-Dec-2020"]).each do |message_id|
envelope = imap.fetch(message_id, "ENVELOPE")[0].attr["ENVELOPE"]
subject = Mail::Encodings.unquote_and_convert_to( envelope.subject, 'utf-8' )
next unless subject.include?("SOME TXT")
body = imap.fetch(message_id, "BODY[]")[0].attr["BODY[]"]
mail = Mail.new(body)
mail.attachments.each do |a|
next unless File.extname(a.filename) == ".pdf"
File.open("/path/to/file/pdfscript/#{a.filename}", 'wb') do |file|
file.write(a.body.decoded)
end
end
end
Right now I have it setup only for "FOLDER1", but I have a lot of folders that I need to get files from. Is it possible to add multiple folders? Something like
imap.examine("FOLDER1", "FOLDER2", "FOLDER3", "FOLDER4")
I solved my problem by making an array
array_of_folders = ["Folder1","Folder2","Folder3","Folder4"]
and made an .each
array_of_folders.each do |folders|
imap.examine(folders) # takes each folder
# Rest of the code
end
The script is working fine for me.
Now, Iam downloading 500 files at a time.
I want to download the files by specifying some range like (10-30) files at one time and (30-60) at another time so on using ruby watir.
These is my code:
require 'watir'
require 'rubygems'
begin
chromedriver_path = File.join(File.absolute_path(File.dirname(__FILE__)),"browser","chromedriver.exe")
Selenium::WebDriver::Chrome.driver_path = chromedriver_path
browser = Watir::Browser.new:chrome
browser.goto "" //url to login
sleep 3
browser.text_field(:name=>"").set "" #e_id
sleep 3
browser.text_field(:name=>"").set "" #pwd
browser.button(:value=>"Login").click #submit
browser.div(:id=>"DivMenu").click
#sleep 3
browser.span(:class =>"down").click
sleep 3
browser.execute_script("document.getElementById('hlGenerateStatusReports').click();")
sleep 3
browser.execute_script("document.getElementById('Report').click();")
sleep 3
optncount = browser.select_list(:id => 'head_ddlClient').options.count
puts optncount
i = 0
while i <= optncount do
puts "Inside the loop i = "+i.to_s
i +=1
browser.select_list(:id => 'ddlClient').option(:index => i).select
sleep 3
browser.button(:value=>"Generate Report").click #submit
sleep 10
end
browser.goto " " //url to logout
rescue Exception => e
puts e.message
puts e.backtrace.inspect
end
I also have download scripts for images etc.
The small part of the one I use here to demonstrate the technique is for 500px.com.
I keep all downloaded files in a textfile and check if allready downloaded against this file. That way you can break off at any moment and resume later.
Of course you could break off if downloaded reached a limit.
I don't publish the whole of the script, just what matters regarding your question.
def download url
filename = "#{url[-32..-1]}.jpg"
if get(url, filename, SAVE_FOLDER)
File.open(PROGRESS_FILE,'a+'){|f|f.puts filename}
end
end
PROGRESS_FILE = './500px.txt'
downloaded = 0
....
response = http.get(path, headers)
json = JSON.parse(response.body)["data"]
processed = File.read(PROGRESS_FILE)
json.each do |item|
url = item['images'].last['url']
signature = url[-32..-1]
filename = "#{signature}.jpg"
# check if the filenames is in the textfile and so was downloaded allready
unless processed[filename]
download url
downloaded += 1
end
end
The file has more than 500 million lines so far and works fast enough (the downloading takes much longer). If I hit a limit I can easily put the lines in a simple database like Sqlite.
I am trying to old delete files from an FTP using Ruby net/sftp, but I keep getting an error saying the file does not exist.
:error=>"Net::SFTP::StatusException (2, \"no such file\")"
I can manually delete files when logging in using the same creds, so I know I have permission.
require 'net/sftp'
ftp = Net::SFTP.start(#ftp_url, #ftp_user, :password => #ftp_pwd)
ftp.dir.entries('somePath').each do |entry|
begin
age_days = (Time.now.to_i - entry.attributes.atime) / 86400
if(age_days > ftp_max_file_age_days)
ftp.remove!(entry.name)
end
rescue Exception => e
# log error here
end
end
I prefer remove! so everything happens synchronously in this case, but I have also tried remove.
I also tried giving it the full path of the file instead of just the entry name (like 'somePath' + entry.name instead of just entry.name). I was thinking perhaps it was because I needed to change the working directory, which apparently net/sftp does not allow.
Thanks in advance!
Check if entry is a directory if yes then use ftp.rmdir. like below -
require 'net/sftp'
ftp = Net::SFTP.start(#ftp_url, #ftp_user, :password => #ftp_pwd)
ftp.dir.entries('somePath').each do |entry|
begin
age_days = (Time.now.to_i - entry.attributes.atime) / 86400
if(age_days > ftp_max_file_age_days)
if File.directory?(entry.name)
ftp.rmdir(entry.name)
else
ftp.remove!(entry.name)
end
end
rescue Exception => e
# log error here
end
end
We were eventually able to delete files using the remove method (instead of remove!.) We made a small change to the way we provide the password.
We confirmed that permissions on the FTP did not change, so I think using non_interactive: true may have been the trick.
require 'net/sftp'
def self.delete_report(endpoint, username, password, report_filename)
SSH_OPTIONS = { non_interactive: true }.freeze
report_filename_base = File.basename(report_filename, '.*')
Net::SFTP.start(endpoint, username, SSH_OPTIONS.merge(password: password)) do |sftp|
sftp.remove(report_filename)
sftp.remove("#{report_filename_base}.fin")
sftp.remove("processed/#{report_filename}")
sftp.remove("processed/#{report_filename_base}.fin")
sftp.remove("failed/#{report_filename}")
sftp.remove("failed/#{report_filename_base}.fin")
sftp.remove("failed/#{report_filename_base}.info")
end
I still don't fully understand why the same method did not work before, but we're able to delete files in subfolders too, as shown in this example.
I am working on a tests scenario that downloads a file from a website and adds it to folder.
For the download part, I am using the code described on the browser-downloads page within the Watir documentation.
The main problem was encountered in my tests when I am waiting for the file to be downloaded:
def verify_csv_file_exists
path = Dir.getwd + "/downloads/"
until File.exist?("#{path}*.csv") == true
sleep 1
end
end
When running the tests, the procedure above never stops, because it cannot see the file in the directory, although the file is downloaded.
Does anyone know a way how I can handle this situation?
Thank you.
You simply check the directory contents before you download the file, then wait until there's a new file added to the directory (by comparing the current content with the previous content). This is how you get the new file name:
This should do the job:
require 'watir-webdriver'
file_name = nil
download_directory = "#{Dir.pwd}/downloads"
download_directory.gsub!("/", "\\") if Selenium::WebDriver::Platform.windows?
downloads_before = Dir.entries download_directory
profile = Selenium::WebDriver::Firefox::Profile.new
profile['browser.download.folderList'] = 2 # custom location
profile['browser.download.dir'] = download_directory
profile['browser.helperApps.neverAsk.saveToDisk'] = "text/csv,application/pdf"
b = Watir::Browser.new :firefox, :profile => profile
b.goto 'https://dl.dropbox.com/u/18859962/hello.csv'
30.times do
difference = Dir.entries(download_directory) - downloads_before
if difference.size == 1
file_name = difference.first
break
end
sleep 1
end
raise "Could not locate a new file in the directory '#{download_directory}' within 30 seconds" if not file_name
puts file_name
You can't use "glob" with File.exists? like File.exists?("*.csv"). It checks whether the file named *.csv exists, not any file with name ends with .csv. You should use exact file name to check if a file exists.
Try it like this instead:
Dir.glob('downloads/*.csv').any?
Also how is sleeping for 1 second supposed to change anything? Is this a multithreaded app?
I am trying to write to a csv file through ftp. Here is what i have so far:
require 'net/ftp'
require 'csv'
users = User.users.limit(5)
csv_string = CSV.generate do |csv|
csv << ["email_addr", "first_name", "last_name"]
users.each do |user|
new_line = [user.email, user.first_name, user.last_name]
csv << new_line
end
end
csv_file = CSV.new(csv_string)
ftp = Net::FTP.new('**SERVER NAME**')
ftp.login(user = "**USERNAME**", passwd = "**PASSWORD**")
ftp.storbinary('STOR ' + 'my_file.csv', csv_file)
ftp.quit()
I get the error "wrong number of arguments (2 for 3)". When i change the line to ftp.storbinary('STOR ' + 'my_file.csv', csv_file, 1024) it says "wrong number of arguments (1 for 0)". I've also tried using storlines instead, but that gave me errors also. Does anybody have any ideas on how to handle this?
In the line
ftp.storbinary('STOR ' + 'my_file.csv', csv_file)
csv_file needs to be an actual File object, not another kind of object.
> (from ruby core)
storbinary(cmd, file, blocksize, rest_offset = nil) { |data| ... }
Puts the connection into binary (image) mode, issues the given server-side
command (such as "STOR myfile"), and sends the contents of the file named file
to the server. If the optional block is given, it also passes it the data, in
chunks of blocksize characters.
require 'net/ftp'
Login to the FTP server
ftp = Net::FTP.new('ftp.sample.com', 'test', 'pass')
OR
ftp = Net::FTP.new('ftp.sample.com')
ftp.login('test', 'pass')
Switch to the desired directory
ftp.chdir('source/files')
Get the file we need and save it to our 'ftp_photos' directory
ftp.getbinaryfile('photos_2009-03-29.zip', 'ftp_photos/photos.zip')
We're done, so we need to close the connection
ftp.close
http://travisonrails.com/2009/03/29/ruby-net-ftp-tutorial
This will help you.