I need to download a file daily from a client that I have SCP but not SSH access to.
The file name will always be /outgoing/Extract/visit_[date]-[timestamp].dat.gz'
For example yesterdays file was called visits_20130604-090003.dat.gz
I can not rely on the fact that the time stamp will always be the same, but the date should always be yesterdays date:
My set up so far:
My home directory contains to sub-directories named downloads_fullname and downloads_wildcard.
It also contains an simple ruby script named foo.rb.
The contents of foo.rb are this`
#! /usr/bin/ruby
require 'net/ssh'
require 'net/scp'
yesterday = (Time.now - 86400).strftime('%Y%m%d')
Net::SCP.start('hostname', 'username') do |scp|
scp.download!('/outgoing/Extract/visits_' + yesterday + '-090003.dat.gz', 'downloads_fullname')
scp.download!('/outgoing/Extract/visits_' + yesterday + '-*.dat.gz', 'downloads_wildcard')
end
When run the downloads_fullname directory contains the file, but the downloads_wildcard directory does not.
Is there any way to use wildcarding in Net::SCP? Or does anybody have any sly workarounds? I tried \*to no avail.
Thank you Tin Man!!!
To anybody else, here is the code I ended up with following Tin Man's lead:
(Tried to post it as a comment but had formatting issues)
#! /usr/bin/ruby
require 'net/sftp'
yesterday = (Time.now - 86400).strftime('%Y%m%d')
Net::SFTP.start('hostname', 'username') do |sftp|
sftp.dir.foreach("/outgoing/Extract") do |file|
if file.name.include? '_' + yesterday + '-'
sftp.download!('/outgoing/Extract/' + file.name, 'downloads/'+ file.name)
end
end
end
I don't think you can get there using scp because it expects you to know exactly which file you want, but sftp will let you get a directory listing.
You can use Net::SFTP to programmatically pick your file and request it. This is the example code:
require 'net/sftp'
Net::SFTP.start('host', 'username', :password => 'password') do |sftp|
# upload a file or directory to the remote host
sftp.upload!("/path/to/local", "/path/to/remote")
# download a file or directory from the remote host
sftp.download!("/path/to/remote", "/path/to/local")
# grab data off the remote host directly to a buffer
data = sftp.download!("/path/to/remote")
# open and write to a pseudo-IO for a remote file
sftp.file.open("/path/to/remote", "w") do |f|
f.puts "Hello, world!\n"
end
# open and read from a pseudo-IO for a remote file
sftp.file.open("/path/to/remote", "r") do |f|
puts f.gets
end
# create a directory
sftp.mkdir! "/path/to/directory"
# list the entries in a directory
sftp.dir.foreach("/path/to/directory") do |entry|
puts entry.longname
end
end
Based on that you can list the directory entries then use find or select to iterate over the returned list to find the one with the current date. Pass that filename to sftp.download! to download it to a local file.
Related
First of all I would like to say that I'm new to Ruby and if I'm not able to give you a good picture of what I'm trying to solve, that is the reason.
I'm trying to convert URLs into images and I've looked around for answers but I can't seem to find an answer that works for me. The file has around 70,000+ links and I'm also trying to name these at the same time. I'm using ruby 2.3.0 if that is relevant.
Code --
require 'open-uri'
require 'tempfile'
require 'uri'
require 'csv'
def downloadFile(path,url)
begin
open(path, "wb+") do |file|
file << open(url).read
end
return true
rescue
return false
end
end
puts Dir.pwd
CSV.foreach("C/Users/b40ssr/RubymineProjects/Bygma/convert/konvertera.CSV", headers:true) do |row|
downloadFile(row[0], row[1])
end
So the error that I'm getting is
C:/Ruby23/lib/ruby/2.3.0/csv.rb:1265:in `initialize': No such file or directory # rb_sysopen - C/Users/b40ssr/RubymineProjects/Bygma/convert/konvertera.CSV (Errno::ENOENT)
I understand that there is something wrong with the directory but I cant seem to figure out what it is.
First of all, you can use relative path or just use "C:/"
Second, You are trying to open each row of CSV file ??
CSV.foreach("C/Users/b40ssr/RubymineProjects/Bygma/convert/konvertera.CSV"). This will iterate over each rows in CSV file.
Do you want to open each CSV file inside a directory ??
From what I see latest Ruby that supported FTPS was 1.8. I found some gems that can connect to FTPS, but they were no updated in several years. Did anyone had to do this recently? What gem did you use?
You can simply use net/ftp standard library.
ftp = Net::FTP.new('cdimage.debian.org')
ftp.login
ftp.list
Or login to protected ftp:
ftp.login('username', 'password')
And for FTPS you can use net/sftp https://github.com/net-ssh/net-sftp
example of code :
require 'net/sftp'
Net::SFTP.start('host', 'username', :password => 'password') do |sftp|
# upload a file or directory to the remote host
sftp.upload!("/path/to/local", "/path/to/remote")
# download a file or directory from the remote host
sftp.download!("/path/to/remote", "/path/to/local")
# grab data off the remote host directly to a buffer
data = sftp.download!("/path/to/remote")
# open and write to a pseudo-IO for a remote file
sftp.file.open("/path/to/remote", "w") do |f|
f.puts "Hello, world!\n"
end
# open and read from a pseudo-IO for a remote file
sftp.file.open("/path/to/remote", "r") do |f|
puts f.gets
end
# create a directory
sftp.mkdir! "/path/to/directory"
# list the entries in a directory
sftp.dir.foreach("/path/to/directory") do |entry|
puts entry.longname
end
end
I have a ruby script where my "config" is in an extra file. It's called ftp_config.rb. Then I have filetrack.rb that downloads files from a ftp server - what files/directories is specified in ftp_config.rb. And finally I got rufus_download.rb that is calling a function from filetrack.rb every day so I get all new files from the server.
Everything works fine just I want to know how to make it so when I edit ftp_config.rb the changes are picked up by the script without the need to restart rufus_download.rb.
currenly
rufus_download.rb contains require_relative 'filetrack'
filetrack.rb contains require_relative 'ftp_config'
Right now if I add new files to be downloaded to ftp_config.rb I need to restart rufus
require_relative returns false if the file you have requested is already loaded to your ruby script and returns true if you haven't
If you want changes to be loaded directly you need to load files
load 'path/to/ftp_config'
every time your script executes it will load / reload the script
EDIT:
you can load by expanding path of the current ruby script:
load ::File.expand_path('../ftp_config.rb', __FILE__)
Assuming that files are in the same folder
EDITEND
hope that helps
You need a gem that monitors filechanges like "sinatra/reloader" for Sinatra and eg filewatcher or listen for desktop apps. After detecting an update you load the script, not require, that only loads a script once.
Here an example of filewatcher.
STDOUT.sync = true
FileWatcher.new(['c:/test/scans']).watch() do |filename, event|
puts filename
if(event == :changed)
puts "File updated: " + filename
end
if(event == :delete)
puts "File deleted: " + filename
end
if(event == :new)
puts "Added file: " + filename
end
end
So, I've been working on a Ruby script that blocks reddit during my school hours (useful stuff). Here's the code:
require 'fileutils'
puts "-----------------------------------"
puts "Welcome to the hosts file modifier!"
puts "-----------------------------------"
puts "Option A: Use modified hosts"
puts "Option B: Use original hosts"
puts "Option C: Do nothing"
puts "Please enter your choice: "
input = gets.chomp.downcase
t = Time.now
# Time.now is used is conjunction with function 'original', in option 'b'
def modified
# This function copies the modified (redditblocking) hosts file from Documents to /etc
puts "Moving original hosts file out of /etc"
FileUtils.mv('/etc/hosts', '/Users/(usernameobscured)/Documents/OriginalHosts/hosts')
puts "Done. Now copying modified hosts to /etc"
FileUtils.cp('/Users/(usernameobscured)/Documents/ModifiedHosts/hosts', '/etc/hosts')
puts "Done"
end
def original
# This function deletes the modified hosts file from /etc (since we have a copy in Documents)
# and then moves the original hosts file back to /etc
puts "Deleting modified hosts file from /etc"
FileUtils.rm_rf('etc/hosts')
puts "Done. Now copying original hosts to /etc"
FileUtils.mv('/Users/(usernameobscured)/Documents/OriginalHosts/hosts', '/etc/hosts')
puts "Done"
end
def nothing
# This does... nothing. Literally.
puts "Doing nothing"
end
if input == 'a'
modified
end
if input == 'b'
# Here's when using Time.now becomes helpful: if the hour of the day is less than 5PM,
# then the original hosts file can't be moved back (don't wanna be on reddit during school hours!)
if t.hour > 17
original
elsif t.hour < 17
puts "Too early to use original hosts file. Come back at 5PM"
end
end
if input == 'c'
# Nothing...
nothing
end
As you can see, it moves a modified hosts file from my Documents folder to /etc. The problem I'm having though, as per OS X/Unix security measures, is that I have to run the script via sudo or logged in as root. This is a minor nuisance, however, it's one that I believe can be fixed within the code. How can I get superuser privileges, OR write access to /etc temporarily, via my ruby script, so that I can simply run the script without sudo/root?
Per Unix security model, it is not possible to gain root access without some sort of external intervention (setuid set to the executable, running as root user). Otherwise we would have a gaping security hole.
I am not clear what is exactly your issue of using sudo or rvmsudo or against setting the script setuid (it is possible to configure sudo to not require password for narrowly defined set of commands).
I would just suggest making the various versions of host files group writable by a group that you are member of.
According to this site : http://ruby.about.com/od/rubyversionmanager/qt/Rvm-And-Sudo.htm
you can start executing the script using the rvmsudo command. In your terminal window or shell script:
rvmsudo ruby blockreddit.rb
Since Heroku does not allow saving dynamic files to disk, I've run into a dilemma that I am hoping you can help me overcome. I have a text file that I can create in RAM. The problem is that I cannot find a gem or function that would allow me to stream the file to another FTP server. The Net/FTP gem I am using requires that I save the file to disk first. Any suggestions?
ftp = Net::FTP.new(domain)
ftp.passive = true
ftp.login(username, password)
ftp.chdir(path_on_server)
ftp.puttextfile(path_to_web_file)
ftp.close
The ftp.puttextfile function is what is requiring a physical file to exist.
StringIO.new provides an object that acts like an opened file. It's easy to create a method like puttextfile, by using StringIO object instead of file.
require 'net/ftp'
require 'stringio'
class Net::FTP
def puttextcontent(content, remotefile, &block)
f = StringIO.new(content)
begin
storlines("STOR " + remotefile, f, &block)
ensure
f.close
end
end
end
file_content = <<filecontent
<html>
<head><title>Hello!</title></head>
<body>Hello.</body>
</html>
filecontent
ftp = Net::FTP.new(domain)
ftp.passive = true
ftp.login(username, password)
ftp.chdir(path_on_server)
ftp.puttextcontent(file_content, path_to_web_file)
ftp.close
David at Heroku gave a prompt response to a support ticket I entered there.
You can use APP_ROOT/tmp for temporary file output. The existence of files created in this dir is not guaranteed outside the life of a single request, but it should work for your purposes.
Hope this helps,
David