I have a simple SFTP script that I am testing to connect to a server and download a file, or files, with a specific date in the file name.
I am using rufus/scheduler to start the SFTP portion of the script every X minutes to see if a new file is on the server.
It all seems to work until I intentionally force an error, such as provide incorrect login credentials. Then I want to be able to capture the exact error or exception and log it using logger. I am not getting error detail or I am not using rescue correctly:
scheduler = Rufus::Scheduler::PlainScheduler.start_new(:frequency => 3.0)
log = Logger.new('sftp.log')
log.level = Logger::INFO
begin
log.info 'starting sftp'
Net::SFTP.start(HOST, ID, :password => PW ) do |sftp|
sftp.dir.glob("./", "20120820*") do |entry|
puts entry.name
file = entry.name
success = sftp.download!(file, file)
end
end
rescue Exception => e
puts e.message # Human readable error
log.error ("SFTP exception occured: " + e.message)
end
scheduler.join
Does adding :verbose => Logger::DEBUG work ?
Net::SFTP.start(HOST, ID, :password => PW, :verbose => Logger::DEBUG ) do |sftp|
Related
I am trying to old delete files from an FTP using Ruby net/sftp, but I keep getting an error saying the file does not exist.
:error=>"Net::SFTP::StatusException (2, \"no such file\")"
I can manually delete files when logging in using the same creds, so I know I have permission.
require 'net/sftp'
ftp = Net::SFTP.start(#ftp_url, #ftp_user, :password => #ftp_pwd)
ftp.dir.entries('somePath').each do |entry|
begin
age_days = (Time.now.to_i - entry.attributes.atime) / 86400
if(age_days > ftp_max_file_age_days)
ftp.remove!(entry.name)
end
rescue Exception => e
# log error here
end
end
I prefer remove! so everything happens synchronously in this case, but I have also tried remove.
I also tried giving it the full path of the file instead of just the entry name (like 'somePath' + entry.name instead of just entry.name). I was thinking perhaps it was because I needed to change the working directory, which apparently net/sftp does not allow.
Thanks in advance!
Check if entry is a directory if yes then use ftp.rmdir. like below -
require 'net/sftp'
ftp = Net::SFTP.start(#ftp_url, #ftp_user, :password => #ftp_pwd)
ftp.dir.entries('somePath').each do |entry|
begin
age_days = (Time.now.to_i - entry.attributes.atime) / 86400
if(age_days > ftp_max_file_age_days)
if File.directory?(entry.name)
ftp.rmdir(entry.name)
else
ftp.remove!(entry.name)
end
end
rescue Exception => e
# log error here
end
end
We were eventually able to delete files using the remove method (instead of remove!.) We made a small change to the way we provide the password.
We confirmed that permissions on the FTP did not change, so I think using non_interactive: true may have been the trick.
require 'net/sftp'
def self.delete_report(endpoint, username, password, report_filename)
SSH_OPTIONS = { non_interactive: true }.freeze
report_filename_base = File.basename(report_filename, '.*')
Net::SFTP.start(endpoint, username, SSH_OPTIONS.merge(password: password)) do |sftp|
sftp.remove(report_filename)
sftp.remove("#{report_filename_base}.fin")
sftp.remove("processed/#{report_filename}")
sftp.remove("processed/#{report_filename_base}.fin")
sftp.remove("failed/#{report_filename}")
sftp.remove("failed/#{report_filename_base}.fin")
sftp.remove("failed/#{report_filename_base}.info")
end
I still don't fully understand why the same method did not work before, but we're able to delete files in subfolders too, as shown in this example.
I'm trying to write a ruby script that logs into a remote server, switches to another user, executes a script and answers questions to that script. Right now, I can log in but it hangs on the execution of the bash script. I'm not sure if I got the prompt part right but it's not getting to that point yet. It "hangs" on the running of the script or it just isn't printing the output of the script to the screen.
Here's what I got for now:
require 'rubygems'
require 'net/ssh'
require 'net/ssh/telnet'
s = Net::SSH::Telnet.new("Host" => "server1", "Username" => "dev", "Password" => "12345", "Prompt" => /[$%#>] \z/n)
puts s.cmd("sudo -s")
puts s.cmd("su - user1")
puts s.cmd("/opt/develop/develop-bin/start.sh")
puts s.waitfor(/Prompt/)
Here is the snippet that works for me. It is taken from the bigger library, so treat it as an example. Fill up the variables command, username, password, servername and sudoer and give a try
class AuthenticationError < StandardError; end
AUTH_METHODS = ['hostbased', 'password', 'keyboard-interactive']
MYPROMPT = SecureRandom.hex # or whatever you want
ret = ""
stderr_data = ""
Net::SSH.start(servername, username, :password => password, :auth_methods => AUTH_METHODS) do |ssh|
ssh.open_channel do |channel|
channel.on_data do |channel, data|
ret += data
raise AuthenticationError, "SUDO ACCESS DENIED" if data.inspect.include?('Sorry, try again.') || data.inspect.include?('not in sudoers')
channel.send_data(password+"\n") if data.inspect.include? MYPROMPT
sleep 0.1
end
channel.on_extended_data do |ch, type, data|
stderr_data+=data
end
channel.request_pty
channel.exec("sudo -p #{MYPROMPT} su - #{sudoer} -c '#{command}'")
channel.wait
end
end
puts ret.gsub(/^"/,'').gsub(/"$/,'')
I have been trying to get port forwarding to work correctly with Net::SSH. From what I understand I need to fork out the Net::SSH session if I want to be able to use it from the same Ruby program so that the event handling loop can actually process packets being sent through the connection. However, this results in the ugliness you can see in the following:
#!/usr/bin/env ruby -w
require 'net/ssh'
require 'httparty'
require 'socket'
include Process
log = Logger.new(STDOUT)
log.level = Logger::DEBUG
local_port = 2006
child_socket, parent_socket = Socket.pair(:UNIX, :DGRAM, 0)
maxlen = 1000
hostname = "www.example.com"
pid = fork do
parent_socket.close
Net::SSH.start("hostname", "username") do |session|
session.logger = log
session.logger.sev_threshold=Logger::Severity::DEBUG
session.forward.local(local_port, hostname, 80)
child_socket.send("ready", 0)
pidi = fork do
msg = child_socket.recv(maxlen)
puts "Message from parent was: #{msg}"
exit
end
session.loop do
status = waitpid(pidi, Process::WNOHANG)
puts "Status: #{status.inspect}"
status.nil?
end
end
end
child_socket.close
puts "Message from child: #{parent_socket.recv(maxlen)}"
resp = HTTParty.post("http://localhost:#{local_port}/", :headers => { "Host" => hostname } )
# the write cannot be the last statement, otherwise the child pid could end up
# not receiving it
parent_socket.write("done")
puts resp.inspect
Can anybody show me a more elegant/better working solution to this?
I spend a lot of time trying to figure out how to correctly implement port forwarding, then I took inspiration from net/ssh/gateway library. I needed a robust solution that works after various possible connection errors. This is what I'm using now, hope it helps:
require 'net/ssh'
ssh_options = ['host', 'login', :password => 'password']
tunnel_port = 2222
begin
run_tunnel_thread = true
tunnel_mutex = Mutex.new
ssh = Net::SSH.start *ssh_options
tunnel_thread = Thread.new do
begin
while run_tunnel_thread do
tunnel_mutex.synchronize { ssh.process 0.01 }
Thread.pass
end
rescue => exc
puts "tunnel thread error: #{exc.message}"
end
end
tunnel_mutex.synchronize do
ssh.forward.local tunnel_port, 'tunnel_host', 22
end
begin
ssh_tunnel = Net::SSH.start 'localhost', 'tunnel_login', :password => 'tunnel_password', :port => tunnel_port
puts ssh_tunnel.exec! 'date'
rescue => exc
puts "tunnel connection error: #{exc.message}"
ensure
ssh_tunnel.close if ssh_tunnel
end
tunnel_mutex.synchronize do
ssh.forward.cancel_local tunnel_port
end
rescue => exc
puts "tunnel error: #{exc.message}"
ensure
run_tunnel_thread = false
tunnel_thread.join if tunnel_thread
ssh.close if ssh
end
That's just how SSH in general is. If you're offended by how ugly it looks, you should probably wrap up that functionality into a port forwarding class of some sort so that the exposed part is a lot more succinct. An interface like this, perhaps:
forwarder = PortForwarder.new(8080, 'remote.host', 80)
So I have found a slightly better implementation. It only requires a single fork but still uses a socket for the communication. It uses IO#read_nonblock for checking if a message is ready. If there isn't one, the method throws an exception, in which case the block continues to return true and the SSH session keeps serving requests. Once the parent is done with the connection it sends a message, which causes child_socket.read_nonblock(maxlen).nil? to return false, making the loop exit and therefore shutting down the SSH connection.
I feel a little better about this, so between that and #tadman's suggestion to wrap it in a port forwarding class I think it's about as good as it'll get. However, any further suggestions for improving this are most welcome.
#!/usr/bin/env ruby -w
require 'net/ssh'
require 'httparty'
require 'socket'
log = Logger.new(STDOUT)
log.level = Logger::DEBUG
local_port = 2006
child_socket, parent_socket = Socket.pair(:UNIX, :DGRAM, 0)
maxlen = 1000
hostname = "www.example.com"
pid = fork do
parent_socket.close
Net::SSH.start("ssh-tunnel-hostname", "username") do |session|
session.logger = log
session.logger.sev_threshold=Logger::Severity::DEBUG
session.forward.local(local_port, hostname, 80)
child_socket.send("ready", 0)
session.loop { child_socket.read_nonblock(maxlen).nil? rescue true }
end
end
child_socket.close
puts "Message from child: #{parent_socket.recv(maxlen)}"
resp = HTTParty.post("http://localhost:#{local_port}/", :headers => { "Host" => hostname } )
# the write cannot be the last statement, otherwise the child pid could end up
# not receiving it
parent_socket.write("done")
puts resp.inspect
I have a problem that I hope you can help me with
I’m trying to use ruby to ssh onto a machine and run a bash script, this part is fairly easy but the bash script requires me to entry a username and password interactively and this is where I’m stuck
So if I run the script manually I see:-
./run_file.sh
Enter username:
Enter password:
So at the enter Username prompt I have to enter the username etc
I’ve got a simple method I use to make the connection and my idea was to pass in an array made up of
[‘command to run’, ‘username’, ‘password’]
but I don’t know how to extend the Net::SSH call to respond to the prompts
# ssh conectivity method
def command_ssh(host, cmd)
require 'net/ssh'
user = LocalConfig::SSH_DETAILS[:user]
pass = LocalConfig::SSH_DETAILS[:pass]
Net::SSH.start(host, user, :password => pass, :paranoid => false, :auth_methods => ['password'], :timeout => 10 )do |ssh |
output = (ssh.exec!(cmd[0]))
return output
end
end
anyone got any ideas
managed to fix this by using the channel function, here's the method I use now
def connect(host,command)
require 'rubygems'
require 'net/ssh'
user = LocalConfig::SSH_DETAILS[:user]
pass = LocalConfig::SSH_DETAILS[:pass
o = ""
Net::SSH.start(host, user, :password => pass, :paranoid => false, :auth_methods => ['password'], :timeout => 30 )do |ssh |
channel = ssh.open_channel do |ch|
ch.exec(command) do |ch2, success|
ch2.send_data "myUserName\n"
ch2.send_data "mPassWord\n"
ch.on_data do |ch2, data|
o += data
end
end
end
channel.wait
return o.to_s
end
end
I am running an EventMachine process using the Twitter streaming API. I always have an issue if the content of the stream is not frequently.
Here is the minimal version of the script:
require 'rubygems'
require 'eventmachine'
require 'em-http'
require 'json'
usage = "#{$0} <user> <password> <track>"
abort usage unless user = ARGV.shift
abort usage unless password = ARGV.shift
abort usage unless keywords= ARGV.shift
def startIt(user,password,keywords)
EventMachine.run do
http = EventMachine::HttpRequest.new("https://stream.twitter.com/1/statuses/filter.json",{:port=>443}).post(
:head =>{ 'Authorization' => [ user, password ] } ,
:body =>{"track"=>keywords},
:keepalive=>true,
:timeout=>-1)
buffer = ""
http.stream do |chunk|
buffer += chunk
while line = buffer.slice!(/.+\r?\n/)
if line.length>5
tweet=JSON.parse(line)
puts Time.new.to_s+"#{tweet['user']['screen_name']}: #{tweet['text']}"
end
end
end
http.errback {
puts Time.new.to_s+"Error: "
puts http.error
}
end
rescue => error
puts "error rescue "+error.to_s
end
while true
startIt user,password,keywords
end
If I search for a keyword like "iphone", everything works well
If I search for a less frequently used keyword, my stream keeps to be closed very rapidely , around 20 sec after the last message.
Note: that http.error is always empty, so it's very hard to understand while the stream is closed...
On the other end, the nerly similar php version is not closed, so seems probably in issue with eventmachine/http-em but I dont' understand which one...
You should add settings to prevent your connection to timeout.
Try this :
http = EventMachine::HttpRequest.new(
"https://stream.twitter.com/1/statuses/filter.json",
:connection_timeout => 0,
:inactivity_timeout => 0
).post(
:head => {'Authorization' => [ user, password ] } ,
:body => {'track' => keywords}
)
Good luck,
Christian