I'm trying to upload a file to a SFTP site using Net::SFTP.
The problem is the Ruby process never returns. It's frozen, stuck, never-ending... I have to Ctrl-C to make it stop.
I can connect to the ftp server in the command line using sftp.
I've tried this code on Linux (Ubuntu) and Snow Leopard and it does the same thing.
Installed the latest version of Net::SSH and Net::SFTP.
From gem list net:
net-sftp (2.0.5)
net-ssh (2.1.4)
Different combinations of hash arguments (see code below)
Tried to call Net::SFTP.start instead of creating a SSH session first.
Server responds with "SFTP protocol version 3" when I login using a command line tool and ask for the version.
I'm running Ruby 1.8.7 (2010-08-16 patchlevel 302) [i686-darwin10.7.0]
Debug output from :verbose (last couple of lines):
D, [2011-07-29T17:30:52.692192 #19399] DEBUG -- tcpsocket[80c06478]: read 36 bytes
D, [2011-07-29T17:30:52.692609 #19399] DEBUG -- tcpsocket[80c06478]: received packet nr 7 type 99 len 12
I, [2011-07-29T17:30:52.692751 #19399] INFO -- net.ssh.connection.session[80bd5da0]: channel_success: 0
D, [2011-07-29T17:30:52.692817 #19399] DEBUG -- net.sftp.session[80bd5c24]: sftp subsystem successfully started
D, [2011-07-29T17:30:52.693625 #19399] DEBUG -- tcpsocket[80c06478]: queueing packet nr 8 type 94 len 28
D, [2011-07-29T17:30:52.693834 #19399] DEBUG -- tcpsocket[80c06478]: sent 52 bytes
Sample code:
require 'rubygems'
require 'net/sftp'
FTP = "someftpsite"
FTP_USER_NAME = "user"
FTP_PASSWORD = "password"
hash = {:password => FTP_PASSWORD, :verbose => :debug, :keys=>[""],:auth_methods => ["password"], :timeout => 5, :port=>22}
begin
Net::SSH.start(FTP,FTP_USER_NAME, hash) do |test|
test.sftp.upload!("localfile","remotefile")
end
rescue Exception => err
puts "error:#{err.inspect}"
end
Edit: 8/22/2011
This seems to be related to the interactiveness of the SFTP server. I solved this by creating a shell script using expect and shelling out from Ruby to run the file. The shell script is created at run time. This seems really hackish, but it's the only way I've been able to make it work using Ruby. If anyone else has a suggestion, I would love to find a better way to do this.
You could use public/private key authentication instead. http://net-ssh.rubyforge.org/ssh/v1/chapter-2.html#s2
Related
What is the proper way to prevent Sinatra from displaying the full backtrace, when it fails to properly run the server (for example, due to the port being already in use)?
This is a sample sinatra app:
# test.rb
require 'sinatra'
require 'bundler/inline'
gemfile do
gem 'sinatra'
gem 'puma'
end
set :bind, "0.0.0.0"
set :port, 3000
get '/' do
"hello"
end
Then, running it with ruby test.rb once, to occupy the port.
Then, running it again in another terminal window, and this full error backtrace is shown:
$ ruby test.rb
== Sinatra (v2.0.4) has taken the stage on 3000 for development with backup from Puma
Puma starting in single mode...
* Version 3.12.0 (ruby 2.5.0-p0), codename: Llamas in Pajamas
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://0.0.0.0:3000
== Someone is already performing on port 3000!
Traceback (most recent call last):
5: from /store/gems/ruby-2.5.0/gems/sinatra-2.0.4/lib/sinatra/main.rb:26:in `block in <module:Sinatra>'
4: from /store/gems/ruby-2.5.0/gems/sinatra-2.0.4/lib/sinatra/base.rb:1464:in `run!'
3: from /store/gems/ruby-2.5.0/gems/sinatra-2.0.4/lib/sinatra/base.rb:1464:in `ensure in run!'
2: from /store/gems/ruby-2.5.0/gems/sinatra-2.0.4/lib/sinatra/base.rb:1439:in `quit!'
1: from /store/gems/ruby-2.5.0/gems/puma-3.12.0/lib/puma/launcher.rb:147:in `stop'
/store/gems/ruby-2.5.0/gems/puma-3.12.0/lib/puma/single.rb:27:in `stop': undefined method `stop' for nil:NilClass (NoMethodError)
Traceback (most recent call last):
3: from /store/gems/ruby-2.5.0/gems/sinatra-2.0.4/lib/sinatra/base.rb:1545:in `block in setup_traps'
2: from /store/gems/ruby-2.5.0/gems/sinatra-2.0.4/lib/sinatra/base.rb:1439:in `quit!'
1: from /store/gems/ruby-2.5.0/gems/puma-3.12.0/lib/puma/launcher.rb:147:in `stop'
/store/gems/ruby-2.5.0/gems/puma-3.12.0/lib/puma/single.rb:27:in `stop': undefined method `stop' for nil:NilClass (NoMethodError)
Since I am using it as an embedded server, I would like the output to simply and with the friendly error that Sinatra is already showing:
== Someone is already performing on port 3000!
and avoid showing the backtrace.
Ruby by default outputs error messages to the STDOUT. But if you're on *nix system you can do this:
ruby test.rb > /dev/null 2>&1
For windows you can probably do
ruby test.rb > NULL
windows powershell
ruby test.rb > $null
but for windows also see Is there a /dev/null on Windows?
But if you want programmatically suppress output when server is running this should work on *nix but not sure on windows
# test.rb
require 'sinatra'
require 'bundler/inline'
gemfile do
gem 'sinatra'
gem 'puma'
end
set :bind, "0.0.0.0"
set :port, 3000
get '/' do
"hello"
end
unless `ps aux | grep sinatra`.match('tcp://0.0.0.0:3000')
STDOUT.reopen('/dev/null', 'w')
STDERR.reopen('/dev/null', 'w')
end
See suppresing output to console with ruby
You can test to see if the port is in use by attempting to listen on the port before allowing Sinatra and Puma to take over. This isn't 100% effective because there's a race condition where you may open and close the port, but before Sinatra/Puma finish initializing some other process comes along and listens on the same port, but it should work for your use-case (which appears to be a cosmetic hack only).
Insert this code anywhere in test.rb:
require 'socket'
include Socket::Constants
begin
# Open and close the port
socket = Socket.new(AF_INET, SOCK_STREAM, 0)
sockaddr = Socket.pack_sockaddr_in(3000, '0.0.0.0')
socket.bind(sockaddr)
socket.listen(1)
socket.close
rescue Errno::EADDRINUSE => error
# Traps the same error that is trapped by Sinatra and exits if raised
puts error.message
exit
end
Start the first one with ruby test.rb:
== Sinatra (v2.0.4) has taken the stage on 3000 for development with backup from Puma
Puma starting in single mode...
* Version 3.12.0 (ruby 2.6.0-p-1), codename: Llamas in Pajamas
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://0.0.0.0:3000
Use Ctrl-C to stop
Start the second one with ruby test.rb:
Address already in use - bind(2) for 0.0.0.0:3000
You can flesh out what you want printed to the console inside the rescue block.
This appears to be caused by an issue with Puma, that is fixed by this PR.
I am using Capybara Selenium to run headless Chrome, which works great, except I cannot figure out how to use remote debugging. When I add --remote-debugging-port=4444 or --remote-debugging-port=9222 or --remote-debugging-port=9521, Selenium no longer connects to the browser to run the test.
How do I get remote debugging to work? Here is my code for reference:
Capybara.register_driver :selenium do |app|
# from https://github.com/SeleniumHQ/selenium/issues/3738
capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(loggingPrefs: {browser: 'ALL'})
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument '--disable-infobars' # hide info bar about chrome automating test
# if we don't use this flag, every selenium test will die with the error:
# "unknown error: Chrome failed to start: exited abnormally"
options.add_argument '--no-sandbox'
# BREAKS THINGS if uncommented
# options.add_argument '--remote-debugging-port=4444'
options.add_argument '--headless'
options.add_argument '--window-size=1600,2400'
options.add_preference('profile.default_content_settings.popups', 0)
options.add_preference('download.default_directory', DownloadHelpers::PATH.to_s)
Capybara::Selenium::Driver.new(
app,
clear_local_storage: true,
clear_session_storage: true,
browser: :chrome,
options: options,
desired_capabilities: capabilities,
)
end
Since chrome 67 and chromedriver 2.39, chromedriver now correctly uses the port you specify with --remote-debugging-port. This removes quite a bit of complexity from my answer above. The steps I now take, which work for my use case of needing to configure download settings using chrome_remote, are as follows:
It makes uses of a nodejs library, crmux - which allows multiple clients to connect to the remote debug port of chrome at the same time.
Get nodejs installed first: Nodejs v9.7.0 works fine
Install crmux by running npm install crmux -g
Before you start chromedriver (Capybara::Selenium::Driver.new), you need to spawn a separate thread that will fire up crmux, which will let both you and chromedriver communicate with chrome itself via the port you specified in Capybara (4444):
crmux --port=4444 --listen=4444
You may want to add a sleep 3 after the spawn command in the main script/thread to give time for crmux to start before you continue with your test startup.
You can then use chrome_remote (for example) to access chrome using port 4444, while capybara is doing its thing.
Updating my ChromeDriver fixed it for me. I didn't have to do anything else. Before it would hang when attempting to start the test.
Specifically I was on ChromeDriver 2.36 and I upgraded to ChromeDriver 2.40. I don't think the Chrome version was a problem, since I was on Chrome 67 both before and after.
Here's how I'm registering the driver:
Capybara.register_driver :headless_chrome do |app|
capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
chromeOptions: { args: %w[headless window-size=1280,960 remote-debugging-port=9222] }
)
Capybara::Selenium::Driver.new(app, browser: :chrome, desired_capabilities: capabilities)
end
After that I ran my test with a debugger (binding.pry) placed where I wanted to inspect. Then when I hit the debugger I navigated to http://localhost:9222/ in a normal instance of Chrome and was able to follow a link to view what was happening in the headless Chrome instance, including the browser console output I needed.
Update: if using versions since Chrome 67/chromedriver 2.39, my alternative answer above provides a simpler solution
The core issue here is Chromedriver also uses the remote debugging port connection to communicate with Chrome. This uses the websocket protocol, which only supports a single-client connected at a time. Normally, when chromedriver starts the chromedriver process, it will choose a random free TCP port number and use this to access the remote debug port. If you specify --remote-debuggging-port=9222 however, Chrome will be opened with the debug port you have requested, but chromedriver will silently continue to try and open a connection using this random port number.
The solution I ended up with was heavily inspired by comment 20 in this chromedriver issue. It required quite a bit of code to get it working, but works solidly. It makes uses of a nodejs library, crmux - which allows multiple clients to connect to the remote debug port of chrome at the same time.
Get nodejs installed first: Nodejs v9.7.0 works fine
Install crmux by running npm install crmux -g
Before you start chromedriver (Capybara::Selenium::Driver.new), you need to spawn a separate thread that will do a few things: look for the remote debug port chromedriver is trying to use to connect to chrome, and then use this to fire up crmux. Once this has happened Capybara etc will work as normal.
My separate thread runs a ruby script that first executes a netstat command repeatedly until it finds the relevant entry for chromedriver (TCP status is SYN_SENT). This separate thread must continue to run in the background while chrome is up and running.
The code for this is:
$chrdrv_wait_timeout = 60
$chrdrv_exe = "chromedriver.exe"
def get_netstat_output
stdout = `netstat -a -b -n`
stat_lines = stdout.split("\n")
stat_lines
end
def try_get_requested_port
socket_state = "SYN_SENT" # i.e. sent with no reply
statout = get_netstat_output
n = statout.length
i = 0
loop do
i += 1
# find lines relating to chromedriver
exe_match = /^ +\[#{$chrdrv_exe}\]$/.match statout[i]
if exe_match != nil
# check preceeding lines which should contain port info
port_match = /TCP.*:([0-9]+)\W+#{socket_state}/.match statout[i-1]
if port_match != nil
return port_match[1].to_i
end
end
break unless i < n
end
return nil
end
def get_tcp_port_requested_by_chromedriver
i = 1
loop do
puts "Waiting for #{$chrdrv_exe}: #{i}"
port = try_get_requested_port
if port != nil
return port
end
break unless i < $chrdrv_wait_timeout
sleep 1
i += 1
end
raise Exception, "Failed to get TCP port requested by #{$chrdrv_exe} (gave up after #{$chrdrv_wait_timeout} seconds)"
end
(I'm working in Windows: for Mac/Linux the netstat syntax/output is probably different so the code will need adjustment; the key thing is you need it to output the executable owner of each connection entry - and parse the bit relating to chromedriver to get the port in question).
Once this has found the random port (I'll use 12225 as an example), the background ruby script can then execute a crmux process, which will reunite chromedriver with chrome itself via the port you specified in Capybara (4444):
crmux --port=4444 --listen=12225
Finally, this separate script saves the discovered listen port to a text file. This allows the main script/thread that is running capybara to know the port number it can use to get access to chrome (via crmux's multiplexed connection) by reading in the port from that file. So you can then use chrome_remote to access chrome using port 12225, for example, while capybara is doing its thing.
I´m doing some tests with default ruby Logger
and I´m getting this error on file rotation
log rotation inter-process lock failed. Text file busy #
rb_file_s_rename - (file.log, file.log.0)
log writing failed. closed stream
log shifting failed. closed stream
My environment.
CentOS 7.2.1511
Ruby: 2.0.0, 2.1.0, 2.3.0, 2.3.1
Note:
This error only occurs on CentOS with ruby versions after 2.0.0. With ruby 2.0.0 works fine!
I made some tests using Ubuntu with all ruby versions after 1.9 and
this error not occurs.
Here my test code
require 'logger'
begin
puts "Starting test"
# rotate when file size equals 20kb keeping 3 old logs
log = Logger.new("file.log", 3, 20000)
while true
puts "Logging..."
log.info "info"
log.error "error"
log.warn "warn"
log.debug "debug"
sleep(1)
end
rescue Interrupt => _
puts "Exiting..."
end
Thanks for help
I have a ruby script that uses the net::ssh gem.
Its a simple script to tunnel through a jump-off server to a destination box. The box is only visible to the jump-off server. Here is the script:
require 'net/ssh/proxy/command'
require "net/ssh"
class TestTunnel
def initialize
proxy = Net::SSH::Proxy::Command.new('ssh admin#10.10.10.10 nc %h %p 2>/dev/null')
p "proxy gotten"
Net::SSH.start("host1", "admin", :password => "admin", :proxy => proxy) do |ssh|
hostname = ssh.exec!("hostname")
p "In #{hostname}"
end
end
end
b=TestTunnel.new
The issue is the script runs in ruby but not in jruby. My framework runs off jruby so I need to use it with jruby.
My solution was to call a bash script from my jruby script to setup the tunnel and run commands there but I would rather have everything done within jruby.
Here are the errors I get:
~/src/main/ruby$ jruby --1.9 scripts/test_tunnel.rb
"proxy gotten"
IO::EAGAINWaitReadable: Resource temporarily unavailable - errno backtraces disabled; run with -Xerrno.backtrace=true to enable
And when I run in ruby:
~/src/main/ruby$ ruby scripts/test_tunnel.rb
"proxy gotten"
"In vz-int-api02\n"
~/src/main/ruby$
When I run with -Xerrno.backtrace=true I can see that jruby is 1.7.9 and I can see that its using ruby 1.9:
~/src/main/ruby$ jruby -Xerrno.backtrace=true --1.9 scripts/test_tunnel.rb
"proxy gotten"
IO::EAGAINWaitReadable: Resource temporarily unavailable -
read_nonblock at org/jruby/RubyIO.java:2856
recv at /usr/local/jruby-1.7.9/lib/ruby/gems/shared/gems/net-ssh-2.9.2/lib/net/ssh/proxy/command.rb:75
fill at /usr/local/jruby-1.7.9/lib/ruby/gems/shared/gems/net-ssh-2.9.2/lib/net/ssh/buffered_io.rb:65
next_packet at /usr/local/jruby-1.7.9/lib/ruby/gems/shared/gems/net-ssh-2.9.2/lib/net/ssh/transport/packet_stream.rb:88
poll_message at /usr/local/jruby-1.7.9/lib/ruby/gems/shared/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:183
loop at org/jruby/RubyKernel.java:1519
poll_message at /usr/local/jruby-1.7.9/lib/ruby/gems/shared/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:178
wait at /usr/local/jruby-1.7.9/lib/ruby/gems/shared/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:215
loop at org/jruby/RubyKernel.java:1519
wait at /usr/local/jruby-1.7.9/lib/ruby/gems/shared/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:213
initialize at /usr/local/jruby-1.7.9/lib/ruby/gems/shared/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:87
start at /usr/local/jruby-1.7.9/lib/ruby/gems/shared/gems/net-ssh-2.9.2/lib/net/ssh.rb:207
initialize at scripts/test_tunnel.rb:8
(root) at scripts/test_tunnel.rb:15
~/src/main/ruby$ jruby -Xerrno.backtrace=true --1.9 --version
jruby 1.7.9 (1.9.3p392) 2013-12-06 87b108a on Java HotSpot(TM) 64-Bit Server VM 1.8.0_72-b15 +indy [linux-amd64]
~/src/main/ruby$
Is there any way to get this to run in jruby like it does in ruby or is my solution to call a bash script from within jruby the only way to do this?
A
This sounds a lot like this bug in net-ssh that was fixed in net-ssh 3.0.2.
The commit message that closed that issue t was "Bugfix: proxy command was using nonblocking io api incorrectly causing rare IO::EAGAIN errors" which sounds like exactly what you are getting.
I've searched in the archive but could not find an answer to my dilemma. I'm coding in Ruby and using watir webdriver framework on my local Mac Yosemite and want to connect to postgres database on a linux box.
I have the required ruby gems installed on my local Mac
* LOCAL GEMS *
dbd-pg (0.3.9)
pg (0.18.4)
dbi (0.4.5, 0.4.4)
I am using the following code.
require 'rubygems'
require 'pg'
require 'dbd/pg'
require 'dbi'
conn = PGconn.connect("10.0.xx.xx","5432",'','',"mydbname","dbuser", "")
res = conn.exec('select * from priorities_map;')
puts res.getvalue(0,0)
conn.close if conn
On running this
I a getting these errors
.initialize': Could not connect to server: Connection refused (PG::ConnectionBad)
Is the server running on host "10.0.xx.xx" and accepting
TCP/IP connections on port 5432?
If I use the code
dbh = DBI.connect("dbi:pg:mydbname:ipaddress", "user", "")
row = dbh.exec('select * from etr_priorities_map;')
puts row.getvalue(0,0)
dbh.disconnect if dbh
I get the error
block in load_driver': Unable to load driver 'pg' (underlying error: wrong constant name pg) (DBI::InterfaceError) from System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
I am new to Ruby. How can I resolve these issues?
The first error, as #Doon said in the comments, comes from the TCP connection and usually means your database is not listening on the network. Most PostgreSQL packages come with a default configuration that only allows local connections, but you can enable connections over the network in the server configuration via the listen_addresses setting. I installed PostgreSQL through Homebrew on my Mac, and my config is at /usr/local/var/postgres/postgresql.conf, but if you installed it some other way the path may be different.
The second error is happening because the "driver" part of the connection string is case-sensitive, and the DBD driver for Postgres is named Pg, not pg. Try this:
dbh = DBI.connect("dbi:Pg:mydbname:ipaddress", "user", "")
Also, unless you have your heart set on using Ruby/DBI, you might want to consider using a more-recently maintained library. Ruby-DBI is very well-written and tested, but it hasn't seen a release since 2010, and Ruby itself has changed pretty significantly in the interim.
If you do want to consider alternatives, I use Sequel for mostly everything, and I highly recommend it, especially for Postgres development, but both DataMapper and ActiveRecord have a large userbase as well.