I'm trying to write in some functionality into the Net::SSH::Telnet Library for a project of mine. Specifically I need to be able to read the exit-status of any command that I write.
Here's the Original source from the Net::SSH::Telnet Library that I'm modifying
#buf = ""
#eof = false
#channel = nil
#ssh.open_channel do |channel|
channel.request_pty { |ch,success|
if success == false
raise "Failed to open ssh pty"
end
}
channel.send_channel_request("shell") { |ch, success|
if success
#channel = ch
waitfor(#options['Prompt'], &blk)
return
else
raise "Failed to open ssh shell"
end
}
channel.on_data { |ch,data| #buf << data }
channel.on_extended_data { |ch,type,data| #buf << data if type == 1 }
channel.on_close { #eof = true }
end
#ssh.loop
And now here are the changes I have made:
#stderr = ""
#exit_code = nil
#buf = ""
#eof = false
#channel = nil
#ssh.open_channel do |channel|
channel.request_pty { |ch,success|
if success == false
raise "Failed to open ssh pty"
end
}
channel.send_channel_request("shell") { |ch, success|
if success
#channel = ch
waitfor(#options['Prompt'], &blk)
return
else
raise "Failed to open ssh shell"
end
}
channel.on_data { |ch,data|
#stdout << data
#buf << data
}
channel.on_extended_data { |ch,type,data|
#stderr << data
#buf << data if type == 1
}
channel.on_request("exit-status") do |ch,data|
#exit_code = data.read_long
end
channel.on_close {
#eof = true
}
end
#ssh.loop
The problem is that when I run this, the #exit_code variable never changes from nil.
The reason I am doing this is because I need three major pieces of functionality:
1. Stateful (interactive) communication (so that if I cd into a directory the next command will occur in that directory)
2. The ability to read the exit status of any command I execute
3. The ability to respond to prompts
I selected Net::SSH::Telnet to adjust because it already offered 1 & 3. I figured I could fix it to support number 2.
If anyone can see how to fix this I would greatly appreciate it. The source for the rest of Net::SSH::Telnet is here:
https://github.com/jasonkarns/net-ssh-telnet/
and I have a forked version of the repo here to which I would gladly accept pull requests.
https://github.com/0x783czar/net-ssh-remotescript
Also if anyone has a suggestion of any other ruby libraries that would offer those three functionalities I would greatly appreciate any suggestions.
You may want to take a look at the source for the Ruby "ssh" gem and "remote_task" gem.
http://rubygems.org/gems/ssh
The relevant source code is here:
https://github.com/seattlerb/ssh/blob/master/lib/ssh.rb
It has code for handling the "cd" command, i/o channels, and the exit status.
Related
I'm new to ruby and I'm trying to make a client to connect to a TCPServer, and it seems that in order to do so I have to call the method close_write every time I finish sending data one way, to let the client/server know that the other end is finished sending data. Whenever I do that then Im not able to write info to the server or client again because the socket is not opened for writing anymore.
This is my code:
client.rb
require "socket"
socket = TCPSocket.open("localhost", 6666)
loop do
input = gets.chomp
socket.puts input # Send data to server
socket.close_write
while(line = socket.gets)
puts line
end # Print sever response
break if input=="EXIT"
end
socket.close
server.rb
require "socket"
server = TCPServer.new 6666
data = Hash.new { |hash, key| hash[key] = {} }
WAITING_SET_VALUE = "1"
WAITING_NEW_COMMAND = "0"
loop do
Thread.start(server.accept) do |session|
thread_status ||= WAITING_NEW_COMMAND
....
puts "Entering If..."
if(thread_status == WAITING_NEW_COMMAND) #Check thread status
puts "thread_status == WAITING_NEW_COMMAND"
puts "CHECKING COMMAND.."
case line.strip
when /^set \w* \w* \d{1,7} \d{1,7}$/
puts "COMMAND SET"
thread_status = WAITING_SET_VALUE
lineArr = line.strip.split(" ")
varName = lineArr[1]
flag = lineArr[2]
ttl = lineArr[3]
size = lineArr[4]
puts "END SET EXECUTION"
session.write "Executed"
session.close_write
...
Is there a way to open the socket for writing again, or a better way to do this back and forth connection between server and client without losing context? Thanks!
When designing a client-server protocol, you have to decide:
How a client knows when a response has more lines/data.
How a client knows when a response is complete.
How a client knows when a response is invalid/valid.
How a client knows when there was some type of server error.
A simple approach is for the server to return a response with the number of lines (as in the code below). However, instead, you could use END or something so that the client knows when there is no more data to read. I would strongly suggest looking into tutorials about Protocols.
Save the below into a file called client_server.rb. First, run the server with ruby ./client_server.rb s and then in a separate terminal run the client with ruby ./client_server.rb c. Next, type in list over and over to see the different responses. I added list so that you don't have to type in set w w 1 1 over and over for testing purposes. Check it out and let me know if you have any questions.
# frozen_string_literal: true
require 'socket'
# Use:
# First, run the server: ruby ./client_server.rb s
# Then, run the client: ruby ./client_server.rb c
# Use "netcat localhost 6666" on the command line to test
# without implementing a client.
# Returns false to close this client socket.
# Returns true to continue reading from this client socket.
def handle_client(client_id, client_socket, command)
# TODO: Define some type of client-server Protocol here.
case command
when /^set \w* \w* \d{1,7} \d{1,7}$/
puts "Running command for client #{client_id}: #{command}"
# This is just for testing purposes.
case rand(3)
when 0
client_socket.puts 'lines 0'
when 1
client_socket.puts 'lines 3'
client_socket.puts 'This is line 1.'
client_socket.puts 'This is line 2.'
client_socket.puts 'This is line 3.'
when 2
client_socket.puts 'The set command returned an error.'
end
when 'list'
puts "Responding to client #{client_id} with list of messages."
# This is just for testing purposes.
case rand(3)
when 0
client_socket.puts 'messages 0'
when 1
client_socket.puts 'messages 2'
client_socket.puts 'This is message 1.'
client_socket.puts 'This is message 2.'
when 2
client_socket.puts 'Unable to retrieve messages due to error.'
end
when 'bye'
puts "Killing client #{client_id}."
return false # Disconnect and kill the thread.
else
client_socket.puts "[ERROR] Invalid command: #{command}"
end
client_socket.flush # Flush all output just in case.
true # Continue connection.
end
case ARGV[0].to_s.downcase
when 's' # server
TCPServer.open(6666) do |server_socket|
global_client_id = 1
loop do
Thread.start(global_client_id, server_socket.accept) do |client_id, client_socket|
puts "Accepting new client #{client_id}."
loop do
command = client_socket.gets
if command.nil?
puts "Client #{client_id} disconnected manually."
break
end
command = command.strip
keep_alive = handle_client(client_id, client_socket, command)
break unless keep_alive
end
client_socket.close
end
global_client_id += 1
end
end
when 'c' # client
TCPSocket.open('localhost', 6666) do |socket|
puts '[Commands]'
puts 'set <word> <word> <num> <num> Run set command.'
puts 'list Get list of messages.'
puts 'exit, bye Exit the client.'
puts
loop do
print '> '
input = $stdin.gets.strip
# TODO: Define some type of client-server Protocol here.
case input
when /EXIT|BYE/i
socket.puts 'bye'
socket.flush
break
when /\Aset .+\z/
socket.puts input
socket.flush
response = socket.gets
if response.nil?
puts 'Server is not running anymore! Disconnecting.'
break
end
response = response.strip
match_data = response.match(/\Alines (?<lines>\d+)\z/)
if match_data
line_count = match_data[:lines].to_i
puts "Received #{line_count} lines from server."
1.upto(line_count) do |i|
line = socket.gets
puts ">> Resp[#{i}] #{line}"
end
else
# Can check for "response == 'ERROR'" or something.
puts "Server error or invalid response from server: #{response}"
end
when 'list'
socket.puts input
socket.flush
response = socket.gets
if response.nil?
puts 'Server is not running anymore! Disconnecting.'
break
end
response = response.strip
match_data = response.match(/\Amessages (?<messages>\d+)\z/)
if match_data
message_count = match_data[:messages].to_i
puts "Received #{message_count} messages from server."
1.upto(message_count) do |i|
line = socket.gets
puts ">> Resp[#{i}] #{line}"
end
else
# Can check for "response == 'ERROR'" or something.
puts "Server error or invalid response from server: #{response}"
end
else
puts "Invalid command: #{input}"
end
end
end
else
puts "Pass in 'c' for client or 's' for server."
end
I am writing a short ruby script that takes a file as an argument and then parses that file.
I have put together a few conditions in the initialize method to ensure that a file path exists and it is readable and if it nots it prints an error message to the user.
However when I run the file with out a file attached along side the message "please add log file path". I also receive the following error messages.
please add log file path
Traceback (most recent call last):
3: from webserver_log_parser.rb:41:in `<main>'
2: from webserver_log_parser.rb:41:in `new'
1: from webserver_log_parser.rb:6:in `initialize'
webserver_log_parser.rb:6:in `exist?': no implicit conversion of nil into String (TypeError)
I would be really grateful if someone could explain why this happens and a way to fix this issue.
def initialize(log_file_path = nil)
puts 'please add log file path' unless log_file_path
puts 'Could not find the file path' unless File.exist?(log_file_path)
puts '${log_file_path} is unreadable' unless File.readable?(log_file_path)
extract_log_file(log_file_path)
end
def extract_log_file(log_file_path)
webpages = Hash.new { |url, ip_address| url[ip_address] = [] }
File.readlines(log_file_path).each do |line|
url, ip_address = line.split
webpages[url] << ip_address
end
sort_results(webpages)
end
def sort_results(results)
total_views = {}
unique_views = {}
results.each do |url, ip_address|
total_views[url] = ip_address.length
unique_views[url] = ip_address.uniq.length
end
display_results(total_views, unique_views)
end
def display_results(views, unique_views)
puts 'The most viewed pages are as follows:'
total_views_sorted = views.sort_by { |_k, count| -count }
total_views_sorted.each { |key, count| puts "#{key} #{count}" }
puts 'The pages with the most unique views are as follows:'
unique_sort = unique_views.sort_by { |_k, count| -count }
unique_sort.each { |key, count| puts "#{key} #{count}" }
end
end
if $PROGRAM_NAME == __FILE__
LogParser.new(ARGV[0])
end```
If you want to directly terminate the script with a message, you can use abort.
def initialize(log_file_path = nil)
abort("please add log file path") unless log_file_path
abort("Could not find the file path") unless File.exist?(log_file_path)
abort("${log_file_path} is unreadable") unless File.readable?(log_file_path)
extract_log_file(log_file_path)
end
When your guard conditions are triggered, you need to stop further processing (no need to check for readability of a file at file_path if you already established that file_path is nil). It could look like this, for example:
def initialize(log_file_path = nil)
unless log_file_path
puts 'please add log file path'
return
end
unless File.exist?(log_file_path)
puts 'Could not find the file path'
return
end
unless File.readable?(log_file_path)
puts '${log_file_path} is unreadable'
return
end
extract_log_file(log_file_path)
end
I am using the redis-semaphore gem, version 0.3.1.
For some reason, I occasionally can't release a stale Redis lock. From my analysis it seems to happen if my Docker process crashed after the lock was created.
I have described my debugging process below and would like to know if anyone can suggest how to further debug.
Assume that we want to create a redis lock with this name:
name = "test"
We insert this variable in two different terminal windows. In the first, we run:
def lock_for_15_secs(name)
job = Redis::Semaphore.new(name.to_sym, redis: NonBlockingRedis.new(), custom_blpop: true, :stale_client_timeout => 15)
if job.lock(-1) == "0"
puts "Locked and starting"
sleep(15)
puts "Now it's stale, try to release in another process"
sleep(15)
puts "Now trying to unlock"
unlock = job.unlock
puts unlock == false ? "Wuhuu, already unlocked" : "Hm, should have been unlocked by another process, but wasn't"
end
end
lock_for_15_secs(name)
In the second we run:
def release_and_lock(name)
job = Redis::Semaphore.new(name.to_sym, redis: NonBlockingRedis.new(), custom_blpop: true, :stale_client_timeout => 15)
release = job.release_stale_locks!
count = job.available_count
puts "Release reponse is #{release.inspect} and available count is #{count}"
if job.lock(-1) == "0"
puts "Wuhuu, we can lock it"
job.unlock
else
puts "Hmm, we can't lock it"
end
end
release_and_lock(name)
This usually plays out as expected. For 15 seconds, the second terminal can't relase the lock, but when run after 15 seconds, it releases. Below is the output from release_and_lock(name).
Before 15 seconds have passed:
irb(main):1:0> release_and_lock(name)
Release reponse is {"0"=>"1580292557.321834"} and available count is 0
Hmm, we can't lock it
=> nil
After 15 seconds have passed:
irb(main):2:0> release_and_lock(name)
Release reponse is {"0"=>"1580292557.321834"} and available count is 1
Wuhuu, we can lock it
=> 1
irb(main):3:0> release_and_lock(name)
Release reponse is {} and available count is 1
Wuhuu, we can lock it
But whenever I see that a stale lock isn't released, and I run release_and_lock(name) to diagnose, this is returned:
irb(main):4:0> release_and_lock(name)
Release reponse is {} and available count is 0
Hmm, we can't lock it
And at this point my only option is to flush redis:
require 'non_blocking_redis'
non_blocking_redis = NonBlockingRedis.new()
non_blocking_redis.flushall
P.s. My NonBlockingRedis inherits from Redis:
class Redis
class Semaphore
def initialize(name, opts = {})
#custom_opts = opts
#name = name
#resource_count = opts.delete(:resources) || 1
#stale_client_timeout = opts.delete(:stale_client_timeout)
#redis = opts.delete(:redis) || Redis.new(opts)
#use_local_time = opts.delete(:use_local_time)
#custom_blpop = opts.delete(:custom_blpop) # false=queue, true=cancel
#tokens = []
end
def lock(timeout = 0)
exists_or_create!
release_stale_locks! if check_staleness?
token_pair = #redis.blpop(available_key, timeout, #custom_blpop)
return false if token_pair.nil?
current_token = token_pair[1]
#tokens.push(current_token)
#redis.hset(grabbed_key, current_token, current_time.to_f)
if block_given?
begin
yield current_token
ensure
signal(current_token)
end
end
current_token
end
alias_method :wait, :lock
end
end
class NonBlockingRedis < Redis
def initialize(options = {})
if options.empty?
options = {
url: Rails.application.secrets.redis_url,
db: Rails.application.secrets.redis_sidekiq_db,
driver: :hiredis,
network_timeout: 5
}
end
super(options)
end
def blpop(key, timeout, custom_blpop)
if custom_blpop
if timeout == -1
result = lpop(key)
return result if result.nil?
return [key, result]
else
super(key, timeout)
end
else
super
end
end
def lock(timeout = 0)
exists_or_create!
release_stale_locks! if check_staleness?
token_pair = #redis.blpop(available_key, timeout, #custom_blpop)
return false if token_pair.nil?
current_token = token_pair[1]
#tokens.push(current_token)
#redis.hset(grabbed_key, current_token, current_time.to_f)
if block_given?
begin
yield current_token
ensure
signal(current_token)
end
end
current_token
end
alias_method :wait, :lock
end
require 'non_blocking_redis'
😜 An awesome bug 👏
The bug
I think it happens if you kill the process when it does lpop on the SEMAPHORE:test:AVAILABLE
Most probably here https://github.com/dv/redis-semaphore/blob/v0.3.1/lib/redis/semaphore.rb#L67
To replicate it
NonBlockingRedis.new.flushall
release_and_lock('test');
NonBlockingRedis.new.lpop('SEMAPHORE:test:AVAILABLE')
Now initially you have:
SEMAPHORE:test:AVAILABLE 0
SEMAPHORE:test:VERSION 1
SEMAPHORE:test:EXISTS 1
After the above code you get:
SEMAPHORE:test:VERSION 1
SEMAPHORE:test:EXISTS 1
The code checks the SEMAPHORE:test:EXISTS and then expects to have SEMAPHORE:test:AVAILABLE / SEMAPHORE:test:GRABBED
Solution
From my brief check I don't think it is possible to make the gem work without a modification. I tried adding an expiration: but somehow it managed to disable the expiration for SEMAPHORE:test:EXISTS
NonBlockingRedis.new.ttl('SEMAPHORE:test:EXISTS') # => -1 and it should have been e.g. 20 seconds and going down
So.. maybe a fix will be
class Redis
class Semaphore
def exists_or_create!
token = #redis.getset(exists_key, EXISTS_TOKEN)
if token.nil? || all_tokens.empty?
create!
else
# Previous versions of redis-semaphore did not set `version_key`.
# Make sure it's set now, so we can use it in future versions.
if token == API_VERSION && #redis.get(version_key).nil?
#redis.set(version_key, API_VERSION)
end
true
end
end
end
end
the all_tokens is https://github.com/dv/redis-semaphore/blob/v0.3.1/lib/redis/semaphore.rb#L120
I'll open a PR to the gem shortly -> https://github.com/dv/redis-semaphore/pull/66 maybe 🤷♂️
Note 1
Not sure how you use the NonBlockingRedis but it is not in use in Redis::Semaphore. You do lock(-1) which does in the code lpop. Also the code never calls your lock.
Random
Here is a helper to dump the keys
class Test
def self.all
r = NonBlockingRedis.new
puts r.keys('*').map { |k|
[
k,
((r.hgetall(k) rescue r.get(k)) rescue r.lrange(k, 0, -1).join(' | '))
].join("\t\t")
}
end
end
> Test.all
SEMAPHORE:test:AVAILABLE 0
SEMAPHORE:test:VERSION 1
SEMAPHORE:test:EXISTS 1
For completeness here is how it looks when you have grabbed the lock
SEMAPHORE:test:VERSION 1
SEMAPHORE:test:EXISTS 1
SEMAPHORE:test:GRABBED {"0"=>"1583672948.7168388"}
I was trying to make a script that downloads all images or videos from a thread in my favourite imageboard: 2ch.hk
I was successful until I wanted to download these files asynchronously (for example, to improve performance)
Here is the code http://ideone.com/k2l4Hm
file = http.get(source).body
require 'net/http'
multithreading = false
Net::HTTP.start("2ch.hk", :use_ssl => true) do |http|
thread = http.get("/b/res/133467978.html").body
sources = []
thread.scan(/<a class="desktop" target="_blank" href=".+">.+<\/a>/).each do |a|
source = "/b#{/<a class="desktop" target="_blank" href="\.\.(.+)">.+<\/a>/.match(a).to_a[1]}"
sources << source
end
i = 0
start = Time.now
if multithreading
threads = []
sources.each do |source|
threads << Thread.new(i) do |j|
file = http.get(source).body #breaks everything
# type = /.+\.(.+)/.match(source)[1]
# open("#{j}.#{type}","wb") { |new_file|
# new_file.write(file)
# }
end
i += 1
end
threads.each do |thr|
thr.join
end
# until downloade=sources.size
#
# end
else
sources.each do |source|
file = http.get(source).body
type = /.+\.(.+)/.match(source)[1]
open("#{i}.#{type}","wb") { |new_file|
new_file.write(file)
}
i += 1
print "#{(((i).to_f / sources.size) * 100).round(2)}% "
end
puts
end
puts "Done. #{i} files were downloaded. It took #{Time.now - start} seconds"
end
I suppose that this line crashes everything.
file = http.get(source).body
Or maybe that's the problem.
threads.each do |thr|
thr.join
end
Error messages are always different, from Bad File Descriptor and IO errors to "You may have encountered a bug in the Ruby interpreter or extension libraries."
If you want to try and run my code, please substitute a link to thread in 4th line with a new thread (from 2ch.hk/b), because the one in my code may be deleted by the time you run my code
Version of ruby: 2.3.1, OS Xubuntu 16.10
You'll probably have much better performance using a ruby http lib that supports parallel requests:
https://github.com/typhoeus/typhoeus
e.g.
hydra = Typhoeus::Hydra.new
10.times.map{ hydra.queue(Typhoeus::Request.new("www.example.com", followlocation: true)) }
hydra.run
The problem with my code is that I can't make multiple requests on a Net::HTTP instance at the same time.
The solution is to open an HTTP connection for each thread.
I use em-ssh gem (which is a net-ssh adapter for EventMachine) to execute some shell commands on multi remote hosts. Commands are executed successfully on the first host, but failed on other hosts, it complains “EventMachine::Ssh::ConnectionTerminated”
and “EventMachine::Ssh::NegotiationTimeout”.
It seems that these hosts are competing for something maybe the port？ Can someone explains this?
def em_ssh_connection(ssh_host,ssh_user,ssh_password,&cb)
EM.run do
EM::Ssh.start("#{ssh_host}","#{ssh_user}",:password => "#{ssh_password}") do|connection|
connection.errback do |err|
#logger.info "#{err} (#{err.class})"
EM.stop
end
connection.callback do |ssh|
#logger.info "succcessfully connected with #{ssh_host},#{ssh_user},#{ssh_password}"
yield ssh
ssh.close
EM.stop
end
end
end
end
def search_servers(ssh_host,ssh_user,ssh_password)
em_ssh_connection(ssh_host,ssh_user,ssh_password) {|ssh|
command = 'sed -n \'/.*server s1 [0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/=\' '+"#{#haproxy_conf}"
myarray = em_ssh_exec(ssh,command)
if myarray[2] == 0
if myarray[0] != ""
#logger.info "all tcp servers(lines): #{myarray[0]}"
lines = myarray[0].split("\n")
if lines.length >0
start_line = lines[0].to_i
stop_line = lines[lines.length-1].to_i
#logger.info "start_line:#{start_line},stop_line:#{stop_line}"
delete_tcp_upstream(ssh,start_line,stop_line)
else
end
else
#logger.info "the stream field in proxy file is empty"
end
else
#logger.info "search all servers command failed:#{myarray[1]}"
end
}
end
def clear_haproxy(&cb)
search_servers(#ssh_host1,#ssh_user1,#ssh_password1)
search_servers(#ssh_host2,#ssh_user2,#ssh_password2)
cb.call
end