I am struggling to implement the INSTREAM command of clamd daemon in Ruby.
Here is the document of clamd
#file = File.open("input.txt")
socket = TCPSocket.new(HOST, PORT)
#writing the command
socket.write("zINSTREAM\0")
#streaming the chunk
socket.write(1024) #size of chunk
socket.write(#file.read(1024)) #chunk of data
#end the streaming
socket.write(0)
puts "Reading from the scoket"
puts socket.recv(1024)
socket.close
But I am always receiving the error response "INSTREAM size limit exceeded. ERROR"
What I doing wrong here?
After the long struggle I found the solution for this.
The size of the chunk must be expressed in 4 byte unsigned integer in network byte order
So
socket.write(1024)
should be
socket.write([1024].pack("N"))
Related
I'm having difficulty to Encrypt large files (bigger than available memory) using GPGME in Ruby.
#!/usr/bin/ruby
require 'gpgme'
def gpgfile(localfile)
crypto = GPGME::Crypto.new
filebasename = File.basename(localfile)
filecripted = crypto.encrypt File.read(localfile), :recipients => "info#address.com", :always_trust => true
File.open("#{localfile}.gpg", 'w') { |file| file.write(filecripted) }
end
gpgpfile("/home/largefile.data")
In this case I got an error of memory allocation:
"read: failed to allocate memory (NoMemoryError)"
Someone can explain me how to read the source file chunk by chunk (of 100Mb for example) and write them passing by the crypting?
The most obvious problem is that you're reading the entire file into memory with File.read(localfile). The Crypto#encrypt method will take an IO object as its input, so instead of File.read(localfile) (which returns the contents of the file as a string) you can pass it a File object. Likewise, you can give an IO object as the :output option, letting you write the output directly to a file instead of in memory:
def gpgfile(localfile)
infile = File.open(localfile, 'r')
outfile = File.open("#{localfile}.gpg", 'w')
crypto = GPGME::Crypto.new
crypto.encrypt(infile, recipients: "info#address.com",
output: outfile,
always_trust: true)
ensure
infile.close
outfile.close
end
I've never used ruby-gpgme, so I'm not 100% sure this will solve your problem since it depends a bit on what ruby-gpgme does behind the scenes, but from the docs and the source I've peeked at it seems like a sanely-built gem so I'm guessing this will do the trick.
I saw an article which suggests the following code for a writer:
output = open("my_pipe", "w+") # the w+ means we don't block
output.puts "hello world"
output.flush # do this when we're done writing data
and a reader:
input = open("my_pipe", "r+") # the r+ means we don't block
puts input.gets # will block if there's nothing in the pipe
But could it happen that open, puts, gets will block the program? Is there some kind of timeout in place? Can one change it? Also, how come w+ means non-blocking call? Which open system call flags is it converted to?
Okay, let me share with you my picture of the world. As rogerdpack said, there are two options: 1) using select in blocking mode, 2) using non-blocking mode (O_NONBLOCK flag, read_nonblock, write_nonblock, select methods). I haven't tried, so these are just speculations.
As to why open, puts and gets may block the thread. open call blocks until there are at least one reader and at least one writer. And that must be the reason why we need to specify r+, w+ for open call. Judging from strace output they both are converted to O_RDWR flag. Then there must be some buffer, where not yet received data are stored. And that must be the reason why write methods may block. Read methods may block because they expect more data to be available, than it really is.
UPD
If a process attempts to read from an empty pipe, then read(2) will block until data is available. If a process attempts to write to a full pipe (see below), then write(2) blocks until sufficient data has been read from the pipe to allow the write to complete.
-- http://linux.die.net/man/7/pipe
The FIFO must be opened on both ends (reading and writing) before data can be passed. Normally, opening the FIFO blocks until the other end is opened also.
Under Linux, opening a FIFO for read and write will succeed both in blocking and nonblocking mode. POSIX leaves this behavior undefined. This can be used to open a FIFO for writing while there are no readers available.
-- http://linux.die.net/man/7/fifo
And here's the implementation I came up with:
#!/home/yuri/.rbenv/shims/ruby
require 'timeout'
data = ((0..15).to_a.map { |v|
(v < 10 ? '0'.ord + v : 'a'.ord + v - 10).chr
} * 4096 * 2).reduce('', :+)
timeout = 10
start = Time.now
open('1.fifo', File::WRONLY | File::NONBLOCK) { |out|
out.flock(File::LOCK_EX)
nwritten = 0
data_len = data.length
begin
delta = out.write_nonblock data
data = data[delta..-1]
nwritten += delta
rescue IO::WaitWritable, Errno::EINTR
timeout_left = timeout - (Time.now - start)
if timeout_left < 0
puts Time.now - start
raise Timeout::Error
end
IO.select nil, [out], nil, timeout_left
retry
end while nwritten < data_len
}
puts Time.now - start
But for my problem at hand I decided to ignore this timeout thing. It probably will suffice to handle just situations when there is no reader on the other end of the pipe (Errno::ENXIO):
open('1.fifo', File::WRONLY | File::NONBLOCK) { |out|
out.flock(File::LOCK_EX)
nwritten = 0
data_len = data.length
begin
delta = out.write_nonblock data
data = data[delta..-1]
nwritten += delta
rescue IO::WaitWritable, Errno::EINTR
IO.select nil, [out]
retry
end while nwritten < data_len
}
P.S. Your feedback is appreciated.
This page should answer all your questions... http://www.ruby-doc.org/core-2.0.0/IO.html
In general, puts can always block the current thread, since they may have to wait for IO to complete for it to return. gets can also block the current thread because it will read and read forever until it hits the first newline, then it will return everything it read. HTH.
Using Getting essid via ioctl in ruby as a template I wanted to get the BSSID rather than the ESSID. However, not being a C developer, there are a few things that I don't understand.
What I have so far which does not work :( ...
NOTE I'm a bit confused because part of me thinks, according to some comments in wireless.h, that the BSSID can only be set via ioctl. However, the ioctl to get exists. That along with my almost complete lack of understanding of the more intermediate C type isms (structs, unions, and stuff ;) ), I simply don't know.
def _get_bssid(interface)
# Copied from wireless.h
# supposing a 16 byte address and 32 byte buffer but I'm totally
# guessing here.
iwreq = [interface, '' * 48,0].pack('a*pI')
sock = Socket.new(Socket::AF_INET, Socket::SOCK_DGRAM, 0)
# from wireless.h
# SIOCGIWAP 0x8B15 /* get access point MAC addresses */
sock.ioctl('0x8B15', iwreq) # always get an error: Can't convert string to Integer
puts iwreq.inspect
end
So, in the meantime, I'm using a wpa_cli method for grabbing the BSSID but I'd prefer to use IOCTL:
def _wpa_status(interface)
wpa_data = nil
unless interface.nil?
# need to write a method to get the src_sock_path
# programmatically. Fortunately, for me
# this is going to be the correct sock path 99% of the time.
# Ideas to get programmatically would be:
# parse wpa_supplicant.conf
# check process table | grep wpa_suppl | parse arguments
src_sock_path = '/var/run/wpa_supplicant/' + interface
else
return nil
end
client_sock_path = '/var/run/hwinfo_wpa'
# open Domain socket
socket = Socket.new(Socket::AF_UNIX, Socket::SOCK_DGRAM, 0)
begin
# bind client domain socket
socket.bind(Socket.pack_sockaddr_un(client_sock_path))
# connect to server with our client socket
socket.connect(Socket.pack_sockaddr_un(src_sock_path))
# send STATUS command
socket.send('STATUS', 0)
# receive 1024 bytes (totally arbitrary value)
# split lines by \n
# store in variable wpa_data.
wpa_data = socket.recv(1024)
rescue => e
$stderr.puts 'WARN: unable to gather wpa data: ' + e.inspect
end
# close or next time we attempt to read it will fail.
socket.close
begin
# remove the domain socket file for the client
File.unlink(client_sock_path)
rescue => e
$stderr.puts 'WARN: ' + e.inspect
end
unless wpa_data.nil?
#wifis = Hash[wpa_data.split(/\n/).map\
{|line|
# first, split into pairs delimited by '='
key,value = line.split('=')
# if key is camel-humped then put space in front
# of capped letter
if key =~ /[a-z][A-Z]/
key.gsub!(/([a-z])([A-Z])/,'\\1_\\2')
end
# if key is "id" then rename it.
key.eql?('id') && key = 'wpa_id'
# fix key so that it can be used as a table name
# by replacing spaces with underscores
key.gsub!(' ','_')
# lower case it.
key.downcase!
[key,value]
}]
end
end
EDIT:
So far nobody has been able to answer this question. I think I'm liking the wpa method better anyway because I'm getting more data from it. That said, one call-out I'd like to make is if anyone uses the wpa code, be aware that it will require escalated privileges to read the wlan socket.
EDIT^2 (full code snippet):
Thanks to #dasup I've been able to re-factor my class to correctly pull the bssid and essids using system ioctls. (YMMV given the implementation, age, and any other possible destabilization thing to your Linux distribution - the following code snippet works with the 3.2 and 3.7 kernels though.)
require 'socket'
class Wpa
attr_accessor :essid, :bssid, :if
def initialize(interface)
#if = interface
puts 'essid: ' + _get_essid.inspect
puts 'bssid: ' + _get_bssid.inspect
end
def _get_essid
# Copied from wireless.h
iwreq = [#if, " " * 32, 32, 0 ].pack('a16pII')
sock = Socket.new(Socket::AF_INET, Socket::SOCK_DGRAM, 0)
sock.ioctl(0x8B1B, iwreq)
#essid = iwreq.unpack('#16p').pop.strip
end
def _get_bssid
# Copied from wireless.h
# supposing a 16 byte address and 32 byte buffer but I'm totally
# guessing here.
iwreq = [#if, "\0" * 32].pack('a16a32')
sock = Socket.new(Socket::AF_INET, Socket::SOCK_DGRAM, 0)
# from wireless.h
# SIOCGIWAP 0x8B15 /* get access point MAC addresses */
sock.ioctl(0x8B15, iwreq) # always get an error: Can't convert string to Integer
#bssid = iwreq.unpack('#18H2H2H2H2H2H2').join(':')
end
end
h = Wpa.new('wlan0')
I'm not very much familiar with Ruby, but I spotted two mistakes:
The hex number for SIOCGIWAP should be given without quotes/ticks.
The initialization of the data buffer ends up with some trailing bytes after the interface name (debugged using gdb). The initialization given below works.
Be aware that your code will break if any of the data structures or constants change (IFNAMSIZ, sa_family, struct sockaddr etc.) However, I don't think that such changes are likely anytime soon.
require 'socket'
def _get_bssid(interface)
# Copied from wireless.h
# supposing a 16 byte address and 32 byte buffer but I'm totally
# guessing here.
iwreq = [interface, "\0" * 32].pack('a16a32')
sock = Socket.new(Socket::AF_INET, Socket::SOCK_DGRAM, 0)
# from wireless.h
# SIOCGIWAP 0x8B15 /* get access point MAC addresses */
sock.ioctl(0x8B15, iwreq) # always get an error: Can't convert string to Integer
puts iwreq.inspect
end
You'll get back an array/buffer with:
The interface name you sent, padded with 0x00 bytes to a total length of 16 bytes.
Followed by a struct sockaddr, i.e. a two-byte identifier 0x01 0x00 (coming from ARPHRD_ETHER?) followed by the BSSID padded with 0x00 bytes to a total length of 14 bytes.
Good luck!
I'm attempting to hack together a Ruby-based (1.9.1) syslog server, and am running into a pretty basic issue right from the get-go.
Here's my (very basic) code:
#!/usr/bin/env ruby
require 'socket'
require 'io/wait'
require 'syslog'
class Server
def initialize
#listener = UDPSocket.new
#listener.bind("192.168.253.5", "514")
getdata
end
def getdata
while true
#text, #sender = #listener.recvfrom(9000)
p #listener
p #text
p #sender
end
end
end
x = Server.new
It all works fine, except that this does not display either the facility or the severity of the message:
#<UDPSocket:fd 5>
"<189>49: *Mar 1 00:24:37.862: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/8, changed state to down"
["AF_INET", 56970, "192.168.253.10", "192.168.253.10"]
Tcpdump shows this info just fine ("local7" facility, "notice" severity):
15:18:01.987542 IP 192.168.253.10.56970 > 192.168.253.5.514: SYSLOG local7.notice, length: 115
How can I inspect the UDP packet that was sent to me so I can glean both facility and severity of the syslog message?
Whenever you are implementing a well-defined network protocol, always look at the RFC:
https://www.rfc-editor.org/rfc/rfc5424
The Priority value is calculated by first multiplying the Facility
number by 8 and then adding the numerical value of the Severity.
so "local7" is 23 according to the RFC. 23 * 8 = 184
the severity of "notice" is 5: 184 + 5 = 189.
And there's 189 right at the beginning of your message - that's the "priority" number referenced by the RFC.
So you'll need to encode the mapping from the RFC between numeric values and the textual description into your program and compute it yourself.
To get the severify and facility:
Severity = Priority % 8
Facility = Priority / 8
I'm trying to build a desktop client that manages some downloads with Ruby. I would like to know how to go about trying to identify how much of the data is downloaded and the size of the entire data that is to be downloaded.
Im trying to do this with Ruby so any help would be useful.
Thanks in advance.
Like Wayne said in his comment, it depends on the protocol that is used to transfer the files. With HTTP for example, the HTTP response will include a Content-Length header which will tell you the length of the file that you are downloading. After you know that you will have to keep track of the number of bytes that you've read from the HTTP connection.
Something like this seems to work (for HTTP), but I wouldn't be surprised if it could be done more elegantly:
require 'net/http'
url = URI.parse('http://www.google.com/index.html')
req = Net::HTTP::Get.new(url.path)
res = Net::HTTP.start(url.host, url.port) do |http|
http.request(req) do |res|
remaining = res.content_length
puts "total length: #{remaining}"
res.read_body do |segment|
puts "read #{segment.length} bytes"
remaining = remaining - segment.length
puts "#{remaining} bytes remaining"
end
end
end
www.google.com/index.html is a bad example since the content gets returned in one segment, but try it on a larger object and you should see multiple "read..." lines.
If you're using Net::HTTP then the length of whatever you're requesting should be in the response header. Net::HTTP mixin NET::HTTPHeader, in it you'll find content_length(). Although it only works if the size is determined before the transfer happens.
Net::HTTPResponse has a method that reads the body in chunks, so you can use that to determine the progress. Start at 0 and add the length of each chunk, compare it to the total size and you're done.
http.request_get('/index.html') {|res|
res.read_body do |segment|
print segment
end
} #Example taken from Ruby-Documentation
If you're using FTP then it should be easier through NET::FTP. Connect to the server, get the size of a given file with size(filename), and then download the file with get, getbinaryfile or gettextfile.
This is the signature of the get method: get(remotefile, localfile = File.basename(remotefile), blocksize = DEFAULT_BLOCKSIZE) {|data| ...}
ftp.get('file.something', 'file.something.local', 1024){ |data|
puts "Downloaded 1024 more bytes"
}