I'm trying to download a file using net/sftp and pass its contents as the stdin for a command-line app. I can do it by first writing the file to disk but I'd rather avoid that step.
Is there any way to control the input to a program invoked with system() in ruby?
Don't use system at all for this sort of thing, system is best for running an external command that you don't need to talk to.
Use Open3.open3 or Open3.open2 to open up some pipes to your external process then write to the stdin pipe just like writing to any other IO channel; if there is any output to deal with, then you can read it straight from the stdout pipe just like reading from any other input IO channel.
Something like this perhaps (using open as mu suggested)?
contents = "Hello, World!"
open('|echo', 'w') { puts contents }
This can also be accomplished with IO.expect
require 'pty'
require 'expect'
str = "RUBY_VERSION"
PTY.spawn("irb") do |reader, writer|
reader.expect(/0> /)
writer.puts(str)
reader.expect(/=> /)
answer = reader.gets
puts "Ruby version from irb: #{answer}"
end
This waits for the spawned process to display "0> " (the end of an irb prompt) and when it sees that prints a defined string. It then looks for the irb to return by waiting for it to display "=> " and grabs the data returned.
Related
I'm trying to pass the array contained in a global variable I've created into my clipboard on my mac.
It is very long so I don't want to highlight, copy & paste on my console.
I want to use embedded unix code, specificially the pbcopy function for the mac laptop console that allows me to pass text into my computers clipboard, ready to paste.
Were I to do this with a file-save, I'd do something like this (in ruby):
stringdata = <<blah blah blah process, lets say it failed and the progress data is stored in this variable so we can skip forward to where the script screwed up in a process when we start up and handle the error instance(s)>>
File.open("temp.txt"){|f| f.write(stringdata)}
`cat temp.txt | pbcopy`
But could I possibly do this without creating a temporary file?
I'm sure this is possible. All things in text are possible. Thanks in advance for the solution
You can just echo it instead if there are no newline characters in the string; otherwise, use the IO class.
Using echo:
system "echo #{stringdata} | pbcopy"
OR
`echo #{stringdata} | pbcopy`
Ruby will then just rip the text from memory, inject it into the shell command which opens a pipe between the echo and pbcopy processes.
Using the IO class:
If you want to do it the Ruby way, we simply create a pipe with pbcopy using the IO class. This creates a shared files between the processes which we write to, and pbcopy will read from.
IO.popen("pbcopy", "w") { |pipe| pipe.puts "Hello world!" }
Here's a simple one-line method you can paste into your IRB console:
def pbcopy(arg); IO.popen('pbcopy', 'w') { |io| io.puts arg }; end
Once it's defined you can simply do
pbcopy stringdata
or copy the result of the last command with
pbcopy _
Of course, you can also put the method definition in an initializer or something, such as .irbrc or .pryrc if you use pry. Here's a prettier and slightly more intelligent version:
def pbcopy(arg)
out = arg.is_a?(String) ? arg : arg.inspect
IO.popen('pbcopy', 'w') { |io| io.puts out }
puts out
true
end
You can use my clipboard gem for a Ruby-API to the system clipboard (which is also platform independet, on macOS it will use the same pbcopy utility under the hood), so that you can use it from IRB:
require 'clipboard'
Clipboard.copy(stringdata);p
Usually, the copy method returns the string which was copied. This the reason for the ;p bit: It is a trick to return nil so that the console would not display the actual string data.
I'm building a daemon that will help me manage my server(s). Webmin works fine, as does just opening a shell to the server, but I'd prefer to be able to control server operations from a UI I design, and also expose some functionality to end users.
The daemon will pick up actions from a queue and execute them. However, since I'll be accepting input from users, I want to make sure they're not permitted to inject something dangerous into a privileged shell command.
Here's a fragment that exemplifies my problem:
def perform
system "usermod -p #{#options['shadow']} #{#options['username']}"
end
A gist that explains more: https://gist.github.com/773292
I'm not positive if typical escaping and sanitizing of inputs is enough for this case, and being a designer, I don't have a ton of security-related experience. I know that this is something that should probably be obvious to me, but its not!
How can I ensure that the web application that will create and serialize the actions can't pass dangerous text into the privileged process that receives the actions?
Thanks for the help
arb
It doesn't look like you need a shell for what you're doing. See the documentation for system here: http://ruby-doc.org/core/classes/Kernel.html#M001441
You should use the second form of system. Your example above would become:
system 'usermod', '-p', #options['shadow'], #options['username']
A nicer (IMO) way to write this is:
system *%W(usermod -p #{#options['shadow']} #{#options['username']})
The arguments this way are passed directly into the execve call, so you don't have to worry about sneaky shell tricks.
If you need not just the exit status but also the result you probably want to use Open3.popen3:
require 'open3'
stdin, stdout, stderr = Open3.popen3('usermod', '-p', #options['shadow'], #options['username'])
stdout.gets
sterr.gets
More information here: Getting output of system() calls in Ruby
I'd suggest looking into the 'shellwords' module. This script:
require 'shellwords'
parts = ['echo', "'hello world'; !%& some stuff", 'and another argument']
command = Shellwords.shelljoin( parts )
puts command
output = `#{ command }`
puts output
outputs the escaped text and the expected output:
echo \'hello\ world\'\;\ \!\%\&\ some\ stuff and\ another\ argument
'hello world'; !%& some stuff and another argument
This is an old question, but since it's pretty much the only real answer you'll find when googling I thought I'd add a caveat. The multi argument version of system seems reasonably safe on Linux, but it is NOT on Windows.
Try system "dir", "&", "echo", "hi!"
on a Windows system. Both dir and echo will be run. Echo could of course just as well be something far less innocuous.
I know this is an old thread, but there is another option that was lightly touched on by Simon Hürlimann.
There is not a lot of information about this topic and I think this might help others in need.
For this example we'll use Open3 which gives you the ability to run commands synchronously or asynchronously, and provides stdout, stderr, exit codes, and PID.
Open3 grants you access to stdout, stderr, exit codes and a thread to wait for the child process when running another program. You can specify various attributes, redirections, current directory, etc., of the program in the same way as for Process.spawn. (Source: Open3 Docs)
I chose to format the output as a CommandStatus object. This contains our stdout, stderr, pid (Of the worker thread) and exitstatus.
class Command
require 'open3'
class CommandStatus
#stdout = nil
#stderr = nil
#pid = nil
#exitstatus = nil
def initialize(stdout, stderr, process)
#stdout = stdout
#stderr = stderr
#pid = process.pid
#exitstatus = process.exitstatus
end
def stdout
#stdout
end
def stderr
#stderr
end
def exit_status
#exitstatus
end
def pid
#pid
end
end
def self.execute(command)
command_stdout = nil
command_stderr = nil
process = Open3.popen3(ENV, command + ';') do |stdin, stdout, stderr, thread|
stdin.close
stdout_buffer = stdout.read
stderr_buffer = stderr.read
command_stdout = stdout_buffer if stdout_buffer.length > 0
command_stderr = stderr_buffer if stderr_buffer.length > 0
thread.value # Wait for Process::Status object to be returned
end
return CommandStatus.new(command_stdout, command_stderr, process)
end
end
cmd = Command::execute("echo {1..10}")
puts "STDOUT: #{cmd.stdout}"
puts "STDERR: #{cmd.stderr}"
puts "EXIT: #{cmd.exit_status}"
While reading the STDOUT/ERR buffers, I use command_stdout = stdout_buffer if stdout_buffer.length > 0 to control whether the command_stdout variable is assigned or not. You should pass nil instead of "" when no data is present. It's more clear when handing data later on.
You probably noticed me using command + ';'. The reason for this is based on the documentation from Kernel.exec (Which is what popen3 uses):
If the string from the first form (exec("command")) follows these
simple rules:
no meta characters
no shell reserved word and no special built-in
Ruby invokes the command directly without shell
You can force shell invocation by adding ";" to the string (because
";" is a meta character)
This simply prevents a Ruby from throwing a 'spawn': No such file or directory error if you pass a malformed command. Instead it will pass it straight to the kernel where the error will be resolved gracefully and appear as STDERR instead of an uncaught exception.
Modern, secure and simple solution (popen will escape arguments for you):
IO.popen(['usermod', '-p', #options['shadow'], #options['username']]).read
(#read will close the IO before returning)
I want to be able to read a currently open file. The test.rb is sending its output to test.log which I want to be able to read and ultimately send via email.
I am running this using cron:
*/5 * * * /tmp/test.rb > /tmp/log/test.log 2>&1
I have something like this in test.rb:
#!/usr/bin/ruby
def read_file(file_name)
file = File.open(file_name, "r")
data = file.read
file.close
return data
end
puts "Start"
puts read_file("/tmp/log/test.log")
puts "End"
When I run this code, it only gives me this output:
Start
End
I would expect the output to be something like this:
Start
Start (from the reading of the test.log since it should have the word start already)
End
Ok, you're trying to do several things at once, and I suspect you didn't systematically test before moving from one step to the next.
First we're going to clean up your code:
def read_file(file_name)
file = File.open(file_name, "r")
data = file.read
file.close
return data
end
puts "Start"
puts read_file("/tmp/log/test.log")
puts "End"
can be replaced with:
puts "Start"
puts File.read("./test.log")
puts "End"
It's plain and simple; There's no need for a method or anything complicated... yet.
Note that for ease of testing I'm working with a file in the current directory. To put some content in it I'll simply do:
echo "foo" > ./test.log
Running the test code gives me...
Greg:Desktop greg$ ruby test.rb
Start
foo
End
so I know the code is reading and printing correctly.
Now we can test what would go into the crontab, before we deal with its madness:
Greg:Desktop greg$ ruby test.rb > ./test.log
Greg:Desktop greg$
Hmm. No output. Something is broken with that. We knew there was content in the file previously, so what happened?
Greg:Desktop greg$ cat ./test.log
Start
End
Cat'ing the file shows it has the "Start" and "End" output of the code, but the part that should have been read and output is now missing.
What happening is that the shell truncated "test.log" just before it passed control to Ruby, which then opened and executed the code, which opened the now empty file to print it. In other words, you're asking the shell to truncate (empty) it just before you read it.
The fix is to read from a different file than you're going to write to, if you're trying to do something with the contents of it. If you're not trying to do something with its contents then there's no point in reading it with Ruby just to write it to a different file: We have cp and/or mv to do those things for us witout Ruby being involved. So, this makes more sense if we're going to do something with the contents:
ruby test.rb > ./test.log.out
I'll reset the file contents using echo "foo" > ./test.log, and cat'ing it showed 'foo', so I'm ready to try the redirection test again:
Greg:Desktop greg$ ruby test.rb > ./test.log.out
Greg:Desktop greg$ cat test.log.out
Start
foo
End
That time it worked. Trying it again has the same result, so I won't show the results here.
If you're going to email the file you could add that code at this point. Replacing the puts in the puts File.read('./test.log') line with an assignment to a variable will store the file's content:
contents = File.read('./test.log')
Then you can use contents as the body of a email. (And, rather than use Ruby for all of this I'd probably do it using mail or mailx or pipe it directly to sendmail, using the command-line and shell, but that's your call.)
At this point things are in a good position to add the command to crontab, using the same command as used on the command-line. Because it's running in cron, and errors can happen that we'd want to know about, we'd add the 2>&1 redirect to capture STDERR also, just as you did before. Just remember that you can NOT write to the same file you're going to read from or you'll have an empty file to read.
That's enough to get your app working.
class FileLineRead
File.open("file_line_read.txt") do |file|
file.each do |line|
phone_number = line.gsub(/\n/,'')
user = User.find_by_phone_number(line)
user.destroy unless user.nil?
end
end
end
open file
read line
DB Select
DB Update
In the cron job you have already opened and cleared test.log (via redirection) before you have read it in the Ruby script.
Why not do both the read and write in Ruby?
It may be a permissions issue or the file may not exist.
f = File.open("test","r")
puts f.read()
f.close()
The above will read the file test. If the file exists in the current directory
The problem is, as I can see, already solved by Slomojo. I'll only add:
to read and print a text file in Ruby, just:
puts File.read("/tmp/log/test.log")
I have to execute a shell command from Ruby script but I have to retrieve the output so that I can use it in the script later on.
Here is my code:
output = system "heroku create" # => true
But the system command returns a boolean and not the output.
Simply said, system "heroku create" has to output to my screen (which it does) but also return the output so I can process it.
You could use
output = `heroku create`
See: http://ruby-doc.org/core/classes/Kernel.html
The Open3 library gives you full access to the standard IO streams (STDIN, STDOUT and STDERR).
It's part of Ruby, so there's no need to install a gem:
require 'open3'
stdin, stdout, stderr = Open3.popen3("heroku create")
puts stdout.read
stdin.close; stdout.close; stderr.close
or you can use the block form which closes the streams implicitely:
require 'open3'
Open3.popen3("heroku create") do |stdin, stdout, stderr|
puts stdout.read
end
See the Open3 documentation for the full details.
Edit: Added extra stream closing details. Thanks Christopher and Gregory.
You can use the following:
output = capture(:stdout) do
system("pwd") # your system command goes here
end
puts output
shortened version:
output = capture(:stdout) { system("pwd") }
Similarly we can also capture standard errors too with :stderr
capture method is provided by active_support/core_ext/kernel/reporting.rb
Looking at that library's code comments, capture is going to be deprecated, so not sure what is the current supported method name is.
I'm looking to expose an interactive command line program via JSON or another RPC style service using Ruby. I've found a couple tricks to do this, but im missing something when redirecting the output and input.
One method at least on linux is to redirect the stdin and stdout to a file then read and write to that file asynchronously with file reads and writes. Another method ive been trying after googling around was to use open4. Here is the code I wrote so far, but its getting stuck after reading a few lines from standard output.
require "open4"
include Open4
status = popen4("./srcds_run -console -game tf +map ctf_2fort -maxplayers 6") do |pid, stdin, stdout, stderr|
puts "PID #{pid}"
lines=""
while (line=stdout.gets)
lines+=line
puts line
end
while (line=stderr.gets)
lines+=line
puts line
end
end
Any help on this or some insight would be appreciated!
What I would recommend is using Xinetd (or similar) to run the command on some socket and then using the ruby network code. One of the problems you've already run into in your code here is that your two while loops are sequential, which can cause problems.
Another trick you might try is to re-direct stderr to stdout in your command, so that your program only has to read the stdout. Something like this:
popen4("./srcds_run -console -game tf +map ctf_2fort -maxplayers 6 2>&1")
The other benefit of this is that you get all the messages/errors in the order they happen during the program run.
EDIT
Your should consider integrating with AnyTerm. You can then either expose AnyTerm directly e.g. via Apache mod_proxy, or have your Rails controller act as the reverse proxy (handling authentication/session validation, then playing back controller.request minus any cookies to localhost:<AnyTerm-daemon-port>, and sending back as a response whatever AnyTerm replies with.)
class ConsoleController < ApplicationController
# AnyTerm speaks via HTTP POST only
def update
# validate session
...
# forward request to AnyTerm
response = Net::HTTP.post_form(URI.parse('http://localhost:#{AnyTermPort}/', request.params))
headers['Content-Type'] = response['Content-Type']
render_text response.body, response.status
end
Otherwise, you'd need to use IO::Select or IO::read_noblock to know when data is available to be read (from either network or subprocess) so you don't deadlock. See this too. Also check that either your Rails is used in a multi-threaded environment or that your Ruby version is not affected by this IO::Select bug.
You can start with something along these lines:
status = POpen4::popen4("ping localhost") do |stdout, stderr, stdin, pid|
puts "PID #{pid}"
# our buffers
stdout_lines=""
stderr_lines=""
begin
loop do
# check whether stdout, stderr or both are
# ready to be read from without blocking
IO.select([stdout,stderr]).flatten.compact.each { |io|
# stdout, if ready, goes to stdout_lines
stdout_lines += io.readpartial(1024) if io.fileno == stdout.fileno
# stderr, if ready, goes to stdout_lines
stderr_lines += io.readpartial(1024) if io.fileno == stderr.fileno
}
break if stdout.closed? && stderr.closed?
# if we acumulated any complete lines (\n-terminated)
# in either stdout/err_lines, output them now
stdout_lines.sub!(/.*\n/m) { puts $& ; '' }
stderr_lines.sub!(/.*\n/m) { puts $& ; '' }
end
rescue EOFError
puts "Done"
end
end
To also handle stdin, change to:
IO.select([stdout,stderr],[stdin]).flatten.compact.each { |io|
# program ready to get stdin? do we have anything for it?
if io.fileno == stdin.fileno && <got data from client?>
<write a small chunk from client to stdin>
end
# stdout, if ready, goes to stdout_lines