I want to write a ruby script that read from a config file that will have filenames, and then when I run the script it will take the tail of each file and output the console.
What's the best way to go about doing this?
Take a look at File::Tail gem.
You can invoke linux tail -number_of_lines file_name command from your ruby script and let it print on console or capture output and print it yourself (if you need to do something with these lines before you print it)
We have a configuration file that contain a list of the log files; for example, like this:
---
- C:\fe\logs\front_end.log
- C:\mt\logs\middle_tier.log
- C:\be\logs\back_end.log
The format of the configuration file is a yaml simple sequence , therefore suppose we named this file 'settings.yaml'
The ruby script that take the tail of each file and output the console could be like this:
require 'yaml'
require 'file-tail'
logs = YAML::load(File.open('settings.yaml'))
threads = []
logs.each do |the_log|
threads << Thread.new(the_log) { |log_filename|
File.open(log_filename) do |log|
log.extend(File::Tail)
log.interval = 10
log.backward(10)
log.tail { |line| p "#{File.basename(the_log,".log")} - #{line}" }
end
}
end
threads.each { |the_thread| the_thread.join }
Note: displaying each line I wanted to prefix it with the name of the file from which it originates, ...this for me is a good option but you can edit the script to change as you like ; is the same for the tails parameters.
if file-tail is missing in your environment, follow the link as #Mark Thomas posts in his answear; i.e you need to:
> gem install file-tail
I found the file-tail gem to be a bit buggy. I would write to a file and it would read the entire file again instead of just thelines appended. This happened even though I had log.backward set to 0. I ended up writing my own and figured that I would share it here in case any one else is looking for a Ruby alternative to the file-tail gem. You can find the repo here. It uses non_blocking io, so it will catch amendments to the file immediately. There is one caveat that can be easily fixed if you can program in the Ruby programming language; log.backward is hard coded to be -1.
Related
I have a stream of data that I’m writing to a named pipe:
named_pipe = '/tmp/pipe' # Location of named pipe
File.mkfifo(named_pipe) # Create named pipe
File.open(named_pipe, 'w+') # Necessary to not get a broken pipe when ⌃C from another process later on
system('youtube-dl', '--newline', 'https://www.youtube.com/watch?v=aqz-KE-bpKQ', out: named_pipe) # Output download progress, one line at a time
Trouble is, while I can cat /tmp/pipe and get the information, I’m unable to read the file from another Ruby process. I’ve tried File.readlines, File.read with seeking, File.open then reading, and other stuff I no longer remember. Some of those hang, others error out.
How can I get the same result as with cat, in pure Ruby?
Note I don’t have to use system to send to the pipe (Open3 would be acceptable), but any solution requiring external dependencies is a no-go.
it looks like File.readlines/IO.readlines, File.read/IO.read need to load the whole temp file first so you don't see any be printed out.
try File#each/IO.foreach which process a file line by line and it does not require the whole file be loaded into memory
File.foreach("/tmp/pipe") { |line| p line }
# or
File.open('/tmp/pipe','r').each { |line| p line }
Due to some OS restrictions I am not able to install ruby and mysql on my prod node. I have to process a Log file written on that node. So I planning to write the contents of the log file to a different file as its written to the log and work on that new file.
def watch_for(file)
f = File.open(file,"r")
f.seek(0,IO::SEEK_END)
while true do
select([f])
line = f.gets
open('myfile.out', 'a+') { |s|
s.puts "#{line}"
}
end
end
But this seems to failing; Any help to this is much appreciated.
It's not a great idea to reopen the output file each time through the loop. If you are just planning on appending and not reading the file, use 'a' rather than 'a+' for the IO mode. Also, there's no good reason to use string interpolation like "#{line}". I rewrote the method to open each file just once:
def watch_for(file)
f = File.open(file,"r")
f.seek(0,IO::SEEK_END)
s = File.open('myfile.out', 'a')
while true do
s.write(f.readline)
end
end
Probably better if this isn't a method at all since there's no way to exit except to kill the process. And, of course, you can just use tail -f $file >> myfile.out as mentioned in this comment.
I want to run some shell scripts remotely as part of my capistrano setup. To test that functionality, I use this code:
execute <<SHELL
cat <<TEST
something
TEST
SHELL
However, that is actually running /usr/bin/env cat <<TEST; something; TEST which is obviously not going to work. How do I tell capistrano to execute the heredoc as I have written it, without converting the newlines into semicolons?
I have Capistrano Version: 3.2.1 (Rake Version: 10.3.2) and do not know ruby particularly well, so there might be something obvious I missed.
I think it might work to just specify the arguments to cat as a second, er, argument to execute:
cat_args = <<SHELL
<<TEST
something
TEST
SHELL
execute "cat", cat_args
From the code #DavidGrayson posted, it looks like only the command (the first argument to execute) is sanitized.
I agree with David, though, that the simpler way might be to put the data in a file, which is what the SSHKit documentation suggests:
Upload a file from a stream
on hosts do |host|
file = File.open('/config/database.yml')
io = StringIO.new(....)
upload! file, '/opt/my_project/shared/database.yml'
upload! io, '/opt/my_project/shared/io.io.io'
end
The IO streaming is useful for uploading something rather than "cat"ing it, for example
on hosts do |host|
contents = StringIO.new('ALL ALL = (ALL) NOPASSWD: ALL')
upload! contents, '/etc/sudoers.d/yolo'
end
This spares one from having to figure out the correct escaping sequences for something like "echo(:cat, '...?...', '> /etc/sudoers.d/yolo')".
This seems like it would work perfectly for your use case.
The code responsible for this sanitization can be found in SSHKit::Command#sanitize_command!, which is called by that class's initialize method. You can see the source code here:
https://github.com/capistrano/sshkit/blob/9ac8298c6a62582455b1b55b5e742fd9e948cefe/lib/sshkit/command.rb#L216-226
You might consider monkeypatching it to do nothing by adding something like this to the top of your Rakefile:
SSHKit::Command # force the class to load so we can re-open it
class SSHKit::Command
def sanitize_command!
return if some_condition
super
end
end
This is risky and could introduce problems in other places; for example there might be parts of Capistrano that assume that the command has no newlines.
You are probably better off making a shell script that contains the heredoc or putting the heredoc in a file somewhere.
Ok, so this is the solution I figured out myself, in case it's useful for someone else:
str = %x(
base64 <<TEST
some
thing
TEST
).delete("\n")
execute "echo #{str} | base64 -d | cat -"
As you can see, I'm base64 encoding my command, sending it through, then decoding it on the server side where it can be evaluated intact. This works, but it's a real ugly hack - I hope someone can come up with a better solution.
Out of curiosity I tried file-tail ruby library to see how it works with ruby code. But, the code doesn't seem to be working.
Here is what I tired(logger.rb):
$:.unshift File.dirname(__FILE__)
filename = 'logger.log'
require "file-tail"
File.open(filename) do |log|
log.extend(File::Tail)
log.interval = 10
log.backward(10)
log.tail { |line| puts line }
end
My logger.log file is in the same directory. Now, when I run: $ ruby logger.rb I see the last 10 lines from my log, but when I open logger.log file append some log data to it, the console doesn't show any progress. I mean it doesn't output the new log I appended.
I thought there may be in an issue with this. So, I tried inheriting and including the File::Tail in the inherited class, like this:
$:.unshift File.dirname(__FILE__)
filename = 'logger.log'
require "file-tail"
class FileTail < File
require "file-tail"
include File::Tail
end
log = FileTail.new(filename)
log.interval = 10
log.backward(10)
log.tail { |line| print line }
However this behaves the same way!!
Any pointers?
I am running on MAC OC X 10.8.5 with ruby-2.0.0-p353 installed.
Also, please let me know if anybody has implemented web version of tail in Ruby?
My Bad. This works when I closed all streams of my logger file. I'd opened the file in my ruby code but, didn't close the file stream. Maybe that's why I didn't see any log output on my console using the file-tail.
So, make sure you close all streams of the log/text file you're running with file-tail program.
I want to be able to read a currently open file. The test.rb is sending its output to test.log which I want to be able to read and ultimately send via email.
I am running this using cron:
*/5 * * * /tmp/test.rb > /tmp/log/test.log 2>&1
I have something like this in test.rb:
#!/usr/bin/ruby
def read_file(file_name)
file = File.open(file_name, "r")
data = file.read
file.close
return data
end
puts "Start"
puts read_file("/tmp/log/test.log")
puts "End"
When I run this code, it only gives me this output:
Start
End
I would expect the output to be something like this:
Start
Start (from the reading of the test.log since it should have the word start already)
End
Ok, you're trying to do several things at once, and I suspect you didn't systematically test before moving from one step to the next.
First we're going to clean up your code:
def read_file(file_name)
file = File.open(file_name, "r")
data = file.read
file.close
return data
end
puts "Start"
puts read_file("/tmp/log/test.log")
puts "End"
can be replaced with:
puts "Start"
puts File.read("./test.log")
puts "End"
It's plain and simple; There's no need for a method or anything complicated... yet.
Note that for ease of testing I'm working with a file in the current directory. To put some content in it I'll simply do:
echo "foo" > ./test.log
Running the test code gives me...
Greg:Desktop greg$ ruby test.rb
Start
foo
End
so I know the code is reading and printing correctly.
Now we can test what would go into the crontab, before we deal with its madness:
Greg:Desktop greg$ ruby test.rb > ./test.log
Greg:Desktop greg$
Hmm. No output. Something is broken with that. We knew there was content in the file previously, so what happened?
Greg:Desktop greg$ cat ./test.log
Start
End
Cat'ing the file shows it has the "Start" and "End" output of the code, but the part that should have been read and output is now missing.
What happening is that the shell truncated "test.log" just before it passed control to Ruby, which then opened and executed the code, which opened the now empty file to print it. In other words, you're asking the shell to truncate (empty) it just before you read it.
The fix is to read from a different file than you're going to write to, if you're trying to do something with the contents of it. If you're not trying to do something with its contents then there's no point in reading it with Ruby just to write it to a different file: We have cp and/or mv to do those things for us witout Ruby being involved. So, this makes more sense if we're going to do something with the contents:
ruby test.rb > ./test.log.out
I'll reset the file contents using echo "foo" > ./test.log, and cat'ing it showed 'foo', so I'm ready to try the redirection test again:
Greg:Desktop greg$ ruby test.rb > ./test.log.out
Greg:Desktop greg$ cat test.log.out
Start
foo
End
That time it worked. Trying it again has the same result, so I won't show the results here.
If you're going to email the file you could add that code at this point. Replacing the puts in the puts File.read('./test.log') line with an assignment to a variable will store the file's content:
contents = File.read('./test.log')
Then you can use contents as the body of a email. (And, rather than use Ruby for all of this I'd probably do it using mail or mailx or pipe it directly to sendmail, using the command-line and shell, but that's your call.)
At this point things are in a good position to add the command to crontab, using the same command as used on the command-line. Because it's running in cron, and errors can happen that we'd want to know about, we'd add the 2>&1 redirect to capture STDERR also, just as you did before. Just remember that you can NOT write to the same file you're going to read from or you'll have an empty file to read.
That's enough to get your app working.
class FileLineRead
File.open("file_line_read.txt") do |file|
file.each do |line|
phone_number = line.gsub(/\n/,'')
user = User.find_by_phone_number(line)
user.destroy unless user.nil?
end
end
end
open file
read line
DB Select
DB Update
In the cron job you have already opened and cleared test.log (via redirection) before you have read it in the Ruby script.
Why not do both the read and write in Ruby?
It may be a permissions issue or the file may not exist.
f = File.open("test","r")
puts f.read()
f.close()
The above will read the file test. If the file exists in the current directory
The problem is, as I can see, already solved by Slomojo. I'll only add:
to read and print a text file in Ruby, just:
puts File.read("/tmp/log/test.log")