I am fairly new to Ruby and programming, less than a year experience. This is ruby 1.9.3 and the newest ffmpeg for Ubuntu from FFmpeg. Files I am using are basic .avi, some v210 .mov, other quicktime/finalcutpro types of files.
I am trying to write an automated probing tool that will help lessen the manual work load for me when I start testing with and dealing with lots of media files. Basically the script goes through a directory and probes each file, extracting the info I need, and writes it out to csv.
Currently, everytime I run it, the actual capture on command line is failing.
def prober(file)
#the_file = file
stdout,stderr,status = Open3.capture3("ffprobe -v quiet -print_format json - show_format -show_streams #{#the_file}")
STDERR.puts stderr
if status.success?
out = stdout
else
STDERR.puts "There was a problem, please try again."
end
#raw_output = JSON.parse(out)
end
I don't know if it is something to do with how I am running the command (I don't really fully understand I/O streams), or something with ffprobe. When it does make it past the probe it is returning #raw_output as nil, which causes JSON parsing to puke, or my other methods for splitting and parsing the metadata fail on nil.
Any help at all would great, I have been stuck on this for a while. Thanks! I can provide more code if needed for clarification,.
After the holidays I came back fresh and found the problem. The filepath string needs to have quotes around it when fed to the stdin for ffprobe, but when I aggregated the files it stripped the quotes. The fix? add quotes around the filepath in the string.
I hope this helps someone, apparently I am the only person in the whole internet to have this problem.
Related
I have a stream of data that I’m writing to a named pipe:
named_pipe = '/tmp/pipe' # Location of named pipe
File.mkfifo(named_pipe) # Create named pipe
File.open(named_pipe, 'w+') # Necessary to not get a broken pipe when ⌃C from another process later on
system('youtube-dl', '--newline', 'https://www.youtube.com/watch?v=aqz-KE-bpKQ', out: named_pipe) # Output download progress, one line at a time
Trouble is, while I can cat /tmp/pipe and get the information, I’m unable to read the file from another Ruby process. I’ve tried File.readlines, File.read with seeking, File.open then reading, and other stuff I no longer remember. Some of those hang, others error out.
How can I get the same result as with cat, in pure Ruby?
Note I don’t have to use system to send to the pipe (Open3 would be acceptable), but any solution requiring external dependencies is a no-go.
it looks like File.readlines/IO.readlines, File.read/IO.read need to load the whole temp file first so you don't see any be printed out.
try File#each/IO.foreach which process a file line by line and it does not require the whole file be loaded into memory
File.foreach("/tmp/pipe") { |line| p line }
# or
File.open('/tmp/pipe','r').each { |line| p line }
I'm really confused on this one, and maybe it's a bug in Ruby 2.6.2. I have files that were written as UTF-8 with BOM, so I'm using the following:
filelist = Dir.entries(#input_dirname).join(' ')
filelist = filelist.split(' ').grep(/xml/)
filelist.each do |indfile|
filecontents_tmp = File.read("#{#input_dirname}/#{indfile}", :encoding =>'bom|utf-8')
puts filecontents_tmp
end
If I put a debug breakpoint at the puts line, my file is read in properly. If I just run the simple script, I get the following error:
in `read': ASCII incompatible encoding needs binmode (ArgumentError)
I'm confused as to why this would work in debug, but not when run normally. Ideas?
Have you tried printing the default encoding when you run the file as opposed to when you debug the file? There are 3 ways to set / change the encoding in Ruby (that I'm aware of), so I wonder if it's different between running the file and debugging. You should be able to tell by printing the default encoding: puts Encoding.default_external.
As for actually fixing the issue, I ran into a similar problem and found this answer which said to add bin mode as an option to the File.open call and it worked for me.
I'm running 2 scripts and after a while, I'm getting Too many open files # error/blob.c/ImageToFile/1832.
Simplified version of the first scripts. It's supposed to read images written to image_pipe, process them, and write them to ocr_pipe for the OCR to read.
# creates 2 named pipes
File.mkfifo(image_pipe) rescue nil
File.mkfifo(ocr_pipe) rescue nil
while image = Image.read(image_pipe)
# do some stuff with `image`...
end
The second script is using ffmpeg to extract frames from a video, writing them to image_pipe
# image_pipe is the same as the script above.
(14..movie.duration).step(0.5) do
`/usr/local/bin/ffmpeg [some options...] #{image_pipe}`
end
I think the issue is RMagick opening too many file descriptors when reading the images in the loop of the first script, but I'm not sure how to stop that from happening. The Magick::Image class doesn't have a close method or anything, afaik.
I did not find the root cause of the issue, but ulferts helped me find a workaround that's acceptable for me.
Instead of letting RMagick open the file itself, we should handle it on our side, and then use .from_blob to create the Magick::Image instance.
while f = File.read(image_pipe)
image = Image.from_blob(f)
# ... do stuff with image.
end
I want to run some shell scripts remotely as part of my capistrano setup. To test that functionality, I use this code:
execute <<SHELL
cat <<TEST
something
TEST
SHELL
However, that is actually running /usr/bin/env cat <<TEST; something; TEST which is obviously not going to work. How do I tell capistrano to execute the heredoc as I have written it, without converting the newlines into semicolons?
I have Capistrano Version: 3.2.1 (Rake Version: 10.3.2) and do not know ruby particularly well, so there might be something obvious I missed.
I think it might work to just specify the arguments to cat as a second, er, argument to execute:
cat_args = <<SHELL
<<TEST
something
TEST
SHELL
execute "cat", cat_args
From the code #DavidGrayson posted, it looks like only the command (the first argument to execute) is sanitized.
I agree with David, though, that the simpler way might be to put the data in a file, which is what the SSHKit documentation suggests:
Upload a file from a stream
on hosts do |host|
file = File.open('/config/database.yml')
io = StringIO.new(....)
upload! file, '/opt/my_project/shared/database.yml'
upload! io, '/opt/my_project/shared/io.io.io'
end
The IO streaming is useful for uploading something rather than "cat"ing it, for example
on hosts do |host|
contents = StringIO.new('ALL ALL = (ALL) NOPASSWD: ALL')
upload! contents, '/etc/sudoers.d/yolo'
end
This spares one from having to figure out the correct escaping sequences for something like "echo(:cat, '...?...', '> /etc/sudoers.d/yolo')".
This seems like it would work perfectly for your use case.
The code responsible for this sanitization can be found in SSHKit::Command#sanitize_command!, which is called by that class's initialize method. You can see the source code here:
https://github.com/capistrano/sshkit/blob/9ac8298c6a62582455b1b55b5e742fd9e948cefe/lib/sshkit/command.rb#L216-226
You might consider monkeypatching it to do nothing by adding something like this to the top of your Rakefile:
SSHKit::Command # force the class to load so we can re-open it
class SSHKit::Command
def sanitize_command!
return if some_condition
super
end
end
This is risky and could introduce problems in other places; for example there might be parts of Capistrano that assume that the command has no newlines.
You are probably better off making a shell script that contains the heredoc or putting the heredoc in a file somewhere.
Ok, so this is the solution I figured out myself, in case it's useful for someone else:
str = %x(
base64 <<TEST
some
thing
TEST
).delete("\n")
execute "echo #{str} | base64 -d | cat -"
As you can see, I'm base64 encoding my command, sending it through, then decoding it on the server side where it can be evaluated intact. This works, but it's a real ugly hack - I hope someone can come up with a better solution.
I want to write a ruby script that read from a config file that will have filenames, and then when I run the script it will take the tail of each file and output the console.
What's the best way to go about doing this?
Take a look at File::Tail gem.
You can invoke linux tail -number_of_lines file_name command from your ruby script and let it print on console or capture output and print it yourself (if you need to do something with these lines before you print it)
We have a configuration file that contain a list of the log files; for example, like this:
---
- C:\fe\logs\front_end.log
- C:\mt\logs\middle_tier.log
- C:\be\logs\back_end.log
The format of the configuration file is a yaml simple sequence , therefore suppose we named this file 'settings.yaml'
The ruby script that take the tail of each file and output the console could be like this:
require 'yaml'
require 'file-tail'
logs = YAML::load(File.open('settings.yaml'))
threads = []
logs.each do |the_log|
threads << Thread.new(the_log) { |log_filename|
File.open(log_filename) do |log|
log.extend(File::Tail)
log.interval = 10
log.backward(10)
log.tail { |line| p "#{File.basename(the_log,".log")} - #{line}" }
end
}
end
threads.each { |the_thread| the_thread.join }
Note: displaying each line I wanted to prefix it with the name of the file from which it originates, ...this for me is a good option but you can edit the script to change as you like ; is the same for the tails parameters.
if file-tail is missing in your environment, follow the link as #Mark Thomas posts in his answear; i.e you need to:
> gem install file-tail
I found the file-tail gem to be a bit buggy. I would write to a file and it would read the entire file again instead of just thelines appended. This happened even though I had log.backward set to 0. I ended up writing my own and figured that I would share it here in case any one else is looking for a Ruby alternative to the file-tail gem. You can find the repo here. It uses non_blocking io, so it will catch amendments to the file immediately. There is one caveat that can be easily fixed if you can program in the Ruby programming language; log.backward is hard coded to be -1.