I'm running 2 scripts and after a while, I'm getting Too many open files # error/blob.c/ImageToFile/1832.
Simplified version of the first scripts. It's supposed to read images written to image_pipe, process them, and write them to ocr_pipe for the OCR to read.
# creates 2 named pipes
File.mkfifo(image_pipe) rescue nil
File.mkfifo(ocr_pipe) rescue nil
while image = Image.read(image_pipe)
# do some stuff with `image`...
end
The second script is using ffmpeg to extract frames from a video, writing them to image_pipe
# image_pipe is the same as the script above.
(14..movie.duration).step(0.5) do
`/usr/local/bin/ffmpeg [some options...] #{image_pipe}`
end
I think the issue is RMagick opening too many file descriptors when reading the images in the loop of the first script, but I'm not sure how to stop that from happening. The Magick::Image class doesn't have a close method or anything, afaik.
I did not find the root cause of the issue, but ulferts helped me find a workaround that's acceptable for me.
Instead of letting RMagick open the file itself, we should handle it on our side, and then use .from_blob to create the Magick::Image instance.
while f = File.read(image_pipe)
image = Image.from_blob(f)
# ... do stuff with image.
end
Related
I have a stream of data that I’m writing to a named pipe:
named_pipe = '/tmp/pipe' # Location of named pipe
File.mkfifo(named_pipe) # Create named pipe
File.open(named_pipe, 'w+') # Necessary to not get a broken pipe when ⌃C from another process later on
system('youtube-dl', '--newline', 'https://www.youtube.com/watch?v=aqz-KE-bpKQ', out: named_pipe) # Output download progress, one line at a time
Trouble is, while I can cat /tmp/pipe and get the information, I’m unable to read the file from another Ruby process. I’ve tried File.readlines, File.read with seeking, File.open then reading, and other stuff I no longer remember. Some of those hang, others error out.
How can I get the same result as with cat, in pure Ruby?
Note I don’t have to use system to send to the pipe (Open3 would be acceptable), but any solution requiring external dependencies is a no-go.
it looks like File.readlines/IO.readlines, File.read/IO.read need to load the whole temp file first so you don't see any be printed out.
try File#each/IO.foreach which process a file line by line and it does not require the whole file be loaded into memory
File.foreach("/tmp/pipe") { |line| p line }
# or
File.open('/tmp/pipe','r').each { |line| p line }
I'm new to ruby file IO. I have a function that takes a File parameter, and I need to make sure that the file is in read-only mode.
def myfunction(file)
raise ArgumentError.new() unless file.kind_of?(File)
#Assert that file is in read-only mode
end
Any help would be appreciated!
If you don't need to raise an error, you can use reopen, I think something like:
file = file.reopen(file.path, "r")
I can't find a way to otherwise verify that there isn't a write stream, but here's a bit of a hack that will work. Although I don't like exception throwing being used in the expected path, you could use close_write:
begin
file.close_write
# you could actually raise an exception here if you want
# since getting here means the file was originally opened for writing
rescue IOError
# This error will be raised if the file was not opened for
# writing, so this is actually the path we want
end
So all you need is 'make sure make sure that the file is in read-only mode', why not just set it as readonly with FileUtils.chmod.
Or if actually you just want to test if it is readonly, use File.writeable?
You can use file.readable?, which returns true or false.
Please check this link.
I have a file called backup.lua, which the program should write to every so often in order to backup its status, in case of a failure.
The problem is that the program writes the backup.lua file completely fine first-time round, but any other times it refuses to write to the file.
I tried removing the file while the program was still open but Windows told me that the file was in use by 'CrysisWarsDedicatedServer.exe', which is the program. I have told the host Lua function to close the backup.lua file, so why isn't it letting me modify the file at will after it has been closed?
I can't find anything on the internet (Google actually tried to correct my search) and the secondary programmer on the project doesn't know either.
So I'm wondering if any of you folks know what we are doing wrong here?
Host function code:
function ServerBackup(todo)
local write, read;
if todo=="write" then
write = true;
else
read = true;
end
if (write) then
local source = io.open(Root().."Mods/Infinity/System/Read/backup.lua", "w");
System.Log(TeamInstantAction:GetTeamScore(2).." for 2, and for 1: "..TeamInstantAction:GetTeamScore(1))
System.LogAlways("[System] Backing up serverdata to file 'backup.lua'");
source:write("--[[ The server is dependent on this file; editing it will lead to serious problems.If there is a problem with this file, please re-write it by accessing the backup system ingame.--]]");
source:write("Backup = {};Backup.Time = '"..os.date("%H:%M").."';Backup.Date = '"..os.date("%d/%m/%Y").."';");
source:write(XFormat("TeamInstantAction:SetTeamScore(2, %d);TeamInstantAction:SetTeamScore(1, %d);TeamInstantAction:UpdateScores();",TeamInstantAction:GetTeamScore(2), TeamInstantAction:GetTeamScore(1) ));
source:close();
for i,player in pairs(g_gameRules.game:GetPlayers() or {}) do
if (IsModerator(player)) then
CMPlayer(player, "[!backup] Completed server backup.");
end
end
end
--local source = io.open(Root().."Mods/Infinity/System/Read/backup.lua", "r"); Can the file be open here and by the Lua scriptloader too?
if (read) then
System.LogAlways("[System] Restoring serverdata from file 'backup.lua'");
--source:close();
Backup = {};
Script.LoadScript(Root().."Mods/Infinity/System/Read/backup.lua");
if not Backup or #Backup < 1 then
System.LogAlways("[System] Error restoring serverdata from file 'backup.lua'");
end
end
end
Thanks all :).
Edit:
Although the file is now written to the disk fine, the system fails to read the dumped file.
So, now the problem is that the "LoadScript" function isn't doing what you expect:
Because I'm psychic, i have divined that you're writing a Crysis plugin, and are attempting to use it's LoadScript API call.
(Please don't assume everyone here would guess this, or be bothered to look for it. It's vital information that must form part of your questions)
The script you're writing attempts to set Backup - but your script, as written - does not separate lines with newline characters. As the first line is a comment, the entire script will be ignored.
Basicallty the script you've written looks like this, which is all treated as a comment.
--[[ comment ]]--Backup="Hello!"
You need to write a "\n" after the comment (and, I'd recommend in other places too) to make it like this. In fact, you don't really need block comments at all.
-- comment
Backup="Hello!"
I am fairly new to Ruby and programming, less than a year experience. This is ruby 1.9.3 and the newest ffmpeg for Ubuntu from FFmpeg. Files I am using are basic .avi, some v210 .mov, other quicktime/finalcutpro types of files.
I am trying to write an automated probing tool that will help lessen the manual work load for me when I start testing with and dealing with lots of media files. Basically the script goes through a directory and probes each file, extracting the info I need, and writes it out to csv.
Currently, everytime I run it, the actual capture on command line is failing.
def prober(file)
#the_file = file
stdout,stderr,status = Open3.capture3("ffprobe -v quiet -print_format json - show_format -show_streams #{#the_file}")
STDERR.puts stderr
if status.success?
out = stdout
else
STDERR.puts "There was a problem, please try again."
end
#raw_output = JSON.parse(out)
end
I don't know if it is something to do with how I am running the command (I don't really fully understand I/O streams), or something with ffprobe. When it does make it past the probe it is returning #raw_output as nil, which causes JSON parsing to puke, or my other methods for splitting and parsing the metadata fail on nil.
Any help at all would great, I have been stuck on this for a while. Thanks! I can provide more code if needed for clarification,.
After the holidays I came back fresh and found the problem. The filepath string needs to have quotes around it when fed to the stdin for ffprobe, but when I aggregated the files it stripped the quotes. The fix? add quotes around the filepath in the string.
I hope this helps someone, apparently I am the only person in the whole internet to have this problem.
I'm just working on my first gem (pretty new to ruby as well), entire code so far is here;
https://github.com/mikeyhogarth/tablecloth
One thing I've tried to do is to create a yaml file which the gem can access as a lookup (under lib/tablecloth/yaml/qty.yaml). This all works great and the unit tests all pass, hwoever when I build and install the gem and try to run under irb (from my home folder) I am getting;
Errno::ENOENT: No such file or directory - lib/tablecloth/yaml/qty.yaml
The code is now looking for the file in ~/lib/tablecloth... rather than in the directory the gem is installed to. So my questions are;
1) How should i change line 27 of recipe.rb such that it is looking in the folder that the gem is installed to?
2) Am I in fact approaching this whole thing incorrectly (is it even appropriate to use static yaml files within gems in this way)?
Well first of all you should refer to the File in the following way:
file_path = File.join(File.dirname(__FILE__),"yaml/qty.yaml")
units_hash = YAML.load_file(filepath)
File.dirname(__FILE__) gives you the directory in which the current file (recipe.rb) lies.
File.join connects filepaths in the right way. So you should use this to reference the yaml-file relative to the recipe.rb folder.
If using a YAML-file in this case is a good idea, is something which is widely discussed. I, myself think, this is an adequate way, especially in the beginning of developing with ruby.
A valid alternative to yaml-files would be a rb-File (Ruby Code), in which you declare constants which contain your data. Later on you can use them directly. This way only the ruby-interpreter has to work and you save computing time for other things. (no parser needed)
However in the normal scenario you should also take care that reading in a YAML file might fail. So you should be able to handle that:
file_path = File.join(File.dirname(__FILE__),"yaml/qty.yaml")
begin
units_hash = YAML.load_file(filepath)
rescue Psych::SyntaxError
$stderr.puts "Invalid yaml-file found, at #{file_path}"
exit 1
rescue Errno::EACCES
$stderr.puts "Couldn't access file due to permissions at #{file_path}"
exit 1
rescue Errno::ENOENT
$stderr.puts "Couldn't access non-existent file #{file_path}"
exit 1
end
Or if you don't care about the details:
file_path = File.join(File.dirname(__FILE__),"yaml/qty.yaml")
units_hash =
begin
YAML.load_file(filepath)
rescue Psych::SyntaxError, Errno::EACCES, Errno::ENOENT
{}
end