I would like to enter a large amount of data in my file.
To do this, I am thinking of using postgresql's asynchronous processing to do the query.
The code is as follows.
raw_conn = ActiveRecord::Base.connection.raw_connection
raw_conn.send_query("SELECT id FROM users")
raw_conn.set_single_row_mode
begin
raw_conn.get_result.stream_each do |f|
# writing process to file
end
end
raw_conn.send_query("SELECT id FROM logins")
raw_conn.set_single_row_mode
begin
raw_conn.get_result.stream_each do |f|
# writing process to file
end
end
When the second send_query is executed, the following error occurs.
another command is already in progress
I had a feeling that the connection pool was the cause, but the documentation did not mention it.
Is there any way to run send_query more than once and get results?
I Resolved.
I tried to reset the connection before sending a new query, and it worked.
begin
raw_conn.get_result.stream_each do |f|
# writing process to file
end
ensure
raw_conn.reset
end
Related
I have some code to make API requests through Full contact to resolve domains for a list of company names, and output these in a csv table. I found the code was terminating whenever it hit a response code other than 200 or 202.
I have the following rescue block written:
def get_parse_all
ORG_ARRAY.each do |company_name|
begin
org_info = get_org_info(company_name)
rescue
next
end
parse_org_info(org_info)
end
end
The issue is, I can't figure out how to still include the skipped company names (due to bad response code)in the output. I am getting a list of the successful calls, but I can't tell which ones were skipped and why.
I have tried to puts "error" before next, but it doesn't appear in the output csv. And if I remove next, I get a nil:NilClass (NoMethodError)
I've read some documentation but I am new to this and I'm quite stuck. If someone could point me in the right direction, that would be greatly appreciated!
Thank you in advance for any help :)
In this case, it sounds like what you're looking to do is iterate over an array and transform each element to another value. The result would be another array, and each element within it would either be org_info or an error.
For this, you would use map instead of each. Remember that each does not return the result of the block, e.g. ORG_ARRAY.each do will always return ORG_ARRAY no matter what you do in the block.
def get_parse_all
ORG_ARRAY.map do |company_name|
begin
parse_org_info(get_org_info(company_name))
rescue => e
e
end
end
end
As mentioned in a comment, you should also avoid using "naked rescue" - rescue a more specific error instead of any error at all.
I'm creating a program that reads lines from a file and tries to connect to a randomly generated "proxy." What I'm trying to do is read the lines, if the connection errors out, either;
Save them to a file called proxies_to_check.txt
Or save them to a file called bad_proxies.txt
It works how it's suppose to, it actually works pretty well I'm kind of impressed with myself. However, while it's saving to the file it saves the ip, with the port, like so:
143.54.67.231:6543
143.23.567.23:3452
9.234.21.124:5432
What I want to do is just save the ip to the file like so:
143.54.67.231
143.23.567.23
9.234.21.124
I've tried a few things, like using a regex, and striping the line (I looked up striping and it doesn't do what I thought it did), how can I go about grabbing all the digits and periods before the semi-colon?
Source:
def check_possibles
puts "Testing possible proxies, this will take awhile..".green.bold
IO.read("possible_proxies.txt").each_line do |proxy|
begin
Timeout::timeout(6) do
begin
open("http://#{proxy.chomp}")
end
File.open("true_proxies.txt", "a+") {|s| s.puts(proxy)}
end
rescue Errno::ENETUNREACH, Errno::EADDRNOTAVAIL
File.open("bad_proxies.txt", "a+"){|s| s.puts("Bad IP => #{proxy}")}
rescue Timeout::Error, Errno::ECONNREFUSED
File.open("proxies_to_check.txt", "a+") {|s| s.puts(proxy)}
next
end
end
end
"143.54.67.231:6543".split(":")[0..-2].join
OR
"143.54.67.231:6543".split(":").first
OR maybe this will helps you if you need RGX
Is it a valid Regular expression for IP address
In my application, the user must upload a text document, the contents of which are then parsed by the receiving controller action. I've gotten the document to upload successfully, but I'm having trouble reading its contents.
There are several threads on this issue. I've tried more or less everything recommended on these threads, and I'm still unable to resolve the problem.
Here is my code:
file_data = params[:file]
contents = ""
if file_data.respond_to?(:read)
contents = file_data.read
else
if file_data.respond_to?(:path)
File.open(file_data, 'r').each_line do |line|
elts = line.split
#
#
end
end
end
So here are my problems:
file_data doesn't 'respond_to?' either :read or :path. According to some other threads on the topic, if the uploaded file is less than a certain size, it's interpreted as a string and will respond to :read. Otherwise, it should respond to :path. But in my code, it responds to neither.
If I try to take out the if statements and straight away attempt File.open(file_data, 'r'), I get an error saying that the file wasn't found.
Can someone please help me find out what's wrong?
PS, I'm really sorry that this is a redundant question, but I found the other threads unhelpful.
Are you actually storing the file? Because if you are not, of course it can't be found.
First, find out what you're actually getting for file_data by adding debug output of file_data.inspect. It maybe something you don't expect, especially if form isn't set up correctly (i.e. :multipart => true).
Rails should enclose uploaded file in special object providing uniform interface, so that something as simple as this should work:
file_data.read.each_line do |line|
elts = line.split
#
#
end
I need to build a huge XML file, about 1-50 MB. I thought that using builder would be effective enough and, well it is, somewhat. The problem is, after the program reaches its last line it doesn't end immediately, but Ruby is still doing something for several seconds, maybe garbage collection? After that the program finally ends.
To give a real example, I am measured the time of building an XML file. It outputs 55 seconds (there is a database behind so it takes long) when the XML was built, but Ruby still processes for about 15 more seconds and the processor is going crazy.
The pseudo/real code is as follows:
...
builder = Nokogiri::XML::Builder.with(doc) do |xml|
build_node(xml)
end
...
def build_node(xml)
...
xml["#{namespace}"] if namespace
xml.send("#{elem_name}", attrs_hash) do |elem_xml|
...
if has_children
if type
case type
when XML::TextContent::PLAIN
elem_xml.text text_content
when XML::TextContent::COMMENT
elem_xml.comment text_content
when XML::TextContent::CDATA
elem_xml.cdata text_content
end
else
build_node(elem_xml)
end
end
end
end
Note that I was using a different approach using my own structure of classes, and the speed of the build was the same, but at the last line the program normally ended, but now I am forced to use Nokogiri so I have to find a solution.
What I can do to avoid that X seconds long overhead after the XML is built? Is it even possible?
UPDATE:
Thanks to a suggestion from Adiel Mittmann, during the creation of my minimal working example I was able to locate the problem. I now have a small (well not that small) example demonstrating the problem.
The following code is causing the problem:
xml.send("#{elem_name}_") do |elem_xml|
...
elem_xml.text text_content #This line is the problem
...
end
So the line executes the following code based on Nokogiri's documentation:
def create_text_node string, &block
Nokogiri::XML::Text.new string.to_s, self, &block
end
Text node creation code gets executed then. So, what exactly is happening here?
UPDATE 2:
After some other tries, the problem can be easily reproduced by:
builder = Nokogiri::XML::Builder.new do |xml|
0.upto(81900) do
xml.text "test"
end
end
puts "End"
So is it really Nokogiri itself? Is there any option for me?
Your example also takes a long time to execute here. And you were right: it's the garbage collector that's taking so long to execute. Try this:
require 'nokogiri'
class A
def a
builder = Nokogiri::XML::Builder.new do |xml|
0.upto(81900) do
xml.text "test"
end
end
end
end
A.new.a
puts "End1"
GC.start
puts "End2"
Here, the delay happens between "End1" and "End2". After "End2" is printed, the program closes immediately.
Notice that I created an object to demonstrate it. Otherwise, the data generated by the builder can only be garbage collected when the program finishes.
As for the best way to do what you're trying to accomplish, I suggest you ask another question giving details of what exactly you're trying to do with the XML files.
Try using the Ruby built-in (sic) Builder. I use it to generate large XML files as well, and it has such an small footprint.
I finally got my code to reference the ruby-alsa library, but I'm stuck again. Eventually, what I'd like to happen is have an audio file play on the server when an action is invoked from the client side. As such, I have the following code in my controller:
File.open('./public/audio/test.wav', 'r') do |f|
ALSA::PCM::Playback.open do |playback|
playback.write do |length|
f.read length
end
end
end
By reading the RDoc for the ALSA library, I get the impression I should be using the write_in_background method from the ALSA::PCM::Playback class. Unfortunately, I can't get the syntax right for getting this method to work. To simply verify I have ALSA working properly, I attempted to use the Playback.write method (note code above). The above code is syntactically correct; however, it plays a sound only for a microsecond, then stops. My guess is the request ends so quickly, it doesn't have enough time to play anything recognizable.
As previously mentioned, my ultimate goal is to have an end-user invoke an action that plays audio back on the server. The file should not stop playing at the end of the HTTP request --it should continue until another action is invoked that stops playback. Knowing that, could somebody please help me with getting the syntax and parameters right for calling the write_in_background method? I'm afraid the ruby-alsa documentation I have at this time hasn't been helpful enough for me (as a complete newbie at Ruby).
Update: If I replace the above call to the write method with a call to the write_to_background method, I get the following runtime error: cannot add pcm handler (Function not implemented)
Update 2: I tried this with a different WAV file and the following code and it plays at warp speed.
File.open('./public/audio/sample.wav', 'r') do |f|
ALSA::PCM::Playback.open do |playback|
playback.write do |length|
#temp = length
f.read length
end
sleep 4
end
end
It appears there may be a combination of things going on here. I believe the first is in reference to the sample rate (length == 44100, which is CD quality). I'll have to look into how to play back audio files at a different rate. Beyond that, however, I'm still stuck on how to get it to play in the background. While the sleep proves ALSA is working, it won't work well in a real-world scenario.
Update 3: Got the sample rate bit working even though it temporarily relies on a magic number:
File.open('./public/audio/sample.wav', 'r') do |f|
ALSA::PCM::Playback.open "default", {:channels => 1, :sample_rate => 11025} do |playback|
playback.write do |length|
#temp = length
f.read length
end
sleep 4
end
end
At this point, I have my ruby code playing audio through ALSA, but I'm not sure how to make it play continuously in the background without requiring the sleep.
How about kicking it off on its own thread?
Thread.new do
File.open('myfile.wav', 'rb') do |f|
ALSA::PCM::Playback.open do |playback|
playback.write do |length|
f.read length
end
end
end
end.join
Edit:
OK, if that doesn't work, I'm guessing it kicks off its own thread already. How about sleeping for the duration? First try:
File.open('myfile.wav', 'rb') do |f|
ALSA::PCM::Playback.open do |playback|
playback.write do |length|
f.read length
end
end
end
sleep 4 # seconds