I have a code in Ruby (as a Logstash plugin) in which I connect to Redis and get information like what is suggested in (How to use redis inside logstash filter?) as below:
require "redis"
$redis = Redis.new(host: "REDIS_HOST", port: REDIS_PORT, db: REDIS_DB)
#my_data = $redis.hget("DATA_KEY", "HASH_FIELD").to_i
Everything works OK generally; however, I want to know:
1- Which possible errors might be faced by the code while connecting to Redis or getting data (e.g. Redis is unreachable or DATA_KEY or HASH_FIELD does not exist)
2- How should we handle such errors?
Using activerecord outside rails, may I be confident all the connection drudgework is performed behind the curtains the same as inside rails?
In rails, activerecord does a great job establishing a connection pool and activating or closing connection as needed.
If I have a ruby daemon which calls a class file with:
ActiveRecord::Base.establish_connection(:production)
# more active_record tasks
can I assume on the following calls a connection from a pool is used?
Yup, calling ActiveRecord::Base.establish_connection will create a connection pool according to: https://github.com/rails/rails/blob/master/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb#L842
UPDATE: I tried changing the storage of the PostgreSQL connection object from a constant POSTGRES to a class variable ##pg. That didn't fix the issue. Then, I tried changing it to a global variable $pg. That seems to have worked! I still would like to eventually implement database connection pooling, however, so that I can have a pool of up to 5 (or so) database connections handling requests instead of just one global connection for all requests. END UPDATE
I'm hosting the Acani Chats REST Server on Heroku.
The first request works OK, but subsequent requests fail to connect to the PostgreSQL database.
I get the following errors:
Rack app error: #<PG::UnableToSend: SSL error: decryption failed or bad record mac>
Rack app error: #<PG::UnableToSend: no connection to the server>
What's going on?
In /config/application.rb, I define the constant POSTGRES to be the PostgreSQL connection object.
Should I be using a global or class variable instead of a constant to hold onto the connection instance?
In /config/routes.rb, I define the Rack call method.
I want to learn how to implement database connection pooling in Ruby for Rack with Puma and PostgreSQL.
Puma is threaded so you need a thread safe pool of connections to PostgreSQL, otherwise concurrent requests will all use the same connection concurrently, which is unexpected.
Please have a look to the connection_pool gem. It should help.
I run several sql statements in a transaction using Ruby pg gem. The problem that I bumped in is that connection times out on these queries due to firewall setup. Solution proposed here does not work, because it requires jdbc connections string, and I'm in Ruby (jRuby is not an option). Moving driver program to AWS to remove firewall is not an option either.
The code that I have is along the following lines:
conn = RedshiftHelper.get_redshift_connection
begin
conn.transaction do
# run my queries
end
ensure
conn.flush
conn.finish
end
I'm now looking into PG asynchronous API. I'm wondering if I can use is_busy to prevent firewall from timing out, or something to that effect. I can't find good documentation on the topic though. Appreciate any hints on that.
PS: I have solved this problem for a single query - I can trigger it asynchronously and track its completion using system STV_INFLIGHT Redshift table.Transaction does not work this way as I have to keep connection open.
Ok, I nailed it down. Here are the facts:
Redshift is based on Postgres 8.0. To check that, connect to Redshift instance using psql and see that it says "server version 8.0"
Keepalive requests are specified on the level of tcp socket (link).
Postgres 8.0 does not support keepalive option when specifying a connection string (link to 9.0 release changes, section E.19.3.9.1 on libpq)
PG gem in Ruby is a wrapper about libpq
Based on the facts above, tcp keepalive is not supported by Redshift. However, PG allows you to retrieve a socket that is used in the established connection. This means that even though libpq does not set keepalive feature, we still can use it manually. The solution thus:
class Connection
attr_accessor :socket, :pg_connection
def initialize(conn, socket)
#socket = socket
#pg_connection = conn
end
def method_missing(m, *args, &block)
#pg_connection.send(m, *args, &block)
end
def close
#socket.close
#pg_connection.close
end
def finish
#socket.close
#pg_connection.close
end
end
def get_connection
conn = PGconn.open(...)
socket_descriptor = conn.socket
socket = Socket.for_fd(socket_descriptor)
# Use TCP keep-alive feature
socket.setsockopt(Socket::SOL_SOCKET, Socket::SO_KEEPALIVE, 1)
# Maximum keep-alive probes before asuming the connection is lost
socket.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_KEEPCNT, 5)
# Interval (in seconds) between keep-alive probes
socket.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_KEEPINTVL, 2)
# Maximum idle time (in seconds) before start sending keep-alive probes
socket.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_KEEPIDLE, 2)
socket.autoclose = true
return Connection.new(conn, socket)
end
The reason why I introduce a proxy Connection class is because of Ruby tendency to garbage-collect IO objects (like sockets) when they get out of scope. This means that we now need connection and socket to be in the same scope, which is achieved through this proxy class. My Ruby knowledge is not deep, so there may be a better way to handle the socket object.
This approach works, but I would be happy to learn if there are better/cleaner solutions.
The link you provided has the answer. I think you just want to follow the section at the top, which has settings for 3 different OS'es, pick the one you are running the code on (the client to the Amazon service).
Look in this section: To change TCP/IP timeout settings - this is the OS that your code is running on (i.e. The client for the Amazon Service is your Server probably)
Linux — If your client is running on Linux, run the following command as the root user.
-- details omitted --
Windows — If your client runs on Windows, edit the values for the following registry settings under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters:
-- details omitted --
Mac — If your client is a Mac, create or modify the /etc/sysctl.conf file with the following values:
-- Details omitted --
I am using Redis Objects with Redis To Go on Heroku. I have a counter on a model, like this:
class Performance < ActiveRecord::Base
include Redis::Objects
counter :tickets_sold, start: 0
end
Accessing this value from Heroku console is working great as well.
irb(main):002:0> Performance.last.tickets_sold.value
Performance Load (3.9ms) SELECT `performances`.* FROM `performances` ORDER BY `performances`.`id` DESC LIMIT 1
=> 0
I confirmed that Redis.current is present:
irb(main):003:0> Redis.current
=> # Redis client v2.2.2 connected to redis://ray.redistogo.com:9023/0 (Redis v2.4.11)
However, accessing the same counter from a template on the website runs into a Errno::ECONNREFUSED error.
Connection refused - Unable to connect to Redis on 127.0.0.1:6379
Why is it trying to connect to the local Redis url? Inspecting Redis.current on the website is also failing with the connection error above. Considering that the same command is working just fine from the Heroku console, I'm a little puzzled as to what's going on here. I hope someone has seen this before and knows how to solve it...