How can I set the connection retry timer in the Ecto Postgrex driver - phoenix-framework

When my application starts, and the database hostname cannot be resolved, the Ecto/Postgrex driver now retries 20 times within the same second, and then Phoenix exits. Here's the error:
failed to connect: ** (DBConnection.ConnectionError) tcp connect (odoodb:5432): non-existing domain - :nxdomain
I want to control the time between retries, and the amount of retries as well. How do I do that? Couldn't find anything in the docs.
BTW: running in docker, so DNS can suddenly start working.
Here are the software versions I use:
{:phoenix, "~> 1.3.0-rc"},
{:phoenix_pubsub, "~> 1.0"},
{:phoenix_ecto, "~> 3.0"},
{:postgrex, ">= 0.0.0"},
{:phoenix_html, "~> 2.6"},
{:phoenix_live_reload, "~> 1.0", only: :dev},
{:gettext, "~> 0.11"},
{:cowboy, "~> 1.0"},
{:distillery, "~> 1.0"},
{:xain, "~> 0.6"},
{:comeonin, "~> 3.0"},
{:ueberauth_identity, "~> 0.2"},
{:ueberauth_github, "~> 0.4"},
{:ueberauth_google, "~> 0.5"},
{:ueberauth_facebook, "~> 0.6"},
{:bamboo, "~> 0.8"},
{:timex, "~> 3.0"},
{:scrivener_ecto, "~> 1.0"},
Here is the code that exits:
defmodule Backend.Application do
use Application
# See http://elixir-lang.org/docs/stable/elixir/Application.html
# for more information on OTP Applications
def start(_type, _args) do
import Supervisor.Spec
# Define workers and child supervisors to be supervised
children = [
# Start the Ecto repository
supervisor(Backend.Web.Repo, []),
# Start the endpoint when the application starts
supervisor(Backend.Web.Endpoint, []),
# Start your own worker by calling: Backend.Worker.start_link(arg1, arg2, arg3)
# worker(Backend.Worker, [arg1, arg2, arg3]),
]
# See http://elixir-lang.org/docs/stable/elixir/Supervisor.html
# for other strategies and supported options
opts = [strategy: :one_for_one, name: Backend.Supervisor]
res = Supervisor.start_link(children, opts)
IO.puts "######### run migrations..."
Ecto.Migrator.run(Backend.Web.Repo, Path.join(["#{:code.priv_dir(:backend)}", "repo", "migrations"]), :up, all: true)
IO.puts "######### migrations done..."
res
end
end
I think it's actually exiting in Migrator.run. Here's the error:
######### run migrations...
08:31:22.774 [info] Application backend exited: exited in: Backend.Application.start(:normal, [])
** (EXIT) an exception was raised:
** (DBConnection.ConnectionError) connection not available because of disconnection
(db_connection) lib/db_connection.ex:925: DBConnection.checkout/2
(db_connection) lib/db_connection.ex:741: DBConnection.run/3
(db_connection) lib/db_connection.ex:1132: DBConnection.run_meter/3
(db_connection) lib/db_connection.ex:584: DBConnection.prepare_execute/4
(ecto) lib/ecto/adapters/postgres/connection.ex:93: Ecto.Adapters.Postgres.Connection.execute/4
(ecto) lib/ecto/adapters/sql.ex:243: Ecto.Adapters.SQL.sql_call/6
(ecto) lib/ecto/adapters/sql.ex:193: Ecto.Adapters.SQL.query!/5
(ecto) lib/ecto/adapters/postgres.ex:86: anonymous fn/4 in Ecto.Adapters.Postgres.execute_ddl/3
(elixir) lib/enum.ex:1755: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto) lib/ecto/adapters/postgres.ex:86: Ecto.Adapters.Postgres.execute_ddl/3
(ecto) lib/ecto/migrator.ex:43: Ecto.Migrator.migrated_versions/2
(ecto) lib/ecto/migrator.ex:142: Ecto.Migrator.run/4
(backend) lib/backend/application.ex:24: Backend.Application.start/2
(kernel) application_master.erl:273: :application_master.start_it_old/4
So I think my question now becomes: how do I wait for a working connection, or how do I make migrator wait for it?
Also I'd still like to be able to make the driver wait longer between retries.

Related

Heroku deployment for Ruby / Sinatra / Postgres app "PG::ConnectionBad"

I’ve deployed my Ruby app on Heroku, however when looking to register or login I’m having an internal server error. My app is built on Sinatra and Rake.
When doing heroku logs -tail I’m having the following error message:
PG::ConnectionBad - connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
I installed the heroku Postgres add-on. I migrated my db with the following command:
heroku pg:push makersbnb DATABASE --app makersbnb-challenge
Here is one of my lib file with my PG connections:
require 'bcrypt'
class User
attr_reader :user_id, :username
def initialize(user_id:, username:)
#user_id = user_id
#username = username
end
def self.register(username:, password: )
if ENV['ENVIRONMENT'] == 'test'
connection = PG.connect(dbname: "makersbnb_test")
else
connection = PG.connect(dbname: 'makersbnb')
end
encrypted_password = BCrypt::Password.create(password)
result = connection.exec_params("INSERT INTO users (username, password)
VALUES ($1, $2)
RETURNING user_id, username, password;",[username, encrypted_password])
User.new(user_id: result[0]['user_id'], username: result[0]['username'])
end
def self.authenticate(username:, password:)
if ENV['ENVIRONMENT'] == 'test'
connection = PG.connect(dbname: "makersbnb_test")
else
connection = PG.connect(dbname: 'makersbnb')
end
result = connection.query(
"SELECT * FROM users WHERE username = $1",
[username]
)
return unless result.any?
return unless BCrypt::Password.new(result[0]['password']) == password
User.new(user_id: result[0]['user_id'], username: result[0]['username'])
end
end
Here is my Gemfile:
source "https://rubygems.org"
ruby '3.1.0'
gem "sinatra"
gem "sinatra-contrib"
gem "rake"
gem 'pg'
gem 'webrick'
gem 'bcrypt'
gem 'rack-flash3'
gem "sinatra-activerecord" # for Active Record models
gem 'capybara', group: :test
gem 'rspec', group: :test
I also created a database.yml file based on other instructions I read but I'm not sure if it applies in this case:
# database.yml file
default: &default
adapter: postgresql
encoding: unicode
pool: 5
test:
<<: *default
database: makersbnb_test
production:
<<: *default
database: makersbnb
Here is the repo
Here is the app
Help!
I will try to explain the problem, but also point some other flaws of Your current approach
To connect to remote database You should specify some options in
connection = PG.connect(dbname: "makersbnb")
You should specify not only dbname option but also:
host
user
password
Those can be retrieved from heroku postgresql config
connection = PG.connect(dbname: "makersbnb", host: <db_host>, user: <db_user>, password: <db_password>)
REMEMBER DON'T SET THOSE VALUES IN YOUR CODE
Instead You can use env variables.
Your config file is not doing anything in this approach
In the script You also don't use ActiveRecord. To do so Your model class should inherit from ActiveRecord::Base, something like this:
class User < ActiveRecord::Base
end
Here You can find some information how to use ActiveRecord without Rails framework: https://www.devdungeon.com/content/ruby-activerecord-without-rails-tutorial
And also some more information how to use it in sinatra specifically:
https://github.com/sinatra-activerecord/sinatra-activerecord

How do I merge two or more results from elasticsearch rails gem?

I'm trying to merge two elasticsearch results into one variable, here my code tries...
class SearchController < ApplicationController
def index
end
def advanced
#results = {}
if !params[:fast_search].empty?
#results[:features] = TestCases.search(params[:fast_search]).results
#results[:steps] = Steps.search(params[:fast_search]).results
#results[:examples] = Examples.search(params[:fast_search]).results
else
unless params[:feature].blank?
features = TestCases.search(query: { match: { function: params[:feature] } }).results
features_tag = Steps.search(query: { match: { tags: params[:tags] } }).results
#results[:features] = features + features_tag
end
unless params[:steps].blank? || params[:scenario].blank?
#results[:steps] = Steps.search(query: { match: { scenario: params[:scenario] } }).results
params[:steps].each do |step|
#results[:steps] += Steps.search(query: { match: { steps: step } }).results
end
#results[:steps] += Steps.search(query: { match: { tags: params[:tags] } }).results
end
unless params[:examples].blank?
params[:examples].each do |example|
#results[:examples] += Examples.search(query: { match: { examples: example } }).results
end
#results[:examples] += Examples.search(query: { match: { tags: params[:tags] } }).results
end
unless params[:bug].blank?
#results[:miscs] = StepsMiscs.search(query: { match: { bug: true } }).results
end
end
render "search/index"
end
end
I also try features.merge(features_tag) but no success either.
It's simple, I just need to merge one and more results from the elasticsearch, but I simply don't know how.
Here's my Gemfile:
source 'https://rubygems.org'
# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '~> 5.0.0', '>= 5.0.0.1'
# Use sqlite3 as the database for Active Record
gem 'sqlite3'
# Use Puma as the app server
gem 'puma', '~> 3.0'
# Use SCSS for stylesheets
gem 'sass-rails', '~> 5.0'
# Use Uglifier as compressor for JavaScript assets
gem 'uglifier', '>= 1.3.0'
# Use CoffeeScript for .coffee assets and views
gem 'coffee-rails', '~> 4.2'
# See https://github.com/rails/execjs#readme for more supported runtimes
gem 'therubyracer', platforms: :ruby
# Use jquery as the JavaScript library
gem 'jquery-rails'
# Turbolinks makes navigating your web application faster. Read more: https://github.com/turbolinks/turbolinks
gem 'turbolinks', '~> 5'
# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder
gem 'jbuilder', '~> 2.5'
# Use Redis adapter to run Action Cable in production
# gem 'redis', '~> 3.0'
# Use ActiveModel has_secure_password
# gem 'bcrypt', '~> 3.1.7'
# Use Capistrano for deployment
# gem 'capistrano-rails', group: :development
# jQuery-Turbolinks
gem 'jquery-turbolinks'
# Mysql
gem 'mysql2'
# Safe Attributes
gem 'safe_attributes'
# Elastic Search
gem 'elasticsearch-model'
gem 'elasticsearch-rails'
gem 'elasticsearch-persistence'
gem 'pry'
#rake
gem 'rake'
#sidekiq
gem 'sidekiq'
group :development, :test do
# Call 'byebug' anywhere in the code to stop execution and get a debugger console
gem 'byebug', platform: :mri
end
group :development do
# Access an IRB console on exception pages or by using <%= console %> anywhere in the code.
gem 'web-console'
end
group :production do
#passenger
gem "passenger", require: "phusion_passenger/rack_handler"
end
# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]
And here's obviously the error that I'm getting:
undefined method `+' for #<Elasticsearch::Model::Response::Results:0x0000000d2719f8>
Thanks!
Here after a lot of search, I've found this:
unless params[:steps].blank? || params[:scenario].blank? || params[:tags].blank?
query = Jbuilder.encode do |json|
json.query do
json.match do
json.scenario do
json.query params[:scenario]
end
end
params[:steps].each do |step|
json.match do
json.tags do
json.query step
end
end
end
json.match do
json.tags do
json.query params[:tags]
end
end
end
end
#results[:steps] = Steps.search(query).results
end
This help me to search more than one match on all my models.
I know that catch only the basics, but at the moment is part for what I need and I hope this could help all the others around here with the same issue!
Thanks you all!

'Cannot find memcached proc' error when running Phoenix app in Elixir

I am currently having a problem running the phoenix server application. This error is displayed when I tried to access 'localhost:4000'. It is complaining that it cannot find memcached proc on runtime as shown below.
[error] #PID<0.558.0> running MyApp.Endpoint terminated Server: localhost:4000 (http) Request: GET /
** (exit) an exception was raised:
** (RuntimeError) cannot find memcached proc
(plug_session_memcached) lib/plug_session_memcached.ex:130: Plug.Session.MEMCACHED.get/3
(plug) lib/plug/session.ex:74: anonymous fn/5 in Plug.Session.fetch_session/1
(plug) lib/plug/debugger.ex:211: Plug.Debugger.maybe_fetch_session/1
(plug) lib/plug/debugger.ex:174: Plug.Debugger.render/6
(plug) lib/plug/debugger.ex:153: Plug.Debugger.__catch__/5
(myapp) lib/myapp/endpoint.ex:1: MyApp.Endpoint.call/2
(plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
(cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4
Here are my dependencies:-
{:coherence, "~> 0.3"},
{:comeonin, "~> 2.4"},
{:cowboy, "~> 1.0"},
{:excoveralls, "~> 0.5", only: :test},
{:gettext, "~> 0.11"},
{:httpoison, "~> 0.10.0"},
{:lager, github: "basho/lager", tag: "3.2.4", override: true},
{:mailgun, "~> 0.1.2"},
{:mariaex, ">= 0.0.0"},
{:mcd, github: "EchoTeam/mcd"},
{:phoenix, "~> 1.2.1"},
{:phoenix_ecto, "~> 3.0"},
{:phoenix_html, "~> 2.6"},
{:phoenix_live_reload, "~> 1.0", only: :dev},
{:phoenix_pubsub, "~> 1.0"},
{:plug_session_memcached, github: "gutschilla/plug-session-memcached"},
{:timex, "~> 2.2.1"},
{:timex_ecto, "~> 1.1.3"}
It is most likely an issue with my Plug.Session memcached settings in endpoint.ex because when i switched to using :cookies as my store, it works as intended but not for :memcached. Any help would be highly appreciated.
This is the code where it throws the argument error in Plug.Session.MEMCACHED
#max_tries 100
def init(opts) do
Keyword.fetch!(opts, :table)
end
def get( conn, sid, table) do
case :mcd.get( table, sid ) do
{:error, :noproc} -> raise ArgumentError, "cannot find memcached proc"
{:error, :notfound} -> {nil, %{}}
{:error, :noconn} -> {nil, %{}}
{:ok, data } -> {sid, data}
end
end
def put( _conn, nil, data, table) do
put_new(data, table)
end
def put( _conn, sid, data, table) do
:mcd.set( table, sid, data )
sid
end
def delete( _conn, sid, table) do
:mcd.delete(table, sid)
:ok
end
defp put_new(data, table, counter \\ 0)
when counter < #max_tries do
sid = :crypto.strong_rand_bytes(96) |> Base.encode64
put( nil, sid, data, table )
end
You didn't start plug_session_memcached or its dependencies.
Adjust your mix.exs like this (Quote from the README)
def application do
[
...
applications: [
...
:lager, :corman # <-- add these both (mcd needs them)
],
included_applications: [
:plug_session_memcached # <--- add this entry
]
]
end

Guard + Spork slowing down tests, not receiving --drb option

I am developing a Rails project via a Vagrant box (Ubuntu 32-bit, Rails 4.0.3, Ruby 2.1.0p0).
I've just tried adding Spork to my project to speed up my tests (using RSpec, Capybara), and am seeing significantly slower tests. If I run simply "rspec .", all my tests pass in 5.83 seconds. However, when I run guard via "guard -c -p", I save one of my spec files, and I get a time of 26.08 seconds.
Note: I have to run "guard -p" to actually get guard to run my tests on file save through Vagrant.
When guard starts running the tests, it shows the args:
["--color", "--failure-exit-code", "2", "--format", "progress", "--format",
"Guard::RSpec::Formatter", "--require", "/home/vagrant/.rvm/gems/ruby-2.1.0/
gems/guard-rspec-4.2.7/lib/guard/rspec/formatter.rb", "spec"]...
I see that "--format" is listed twice, and "--drb" is not showing up at all.
Here's my Guardfile:
guard 'spork', :rspec_env => { 'RAILS_ENV' => 'test' }, :test_unit => false do
watch('config/application.rb')
watch('config/environment.rb')
watch('config/environments/test.rb')
watch(%r{^config/initializers/.+\.rb$})
watch('Gemfile.lock')
watch('spec/spec_helper.rb') { :rspec }
end
guard :rspec, :cmd => 'rspec --drb' do
watch(%r{^spec/.+_spec\.rb$})
watch(%r{^lib/(.+)\.rb$}) { |m| "spec/lib/#{m[1]}_spec.rb" }
watch('spec/spec_helper.rb') { "spec" }
# Rails example
watch(%r{^app/(.+)\.rb$}) { |m| "spec/#{m[1]}_spec.rb" }
watch(%r{^app/(.*)(\.erb|\.haml|\.slim)$}) { |m| "spec/#{m[1]}#{m[2]}_spec.rb" }
watch(%r{^app/controllers/(.+)_(controller)\.rb$}) { |m| ["spec/routing/#{m[1]}_routing_spec.rb", "spec/#{m[2]}s/#{m[1]}_#{m[2]}_spec.rb", "spec/acceptance/#{m[1]}_spec.rb"] }
watch(%r{^spec/support/(.+)\.rb$}) { "spec" }
watch('config/routes.rb') { "spec/routing" }
watch('app/controllers/application_controller.rb') { "spec/controllers" }
end
guard 'livereload' do
watch(%r{app/views/.+\.(erb|haml|slim)$})
watch(%r{app/helpers/.+\.rb})
watch(%r{public/.+\.(css|js|html)})
watch(%r{config/locales/.+\.yml})
# Rails Assets Pipeline
watch(%r{(app|vendor)(/assets/\w+/(.+\.(css|js|html|png|jpg))).*}) { |m| "/assets/#{m[3]}" }
end
Here is my spec_helper.rb:
require 'rubygems'
require 'spork'
#uncomment the following line to use spork with the debugger
#require 'spork/ext/ruby-debug'
Spork.prefork do
# Loading more in this block will cause your tests to run faster. However,
# if you change any configuration or code from libraries loaded here, you'll
# need to restart spork for it take effect.
# This file is copied to spec/ when you run 'rails generate rspec:install'
ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../../config/environment", __FILE__)
require 'rspec/rails'
require 'rspec/autorun'
require 'capybara/rspec'
# Requires supporting ruby files with custom matchers and macros, etc,
# in spec/support/ and its subdirectories.
Dir[Rails.root.join("spec/support/**/*.rb")].each { |f| require f }
# Checks for pending migrations before tests are run.
# If you are not using ActiveRecord, you can remove this line.
ActiveRecord::Migration.check_pending! if defined?(ActiveRecord::Migration)
RSpec.configure do |config|
config.use_transactional_fixtures = true
config.infer_base_class_for_anonymous_controllers = false
config.order = "random"
#Devise
config.include Devise::TestHelpers, type: :controller
#Capybara
config.include Capybara::DSL
end
end
Spork.each_run do
# This code will be run each time you run your specs.
end
The only thing in my .rspec :
--color
The relevant part of my Gemfile:
group :development, :test do
gem 'rspec-rails', '~> 2.0'
gem 'factory_girl_rails'
gem 'guard-rspec'
gem 'guard-livereload'
gem 'spork-rails'
gem 'guard-spork'
end
group :test do
gem 'shoulda-matchers'
gem 'capybara'
end
I've noticed that if I have guard running, then save a file, at times I will get an error:
Could not start Spork server for rspec after 30 seconds. I will continue
waiting for a further 60 seconds.
I'm not sure why it's taking so long after installing Spork, especially when rspec is much faster going through the same Vagrant box. Any ideas?
There's a new --listen-on option since Guard 2.5: https://github.com/guard/guard/releases/tag/v2.5.0
From Guard's README:
Use Listen's network functionality to receive file change events from the network. This is most useful for virtual machines (e.g. Vagrant) which have problems firing native filesystem events on the guest OS.
Suggested use:
On the host OS, you need to listen to filesystem events and forward them to your VM using the listen script:
$ listen -f 127.0.0.1:4000
Remember to configure your VM to forward the appropriate ports, e.g. in Vagrantfile:
config.vm.network :forwarded_port, guest: 4000, host: 4000
Then, on your guest OS, listen to the network events but ensure you specify the host path
$ bundle exec guard -o '10.0.2.2:4000' -w '/projects/myproject'

Guard Silently Exits When Using WDM Gem

Whenever I try to use wdm in conjunction with guard on Windows, guard silently exits. However, if I force polling, then guard works correctly. Probably the best way to illustrate this is with an example. This is what happens when I run guard with wdm:
C:/path/to/code>bundle exec guard init rspec
11:47:02 - INFO - rspec guard added to Guardfile, feel free to edit it
C:/path/to/code>bundle exec guard
11:48:06 - INFO - Guard is using TerminalTitle to send notifications.
11:48:06 - INFO - Guard::RSpec is running
11:48:06 - INFO - Guard is now watching at 'C:/path/to/code'
C:/path/to/code>
However, if I force guard to poll for changes, then it behaves correctly by ending up at a guard console:
C:/path/to/code>bundle exec guard -p
11:50:03 - INFO - Guard is using TerminalTitle to send notifications.
11:50:03 - INFO - Guard::RSpec is running
11:50:03 - INFO - Guard is now watching at 'C:/path/to/code'
[1] guard(main)>
I'm hoping that someone might have some light to shed on this problem. I have tried searching around quite a bit, but have not been able to find any answers to this question.
Relevant Excerpts from my Gemfile:
source 'http://rubygems.org'
ruby '1.9.3'
require "rbconfig"
gem 'rails', '3.2.13'
gem "figaro", ">= 0.5.3"
gem "rspec-rails", ">= 2.12.2", :group => [:development, :test]
gem "factory_girl_rails", ">= 4.2.0", :group => [:development, :test]
group :test do
gem "capybara", ">= 2.0.2"
gem "database_cleaner", ">= 0.9.1"
gem "email_spec", ">= 1.4.0"
gem "guard-rspec"
gem "wdm", ">= 0.1.0" if RbConfig::CONFIG["target_os"] =~ /mswin|mingw|cygwin/i
end
My Guardfile:
# A sample Guardfile
# More info at https://github.com/guard/guard#readme
guard :rspec do
watch(%r{^spec/.+_spec\.rb$})
watch(%r{^lib/(.+)\.rb$}) { |m| "spec/lib/#{m[1]}_spec.rb" }
watch('spec/spec_helper.rb') { "spec" }
# Rails example
watch(%r{^app/(.+)\.rb$}) { |m| "spec/#{m[1]}_spec.rb" }
watch(%r{^app/(.*)(\.erb|\.haml|\.slim)$}) { |m| "spec/#{m[1]}#{m[2]}_spec.rb" }
watch(%r{^app/controllers/(.+)_(controller)\.rb$}) { |m| ["spec/routing/#{m[1]}_routing_spec.rb", "spec/#{m[2]}s/#{m[1]}_#{m[2]}_spec.rb", "spec/acceptance/#{m[1]}_spec.rb"] }
watch(%r{^spec/support/(.+)\.rb$}) { "spec" }
watch('config/routes.rb') { "spec/routing" }
watch('app/controllers/application_controller.rb') { "spec/controllers" }
# Capybara features specs
watch(%r{^app/views/(.+)/.*\.(erb|haml|slim)$}) { |m| "spec/features/#{m[1]}_spec.rb" }
# Turnip features and steps
watch(%r{^spec/acceptance/(.+)\.feature$})
watch(%r{^spec/acceptance/steps/(.+)_steps\.rb$}) { |m| Dir[File.join("**/#{m[1]}.feature")][0] || 'spec/acceptance' }
end
I don't have an answer to this, but I thought I might collect some of the issues that relate to it.
The primary issue is Problems on Windows in the listen gem.
There is a related issue on guard is Guard 2.0.3 daemon doesn't stay running on windows which was closed in favour of the listen issue.
The wdm gem also has an open issue: fail with listen gem.
Sorry I can't solve it, but at least there are on-going issues for this problem. The few people that work on Ruby for Windows are such gems! I apologise for the pun :)

Resources