Getting DB Configuration from require - ruby

Right now I've got a simple Sinatra app that connects to a DB.
Right now I have the connection strings hard-coded inside the repo itself. My aim is to return the connection information from a different file (which will eventually be outside the repo).
This is what I've tried so far:
app.rb
class MySinatraApp < Sinatra::Application
configure do
conf = require_relative 'configuration'
# set public dirs, sessions, etc.
set :mongourl, conf[:mongourl]
set :mongodb, conf[:mongodb]
end
end
...And finally in the configuration.rb
env = ENV['RACK_ENV'].to_sym
case env
when :production
return {
"mongourl" => 'mongodb://localhost/prod',
"mongodb" => 'prod'
}
when :development
return {
"mongourl" => 'mongodb://localhost/dev',
"mongodb" => 'prod'
}
else
return {
"mongourl" => 'mongodb://localhost/test',
"mongodb" => 'test'
}
end
The idea would be to just return whatever hash I need to use, however when I attempt the above, it errors out with these errors:
<my_project_dir>/configuration.rb: <my_project_dir>/configuration.rb:9: Invalid return (SyntaxError)
<my_project_dir>/configuration.rb:16: Invalid return
<my_project_dir>/configuration.rb:21: Invalid return
From the error, you are unable to return the way I'm doing it now. I am assuming this is because it isn't in a function call of some sort? How do people normally include configuration files in Ruby/ Sinatra?

You’re right that you can’t do this because you can’t return from a required file in this way (at least not at the moment). Your solution is using a top level constant rather than a global (global variables look like $foo, constants look like Foo or FOO) but it pretty much amounts to the same thing.
A common solution to doing configuration like this in Sinatra is with ConfigFile from Sinatra-contrib.
Create a config Yaml file:
production:
mongourl: mongodb://localhost/prod
mongodb: prod
develepment:
mongourl: mongodb://localhost/dev
mongodb: prod
test:
mongourl: mongodb://localhost/test
mongodb: test
Then in your Sinatra file (install the sinatra-contrib gem first):
require 'sinatra'
require 'sinatra/config_file'
config_file 'path/to/config.yml'
The settings from the config file will now be available according to the whatever environment is running.
Another way to do this kind of thing is to put all the configuration data into environment variables and then access them with ENV. Doing this prevents you from accidentally checking in production credentials to your version control, but it means you have to manage all the config vars somehow. Have a look at dotenv or Foreman for tools that can help with this if you want to go this way.

If anyone has a better way of including DB connection strings, please share!
I'm not going to say this is a better way, but I do it this way because it's simple.
More or less it's the same as sinatra/config_file.
So you have your config file with the different values for each environment
$cat config.yaml
development:
mongourl: mongodb://localhost/test
mongodb: dev
production:
mongourl: mongodb://localhost/test
mongodb: prod
And now just use it:
configure do
env = ENV['RACK_ENV']
AppConfig = YAML.load_file(File.expand_path("../config.yaml", File.dirname(__FILE__)))[env]
set :mongourl, AppConfig['mongourl']
set :mongodb, AppConfig['mongodb']
end

It would seem I can just declare a variable in the root of the require'ed file, it automatically is global (I think). In either case, this is how I changed the files to work:
app.rb
class MySinatraApp < Sinatra::Application
configure do
require_relative 'configuration'
# set public dirs, sessions, etc.
set :mongourl, CONFIG["mongourl"]
set :mongodb, CONFIG["mongodb"]
end
end
configuration.rb
env = ENV['RACK_ENV'].to_sym
case env
when :production
CONFIG = {
"mongourl" => 'mongodb://localhost/prod',
"mongodb" => 'prod'
}
when :development
CONFIG = {
"mongourl" => 'mongodb://localhost/dev',
"mongodb" => 'prod'
}
else
CONFIG = {
"mongourl" => 'mongodb://localhost/test',
"mongodb" => 'test'
}
end
Not a super-big fan of having connection strings being in a global variable, but it will work for now I think. If anyone has a better way of including DB connection strings, please share!

Related

Simple Ruby server setup using Thin as a simple rhtml server (like Webrick)

my simple Webrick server serves up static html and rhtml embedded ruby files. How would I achieve the same with a multithreaded or multiprocess Ruby server like Thin?
Webrick setup:
#!/usr/bin/ruby
# simple_servlet_server.rb
require 'webrick'
include WEBrick
s = HTTPServer.new(:Port => 8000)
# Add a mime type for *.rhtml files
HTTPUtils::DefaultMimeTypes.store('rhtml', 'text/html')
s.mount('/', HTTPServlet::FileHandler, './public_html')
['TERM', 'INT'].each do |signal|
trap(signal){ s.shutdown }
end
s.start
I have installed Thin and Rack, my config.ru reads HTML but wont render rhtml docs?:
use Rack::Static,
:urls => ["/images", "/js", "/css"],
:root => "public"
run lambda { |env|
[
200,
{
'Content-Type' => 'text/html',
'Cache-Control' => 'public, max-age=86400'
},
File.open('./public_html', File::RDONLY)
]
HTTPUtils::DefaultMimeTypes.store('rhtml', 'text/html')
}
TL;DR; - Dynamic ruby content (i.e., .rhtml files) needs to be rendered by a template engine; The built-in engine used by WebRick is Ruby's ERB engine; Render dynamic files before sending the Rack response.
I would recommend that you separate the static files from the dynamic files.
This would allow you to optimize the static file service using either the reverse proxy (i.e., nginx) or a static file enabled Ruby server (i.e., iodine).
A common approach is to:
Place static files under ./public.
Place dynamic files under ./app/views.
Your Rack application will need to process the dynamic files using the appropriate template engine processor.
For your .rhtml files I would assume that this would be the built-in ERB template engine (there's a nice post about it here).
Assuming you placed your dynamic files and static files as mentioned, you might start up with a config.ru file that looks like this:
require 'erb'
require 'pathname'
ROOT ||= Pathname.new(File.dirname(__FILE__)).expand_path
module APP
VIEWS = ROOT.join('app', 'views')
def self.call(env)
# prevent folder trasversal (security) and extract file name
return [403, {}, ["Not Allowed!"]] if env['PATH_INFO'].index("..") || env['PATH_INFO'].index("//")
filename = VIEWS.join(env['PATH_INFO'][1..-1])
# make sure file exists
return [404, {}, ["Not Found"]] unless File.exists?(filename)
# process dynamic content
template = IO.binread filename
data = ERB.new(template).result(binding)
return [200, {}, [data]]
end
end
run APP
Next you can run the application with iodine, which will handle the static file service part (in this example, a single threaded worker per core):
iodine -w -1 -t 1 -www public
Of course, you can use the Rack::Static middleware, but it should prove to be significantly slower (bench mark it yourself or test it against Thin)...
...I'm iodine's author, so I might be biased.
EDIT
P.S. (side note about security and performance)
I would reconsider the template engine.
ERB is fast and effective, however, it also allows code to be executed within the template.
This could result in difficult project maintenance as code leaks into the template, making the code less readable and harder to maintain.
I would consider switching to Mustache templates, which prevent code from running within the template.
Changing the template engine could also improve performance. Consider the following benchmark, using iodine flavored mustache templates (which offer aggressive HTML escaping):
require 'iodine'
require 'erb'
require 'benchmark'
module Base
ERB_ENG = ERB.new("<%= data %> <%= Time.now %>")
MUS_ENG = Iodine::Mustache.new(nil, "{{ data }} {{ date }}")
def self.render_erb(data)
ERB_ENG.result(binding)
end
def self.render_mus(data)
h = {data: data, date: Time.now.to_s }
MUS_ENG.render(h)
end
end
puts "#{Base.render_mus("hello")} == #{Base.render_erb("hello")} : #{(Base.render_mus("hello") == Base.render_erb("hello"))}"
TIMES = 100_000
Benchmark.bm do |bm|
bm.report("Ruby ERB") { TIMES.times { Base.render_erb("hello") } }
bm.report("Iodine ") { TIMES.times { Base.render_mus("hello") } }
end
The results, on my machine, were (lower is better):
user system total real
Ruby ERB 1.701363 0.006301 1.707664 ( 1.709132)
Iodine 0.256918 0.000750 0.257668 ( 0.258190)
Again, since I'm iodine's author, I'm biased. Play around and find what's best for you.

Cannot get rspec to test both parts of an if else puppet statement

I am writing unit tests for our puppet code (puppet 3.8). I have a variable set by data in hiera. For example, I have this code in puppet:
# globals value coming from hiera
$status = $globals['yum']['status']
if $status =~ /on/ {
service { 'yum-cron':
ensure => 'running',
enable => true,
hasrestart => true,
require => [ Package['yum-cron'], File['/var/lock/subsys/'] ]
}
} else {
service { 'yum-cron':
ensure => 'stopped',
enable => false,
hasrestart => true,
require => Package['yum-cron'],
}
file {'/var/lock/subsys/yum-cron':
ensure => 'absent',
require => Package['yum-cron'],
}
}
In my rspec test file, I have the following code to test both the parts of the if/else:
context 'If the globals yum status = on' do
it 'The service resource yum-cron should exist' do
is_expected.to contain_service('yum-cron').with(
ensure: 'running',
enable: true,
hasrestart: true,
require: ['Package[yum-cron]', 'File[/var/lock/subsys/]' ]
)
end
end
context 'If the globals yum status = off' do
let(:status) {'off'}
it 'The service resource yum-cron should NOT exist' do
is_expected.to contain_service('yum-cron').with(
ensure: 'stopped',
enable: false,
hasrestart: true,
require: 'Package[yum-cron]'
)
end
end
No matter what I do in my xxx_setup.rb file to test both parts of the if/else statement, only the part that matches the value coming from hiera tests successfully. Because the value from hiera sets the value of $status to "on", that section evaluates successfully in the rspec test code. But the section where I try to test for the value of $status to be "off" fails no matter how I try to set the value of the status variable in rspec. When the puppet catalog is generated, it seems to generate only the section that matches what is in hiera and not what I set the $status variable to in rspec.
What am I missing?
let(:status) in your rspec code is just setting a local variable status, it is not setting the global $status. What's more, your puppet code is setting that $status global at the top of the file, so even if you could set it in your rspec code, it would be overwritten.
You say $globals is getting it's values from hiera. I've never used that before, but if you're using rspec-puppet gem, it looks like you can define the path to your hiera yaml file. So you could then possibly overwrite the value after that, or have separate hiera yaml files for each test.
Many thanks to #supremebeing7 for setting me on the right path. I created a second hiera yaml file which contained alternate values for which I wanted to test in my rspec code. I added the following code to the section in my rspec file where I needed to test with the alternate value:
context 'If the globals yum status = off' do
let(:hiera_config) {
'spec/fixtures/alt_hiera/hiera.yaml' }
hiera = Hiera.new({ :config =>
'spec/fixtures/alt_hiera/hiera.yaml' })
globals = hiera.lookup('globals', nil, nil)
This alternate hiera file had the value of "off" set for $globals['yum']['status'] and my test passed.

sinatra app on puma server set to port 80 via non-bootstraped executable

I'm trying to bind my app to port 80, it binds to '0.0.0.0' fine but using
set :port,80
doesn't change port its set to.
I can't use anything that requires me to externally set the port because I'm starting my app from inside an executable like this:
if options[:daemonize]
use Rack::Deflater
pid = fork {Rack::Handler::pick(['puma']).run VCK::App}
File.open(options[:pid], 'w') {|f| f.write(pid)}
Process.detach(pid)
else
use Rack::Deflater
Rack::Handler::pick(['puma']).run VCK::App
end
I've looked but beyond using set I can't find how else to set the port.
I'm using chef to provision the application and start it which should have super user access.
P.s. Just in case it wasn't obvious, when I say non-bootstraped I mean by not doing a system command that does runs rackup or something.
After digging away through both rack and puma source code I was able to get an answer.
You just pass it in when you use .run like this:
Rack::Handler::pick(['puma']).run VCK::App, {:Port => 80}
so my code ends up like this:
server_config = {
:Host => options[:host],
:Port => options[:port],
:Threads => "0:#{options[:threads]}",
:Verbose => options[:verbose],
:environment => options[:environment]
}
if options[:daemonize]
use Rack::Deflater
pid = fork {Rack::Handler::pick(['puma']).run VCK::App,server_config}
File.open(options[:pid], 'w') {|f| f.write(pid)}
Process.detach(pid)
else
use Rack::Deflater
Rack::Handler::pick(['puma']).run VCK::App
end
which in hind-site is kind of obvious.
Also I'd like to state that this line I found in the rack handler code to retrieve the server object is VERY clever:
klass.split("::").inject(Object) { |o, x| o.const_get(x) }

compass/sass remote themeing via sftp/scp with alternate port

I am trying to get compass/sass to watch changes on my local computer and reflect those changes remotely using a custom config.rb script. net::sftp works but my server requires a custom ssh port. I couldn't find any mods to make sftp work with an alternate port so im trying net:scp now, the problem is I dont know the proper command structure to upload using net:scp and wanted to see if someone can help me. Here is my code:
# Require any additional compass plugins here.
require 'net/ssh'
require 'net/scp'
# SFTP Connection Details - Does not support alternate ports os SSHKeys, but could with mods
remote_theme_dir_absolute = '/home2/trinsic/public_html/scottrlarson.com/sites/all/themes/ gateway_symbology_zen/css'
sftp_host = 'xxx.xxx.xxx.xxx' # Can be an IP
sftp_user = 'user' # SFTP Username
sftp_pass = 'password' # SFTP Password
# Callback to be used when a file change is written. This will upload to a remote WP install
on_stylesheet_saved do |filename|
$local_path_to_css_file = css_dir + '/' + File.basename(filename)
Net::SSH.start( sftp_host, sftp_user, {:password => sftp_pass,:port => 2222} ) do ssh.scp.upload! $local_path_to_css_file, remote_theme_dir_absolute + '/' + File.basename(filename)
end
puts ">>>> Compass is polling for changes. Press Ctrl-C to Stop"
end
#
# This file is only needed for Compass/Sass integration. If you are not using
# Compass, you may safely ignore or delete this file.
#
# If you'd like to learn more about Sass and Compass, see the sass/README.txt
# file for more information.
#
# Change this to :production when ready to deploy the CSS to the live server.
environment = :development
#environment = :production
# In development, we can turn on the FireSass-compatible debug_info.
firesass = false
#firesass = true
# Location of the theme's resources.
css_dir = "css"
sass_dir = "sass"
extensions_dir = "sass-extensions"
images_dir = "images"
javascripts_dir = "js"
# Require any additional compass plugins installed on your system.
#require 'ninesixty'
#require 'zen-grids'
# Assuming this theme is in sites/*/themes/THEMENAME, you can add the partials
# included with a module by uncommenting and modifying one of the lines below:
#add_import_path "../../../default/modules/FOO"
#add_import_path "../../../all/modules/FOO"
#add_import_path "../../../../modules/FOO"
##
## You probably don't need to edit anything below this.
##
# You can select your preferred output style here (can be overridden via the command line):
# output_style = :expanded or :nested or :compact or :compressed
output_style = (environment == :development) ? :expanded : :compressed
# To enable relative paths to assets via compass helper functions. Since Drupal
# themes can be installed in multiple locations, we don't need to worry about
# the absolute path to the theme from the server root.
relative_assets = true
# To disable debugging comments that display the original location of your selectors. Uncomment:
# line_comments = false
# Pass options to sass. For development, we turn on the FireSass-compatible
# debug_info if the firesass config variable above is true.
sass_options = (environment == :development && firesass == true) ? {:debug_info => true} : {}
I get an error when I run the command: compass watch:
NoMethodError on line ["17"] of K: undefined method `upload!' for #<Net::SSH::Co
nnection::Session:0x000000036bb220>
Run with --trace to see the full backtrace
I needed a solution for this too but did not find any satisfying answer anywhere.
After reading the Ruby Net::ssh documentation and some source of Compass, this is my solution to upload CSS and sourcemap to a remote SSH server with non-standard port and forced public-key authorisation:
First make sure you have the required gems installed
sudo gem install net-ssh net-sftp
then add this to your config.rb
# Add this to the first lines of your config.rb
require 'net/ssh'
require 'net/sftp'
...
# Your normal compass config comes here
...
# At the end of your config.rb add the config for the upload code
remote_theme_dir_absolute = '/path/to/my/remote/stylesheets'
sftp_host = 'ssh_host' # Can be an IP
sftp_user = 'ssh_user' # SFTP Username
on_stylesheet_saved do |filename|
# You can use the ssh-agent for authorisation.
# In this case you can remove the :passphrase from the config and set :use_agent => true.
Net::SFTP.start(
sftp_host,
sftp_user ,
:port => 10022,
:keys_only => true,
:keys => ['/path/to/my/private/id_rsa'],
:auth_methods => ['publickey'],
:passphrase => 'my_secret_passphrase',
:use_agent => false,
:verbose => :warn
) do |sftp|
puts sftp.upload! css_dir + '/app.css', remote_theme_dir_absolute + '/' + 'app.css'
end
end
on_sourcemap_saved do |filename|
# You can use the ssh-agent for authorisation.
# In this case you can remove the :passphrase from the config and set :use_agent true.
Net::SFTP.start(
sftp_host,
sftp_user ,
:port => 10022,
:keys_only => true,
:keys => ['/path/to/my/private/id_rsa'],
:auth_methods => ['publickey'],
:passphrase => 'my_secret_passphrase',
:use_agent => false,
:verbose => :warn
) do |sftp|
puts sftp.upload! css_dir + '/app.css.map', remote_theme_dir_absolute + '/' + 'app.css.map'
end
end
It was quite some trial and error until this worked for me.
Some points of failure were:
If no ssh-agent is available connection will fail until you set :ssh_agent => false explicitly
If you do not limit the available keys with :keys all available keys will be tried one after another. If you use the ssh-agent and have more than 3 keys installed chanches are high that the remote server will close the connection if you try too much keys that are not valid for the server you currently connect.
On any connection issue set verbosity level to :verbose => :debug to see what is going on. Remember to stop the compass watch and restart to ensure configuration changes apply.

Ruby backup gem - shared configurations?

I'm using meskyanichi's backup gem. By and large it does what I need it to, but I need to have multiple backups (e.g., hourly, daily, weekly). The configurations are mostly the same but have a few differences, so I need to have multiple configuration files. I'm having trouble finding a sane way to manage the common bits of configurations (i.e., not repeat the common parts).
The configuration files use a lot of block structures, and from what I can tell, each backup needs to have a separate config file (e.g. config/backup/hourly.rb, config/backup/daily.rb, etc). A typical config file looks like this:
Backup::Model.new(:my_backup, 'My Backup') do
database MySQL do |db|
db.name = "my_database"
db.username = "foo"
db.password = "bar"
# etc
end
# similar for other config options
end
Then the backup is executed a la bundle exec backup perform -t my_backup -c path/to/config.rb.
My first swag at enabling a common config was to define methods that I could call from the blocks:
def my_db_config db
db.name = "my_database"
# etc
end
Backup::Model.new(:my_backup, 'My Backup') do
database MySQL do |db|
my_db_config db
end
#etc
end
But this fails with an undefined method 'my_db_config' for #<Backup::Database::MySQL:0x10155adf0>.
My intention was to get this to work and then split the common config functions into another file that I could require in each of my config files. I also tried creating a file with the config code and requireing it into the model definition block:
# common.rb
database MySQL do |db|
db.name = "my_database"
#etc
end
# config.rb
Backup::Model.new(:my_backup, 'My Backup') do
require "common.rb" # with the right path, etc
end
This also doesn't work, and from subsequent research I've discovered that that's just not the way that require works. Something more in line with the way that C/C++'s #include works (i.e., blindly pasting the contents into whatever scope it is called from) might work.
Any ideas?
The gem seems to modify the execution scope of the config blocks. To work around this, you could wrap your functions in a class:
class MyConfig
def self.prepare_db(db)
db.name = "my_database"
# etc
db
end
end
Backup::Model.new(:my_backup, 'My Backup') do
database MySQL do |db|
db = MyConfig.prepare_db(db)
end
#etc
end
You could get a bit more fancy and abstract your default config merge:
class BaseConfig
##default_sets =
:db => {
:name => "my_database"
},
:s3 => {
:access_key => "my_s3_key"
}
}
def self.merge_defaults(initial_set, set_name)
##default_sets[set_name].each do |k, v|
initial_set.send("#{k}=".to_sym, v)
end
initial_set
end
end
Backup::Model.new(:my_backup, 'My Backup') do
database MySQL do |db|
db = BaseConfig.merge_defaults(db, :db)
end
store_with S3 do |s3|
s3 = BaseConfig.merge_defaults(s3, :s3)
end
end
In newest versions of backup gem you can simple use main config file like this:
Genrate main config file:
root#youhost:~# backup generate:config
Modify file /root/Backup/config.rb like this:
Backup::Storage::S3.defaults do |s3|
s3.access_key_id = "youkey"
s3.secret_access_key = "yousecret"
s3.region = "us-east-1"
s3.bucket = "youbacket"
s3.path = "youpath"
end
Backup::Database::PostgreSQL.defaults do |db|
db.name = "youname"
db.username = "youusername"
db.password = "youpassword"
db.host = "localhost"
db.port = 5432
db.additional_options = ["-xc", "-E=utf8"]
end
Dir[File.join(File.dirname(Config.config_file), "models", "*.rb")].each do |model|
instance_eval(File.read(model))
end
Create model file:
root#youhost:~# backup generate:model --trigger daily_backup \
--databases="postgresql" --storages="s3"
Then modify /root/Backup/models/daily_backup.rb like this:
# encoding: utf-8
Backup::Model.new(:daily_backup, 'Description for daily_backup') do
split_into_chunks_of 250
database PostgreSQL do |db|
db.keep = 20
end
store_with S3 do |s3|
s3.keep = 20
end
end
With this you can simply create daily, monthly or yearly archives.

Resources