Inserting an Array into Postgres Database from a Ruby File - ruby

I'm trying to transfer all the information from my Ruby file into a Postgres database. I am able to transfer the information when I do not have an array column, so I am assuming the error message I am getting is because of the array column I am trying to add. The error message I am getting is:
in `exec_prepared': ERROR: missing dimension value (PG::InvalidTextRepresentation)
Here is the code I used to connect my Ruby file to my Postgres database:
require 'pg'
class Postgres
# Create the connection instance. Scraping is the name of the database I am adding this information to
def connect
#conn = PG.connect(:dbname => 'scraping')
end
# Create our venue table
def createVenueTable
#conn.exec("CREATE TABLE venues (venue_number varchar(15) UNIQUE,...,img_array varchar[]);")
end
...
def prepareInsertVenueStatement
#conn.prepare("insert_venue", "insert into venues(venue_number,...,img_array) values ($1,...,$24)")
end
# Add a venue with the prepared statement.
def addVenue(venue_number,...,img_array)
#conn.exec_prepared("insert_venue", [venue_number,...,img_array])
end
end
When I check my Postgres database, the img_array column is made, however, I am unable to populate it. Please help! Thank you.

I would suggest using serialization to handle this so that you are actually just writing a string rather than an actual array.
require 'pg'
require 'yaml'
class Postgres
# Create the connection instance. Scraping is the name of the database I am adding this information to
def connect
#conn = PG.connect(:dbname => 'scraping')
end
# Create our venue table
def createVenueTable
#changed img_array to a varchar(8000) for storing serialized Array
#conn.exec("CREATE TABLE venues (venue_number varchar(15) UNIQUE,...,img_array varchar(8000));")
end
...
def prepareInsertVenueStatement
#conn.prepare("insert_venue", "insert into venues(venue_number,...,img_array) values ($1,...,$24)")
end
# Add a venue with the prepared statement.
def addVenue(venue_number,...,img_array)
#conn.exec_prepared("insert_venue", [venue_number,...,serialized(img_array)])
end
#serialize the Object
def serialized(obj)
YAML.dump(obj)
end
#deserialize the Object
def deserialized(obj)
YAML.load(obj)
end
end
Abstracted Usage Example just to show serialization
a = [1,2,4,5]
serialized = YAML.dump(a)
#=> "---\n- 1\n- 2\n- 3\n- 4\n- 5\n"
YAML.load(serialized)
#=> [1,2,3,4,5]
#Also works on Hash Objects
h = {name: "Image", type: "jpeg", data:[1,2,3,4,5]}
serial = YAML.dump(h)
#=> "---\n:name: Image\n:type: jpeg\n:data:\n- 1\n- 2\n- 3\n- 4\n- 5\n"
YAML.load(serial)
#=> {:name=>"Image", :type=>"jpeg", :data=>[1, 2, 3, 4, 5]}
Hope this helps you out with handling this issue.
If you need to store more than 8000 characters you can switch to varchar(MAX) or text column definitions. I would recommend varchar(MAX) because the data will be stored as a standard varchar until it exceeds 8000 character at which point the db basically converts it to a text column under the hood.

Related

how to fetch WordPress Data Base Table via Ruby script with dynamic active record classes?

I currently have a website that is built in WordPress, and have over 1000 customers registered. Would I be able to extract the information from the WordPress db in ruby code
Need way that's seamless for my customers? Without requiring registering again?
Note:
This code should create active record classes or models dynamically instead of manually defining active record models
With out having the Gem You can achieve this via Ruby script itself
I assumed you have rails application from there you can create rake task to fetch the data from WordPress Database
Step 1:
value = {host: "192.*.*.*", username: '', password: '', database: ''}
ActiveRecord::Base.establish_connection(
adapter: 'mysql2',
encoding: 'utf8',
pool: 5,
host: value[:host],
username: value[:username],
password: value[:password],
database: value[:database]
)
Step 2:
Then Define the database tables needs to be fetched
database_tables = {:"database_name" => ["SaveContactForm7_1", "SaveContactForm7_2","SaveContactForm7_3","SaveContactForm7_4"]}
Step 3:
tables = database_tables[key]
Key refers to the database name
Step 4:
tables.each do |table|
MODEL_CLASS= table
Object.const_set(MODEL_CLASS, Class.new(ActiveRecord::Base) { def self.name() MODEL_CLASS end;def self.table_name() MODEL_CLASS end })
records = MODEL_CLASS.constantize.all
results = []
records.each do |record|
set = {}
columns = MODEL_CLASS.constantize.column_names
p columns
columns.each do |column|
p record.send(column.to_sym)
p set[:mobile] = record.send(column.to_sym)
end
results << set
p record
end
p "Task done...."
p results
end
As a result you can see the array of hashes. you can insert into your active record models
Please Feel free to give your suggestion for this

Ruby activerecord result to array

My aim is to take the result of my activerecord search and print it into a nice array but the print part is where I am having trouble.
I first build my oracle connection with the following which works in isolation.
def oracle_connection(adapter, database, username, password)
begin
ActiveRecord::Base.establish_connection(
adapter: adapter,
database: database,
username: username,
password: password)
end
end
I then create my query with the following function:
def query
"select * from owner.appn where appn_id = #{$id}"
end
And here is the part where I am asking the question on. I want to pass the result of the query being returned out into an 2D array. Below is what I currently have to execute the active connection query.
def oracle_query_into_array(query)
result_set = ActiveRecord::Base.connection.execute(query)
if result_set.present?
#add logic here
else
return nil
end
end
Thanks
I'm assuming you have reasons to use the underlying connection calls rather than the abstractions that are common practise.
With the ActiveRecord::Base.connection.execute(query) I would expect this to return true if it executes. What you want is a cursor on the data, so try this:
result = ActiveRecord::Base.connection.exec_query(query)
puts result.to_a
=> [array of results]
A usual abstraction (ActiveRecord::Base) would take the form of creating a model to represent your data, so in your case, this could look like:
class Appn < ActiveRecord::Base
end
This will be automatically mapped to a table within your connection called Appnn allowing you to update the above code to:
results = Appn.where(appn_id: $id)
puts results.to_a
=> [array of results]

What's the best way to populate foreign keys using a rake task?

I need to import a big csv file into a Rails project. I'm using:
ruby 2.1.2p95
Rails 4.1.1
mysql Ver 14.14
I'm trying to do this as a rake task, by first creating records in the independent tables, get the record id's for those records, and use these for the foreign keys to populate records in the dependent tables.
I'm trying to create just the independent records, and print out these foreign id's, operating on a small test file first.
However, I'm getting this end-of-file I don't understand:
rake aborted!
SyntaxError: /Users/rickcasey/Projects/Programming/WikiFrac/wfrails/lib/tasks/import_partial.rake:94:
syntax error, unexpected keyword_end, expecting end-of-input
This is what my rake script looks like:
#lib/tasks/import_partial.rake
require 'csv'
# Independent tables:
# Companies
# Counties
# Fields
# Formations
# Gastypes
# Wells
#
# Dependendecies and foreign key field used to find correct record id:
# Facilities.company_id -> Companies.company_name
# Facilities.field_id -> Fields.field_name
# Facilities.county_id -> Counties.county_name
# Wells.gastype_id -> GasTypes.gas_type
task :import_partial => :environment do
csv.foreach('public/partial.csv', :headers => true) do |row|
# create records in independent tables
# create the Company object
this_company_name = row.to_hash.slice(*%w[county_name])
if !(Company.exists?(company_name: this_company_name))
Companies.create(row.to_hash.slice(*%w[company_name operator_num]))
end
thecompany = Company.find(this_company_name)
company_id = thecompany.id
# create the County object
this_county_name = row.to_hash.slice(*%w[county])
if !(County.exists?(county_name: this_county_name))
Counties.create(county_name: this_county_name)
end
thecounty = County.find(this_county_name)
county_id = thecounty.id
# create the GasType object
this_gastype_name = row.to_hash.slice(*%w[gas_type])
if !(GasType.exists?(gastype_name: this_gastype_name))
GasType.create(gastype_name: this_gastype_name)
end
thegastype = GasType.find(this_gastype_name)
gastype_id = thegastype.id
# create the Field object
this_field_name = row.to_hash.slice(*%w[field])
if !(Field.exists?(field_name: this_field_name))
Field.create(field_name: this_field_name, field_code: field_code)
end
thefield = Field.find(this_field_name)
field_id = thefield.id
# create the Formations object
this_formation_name = row.to_hash.slice(*%w[formation])
if !(Formation.exists?(formation_name: this_formation_name))
Counties.create(formation: this_formation_name, formation_code: formation_code)
end
theformation = Formation.find(this_formation_name)
formation_id = theformation.id
# debugging:
puts "company_id:", company_id
puts "county_id:", county_id
puts "gastype_id:", gastype_id
puts "field_id:", field_id
# create records in dependent tables:
# Use the record id's from above independent table create records containing foreign keys:
#Facilities.create(row.to_hash.slice(*%w[dir_e_w dir_n_s dist_e_w dist_n_s facility_name facility_num ground_elev lat long meridian qtrqtr range sec twp utm_x utm_y])
#Wells.create(row.to_hash.slice(*%w[api_county_code api_seq_num first_prod_date form_status_date formation_status sidetrack_num spud_date status_date td_date test_date wbmeasdepth wbtvd well_bore_status well_name])
end
end
end
Any suggestions much appreciated...
You have 1 too many end statements at the end of your file.

cassandra Ruby : multiple values for a block parameter (2 for 1)

I am trying to follow a tutorial on big data, it wants to reads data from a keyspace defined with cqlsh.
I have compiled this piece of code successfully:
require 'rubygems'
require 'cassandra'
db = Cassandra.new('big_data', '127.0.0.1:9160')
# get a specific user's tags
row = db.get(:user_tags,"paul")
###
def tag_counts_from_row(row)
tags = {}
row.each_pair do |pair|
column, tag_count = pair
#tag_name = column.parts.first
tag_name = column
tags[tag_name] = tag_count
end
tags
end
###
# insert a new user
db.add(:user_tags, "todd", 3, "postgres")
db.add(:user_tags, "lili", 4, "win")
tags = tag_counts_from_row(row)
puts "paul - #{tags.inspect}"
but when I write this part to output everyone's tags I get an error.
user_ids = []
db.get_range(:user_tags, :batch_size => 10000) do |id|
# user_ids << id
end
rows_with_ids = db.multi_get(:user_tags, user_ids)
rows_with_ids.each do |row_with_id|
name, row = row_with_id
tags = tag_counts_from_row(row)
puts "#{name} - #{tags.inspect}"
end
the Error is:
line 33: warning: multiple values for a block parameter (2 for 1)
I think the error may have came from incompatible versions of Cassandra and Ruby. How to fix it?
Its a little hard to tell which line is 33, but it looks like the problem is that get_range yields two values, but your block is only taking the first one. If you only care about the row keys and not the columns then you should use get_range_keys.
It looks like you do in fact care about the column values because you fetch them out again using db.multi_get. This is an unnecessary additional query. You can update your code to something like:
db.get_range(:user_tags, :batch_size => 10000) do |id, columns|
tags = tag_counts_from_row(columns)
puts "#{id} - #{tags.inspect}"
end

Writing to excel files in ruby using roo gem

I am parsing Excel and Excelx file using Roo gem. But I am not sure how to write in those files. set_value(row, column, text) method is not working.
Code
#oo = Excelx.new('tes.xlsx')
#oo.default_sheet = #oo.sheets.first
def return_column
keywords = ["website", "url"]
keywords.each do |keyword|
1.upto(#oo.last_column) do |n|
data = #oo.cell(1, n)
return n if data.downcase=~/#{keyword}/i
end
end
end
def return_rows
n = return_n
2.upto(#oo.last_row) do |row|
data = #oo.cell(row, n)
stack << data
end
end
def appender
#oo.set_value(1,11, "hey")
end
appender
The Error Message I'm getting is
/.rvm/gems/ruby-1.8.7-p352/gems/roo-1.10.1/lib/roo/generic_spreadsheet.rb:441:in `method_missing': private method `set_value' called for #<Excelx:0x101221f08> (NoMethodError)
from /Users/bhushan/.rvm/gems/ruby-1.8.7-p352/gems/roo-1.10.1/lib/roo/excelx.rb:168:in `method_missing'
from parser.rb:32:in `appender'
from parser.rb:35
No answers here actually answer the question of how to do this with Roo, so I'll add the solution that I just tested in our app.
Roo recently added functionality for editing cells: https://github.com/roo-rb/roo/blob/master/lib/roo/csv.rb#L42
You can use it like such:
sheet.set_value(1, 5, 'TEST', nil) # to set the 1st row, 5th column to the string 'TEST'
Notes:
The last argument nil is not used in the function but has no default so it's required.
This is only added in version 2.7.0.
Try 'set' method instead of 'set_value' method in Excelx or OpenOffice object. For more refer API http://rubydoc.info/gems/roo/1.10.1/frames and I think roo gem specializes in reading excel contents than writing. For instance using set method will not save back to the spreadsheet file. It saves on the buffer I think. Try some other gems for writing
You can set the value of a column by pushing a string into it.
sheet.row(0).push 'some value'
The code below writes to a spreadsheet
require 'spreadsheet'
class Util::Table < ActiveRecord::Migration
def self.create_import_template
# create an xls workbook template for data importing based on models in activerecord
#format = Spreadsheet::Format.new(:weight => :bold)
#template_folder = File.join(Dir.home, 'Dropbox', 'horizon', 'data', 'templates')
#template_file = File.join(#template_folder, "data_import_template_#{Time.now.round(3).to_s.chomp(' -0700').gsub(':','-').gsub(' ','_').chop.chop.chop}.xls")
#book = Spreadsheet::Workbook.new
ActiveRecord::Base.send(:subclasses).each {|model| add_worksheet_to_template(model)}
#book.write #template_file
end
def self.add_worksheet_to_template(model)
# create a tab for each model that you wish to import data into
write_sheet = #book.create_worksheet :name => model
write_sheet.row(0).set_format(0, #format)
model.columns.each_with_index do |c,i|
column = ""
column << "*" unless c.null # indicate required field
column << c.name
write_sheet.row(0).set_format(i+1, #format)
write_sheet.row(0).push column
end
end
end
you can use set method
sheet.set(row, col, value)

Resources