I'm using happy mapper for object mapping in ruby , then parse the xml data in the .xml files i obtain from the api.
I get a zip file in the api response and extract it to get 5-6 files with same xml format of data.
The data in each file is around 2-3 mb.
I want to save this data in files keeping in mind that i should be able to perform search operations over it.
I don't want to use relational db rather would be looking to save the data in files.
What should be the better approach to save the data which will be efficient enough for the later search operations to be performed on that data.
require 'json'
require 'happymapper'
file_contents = File.read('/home/GhostRider/x.xml')
class Message
include HappyMapper
tag 'Message'
element :color, String, :tag => 'Colour'
element :bg_color, String, :tag => 'BgColour'
end
class Status
include HappyMapper
tag 'Status'
element :text, String, :tag => 'Text'
element :color, String, :tag => 'Colour'
element :bg_color, String, :tag => 'BgColour'
has_one :message, Message
end
class Line
include HappyMapper
tag 'Line' # if you put class in module you need tag
element :name, String, :tag => 'Name'
element :color, String, :tag => 'Colour'
element :bg_color, String, :tag => 'BgColour'
element :url, String, :tag => 'Url'
has_one :status, Status
end
class Lines
include HappyMapper
tag 'Lines' # if you put class in module you need tag
has_many :lines, Line
end
item = Lines.parse(file_contents, :single => true)
item.lines.each do |i|
puts i.name, i.color, i.url, i.status.text, i.status.message.color
end
I need to save this data obtained.
The better approach is to parse the xml using xml selectors like nokogiri or xml simple or default Hash.to_xml(xml file) from rails. Then define the ruby classes separately which helps in keeping better control over the classes and perform the operations.
Related
I am running Ruby 2.1 and Mongoid 5.0 (no Rails).
I want to track on a before_save callback whether or not an embedded field has changed.
I can use the document.attribute_changed? or document.changed methods to check normal fields, but somehow these don't work on relations (embed_one, has_one, etc).
Is there a way of detecting these changes before saving the document?
My model is something like this
class Company
include Mongoid::Document
include Mongoid::Attributes::Dynamic
field :name, type: String
#...
embeds_one :address, class_name: 'Address', inverse_of: :address
#...
before_save :activate_flags
def activate_flags
if self.changes.include? 'address'
#self.changes never includes "address"
end
if self.address_changed?
#This throws an exception
end
end
One example of how I save my document is:
#...
company.address = AddressUtilities.parse address
company.save
#After this, the callback is triggered, but self.changes is empty...
#...
I have read the documentation and Google the hell out of it, but I can't find a solution?
I have found this gem, but it's old and doesn't work with the newer versions of Mongoid. I want to check if there is another way of doing it before considering on trying to fix/pull request the gem...
Adding these two methods to your Model and calling get_embedded_document_changes should provide you an hash with the changes to all its embedded documents:
def get_embedded_document_changes
data = {}
relations.each do |name, relation|
next unless [:embeds_one, :embeds_many].include? relation.macro.to_sym
# only if changes are present
child = send(name.to_sym)
next unless child
next if child.previous_changes.empty?
child_data = get_previous_changes_for_model(child)
data[name] = child_data
end
data
end
def get_previous_changes_for_model(model)
data = {}
model.previous_changes.each do |key, change|
data[key] = {:from => change[0], :to => change[1]}
end
data
end
[ source: https://gist.github.com/derickbailey/1049304 ]
considering this simple Page Object:
require 'watir-webdriver'
require 'page-object'
class SomePage
include PageObject
text_field :first_name, :name => "fname"
text_field :last_name, :name => "lname"
text_field :birth_date, :name => "birthday"
button :submit, :type => "submit"
end
browser = Watir::Browser.new
page = SomePage.new(browser)
is there a way to iterate over all the text fields (or any elements) to access their "identifier" (i.e. :username, :password or :birth)?
something like:
page.text_fields.each do |text_field|
puts text_field.identifier.inspect
end
=> :first_name
=> :last_name
=> :birth_date
I'm just looking to see if I could turn this:
page.first_name = #user.first_name
page.last_name = #user.last_name
etc...
into this:
page.text_fields.each do |text_field|
attribute = text_field.attribute
text_field = #user[attribute]
end
Anybody knows what I mean?
The answers already provided definitely will suite your needs, but I would like to add one additional option that uses a feature built into the page-object gem.
It's call populate_page_with() in page_populator.rb. It's used like this so.
Start with a hash that has the element names(I use a CSV file that is loaded into a hash), as defined on your page object, in the keys. The value of the hash contains the value you wish to populate each element with.
form_data = {first_name: #user.first_name, last_name: #user.last_name, birth_date: #user.birth_date}
#page.populate_page_with form_data
That's it. The populate_page method will find the right elements on the page and populate them with whatever value you have set in your source hash. This works for checkbox, radio buttons and text.
This is a very nice time saving method that Cheezy put in for us!
Thanks Cheezy!
The names (:first_name, :last_name, :birth_date) are only used to generate the method names such as first_name=, last_name= and birth_date=. The name is not stored or retained for later use.
That said, you could iterate through the page's instance methods to find the text fields. The following text_fields method will:
Get all of the class instance methods.
Find the methods that end with "_element".
Create an array that includes the element names and element.
The page object would be:
class SomePage
include PageObject
text_field :first_name, :name => "fname"
text_field :last_name, :name => "lname"
text_field :birth_date, :name => "birthday"
button :submit, :type => "submit"
def text_fields
self.class.instance_methods(false)
.grep(/_element$/)
.map { |m|
element = self.send(m)
[m[/(.+)_element$/, 1].to_sym, element] if element.kind_of?(PageObject::Elements::TextField)
}.compact
end
end
You could then iterate through the text fields with access to their name (or attribute) and the TextField element:
page = SomePage.new(browser)
page.text_fields.each do |attribute, text_field|
text_field.value = #user[attribute]
end
I do exactly the "opposite" :
#user.each do | key, value |
unless value.empty?
browser.text_field(label: key).set value
end
end
I make the job done for the datas I have, and not the fields. It allows to test form fill with only some fields.
This is a get started example of Ruby data mapper. However, how does this example illustrate the power of datamapper that separate the application logic and data persistence?
Any better example can give us that the separation can lead us do unit test more easily?
class Post
include DataMapper::Resource
property :id, Serial # An auto-increment integer key
property :title, String # A varchar type string, for short strings
property :body, Text # A text block, for longer string data.
property :created_at, DateTime # A DateTime, for any date you might like.
end
# create makes the resource immediately
#post = Post.create(
:title => "My first DataMapper post",
:body => "A lot of text ...",
:created_at => Time.now
)
# Or new gives you it back unsaved, for more operations
#post = Post.new(:title => ..., ...)
#post.save # persist the resource
Thanks!
I have a mongoid model
class MyMongoidModel
include Mongoid::Document
include Mongoid::Timestamps
field :name, :type => String
field :data_id, :type => Integer
has_and_belongs_to_many :the_other_model, :class_name => 'class_name_model'
has_many :model2
def self.all
[
#.... the hardcoded data that will never be changed
]
end
end
it's used by the other model and it uses them as well. However, it contains the data that won't be changed for a very long time, let's say, at all. Thus, I don't want to retrieve it from db, I want it to be hardcoded and, at the same time, I want it acts like a normal mongoid model. Using caching is not what I'm looking for.
I hope you understand what I mean.
How do accomplish it?
There's a great gem called active_hash that provides this functionality for ActiveRecord: defining a fixed set of data as models you can reference/relate to normal models, but have it defined in code and loaded in memory (not stored/retrieved from DB).
https://github.com/zilkey/active_hash
Interestingly, since Mongoid and ActiveRecord both share common ActiveModel basis, you may be able to use active_hash with a Mongoid document now.
For example:
class Country < ActiveHash::Base
self.data = [
{:id => 1, :name => "US"},
{:id => 2, :name => "Canada"}
]
end
class Order
include Mongoid::Document
include Mongoid::Timestamps
has_one :country
end
I'm developing a RESTful API using Sinatra and DataMapper. When my models fail validation, I want to return JSON to indicate what fields were in error. DataMapper adds an 'errors' attribute to my model of type DataMapper::Validations::ValidationErrors. I want to return a JSON representation of this attribute.
Here's a single file example (gotta love Ruby/Sinatra/DataMapper!):
require 'sinatra'
require 'data_mapper'
require 'json'
class Person
include DataMapper::Resource
property :id, Serial
property :first_name, String, :required => true
property :middle_name, String
property :last_name, String, :required => true
end
DataMapper.setup :default, 'sqlite::memory:'
DataMapper.auto_migrate!
get '/person' do
person = Person.new :first_name => 'Dave'
if person.save
person.to_json
else
# person.errors - what to do with this?
{ :errors => [:last_name => ['Last name must not be blank']] }.to_json
end
end
Sinatra::Application.run!
In my actual app, I'm handling a POST or PUT, but to make the problem easy to reproduce, I'm using GET so you can use curl http://example.com:4567/person or your browser.
So, what I have is person.errors and the JSON output I'm looking for is like what's produced by the hash:
{"errors":{"last_name":["Last name must not be blank"]}}
What do I have to do to get the DataMapper::Validations::ValidationErrors into the JSON format I want?
So, as I was typing this up, the answer came to me (of course!). I've burned several hours trying to figure this out, and I hope this will save others the pain and frustration I've experienced.
To get the JSON I'm looking for, I just had to create a hash like this:
{ :errors => person.errors.to_h }.to_json
So, now my Sinatra route looks like this:
get '/person' do
person = Person.new :first_name => 'Dave'
if person.save
person.to_json
else
{ :errors => person.errors.to_h }.to_json
end
end
Hope this helps others looking to solve this problem.
I know, I am answering this late, but, in case you are just looking for just validation error messages, you can use object.errors.full_messages.to_json. For example
person.errors.full_messages.to_json
will result in something like
"[\"Name must not be blank\",\"Code must not be blank\",
\"Code must be a number\",\"Jobtype must not be blank\"]"
This will rescue on client side from iterating over key value pair.