When simply displaying large amounts of data (over 100k records) my code works well, and I paginate on the server.
However, when I need to sort this data I'm stuck. I'm only sorting on the page, and NOT sorting on ALL the records related to this one customer.
How can I paginate but also sort across all the records of my customer and NOT simply sort the records returned from the server side pagination?
I'm also using BootStrap Table to display all my data.
Here is my code that gets all the customers:
def get_customers
#data_to_return = []
#currency = current_shop.country_currency
customers = current_shop.customers.limit(records_limit).offset(records_offset)#.order("#{sort_by}" " " "#{sort_order}")
customers.each do |customer|
#data_to_return.push(
state: false,
id: customer.id,
email: customer.email,
accepts_marketing: customer.accepts_marketing,
customer_status: customer.customer_status,
tags: customer.tags)
end
sort_customers
end
And then this is the sort_customers method:
def sort_customers
fixed_data = data_to_return.sort_by {|hsh| hsh[sort_by]}
customer_size = current_shop.customers.length
if sort_order == "ASC"
fixed_data
else
fixed_data.reverse!
end
render json: {"total": customer_size, "rows": fixed_data}
end
In the above code you can see that data_to_return is coming from get_customers and its limited. But I don't want to return ALL the customers for many reasons.
How can I sort across all the records, but only return the paginated subset?
You should actually sort at the model/query level, not at the ruby level.
The difference is basically:
# sort in ruby
relation.sort_by { |item| foo(item) }
# sort in database - composes with pagination
relation.order('column_name ASC/DESC')
In the first case, the relation is implicitly executed, enumerated and converted to array before calling sort_by. If you did pagination (manually or with kaminari), you will get just that page of data.
In the second case, you are actually composing the limit, offset and where (limit and offset are anyways used under the hood by kaminari, where is implicit when you use associations) with a order so your database would execute
SELECT `customers`.`*` FROM `customers`
WHERE ...
OFFSET ...
LIMIT ...
ORDER BY ...
which will return the correct data.
A good option is to define scopes in the model, like
class Customer < ApplicationRecord
scope :sorted_by_email, ->(ascending = true) { order("email #{ascending ? 'ASC' : 'DESC'}") }
end
# in controller
customers = current_shop.customers.
limit(records_limit).
offset(records_offset).
sorted_by_email(false)
You can resolve sorting and paginate issue using Data Tables library, which is client side. It's a Jquery library. Using this you need to load all data into page, then it would work very well.
Below are the references please check.
Data tables jquery libray
Data tables gem for rails
You can try these, they will work very well. You can customise it as well
If the answer is helpful, you can accept it.
Related
I'm using Mongoid to access a MongoDB database, however I'm running into an odd problem. It seems like I can only query for records using find_by, as find will always return nil:
invoices = Invoice.find({})
p "invoices"
p invoices
puts ''
invoice = Invoice.find_by({ _id: <ObjectId> })
p "invoice"
p invoice
puts ''
The second query using find_by will return a single record. According to the documentation, find should be returning every record that satisfies the query.
Does anyone have an idea what could be causing this?
Be careful not to confuse the Moped syntax with the Mongoid syntax. For Mongoid, the docs describe the find method:
Find a document or multiple documents by their ids. Will raise an error by default if any of the ids do not match
If you really want every record, Invoice.all can do the trick. (Also be careful with your find_by method. The Mongoid syntax varies from mongo's a bit, so you don't have to have the curlies around your params.)
I am building a messaging module into an existing web app. We are storing the messages in mongo with a data structure that something looks like:
{
id: "",
inResponseToMessageId: ""
to: []
cc: []
bcc: []
subject: ""
body: ""
owners: [{id:4, status:"read", folder:"inbox"}, {id:1, status:'unread', folder:'inbox'}]
dateSent:
}
The client would like us to be able to display messages in both a conversation view and a singleton view.
I am having trouble figuring out an efficient query that can
Return results grouped by message thread.
Work well with pagination.
Sortable by date and subject.
I can modify the data structure however I need in order to get this to work well.
Below are a few possible solutions but they all seem to fall short:
Store messages as children of a thread object.
Add a threadId to each message and then query and group by threadId.
Create some type of meta information object that helps.
The problem with the standard mongo group or $group function is that I imagine it will perform very poorly when the collection is large. We are expecting there to be hundreds of millions of messages in the collection.
Put a threadId on your messages.
Return results grouped by message thread.
You can find messages by thread like
db.messages.find({ "threadId" : id })
I don't think there's any need to group them in the sense of the $group operator.
Work well with pagination.
Pagination for the messages in what order? Pagination works well if you have a sort on a unique field. dateSent should be unique if you keep it to millisecond precision, so you can paginate on it.
// page 1
db.messages.find({ "threadId" : id }).sort({ "dateSent" : -1 }).limit(25)
// page 2
db.messages.find({ "threadId" : id, "dateSent" : { "$gt" : <25th date sent> } }).sort({ "dateSent" : -1 }).limit(25)
Sortable by date and subject.
Who sorts messages by subject? Anyway, this is just a matter of creating the right indexes if you want to retrieve messages in date or subject order. Depending on your requirements you might be doing this sorting for a client view, where it might not be necessary to have the database sort the results. The client could do it for the returned subset instead.
My query is this:
DB[:expense_projects___p].where(:project_company_id=>user_company_id).
left_join(:expense_items___i, :expense_project_id=>:project_id).
select_group(:p__project_name, :p__project_id).
select_more{count(:i__item_id)}.
select_more{sum(:i__amount)}.to_a.to_json
which works.
However, payment methods include cash, card and invoice. So I would like to sum each of those for summary purposes to achieve a discrete total for payments by cash, card, and invoice repsectively. I included the following line into the query
select_more{sum(:i__amount).where(:i__mop => 'card')}.
and the error message was
NoMethodError - undefined method `where' for #<Sequel::SQL::Function:0x007fddd88b5ed0>:
so I created the dataset separately with
ds1 = expense_items.where(:mop=>'card', :expense_company_id=>user_company_id).sum(:amount)
and appended it, at the end of the original query, with
select_append{ds1}
which achieved partial success as the returned json is now:
{"project_name":"project 2","project_id":2,"count":4,"sum":"0.40501E3","?column?":"0.2381E2"}
as can be seen there is no name for this element which I need in order to reference it in my getJSON call. I tried to add an identifier by adding ___a
to the ds1 query as below
ds1 = expense_items.where(:mop=>'card', :expense_company_id=>user_company_id).sum(:amount___a)
but that failed.
In summary, is this the right approach and, in any case, how can I provide an identifier when doing a sequel sum query? In other words sum(:a_column).as(a_name)
Many thanks.
Dataset#sum returns the sum, not a modified dataset. You probably want something like:
ds1 = expense_items.where(:mop=>'card', :expense_company_id=>user_company_id).select{sum(:amount)}
select_append{ds1.as(:sum)}
I am not sure for approach ( better ask Jeremy Evans ) but it work.
You just change: .sum(... to .select_more{:amount___a).as(:desired_name)}
ds1 = expense_items.where(:mop=>'card', :expense_company_id=>user_company_id).select_more{:amount___a).as(:desired_name)}
and actually get that desired_name in db response.
I have Team and Players classes and want to return the data in one JSON string which contains Team info but at the same time it displays all the information about the players.
class Team < ActiveRecord::Base
has_many :players
end
class Players < ActiveRecord::Base
belongs_to :team
end
I know how to retrieve the information about team and players but not in the same query. Another problem is I don't how to merge the result JSONs in one JSON.
team = Team.last.to_json
player = team.players.to_json
How can I query the info about Team and Players in the same query. I tried:
#team = Team.includes(:players).where(players: {team_id: Team.last}).last.to_json
and it only returns me information about the team. I want a JSON like:
-id
-name
-players
-player
-player
In case it's impossible, how can I merge into one JSON all the information from the two queries.
You can write a "join" to incorporate the players in the team with the team information. At that point you'll have a structure that has the information needed to create the JSON. See "12 Joining Tables" from the Active Record documentation for more information.
Or, you can make two separate queries, then create a bit more complex JSON hash or array allowing you to output both sets of data into one larger serialized object. For instance:
require 'json'
team = {
'name' => 'bears'
}
players = {
'1' => 'fred',
'2' => 'joe'
}
puts ({
'team' => team,
'players' => players
}).to_json
Here's the output:
{"team":{"name":"bears"},"players":{"1":"fred","2":"joe"}}
Here's the data returned back to the Ruby object:
data = '{"team":{"name":"bears"},"players":{"1":"fred","2":"joe"}}'
JSON[data]
# => {"team"=>{"name"=>"bears"}, "players"=>{"1"=>"fred", "2"=>"joe"}}
Also, since you're using Sinatra, it's not necessary to use Active Record. Sequel is a very good ORM, and is my personal favorite when working with Sinatra. You might find it easier to work with.
Another option to manual serialization is to use ActiveModel::Serializer which allows you to define relationships between objects and gives you finer grained choices of what to include when you serialize, what to filter out and what related objects to preload. An alternative could also be Rabl which also has quite a nice API.
If you're just playing around with a small amount of JSON this might be overkill, but it's a nice practice to be more organized
I have a set of legacy database tables that i cannot normalize out to what should have been done in the first place. e.g one big table with 200 columns.
I'm building an API and would like to represent this data to the consumer in a better state, and perhaps address the database issues at a later stage, there are many backend systems that reply on the data and changes are not easy.
I wanted to represent the current database schema using Active Record, however perform a model transformation into a new model that will be used for presentation only to an API consumer as json data.
current database schema:
Products table (200 columns)
New Model:
Product
+ Pricing
+ Assets
+ Locations
+ Supplier
I could hard-code a json string in a template, but feel that would not be a very poor approach.
What approach or gem would you recommend to tackle this best?
I have looked at :
RABL
ActiveModel::Serializers
If you define an as_json method that returns a hash, ActiveRecord will take care of the serialization for you. E.g.
class Product < ActiveRecord::Base
def as_json options = {}
{
product: <product value>,
pricing: <pricing value>,
# ... etc.
}
end
end
Now you can do:
> Product.first.to_json
=> "{\"product\":<product_value> ... }"
You can even render these as json from the controllers via:
render json: #model