I am working on an e-commerce application on which we are predicting around 2 millions products.
We are using Solr for searching, suggestions and filtering.
But for showing the product card data(which has detail like name, image, price, discount and rating) on the Product Listing page(PLP), we are going to Database after fetching the product ids from the Solr.
For E.g:
User went on the our e-commerce application and there on the search box typed "I phone 13 pro".
While user was typing we are showing suggestions to him/her after fetching it from the solr, no db involved yet.
After that user either click or hit enter then we are showing the Product listing page where we first reaching to the solr and fetching the product ids(say 24 per page we are showing) matching the entered text and then we are going to the DB with these ids and fetching the product objects and passing it to the UI with the necessary detail(using DTO object).
Now my question here is:
As I mentioned above we are hitting DB for the Product Listing Page(PLP) for fetching 24 product per page, here I am thinking it may slow down the page when our catalog and user started increasing?
Note:- we are also using Hibernate h2 level caching to stop hitting DB again and again for the same product but cache has limit up to 0.5 million.
Why we are hitting to DB instead of fetching everything from the Solr as we have all product data info which we wanted to show above on the PLP page?
Ans- As we are now keeping important fields of the product object in the Solr not all fields, So we are thinking may be in future we wanted to show something which we don't have in the Solr then we have to Re-index the full catalog to add new field into it and it may be very time-consuming.
I am in a favour of using Solr for showing PLP page instead of going to DB.Of course keeping Re-indexing in mind but not sure it is the best approach or not?
Please provide your suggestion or thoughts on this.
Thanks in advance.
I have multiple Orders which have related Payments and related InvoiceCorrections.
I want to fetch all orders where the
order->payments->sum('amount')differs from(order->amount - order->invoiceCorrections->sum('amount')).
What would be the best way to archive this, while keeping a good performance?
The payments should have their own column containing correction information which gets updated via a model observer, otherwise your queries are going to get very complex and messy very fast.
We are about to begin working on an addition to Magento 1.14.2 EE that will allow us to merchandise and sort products within categories at a store level. currently we do this by having 3 entirely separate root category trees, our editors are finding this cumbersome, and our indexing takes 3x the time it should take to reindex a single tree.
The plan is to add a store_id column to the catalog_category_product table which currently stores the product_id, category_id and products position within the category in question.
So my questions are fairly general at this point, has anyone attempted this previously and are there any obvious pitfalls that we are likely to encounter as a result of attempting this? The solution to us seems fairly obvious yet it hasn't been implemented yet by Magento, surely this is a piece of functionality that would be useful for any company that has a presence in multiple countries.
I'm working on a portal based on Orchard CMS. We're using Orchard to manage the "normal" content of the site, as well as to model what's essentially data for a small application embedded in it.
We figured that doing it that way is "recommended" for working in Orchard, and that it would save us duplicating a bunch of effort in features that Orchard already provides, mainly generating a good enough admin UI. This is also why we're using fields wherever possible.
However, for said application, the client wants to be able to display the data in the regular UI in a garden-variety datagrid that can be filtered, sorted, and paged.
I first tried to implement this by cobbling together a page with a bunch of form elements for the filtering, above a projection with filters bound to query string parameters. However, I ran into the following issues with this approach:
Filters for numeric fields crash when the value is missing - as would be pretty common to indicate that the given field shouldn't be considered when filtering. (This I could achieve by changing the implementation in the Orchard source, which would however make upgrading trickier later. I'd prefer to keep anything I haven't written untouched.)
It seems the sort order can only be defined in the administration UI, it doesn't seem to support tokens to allow for the field to sort by to be changed when querying.
So I decided to dump that approach and switched to trying to do this with just MVC controllers that access data using IContentQuery. However, there I found out that:
I have no clue how, if at all, it's possible to sort the query based on field values.
Or, for that matter, how / if I can filter.
I did take a look at the code of Orchard.Projections, however, how it handles sorting is pretty inscrutable to me, and there doesn't seem to be a straightforward way to change the sort order for just one query either.
So, is there any way to achieve what I need here with the rest of the setup (which isn't little) unchanged, or am I in a trap here, and I'll have to move every single property I wish to use for sorting / filtering into a content part and code the admin UI myself? (Or do something ludicrous, like create one query for every sortable property and direction.)
EDIT: Another thought I had was having my custom content part duplicate the fields that are displayed in the datagrids into Hibernate-backed properties accessible to query code, and whenever the content item is updated, copy values from these fields into the properties before saving. However, again, I'm not sure if this is feasible, and how I would be able to modify a content item just before it's saved on update.
Right so I have actually done a similar thing here to you. I ended up going down both approaches, creating some custom filters for projections so I could manage filters on the frontend. It turned out pretty cool but in the end projections lacked the raw querying power I needed (I needed to filter and sort based on joins to aggregated tables which I think I decided I didn't know how I could do that in projections, or if its nature of query building would allow it). I then decided to move all my data into a record so I could query and filter it. This felt like the right way to go about it, since if I was building a UI to filter records it made sense those records should be defined in code. However, I was sorting on users where each site had different registration data associated to users and (I think the following is a terrible affliction many Orchard devs suffer from) I wanted to build a reusable, modular system so I wouldn't have to change anything, ever!
Didn't really work out quite like I hoped, but to eventually answer the question in your title: yes, you can query fields. Orchard projections builds an index that it uses for querying fields. You can access these in HQL, get the ids of the content items, then call getmany to get them all. I did this several years ago, and I cant remember much but I do remember having a distinctly unenjoyable time with it haha. So after you have an nhibernate session you can write your hql
select distinct civr.Id
from Orchard.ContentManagement.Records.ContentItemVersionRecord civr
join civ.ContentItemRecord cir
join ci.FieldIndexPartRecord fipr
join fipr.StringFieldIndexRecord sfir
This just shows you how to join to the field indexes. There are a few, for each different data type. This is the string one I'm joining here. They are all basically the same, with a PropertyName and value field. Hql allows you to add conditions to your join so we can use that to join with the relevant field index records. If you have a part called Group attached directly to your content type then it would be like this:
join fipr.StringFieldIndexRecord sfir
with sfir.PropertyName = 'MyContentType.Group.'
where sfir.Value = 'HR'
If your field is attached to a part, replace MyContentType with the name of your part. Hql is pretty awesome, can learn more here: https://docs.jboss.org/hibernate/orm/3.3/reference/en/html/queryhql.html But I dunno, it gave me a headache haha. At least HQL has documentation though, unlike Orchard's query layer. Also can always fall back to pure SQL when HQL wont do what you want, there is an option to write SQL queries from the NHibernate session.
Your other option is to index your content types with lucene (easy if you are using fields) then filter and search by that. I quite liked using that, although sometimes indexes are corrupted, or need to be rebuilt etc. So I've found it dangerous to rely on it for something that populates pages regularly.
And pretty much whatever you do, one query to filter and sort, then another query to getmany on the contentmanager to get the content items is what you should accept is the way to go. Good luck!
You can use indexing and the Orchard Search API for this. Sebastien demoed something similar to what you're trying to achieve at Orchard Harvest recently: https://www.youtube.com/watch?v=7v5qSR4g7E0
I am doing a ajax based search which suggests the values from the database tables as the user inputs fields in the search box. It does work but its really very slow, takes more than 10 seconds minimum for any suggestions to load. how do I optimize this to make it fast..
Like Index it/save it in cache/crawl?
Right now autoSearch() js function is fired which retrieves data from the related table to load the suggestions. how do I make this faster?
I dont know if this will make a difference but I am using grails/groovy for application development.
Thanks
Priyank
Have you added an index to any searched fields (or checked in the database to make sure the examined fields are indexed)?
The Grails GORM does not automatically create indices for text fields unless you add the appropriate static mapping to your domain class.