In our application we offer list queries with sorting and filtering and now we were discussing if we should also offer single item queries with the id as parameter.
I've googled a bit and found that very often both queries, list and single item, are implemented.
I'm just wondering why. Wouldn't/Shouldn't a single item query with id as parameter be the same as a list query filtered for id? In both cases the UI should be able to select all properties throughout the graph, right?
Why would the backend care to offer a single item query when the list is already sufficient?
Looking for best practices here.
Related
I have a search database of car models: "Nissan Gtr", "Huynday Elantra", "Honda Accord", etc...
Now I also have a user list and the types of cars they like
user1 likes: carId:1234, carId:5678 etc...
Given user 1 I would like to return all the cars he likes, it can be 0 to even hundreads.
What the best way to model this in Solr or potentially another "nosql" system that can help with this problem.
I'm using Solr but I have the opportunity to use another system if I can and if it makes sense.
EDIT:
Solr solution is to slow for Join (Maybe we can try nested). And the current MySQL solution which uses join tables has over 2 billion rows.
so, you just want to store a mapping between User->Cars, and retrieve the cars based on the user...sounds very simple:
Your docs are Users: contain id (indexed), etc fields
one of the field is 'carsliked', multivalued, which contains the set of car ids he likes
you have details about each care in a different collection for example.
given a user id, you retrieve the 'carsliked' field, and get the car details with a cross collection join
You could also use nested object to store each liked car (with all the info about it) inside each user, but is a bit more complex. As a plus, you don't need the join on the query.
Solr would allow you many more things, for example, given a car, which users do like it? Elasticsearch will work exactly the same way (and probably many other tools, given how simple your use case seems).
I'm working on a portal based on Orchard CMS. We're using Orchard to manage the "normal" content of the site, as well as to model what's essentially data for a small application embedded in it.
We figured that doing it that way is "recommended" for working in Orchard, and that it would save us duplicating a bunch of effort in features that Orchard already provides, mainly generating a good enough admin UI. This is also why we're using fields wherever possible.
However, for said application, the client wants to be able to display the data in the regular UI in a garden-variety datagrid that can be filtered, sorted, and paged.
I first tried to implement this by cobbling together a page with a bunch of form elements for the filtering, above a projection with filters bound to query string parameters. However, I ran into the following issues with this approach:
Filters for numeric fields crash when the value is missing - as would be pretty common to indicate that the given field shouldn't be considered when filtering. (This I could achieve by changing the implementation in the Orchard source, which would however make upgrading trickier later. I'd prefer to keep anything I haven't written untouched.)
It seems the sort order can only be defined in the administration UI, it doesn't seem to support tokens to allow for the field to sort by to be changed when querying.
So I decided to dump that approach and switched to trying to do this with just MVC controllers that access data using IContentQuery. However, there I found out that:
I have no clue how, if at all, it's possible to sort the query based on field values.
Or, for that matter, how / if I can filter.
I did take a look at the code of Orchard.Projections, however, how it handles sorting is pretty inscrutable to me, and there doesn't seem to be a straightforward way to change the sort order for just one query either.
So, is there any way to achieve what I need here with the rest of the setup (which isn't little) unchanged, or am I in a trap here, and I'll have to move every single property I wish to use for sorting / filtering into a content part and code the admin UI myself? (Or do something ludicrous, like create one query for every sortable property and direction.)
EDIT: Another thought I had was having my custom content part duplicate the fields that are displayed in the datagrids into Hibernate-backed properties accessible to query code, and whenever the content item is updated, copy values from these fields into the properties before saving. However, again, I'm not sure if this is feasible, and how I would be able to modify a content item just before it's saved on update.
Right so I have actually done a similar thing here to you. I ended up going down both approaches, creating some custom filters for projections so I could manage filters on the frontend. It turned out pretty cool but in the end projections lacked the raw querying power I needed (I needed to filter and sort based on joins to aggregated tables which I think I decided I didn't know how I could do that in projections, or if its nature of query building would allow it). I then decided to move all my data into a record so I could query and filter it. This felt like the right way to go about it, since if I was building a UI to filter records it made sense those records should be defined in code. However, I was sorting on users where each site had different registration data associated to users and (I think the following is a terrible affliction many Orchard devs suffer from) I wanted to build a reusable, modular system so I wouldn't have to change anything, ever!
Didn't really work out quite like I hoped, but to eventually answer the question in your title: yes, you can query fields. Orchard projections builds an index that it uses for querying fields. You can access these in HQL, get the ids of the content items, then call getmany to get them all. I did this several years ago, and I cant remember much but I do remember having a distinctly unenjoyable time with it haha. So after you have an nhibernate session you can write your hql
select distinct civr.Id
from Orchard.ContentManagement.Records.ContentItemVersionRecord civr
join civ.ContentItemRecord cir
join ci.FieldIndexPartRecord fipr
join fipr.StringFieldIndexRecord sfir
This just shows you how to join to the field indexes. There are a few, for each different data type. This is the string one I'm joining here. They are all basically the same, with a PropertyName and value field. Hql allows you to add conditions to your join so we can use that to join with the relevant field index records. If you have a part called Group attached directly to your content type then it would be like this:
join fipr.StringFieldIndexRecord sfir
with sfir.PropertyName = 'MyContentType.Group.'
where sfir.Value = 'HR'
If your field is attached to a part, replace MyContentType with the name of your part. Hql is pretty awesome, can learn more here: https://docs.jboss.org/hibernate/orm/3.3/reference/en/html/queryhql.html But I dunno, it gave me a headache haha. At least HQL has documentation though, unlike Orchard's query layer. Also can always fall back to pure SQL when HQL wont do what you want, there is an option to write SQL queries from the NHibernate session.
Your other option is to index your content types with lucene (easy if you are using fields) then filter and search by that. I quite liked using that, although sometimes indexes are corrupted, or need to be rebuilt etc. So I've found it dangerous to rely on it for something that populates pages regularly.
And pretty much whatever you do, one query to filter and sort, then another query to getmany on the contentmanager to get the content items is what you should accept is the way to go. Good luck!
You can use indexing and the Orchard Search API for this. Sebastien demoed something similar to what you're trying to achieve at Orchard Harvest recently: https://www.youtube.com/watch?v=7v5qSR4g7E0
Let's say we have two index types: members and restaurants. Both contain city attribute.
I want to filter members (e.g. by name) and would like to include list of restaurant names from the members' hometown/city in the results.
Is it possible to do this using just one ES query? I guess it should be similar to DB join.
Thanks.
ES doesn't have the concepts of joins. This is due to it being an index rather than a relational database. Your best best to make two calls. One to get the member's documents, then another to get the restaurants.
Unless you have odd circumstances, this should still be very efficient.
In our GSA index of 500K documents half of the documents are coming from an internal bug tracking system.
We have been hearing some power users complain about results from the bug tracking system pushing down other useful results from many other sources.
We discussed about using result biasing to lower the importance of bug tracking documents but I am not very keen on this approach as I believe we should let GSA do its magic and decide on the relevancy of the results.
Instead what I want to provide users as an option is a UI (checkbox for each collection) where they can pick what collections they want to perform the search.
My non-default collections does not include everything that is under the default_collection. So when user checks each and every checkbox they may think that that is everything in the index while it is not.
Because of this I want the checkboxes to behave as exclude rather than include (i,e. check to exclude this collection).
Finally my question: Is there a way to search in the default collection but filter out results that belong to a specific collection (bug tracking collection).
When you want to use multiple collections you do &site=col1|col2|col3..
What I am after is something like &site=default_collection-col1 (that's a minus in between).
Is there a way to do this?
Any alternative approaches to this problem?
Personally, I would rethink the design of your collections and build more modular collections that you can include. That way as you mentioned you can include OR queries in your site include.
http://www.google.com/support/enterprise/static/gsa/docs/admin/70/gsa_doc_set/xml_reference/request_format.html#1076953
A less ideal but more specific solution to your problem is going to be do an exclude by URL in your search query, be aware this can appear in results query search box and looks ugly, but this can be fixed using a simple XSLT change.
To exclude results for a specific site (http://www.google.com/support/enterprise/static/gsa/docs/admin/70/gsa_doc_set/xml_reference/request_format.html#1076964) I would use this sparingly and opt for better design of the collections.
By far the best way to do this is in your collection config. Just create a new collection that has the same include pattern as your default collection and add the pattern from your bug tracking collection as an exclude pattern.
There's no way to do what you're asking purely using query parameters unless you list out every individual collection using the '|' except the one you want and then you're likely to run in to URL length issues.
Update your frontend to exclude the url patterns mentioned for the bugtracking collection.
check this url on your box
http://yourGSAEnterpriseCcontroller:8000/EnterpriseController/serve_remove.html
I have two collections. One a list of items with id and content, lets call this list ItemList. I have another collection which tells me whether a user has picked an item. Calling this list Collected it will have the user id and item id. Both the number of users and items are really large. What is the optimal way to query for items from ItemList for a user which are not in list Collected.
Here are a few ideas I have:
Use joins of a relational database to solve this. My only query is will this handle really large dataset.
Use blooms filter to store the collected item list and while querying for items check if it is not in the filter.
If the above thought will not scale could you provide me with algorithms that would. These can't be in-memory solutions as I would definitely require to persist data.
You can use bitsets, also.
Load into some bitset ItemList (ItemID is index in the bitset).
Load into another bitset IDs_for_user for this user.
Perform opearation: Resut = ItemList ANDNOT IDs_for_user.
Free bitset library you can get here:
http://bmagic.sourceforge.net/