I'm writing a small torrent indexer in Ruby (here) and would love to support MongoDB as an option for the database. Currently, I have the database set up with a many-to-many relationship between tags and torrents.
How would I format a query that gets all the torrent_ids from a map table that match to all the tags in a given list?
I did this in SQL like this:
select torrent_id, count(*) num from tagmap where tag_id in (tag1, tag2, tag3, tag4) group by torrent_id having num = 4"
EDIT: I am right now working only with the collection with torrent_id and tag_id. That's all it has in there. So I'm mapping ids to ids and naught more.
It's better to create a collection to create the mapping consisting tag_id's and torrent_id's. Whenever you add a torrent, also add the torrents tags to the torrenttags collection. Index should be on tag_id.
You can use the following query syntax to get a list of torrents matching more than one tag.
db.tagmap.find({tag_id:{$in: ['tag1','tag2','tag3','tag4']}});
For Aggregation (group by, count) you need to use MapReduce
Related
I have a few records in my elasticsearch collection and i want to use a GroupBy aggregation in elasticsearch querystring.
I want to know if it is possible, because i tried to google it always give result about this
i want to use this something like this in the query string , which can
give me records in the group.
For i.e.
http://localhost:9200/_all/tweets/_count?q=user:Pu*+user:Kim*
This will give me count of all the records which has name starts from Pu and Kim,
But i want to know that how many records are there has name starting with Pu
and Kim,
aggregations need to be specified in addition in the search request, you cannot specify them as part of a query string query.
You could also just execute two queries to find out this particular requirement...
I'm new to rethinkdb and i love it, but i found some problems when i tried to optimize my query and make it work on bigger datasets.
The problem is simple.
I need to filter my "event" table by timestamp (row.to) , by tag (row.tags), order by timestamp (row.from) and then slice for pagination.
row.tags has a multi index and works well!
row.from and row.to are start/end time of Event.
The slow query (testeded on 100k entries) is this:
r.db("test").table("event")
.getAll(r.args(["148a6e03-b6c3-4092-afa0-3b6d1a4555cd","7008d4b0-d859-49f3-b9e0-2e121f000ddf"]), {"index": "tags"})
.filter(function(row) {return row("to").ge(r.epochTime(1480460400));})
.orderBy(r.asc("from"))
.slice(0,20)
I created an index on 'from' and tried to do
.orderBy(r.asc("from"),{index:'from'})
but i get
e: Indexed order_by can only be performed on a TABLE or TABLE_SLICE in:
I already read about problems about index intersection in Rethinkdb, but maybe i miss something, maybe there is a way of doing this simple task.
Thank you.
The reason RethinkDB complains is this:
getAll returns a selection. When filter is applied to a selection it returns a selection. When orderBy is applied to a selection the index parameter can't be used (it can only be used when orderBy is applied to a table).
orderBy can be applied to a table, sequence or selection. Only when it's applied to table can the index parameter be used. This makes sense as the index is updated when rows are added and removed from the table.
In your case, you are applying orderBy on a result of filter which is a selection. In order to sort a selection the database needs to:
read all elements into memory (by default max is 100,000 elements)
sort them using the provided function or field
and it can't use index in this case.
The way to improve your query might be to sort the table first and then apply the filter. You will be able to use the index in this case.
I am using JDBC river to pull data to ElasticSearch from Oracle database.
As mentioned in following link, left join can be used to get multiple values of one column of same primary id record in single json array. But if there is only one records after left join, river doesn't create array, rather puts the value in the json field.
This is causing problem to NEST to understand the type of object.
https://github.com/jprante/elasticsearch-jdbc#structured-objects
So, is there any way to force some fields to be array even it has just one value?
There is a way to to do this using the bracket notation as described here JDBC river Bracket Notation
So basically in your SQL query if you have
Select tag as tag.name from tags
you need to change it to
Select tag as tag[name] from tags
hope this helps
I have a question regarding the setup of my elasticsearch database index... I have created a table which I have rivered to index in elasticsearch. The table is built from a script that queries multiple tables to denormalize data making it easier to index by a unique id 1:1 ratio
An example of a set of fields I have is street, city, state, zip, which I can query on, but my question is , should I be keeping those fields individually indexed , or be concatenating them as one big field like address which contains all of the previous fields into one? Or be putting in the extra time to setup parent-child indexes?
The use case example is I have a customer with billing info coming from one direction, I want to query elasticsearch to see if that customer already exists, or at least return the closest result
I know this question is more conceptual than programming, I just can't find any information of best practices.
Concatenation
For the first part of your question: I wouldn't concatenate the different fields into a field containing all information. Having multiple fields gives you the advantage of calculating facets and aggregates on those fields, e.g. how many customers are from a specific city or have a specific zip. You can still use a match or multimatch query to query for information from different fields.
In addition to having the information in separate fields I would use multifields with an analyzed and not_analyzed part (fieldname.raw). This again allows for aggregates, facets and sorting.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/mapping-multi-field-type.html
Think of 'New York': if you analyze it will be stored as ['New', 'York'] and you will not be able to see all People from 'New York'. What you'd see are all people from 'New' and 'York'.
_all field
There is a special _all field in elasticsearch which does the concatenation in the background. You don't have to do it yourself. It is possible to enable/disable it.
Parent Child relationship
Concerning the part whether to use nested objects or parent child relationship: I think that using a parent child relationship is more appropriate for your case. Nested objects are stored in a 'flattened' way, i.e. the information from the nested objects in arrays is stored as being part of one object. Consider the following example:
You have an order for a client:
client: 'Samuel Thomson'
orderline: 'Strong Thinkpad'
orderline: 'Light Macbook'
client: 'Jay Rizzi'
orderline: 'Strong Macbook'
Using nested objects if you search for clients who ordered 'Strong Macbook' you'd get both clients. This because 'Samuel Thomson' and his orders are stored altogether, i.e. ['Strong' 'Thinkpad' 'Light' 'Macbook'], there is no distinction between the two orderlines.
By using parent child documents, the orderlines for the same client are not mixed and preserve their identity.
I am struck in a data operation where I want to sort results of a query by a Ref field.
Lets say I have the following Data Objects.
EmployeeDO {Long id, String name, Ref refCompany}
CompanyDO {Long id, String name}
Now i want to query employees arranged by company name.
I tried the query
Query<EmployeeDO> query = ofy().load().type(EmployeeDO.class).order("refCompany");
Obviously this did not sort results with company name, but this compiled successfully too.
Please suggest if such sorting is possible by this way or some other workaround can be tried?
You can order by refCompany if you #Index refCompany, but it won't sort by company name - it will index by the key (if you aren't using #Parent, just an id order).
There are two 'usual' choices:
Load the data into ram and sort there. This is what an RDBMS would do internally. It's not exactly true that GAE doesn't support joins; it's just that you're the query planner.
Denormalize and pre-index the companyName. Put #Index companyName in the EmployeeDO. This is what you would do with an RDBMS if you the magic sorting performed poorly (say, there are too many employees).