Is it possible to use RocksDB efficiently for range queries on numbers?
For example if I have billions of tuples (price, product_id) can I use RocksDB to retrieve all products that have 10 <= price <= 100? Or it can't be used for that?
I am confused because I can't find any specific docs about number keys and range queries. However I also read that RocksDB is used as a database engine for many DBMS and that suggests that it's possible to query it efficiently for this case.
What is the recommended way to organize the above tuples in a key-value store like RocksDB in order to get arbitrary ranges (not known in advance)?
What kind of keys would you use? What type of queries would you use?
Yes, rocksdb supports efficient range queries [even for arbitrary ranges that are not known in advance]
range queries.
https://github.com/facebook/rocksdb/wiki/Prefix-Seek
number keys
There are no docs on how to model your data like that - if you don't know how to model that already you shouldn't be using rocksdb in the first place as it is too low level
What is the recommended way to organize the above tuples in a key-value store like RocksDB in order to get arbitrary ranges (not known in advance)?
In your example - it is creating an index on price to lookup the product id
So you would encode the price as a byte array and use that as the key and then the product id as a byte array as the value
Example format
key => value
priceIndex:<price>#<productId> => <productId>
Then you will
Create an iterator
Seek to the lower bound of your price [priceIndex:10 in this case]
Set upper bound on the options [priceIndex:100 in this case]
Loop over until iterator is valid
This will give you all the key value pairs that are in the range - which in your case would be all the price, product id tuples that are within the price range
Care must be taken since many products can have the same price and rocksdb keys are unique - so you can suffix the price with the product id as well to make the key unique
Related
In the ElasticSearch documentation for the Cardinality Aggregation under the heading "Pre-computed hashes" I see the following:
On string fields that have a high cardinality, it might be faster to
store the hash of your field values in your index and then run the
cardinality aggregation on this field. This can either be done by
providing hash values from client-side or by letting Elasticsearch
compute hash values for you by using the mapper-murmur3 plugin.
Pre-computing hashes is usually only useful on very large and/or
high-cardinality fields as it saves CPU and memory. However, on
numeric fields, hashing is very fast and storing the original values
requires as much or less memory than storing the hashes. This is also
true on low-cardinality string fields, especially given that those
have an optimization in order to make sure that hashes are computed at
most once per unique value per segment.
I'm curious about the part where it says, "[this can be done] by providing hash values from client-side," because it doesn't elaborate at all on that point, but goes on to discuss numeric fields.
If I wanted to pre-compute hashes on the client, would using something like xxhash and putting the result in an appropriate number field be sufficient? (And, of course, having cardinality target that field.) Or would I need to use another type of field for the hash value?
Pre-computing hashes for high-cardinality string fields will speed up the cardinality aggregation, because hashes don't have to be computed in real-time. No need to do it on numeric fields, though!
For string fields, they advise to use the mapper-murmur3 plugin. Those hashes will be alphanumeric and should be stored in keyword fields (not a numeric field type!), that you then use in your cardinality aggregation.
I've personally seen 10x+ improvements when computing the cardinality of high-cardinality string fields with pre-computed hashes. Worth a try!
Hello everyone I have prometheus as a label returns the amount. The metric value is the number of payments. How do I withdraw the total amount a day to the dashboard? i.e. value_metric*sum
As far as I know, there is no way to do that because labels aren't meant to be used in calculations. Labels and their values are essentially the index of Prometheus' NoSQL TSDB, they're used to create relations and join pieces of data together. You wouldn't store values and do math with column names of a relational database, would you?
Another problem is that labels with high cardinality greatly increase database size. Here is an extraction from Prometheus best practices:
CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.
Though I see that you use somewhat fixed values in labels, maybe a histogram would fit your needs.
I'm planning to store rooms availability in a redis database. The json object looks as such:
{
BuildingID: "RE0002439",
RoomID: "UN0002384391290",
SentTime: 1572616800,
ReceivedTime: 1572616801,
Status: "Occupied",
EstimatedAvailableFrom: 1572620400000,
Capacity: 20,
Layout: "classroom"
}
This is going to be reported by both devices and apps (tablet outside the room, sensor within the room in some rooms, by users etc.) and vary largely as we have hundreds of buildings and over 1000 rooms.
My intention is to use a simple key value structure in Redis. The main query would be which room is available now, but other queries are possible.
Because of that I was thinking that the key should look like
RoomID,Status,Capacity
My question is is it correct assumption because this is the main query we expect to have these all in the key? Should there be other fields in the key too or should the key be just a number with Redis increment, as if it was SQL?
There are plenty of questions I could find about hierarchy but my object has no hierarchy really.
Unless you will use the redis instance exclusively for this, using keys with pattern matching for common queries is not a good idea. KEYS is O(N) and SCAN too when called multiple times to traverse the whole keyspace.
Consider RediSearch module, it would give you a lot of power on this use case.
If RediSearch is not an option:
You can use a single hash key to store all rooms, but then you have to store the whole json string as value, and whenever you want to modify a field, you need to get, then modify then set.
You are probably better off using multiple data structures, here an idea to get you started:
Store each room as a hash key. If RoomID is unique you can use it as key, or pair it with building id if needed. This way, you can edit a field value in one operation.
HSET UN0002384391290 BuildingID RE0002439 Capacity 20 ...
Keep a set with all room IDs. SADD AllRooms UN0002384391290
Use sets and sorted sets as indexes for the rest:
A set of available rooms: Use SADD AvailableRooms UN0002384391290 and SREM AvailableRooms UN0002384391290 to mark rooms as available or not. This way your common query of all rooms available is as fast as it gets. You can use this in place of Status inside the room data. Use SISMEMBER to test is a given room is available now.
A sorted set with capacity: Use ZADD RoomsByCapacity 20 UN0002384391290. So now you can start doing nice queries like ZRANGEBYSCORE RoomsByCapacity 15 +inf WITHSCORES to get all rooms with a capacity >=15. You then can intersect with available rooms.
Sets by layout: SADD RoomsByLayout:classroom UN0002384391290. Then you can intersect by layout, like SINTER AvailableRooms RoomsByLayout:classroom to get all available classrooms.
Sets by building: SADD RoomsByBuilding:RE0002439 UN0002384391290. Then you can intersect by buildings too, like SINTER AvailableRooms RoomsByLayout:classroom RoomsByBuilding:RE0002439 to get all available classrooms in a building.
You can mix sets with sorted sets, like ZINTERSTORE Available:RE0002439:ByCap 3 RoomsByBuilding:RE0002439 RoomsByCapacity AvailableRooms AGGREGATE MAX to get all available rooms scored by capacity in building RE0002439. Sorted sets only allow ZINTERSTORE and ZUNIONSTORE, so you need to clean up after your queries.
You can avoid sorted sets by using sets with capacity buckets, like Rooms:Capacity:1-5, Rooms:Capacity:6-10, etc.
Consider adding coordinates to your buildings, so your users can query by proximity. See GEOADD and GEORADIUS.
You may want to allow reservations and availability queries into the future. See Date range overlap on Redis?.
A field in an index I'm building gets regularly appended to. I'd like to be able to query elasticsearch to count the number of current items. Is it possible to query for the length of an array field? I can write something to bring back the field and count the items but some of them have a large number of entries and so am looking for something that is done in place in ES.
Here's what I recommend. Perform a terms aggregation over _uid. Then perform another aggregation over all the fields in the array field and sum the doc_counts.
But such an operation really depends on the number of records that you need to query otherwise, this can be an expensive operation.
Another option you have is to store count of array elements as another field and query it directly and given the fact that you are already storing large arrays in your document, having an integer field for the size seems to be a fair trade-off. In case you need the count to filter records I would recommend using scripted filter as explained here
I've been having some difficulty scaling up the application and decided to ask a question here.
Consider a relational database (say mysql). Let's say it allows users to make posts and these are stored in the post table (has fields: postid, posterid, data, timestamp). So, when you go to retrieve all posts by you sorted by recency, you simply get all posts with posterid = you and order by date. Simple enough.
This process will use timestamp as the index since it has the highest cardinality and correctly so. So, beyond looking into the indexes, it'll take literally 1 row fetch from disk to complete this task. Awesome!
But let's say it's been 1 million more posts (in the system) by other users since you last posted. Then, in order to get your latest post, the database will peg the index on timestamp again, and it's not like we know how many posts have happened since then (or should we at least manually estimate and set preferred key)? Then we wasted looking into a million and one rows just to fetch a single row.
Additionally, a set of posts from multiple arbitrary users would be one of the use cases, so I cannot make fields like userid_timestamp to create a sub-index.
Am I seeing this wrong? Or what must be changed fundamentally from the application to allow such operation to occur at least somewhat efficiently?
Indexing
If you have a query: ... WHERE posterid = you ORDER BY timestamp [DESC], then you need a composite index on {posterid, timestamp}.
Finding all posts of a given user is done by a range scan on the index's leading edge (posterid).
Finding user's oldest/newest post can be done in a single index seek, which is proportional to the B-Tree height, which is proportional to log(N) where N is number of indexed rows.
To understand why, take a look at Anatomy of an SQL Index.
Clustering
The leafs of a "normal" B-Tree index hold "pointers" (physical addresses) to indexed rows, while the rows themselves reside in a separate data structure called "table heap". The heap can be eliminated by storing rows directly in leafs of the B-Tree, which is called clustering. This has its pros and cons, but if you have one predominant kind of query, eliminating the table heap access through clustering is definitely something to consider.
In this particular case, the table could be created like this:
CREATE TABLE T (
posterid int,
`timestamp` DATETIME,
data VARCHAR(50),
PRIMARY KEY (posterid, `timestamp`)
);
The MySQL/InnoDB clusters all its tables and uses primary key as clustering key. We haven't used the surrogate key (postid) since secondary indexes in clustered tables can be expensive and we already have the natural key. If you really need the surrogate key, consider making it alternate key and keeping the clustering established through the natural key.
For queries like
where posterid = 5
order by timestamp
or
where posterid in (4, 578, 222299, ...etc...)
order by timestamp
make an index on (posterid, timestamp) and the database should pick it all by itself.
edit - i just tried this with mysql
CREATE TABLE `posts` (
`id` INT(11) NOT NULL,
`ts` INT NOT NULL,
`data` VARCHAR(100) NULL DEFAULT NULL,
INDEX `id_ts` (`id`, `ts`),
INDEX `id` (`id`),
INDEX `ts` (`ts`),
INDEX `ts_id` (`ts`, `id`)
)
ENGINE=InnoDB
I filled it with a lot of data, and
explain
select * from posts where id = 5 order by ts
picks the id_ts index
Assuming you use hash tables to implement your Data Base - yes. Hash tables are not ordered, and you have no other way but to iterate all elements in order to find the maximal.
However, if you use some ordered DS, such as a B+ tree (which is actually pretty optimized for disks and thus data bases), it is a different story.
You can store elements in your B+ tree ordered by user (primary order/comparator) and date (secondary comparator, descending). Once you have this DS, finding the first element can be achieved in O(log(n)) disk seeks by finding the first element matching the primary criteria (user-id).
I am not familiar with the implementations of data bases, but AFAIK, some of them do allow you to create an index, based on a B+ tree - and by doing so, you can achieve finding the last post of a user more efficiently.
P.S.
To be exact, the concept of "greatest" element or ordering is not well defined in Relational Algebra. There is no max operator. To get the max element of a table R with a single column a one should actually create the Cartesian product of that table and find this entry. There is no max nor sort operator in strict relational algebra (though it does exist in SQL)
(Assuming set, and not multiset semantics):
MAX = R \ Project(Select(R x R, R1.a < R2.a),R1.a)