Say I want to store user preferences...something simple like this:
{
"favoriteColor": "green",
"bestFriends": [
"Tom",
"Jenny",
"Horton"
]
}
What's the best, most performant way to store this in redis cache (optimized for reads)?
Imagine UserId = 123
NOTE: Below I'm using the Redis documentation's way of representing the various structures. See here.
Simple, flat, key/value pairs right in the root?
user-123-favoriteColor = green (this is a STRING type)
user-123-bestFriends =
1) "bestFriends" (SET TYPE)
2) "Tom"
3) "Jenny"
4) "Horton"
Hierarchical structure (hash of values)
user-123 =
1) "favoriteColor" (STRING type)
2) "green"
3) "bestFriends" (SET TYPE)
4) "Tom"
5) "Jenny"
6) "Horton"
And a related question...is there any reason not to store user preferences in redis vs the domain sql database?
And one more related question...is it a bad idea to store all users under one root key called "users"?
Hierarchical structure should be preferred.
This answer gives a lot of explanation and helped me.
From the horses mouth:
Use hashes when possible
Small hashes are encoded in a very small space, so you should try representing your data using hashes every time it is possible. For instance if you have objects representing users in a web application, instead of using different keys for name, surname, email, password, use a single hash with all the required fields.
If you want to know more about this, read the next section.
So the answer is yes, use hashes where possible.
The answer here is a good way to do it.
Related
My question is theoretical,
I'm trying to make a design for a mapreduce example in Big data processing.
The case which I have requires a pair of keys to be mapped to a pair of values.
for example if we have below text:
"Bachelors in Engineering has experience of 5 years"
I am trying to count the words Engineering & Experience in a way where I would have a value for each word separately.
So my key would be (Engineering,Experience) and my value would be (1,1) as per the above given text example.
Note that there is a relationship between both key values in my homework, therefore I want them both in one set of a key-value to determine if both keys are mentioned in one text file, or only one key is mentioned, or none is mentioned.
Please let me know if above case is possible to do in map-reduce of big data or not..
Having a string key of "(Engineering,Experience)" is no different than just having a String of one of those words.
If you want to have some more custom type, then you will want to subclass the Writable and maybe the WritableComparable interfaces.
Simlarly, for the value, you could put the entire tuple as Text and parse it later, or you can create your own Writable subclass that can store two integers.
Thanks for the Answer, but I figured I could use "Engineering Experience" as a string for the key.
I have a lot of javascript objects like:
var obj1 = {"key1" : value1, "key2" : value2, ...}
var obj2 = {"key3" : value3, "key4" : value4, ...}
and so on...
Following are the two approaches :
Store each object as Redis Hash i.e. one-to-one mapping.
Have one Redis Hash(bucketing can be done for better performance), store each object as stringified object in each key of hash i.e. for each object having a key value pair in the Redis Hash. Parse the object when we need to use the object.
1) -> Takes more space than 2) but has better performance than 2)
2) -> Takes less space than 1) but has worse performance than 1)
Is there a way to determine which approach would be better in the long run?
Update: This data is used on the client side (AngularJS), so all parsing of stringified JSON is done in the frontend.
This would probably be solved by deciding which method minimises the number of steps required to extract the required data from redis.
Case 1: Lots of nested objects
If your objects have a lot of nesting, ie objects within objects, like this,
obj = {key1:{key2:value1, key:3{key4:value2}}}
You should probably stringify and store them.
Because Redis does not allow nesting of data structures. You can't store a hash within another hash.
And storing the name of hash2 as a key within hash1 and querying hash2 after getting hash1 and so on is unnecessarily complex and has a lot of queries. In this case all you have to do is get the entire string from Redis and JSON.parse it. and you can get whatever data you want from the Object.
Case 2: No nested objects.
But on the other hand, if there is no nesting of objects and you store it as a string, you have to JSON.parse() every time you get the data from Redis. And parsing JSON is blocking and is CPU intensive. Node.js: does JSON.parse block the event loop?
Redis documentation also says that hashes are encoded in a very small space, so you should try representing your data using hashes every time it is possible. http://redis.io/topics/memory-optimization
So, in this case, you could probably go ahead and store them all as individual hashes as querying a particular value will be a lot easier.
---------Update---------
Even if the JSON parsing is done on the client, try not to do an extra computation needlessly :)
But nested objects are easier to store and query as a string. Otherwise, you'll have to query more than one hash table. In this case storing as stringified object might just be better for performance.
Redis stores small hashes very efficiently. So much that storing multiple small hashmaps is more memory efficient than one big hashmap.
the number of keys deciding about the encoding to use can be found in redis.conf
hash-max-zipmap-entries 512
also the value of each key should be hash-max-zipmap-value 64
So, you can now decide on the basis of nesting of your objects, number of Hash Keys below which Redis is more memory efficient and the value assigned to your keys.
Do go through http://redis.io/topics/memory-optimization
For example, key 1 will have values "A","B","C" but key 2 will have value "D". If I use
Map<String, List<String>>
I need to populate the List<String> even when I have only single String value.
What data structure should be used in this case?
Map<String,List<String>> would be the standard way to do it (using a size-1 list when there is only a single item).
You could also have something like Map<String, Object> (which should work in either Java or presumably C#, to name two), where the value is either List<String> or String, but this would be fairly bad practice, as there are readability issue (you don't know what Object represents right off the bat from seeing the type), casting happens during runtime, which isn't ideal, among other things.
It does however depend what type of queries you plan to run. Map<String,Set<String>> might be a good idea if you plan of doing existence checks in the List and it can be large. Set<StringPair> (where StringPair is a class with 2 String members) is another consideration if there are plenty of keys with only 1 mapped value. There are plenty of solutions which would be more appropriate under various circumstances - it basically comes down to looking at the type of queries you want to perform and picking an appropriate structure according to that.
I am dealing with a csv file that has some customer information (email, name, address, amount, [shopping_list: item 1, item 2]).
I would like work with the data and produce some labels for printing... as well as to gather some extra information (total amounts, total items 1...)
My main concern is to find the appropriate structure to store the data in ruby for future manipulation. For now I have thought about the following possibilities:
multidimensional arrays: pretty simple to build, but pretty hard to access the data in a beautiful ruby way.
hashes: having the email as key, and storing the information in different hashes (one hash for name, another hash for address, another hash for shopping list...)
(getting the cvs data in to a Database and working with the data from ruby??)
I would really appreciate your advice and guidance!!
Once you have more than a couple pieces of information that you need to group together, it's time to consider moving from a generic hash/array to something more specialized. A good candidate for what you've described is Ruby's struct module:
Customer = Struct.new(:email, :name, :address) # etc.
bill = Customer.new('bill#asdf.com', 'Bill Foo', '123 Bar St.')
puts "#{bill.name} lives at #{bill.address} and can be reached at #{bill.email}"
Output:
Bill Foo lives at 123 Bar St. and can be reached at bill#asdf.com
Struct#new simply creates a class with an attr_accessor for each symbol you pass in. Well, it actually creates a bit more than that, but for starters, that's all you need to worry about.
Once you've got the data from each row packed into an object of some sort (whether it's a struct or a class of your own), then you can worry about how to store those objects.
A hash will be ideal for random access by a given key (perhaps the customer's name or other unique ID)
A one-dimensional array works fine for iterating over the entire set of customers in the same order they were inserted
I need to be able to lookup based on the full key or part of the key..
e.g. I might store keys like 10,20,30,40 11,12,30,40, 12,20,30,40
I want to be able to search for 10,20,30,40 or 20,30,40
What is the best data structure for achieving this..best for time.
our programming language is Java..any pointers for open source projects will be appreciated..
Thanks in advance..
If those were the actual numbers I'd be working with, I'd use an array where a given index contains an array of all records that contain the index. If the actual numbers were larger, I'd use a hash table employed the same way.
So the structure would look like (empty indexes elided, in the case of the array implementation):
10 => ((10,20,30,40)),
11 => ((11,12,30,40)),
12 => ((11,12,30,40), (12,20,30,40)),
20 => ((10,20,30,40), (12,20,30,40)),
30 => ((10,20,30,40), (11,12,30,40), (12,20,30,40)),
40 => ((10,20,30,40), (11,12,30,40), (12,20,30,40)),
It's not clear to me whether your searches are inclusive (OR-based) or exclusive (AND-based), but either way you look up the record groups for each element of the search set; for the inclusive search you find their union, and for the exclusive search you find their intersection.
Since you seen to care about retrieval time over other concerns (such as space), I suggest you use a hashtable and you enter your items several times, once per subkey. So you'd put("10,20,30,40",mydata), then put("20,30,40",mydata) and so on (of course this would be a method, you're not going to manually call put so many times).
Use a tree structure. Here is an open source project that might help ... written in Java :-)
http://suggesttree.sourceforge.net/