We are receiving a substantial blob of data from a server and storing it the store in ngrx. Some of it should be organized as a map from keys to values. So, we wrote:
Immutable.Map<string, string>()
as the data type. (There are other places where the types would be <string, some_serializable_class> We enabled all of the runtime checks, and ngrx reminded us that these maps are not serializable. It would not take very much code to safely serialize and deserialize them, of course.
Is there a place in the ngrx architecture to put our own serialization? If we do so, will the runtime check be aware?
#ngrx/entity turns out to be the intended solution to this problem?
Related
What is Redis Value operations in Spring boot?
Is it like we can directly store Key-value pair in Redis database without creating the entity and stuff just by using RedisTemplate<String, Object> ?
Also, if we use ValueOperations how will it impact the performance?
When using Redis, you should think about what data format/datatype suits your needs best, similar to what you would do when coding in any general programming language. All those operations, ValueOperations, ListOperations, SetOperations, HashOperations, StreamOperations are the support provided for interacting with the mentioned datatypes. They are provided by the RedisTemplate.
When you are using ValueOperations, you are more or less treating your whole Redis instance as a giant hash map. For example, you can store entries in Redis like current_user = "John Doe". However, you can also do something silly such as keeping a string representation of a huge hashmap against a key, top_users = <huge_string_representing_a_hash_map> when thinking from the perspective of the second case, what if you want to get the value for one key in the mentioned hash map. Then, the task becomes more or less impossible without transferring the whole hash map in RAM. Yet, if you have used Redis Hashes and HashOperations that would have been a more trivial task.
Going back to your question, if you want to store a simple object using ValueOperations. That wouldn't degrade the performance. In contrast, if you are moving huge maps around, you'll utilise a lot of your network bandwidth and RAM capacity.
In summary, choose your Redist data types carefully to suit your needs.
https://redis.io/topics/data-types
Writing some code in dart I found recently a case when it is simpler to use certain objects as map keys. Actually this code is executed in web mode (compiled to javascript) but eventually it will be used also in dart VM mode.
Does using objects as keys of map can have significant memory / performance impact in dart?
I didn't found a lot of resources on benchmarking that (https://dart.dev/articles/benchmarking is defunct), so any directions are also welcome.
There are some considerations:
Memory
Since each object used as keys will be refereed by the Map, each object cannot be garbage collected unless the Map itself can be garbage collected or the key has been removed from the Map. The same can be said for the associated value for each key.
As of storage used by the Map for keeping the keys and values, it should not make any difference at all if you are using e.g. String as a key or a custom object since both kind of keys will just be saved as references.
Performance
The performance of operations on a Map are very much a question about the performance of the == operator and hashCode property since both are used for several of the operations used on a Map.
I can recommend reading about them here:
https://api.dart.dev/stable/2.7.2/dart-core/Object/operator_equals.html
https://api.dart.dev/stable/2.7.2/dart-core/Object/hashCode.html
Conclusion
A lot of projects are using custom objects as keys for Maps and usually there are no problems of during that. In fact, using a custom object are no more different than using e.g. a String as key since String are also just a normal class with its own == operator and hashCode property.
I'm learning about redis/memcache and redis is clearly the more popular option. My question is about supported data types. At my company we use the memcashier library which is built in memcached. We store temporary user data when they're making a purchase in memcache. We can easily update this object as things are added to the cart or more info about the user is given. This appears to be the same functionality as a hash in redis. I don't understand how this is only a basic string data type and how it's less powerful than a hash.
If you are using strings, that's fine - but any change involves loading the data to your application, parsing it, modifying it, and serializing it back to Redis/Memcache.
This has two problems: it's slow and non atomic. You can have two servers modifying the same object arriving in an inconsistent state - such as double or missing items in a shopping cart. And again, it's slow.
With a Redis hash key, you can atomically modify specific fields of the object without loading the entire object into memory. Instead of read, parse, modify, save - you just update.
Besides, Redis has many many data structures that can create very flexible data stores with different properties, whereas Memcache can only store strings.
BTW Redis has a module that allows you to store JSON objects just as you would a string, and manipulate them directly and atomically without getting them to the client. See Rejson.io for details.
Memcached doesn't support complex datastructures
In redis you have Lists, Sets, SortedSets, HashTables , and more.
Each data-structure mentioned above supports mutation of one or more of its elements atomically and without replacing the entire data-structure/value.
Memcached on the other hand , is a simple key-value store - that means every operation involving an attribute change within a complex object is a read-modify-write. If you just go around blindly replacing fields in objects then you are risking race-conditions and operations atomicity issues (which you can get away from by using CAS )
If the library abstracts that complexity, well - that's great but it's still less efficient than mutating only the relevant field(s)
This answer only relates to your usecase. Redis holds many other virtues over memcached, which are not relevant to this question.
I have a simple Person object that contains some basic information about a person. If I want to send it by JMS:
I can convert this object into JSON, then deliver it as a String object.
I can use Person object as the payload directly.
I'm using ActiveMQ as JSM provider. Which way is faster?
And what if I need to send a Map or List as the payload?
It's all about the performance of serialization, not much about jms/activemq. So an ObjectMessage is a binary blob at transport that uses java serialization and for the string message, you can choose whatever serialization processor you want.
This article with runnable benchmarks shows that json serialization can be as fast as java object serialization. Although the article is obviously biased, you can note that also jackson/JSON serialization and java serialization is pretty close in terms of performance.
I guess you can measure yourself, with your kind of data. Either way, it's likely a micro optimization. If serialization speed truly matters that much, see if you can optimize in terms of size/quantity in terms objects sent.
As a final note, if you deal with very large payloads, the size and therefore the transport time will contribute to performance. In that case, you may want to make sure your json is not indented and possibly also compressed.
according to Apache AVRO project, "Avro is a serialization system". By saying data serialization system, does it mean that avro is a product or api?
also, I am not quit sure about what a data serialization system is? for now, my understanding is that it is a protocol that defines how data object is passed over the network. Can anyone help explain it in an intuitive way that it is easier for people with limited distributed computing background to understand?
Thanks in advance!
So when Hadoop was being written by Doug Cutting he decided that the standard Java method of serializing Java object using Java Object Serialization (Java Serialization) didn't meet his requirements for Hadoop. Namely, these requirements were:
Serialize the data into a compact binary format.
Be fast, both in performance and how quickly it allowed data to be transfered.
Interoperable so that other languages plug into Hadoop more easily.
As he described Java Serialization:
It looked big and hairy and I though we needed something lean and mean
Instead of using Java Serialization they wrote their own serialization framework. The main perceived problems with Java Serialization was that it writes the classname of each object being serialized to the stream, with each subsequent instance of that class containing a 5 byte reference to the first, instead of the classname.
As well as reducing the effective bandwidth of the stream this causes problems with random access as well as sorting of records in a serialized stream. Thus Hadoop serialization doesn't write the classname or the required references, and makes the assumption that the client knows the expected type.
Java Serialization also creates a new object for each one that is deserialized. Hadoop Writables, which implement Hadoop Serialization, can be reused. Thus, helping to improve the performance of MapReduce which accentually serializes and deserializes billions of records.
Avro fits into Hadoop in that it approaches serialization in a different manner. The client and server exchange a scheme which describes the datastream. This helps make it fast, compact and importantly makes it easier to mix languanges together.
So Avro defines a serialization format, a protocol for clients and servers to communicate these serial streams and a way to compactly persist data in files.
I hope this helps. I thought a bit of Hadoop history would help understand why Avro is a subproject of Hadoop and what its meant to help with.
If you have to store in a limited file the information like the hierarchy or data structure implementation details and pass that information over a network, you use data serialization. It is close to understanding xml or json format. The benefit is that the information which is translated into any serialization format can be deserialized to regenerate the classes, objects, data structures whatever that was serialized.
actual implementation-->serialization-->.xml or .json or .avro --->deserialization--->imlementation in original form
Here is the link to the list of serialization formats. Comment if you want further information! :)