I have core data nested contexts setup. Main queue context for UI and saving to SQLite persistent store. Private queue context for syncing data with the web service.
My problem is the syncing process can take a long time and there are the chance that the syncing object is deleted in the Main queue context. When the private queue is saved, it will crash with the "Core Data could not fulfill faulted" exception.
Do you have any suggestion on how to check this issue or the way to configure the context for handle this case?
There is no magic behind nested contexts. They don't solve a lot of problems related to concurrency without additional work. Many people (you seem to be one of those people) expect things to work out of the box which are not supposed to work. Here is a little bit of background information:
If you create a child context using the private queue concurrency type then Core Data will create a queue for this context. To interact with objects registered at this context you have to use either performBlock: or performBlockAndWait:. The most important thing those two methods do is to make sure to invoke the passed block on the queue of the context. Nothing more - nothing less.
Think about this for a moment in the context of a non Core Data based application. If you want to do something in the background you could create a new queue and schedule blocks to do work on that queue in the background. If your job is done you want to communicate the result of the background operations to another layer inside your app logic. What happens when the user deleted the object/data in the meantime which is related to the results from the background operation? Basically the same: A crash.
What you experience is not a Core Data specific problem. It is a problem you have as soon you introduce concurrency. What you need is to think about a policy or some kind of contract between your child and parent contexts. For example, before you delete the object from the root context you should cancel all of the operations/blocks which are running on other queues and wait for the cancellation to finish before you actually delete the object.
Related
I have a spring boot application (let's say it's called app-1) that is connected to a kafka cluster and that consumes from a specific topic, let's say the topic is called "foo". Topic foo always receives a message when another application (let's say it's called app-2) has imported a new foo-item into the database.
The topic is primarily meant to be used in a third application (let's say it's called app-3) which sends out some e-Mail notification to people that may be interested in this new foo-item. App-3 is clustered, meaning there are multiple instances of it running at the same time. Kafka automatically balances the foo-topic messages between all these instances because they use the same consumer-id. This is good and in the case of app-3 it is actually desired.
In the case of app-2, however, the messages from the foo-topic are used for cache eviction. The logic is, basically, that if there is a new foo-item then the currently existing caches should probably be cleared, because their content depends on the foo-items. The issue is that app-2 is also clustered, which means that by default kafka-logic, every instance will only receive some of the messages sent to the foo-topic. This does not work correctly for this specific app tho, because whenever there is a new foo-item, all of the instances need to know about it because all of them need their clear their local caches.
From what I understand I have these two options if I want to keep the current logic:
Introduce a distributed cache for all instances of app-2 so that they all share the same cache. Then it does not matter if only one instance receives a foo-item, because the cache eviction will also affect the cache of the other instances; even though they never learned about the foo-item. I would like to avoid this solution, as a distributed cache would add a noticeable amount of complexity and also overhead.
Somehow manage to use a different consumer-id for each instance of app-2. Then they would be considered different consumers by kafka and they all would get each foo-topic message. However, I don't even know how to programmatically do this. The code of the application is not aware of replicated instances, there is no way to access any information about what node it is. If I use a randomly generated string on startup, then each time such instance restarts it would be considered a new consumer and would have to re-process all previous messages. That would be incorrect behavior as well.
Here is my bottom line question: Is it possible to make all instances of app-2 receive all messages from the foo-topic without completely breaking the way kafka is supposed to work? I know that it is probably very unconventional to use kafka-messages for cache eviction and I am entirely able to find an alternative mechanism for the cache eviction logic that does not depend on kafka-topic messages. However, the applications are for demonstration purposes and I thought it would be cool if more than one app read from this topic. But if I end up having to hack a dirty workaround to make it work then it's also bad for demonstration purposes and I would rather implement an alternative way of cache eviction.
As you mentioned, you could use different consumer ids with random strings.
If notifications are being read from the beginning, then you probably have ConsumerConfig.AUTO_OFFSET_RESET_CONFIG set to "earliest" somewhere in your consumer configuration. If this is the case, removing it will probably solve your problems - when the app will start it will only receive notification sent after the consumer started listening.
I have a persistent actor which receives many messages. Fist message is CREATE (case class) and next messages are UPDATEs (case classes). So if it receives CREATE then it should not go into persistence to run recovery because the storage is empty for this actor. It's performance wasting from my perspective.
Is there any possibility to do not call recovery for particular input message (the first one which is CREATE), please?
A persistent actor will always have to hit the database, because there is no other way to know whether it having existed before - it could have been created in a previous instance of the application that was stopped or it could have been created on a different node in a cluster.
In general a good pattern for performance is to keep the actor in memory after it has been hit the first time, as that will allow as fast responses as possible. The most common way to do this is using Cluster Sharding (which you can read more about in the docs here: https://doc.akka.io/docs/akka/current/cluster-sharding.html?language=scala#cluster-sharding
I have never heard of anyone seeing the hit for an empty persistent actor as a performance problem and I'm not sure it is possible to solve that in a general way, so if you have such a problem and somehow can know the actor was never created before you can not do that with Akka Persistence but would have to build a special solution for that yourself.
We are using microservices, cqrs, event store using nodejs cqrs-domain, everything works like a charm and the typical flow goes like:
REST->2. Service->3. Command validation->4. Command->5. aggregate->6. event->7. eventstore(transactional Data)->8. returns aggregate with aggregate ID-> 9. store in microservice local DB(essentially the read DB)-> 10. Publish Event to the Queue
The problem with the flow above is that since the transactional data save i.e. persistence to the event store and storage to the microservice's read data happen in a different transaction context if there is any failure at step 9 how should i handle the event which has already been propagated to the event store and the aggregate which has already been updated?
Any suggestions would be highly appreciated.
The problem with the flow above is that since the transactional data save i.e. persistence to the event store and storage to the microservice's read data happen in a different transaction context if there is any failure at step 9 how should i handle the event which has already been propagated to the event store and the aggregate which has already been updated?
You retry it later.
The "book of record" is the event store. The downstream views (the "published events", the read models) are derived from the book of record. They are typically behind the book of record in time (eventual consistency) and are not typically synchronized with each other.
So you might have, at some point in time, 105 events written to the book of record, but only 100 published to the queue, and a representation in your service database constructed from only 98.
Updating a view is typically done in one of two ways. You can, of course, start with a brand new representation and replay all of the events into it as part of each update. Alternatively, you track in the metadata of the view how far along in the event history you have already gotten, and use that information to determine where the next read of the event history begins.
Inside your event store, you could track whether read-side replication was successful.
As soon as step 9 suceeds, you can flag the event as 'replicated'.
That way, you could introduce a component watching for unreplicated events and trigger step 9. You could also track whether the replication failed multiple times.
Updating the read-side (step 9) and flagigng an event as replicated should happen consistently. You could use a saga pattern here.
I think i have now understood it to a better extent.
The Aggregate would still be created, answer is that all the validations for any type of consistency should happen before my aggregate is constructed, it is in case of a failure beyond the purview of the code that a failure exists while updating the read side DB of the microservice which needs to be handled.
So in an ideal case aggregate would be created however the event associated would remain as undispatched unless all the read dependencies are updated, if not it remains as undispatched and that can be handled seperately.
The Event Store will still have all the event and the eventual consistency this way is maintained as is.
I have a NSOperationQueue set to NSOperationQueueDefaultMaxConcurrentOperationCount. It is filled with NSOperation objects (nothing weird so far). I subclassed the NSOperation to do some background tasks.
Download data from the internet.
Parse the data so I can read it.
Create a NSManagedObject:
[NSEntityDescription insertNewObjectForEntityForName:#"Channel" inManagedObjectContext:context];
Save it with the context.
[managedObjectContext save:&error]
I like this all to happen in the background so the UI won't get blocked. I read this article about concurrency with core data, and as far as I understood it. The best way would be to create a new NSManagedObjectContext in every NSOperation, but share the same persistent store coordinator.
That's easily done, however, when it comes to saving the context it says in the documentation it is error prone to do so. So my question is the following:
If I have different operations running in the NSOperationQueue, could those operations interfere with each other while saving the managed object context? Or does it wait to execute the following operation till the saving has been complete?
Can I safely save the context in a NSOperation? Or is it really bad practice?
I hope someone can shine a light on this matter, because I am really stuck at the moment.
What you need to do is the following:
Create a managed object context for each NSOperation. Create this new context on the main method, because this is when it's executing on the right thread.
Assign the context persistent store coordinator.
Create an observer to receive the NSManagedObjectContextDidSaveNotification. This is the only way the main context will know at the time the changes were made on the NSOperation's context. Make sure the merge call is made on the thread/block the merging context lives in. If you are merging with the main thread's context, call the mergeChangesFromContextDidSaveNotification: method on the main thread with the notification from the NSOperation's context.
Also, ask yourself if you really want to have all these operations working concurrently. Per the documentation:
The default maximum number of operations is determined dynamically by the NSOperationQueue object based on current system conditions.
You do not have control over how many NSOperations will be operating at the same time. If this is not what you want, you might be better if you just go with a serial NSOperationQueue (maxConcurrentOperation=1), considering the fact that you are going to be locking the database to do the save, and also because you have networking being done as well.
You can safely save inside the NSOperation's main method, if you take the precautions mentioned above.
Morning,
I'm using the SimpleEvent bus to send data from my centralized data reviver to the Widgets. This works really fine, I get one set of new Data form the server, the success method of the RPC call puts it on the Eventbus, each widget looks if the data is for it, if yes it 'displays' it, if not, it does nothing.There is only one data set per request and the widgets don't depend on other data being already sent.
Now I have a Tree widget. The child nodes of the Tree are created throw this data sets too, and this child nodes register itself to the Eventbus to revive the data for their child nodes. The data shall be received in on rush (for performance reasons obv), so I will get multiple data sets which are put on the Eventbus at the 'same time' (in a for loop). I only control the order in which they are put there (first the root, then the data for the first child......). How does the Eventbus now proceeds the events?
Does he wait till the first event is completed, so the first child of
the tree already finished creation and register itself to the
Eventbus, to revive the data to create it's child's.
Does he handle them simultaneous, so a widget isn't even registered to the Eventbus.
Does he mix up the order?!?!
Current solution approaches:
The best solution I can think of, is to only put new events on the
Eventbus when the previous got completed. However I found a method
which does so, or if it is the standard behavior of the Eventbus .
Fire a request processing finished event, when a event was processed by a widget. Yucks... this leads to a lot of additional code and causes big problems, when data is put on the Eventbus which doesn't belong to any widget....
Register a static variable which is set to true when the request got handled and the Eventbus waits this long till he puts the next request on the Eventbus (Quiet similar to two, but way worse coding style and the same problems)
All events are handled by the root tree element, which sends them upwards to the respective child's.
Which solution would you prefer and why?
Regards,
Stefan
PS: my favorite answer would be that 1. is the standard behavior of the Eventbus^^
PPS: The solution should also be working on when introducing Webworkers.
The EventBus#fireEvent is synchronous. It's by design. You can pass an event to the bus, have handlers possibly modify it, and when execution returns to your method you can check the event; this is used for PlaceChangeRequestEvent and its setMessage for instance.
FYI, if a handler throws an exception, it won't prevent other handlers from being executed. The fireEvent will then wrap the exceptions (plural; several handlers can throw) in an UmbrellaException.
Although EventBus is a nice way of de-coupling parts of your application it doesn't mean it should be "overused".
I also think you should be careful not to circumvent the asynchronous behavior of your client-side code by introducing synchronous/blocking like behavior.
Javascript is single threaded so I don't think you can have two events at the same time. They will be executed one after the other.
If you fire an event on the EventBus (i.e. SimpleEventBus) it will just iterator through the list of attached handlers and execute them. If no handler is attached nothing happens.
I personally would prefer the 4th. approach especially if you plan to use a CellTree some time in the future. The Tree widget/CellTree widget handles the event and constructs its structure by traversing through the object.