I am working on a Spring application which has an existing mongo collection with a huge amount of data.
I need to create an index on that collection. I will use Mongobee/Mongock like migration framework to create the index.
I want to know that will this index creation affect the duration of Spring application's deployment? What if I set the background property as true for index creation ?
Basically, my desired scenario would be that :-
application's deployment should not affected by the index creation in any way
index creation should happen in background and meanwhile mongo should be able to serve queries on that collection
Please always be sure to include the version that you are running when asking questions like this. There are often important behavioral changes between the versions that are relevant for answers.
Generally speaking, background index creation does what you are looking for. Which is to say that such index builds do not lock the collection and allow the application to continue functioning while the index build is in progress.
That said, the concepts of foreground and background index builds no longer exist. As of 4.2 all index builds are effectively done in the background (ignoring the background argument if provided when issuing the command to create the index). You can read more about that here.
It may also be worth mentioning that you may also choose to build indexes in a rolling manner on a replica set. Clusters in Atlas can choose to use this automatically or you can perform this technique manually otherwise. It is uncommon for this to provide much benefit though, particularly since the new index build method was introduced in version 4.2.
while #user20042973's answer is right and very useful, it only applies to MongoDB.
However, If I am not wrong, you're also worry about Mongock's behaviour and how it may affect your deployment.
What #user20042973 explained above, combined with the use of runner-type: applicationrunner in Mongock, will provide what you are looking for:
Application starts and serves requests, without waiting for Mongock to finish.
MongoDB is available while building the index(for MongoDB version +=4.2)
The deployment of all the instances of your service will start and serve without waiting for the Mongock's lock(not to be confused with MongoDB's lock).
However, it is worth mentioning the following:
You mention the mongock configuration property indexCreation. Well, nothing to do here, it's for the internal Mongock's structure. This is for uses cases where the application doesn't have rights to create the indexes.
If the ChangeUnit fails creating the index and throws an exception, the application is aborted.
If, instead of using runner-type: applicationrunner, you use runner-type: initializingbean, you get the opposite behaviour. Your application won't start until Mongock finishes
runner-type: applicationrunner is the default
Related
I was looking at saving some data to my Room database and was reevaluating as there are some places in my repositories where I am extending AsyncTask (I'm still using Java) and wanted to check on the state of things to see if it was a good time to swap them out. I saw this reference in the Android developer site on Approaches to background work.
All persistent work: You should use WorkManager for all forms of
persistent work. Immediate impersistent work: You should use Kotlin
coroutines for immediate impersistent work. For Java programming
language users, see Threading on Android for recommended options.
Long-running and deferrable impersistent work: You should not use
long-running and deferrable impersistent work. You should instead
complete such tasks through persistent work using WorkManager.
I started using WorkManager for an API which needed to be called, but for which I could not rely on network connectivity. Because I'm using Room, which is persistent, it seems like I should be using WorkManager.
It defines persistent work as:
Persistent work: Remains scheduled through app restarts and device reboots.
A database insert/update/delete is persistent by this definition. Scheduled throws me off a little, as I want it to be immediate, but according to this chart that would still apply.
Is anybody using WorkManager as the mechanism for CUD operations in their repositories and if so, do they have an example?
It would be great to see how this all works in an update fragment. If a single item is selected and I am viewing it in a fragment, when changes are made I would need to update the database using a Worker class and view the data using a LiveData object, correct?
Inserts and returning the id (or object) would be interesting to see as well.
I would like to monitor my pushs' to the clients with the famous
UI.access() ... sequence on the server side.
Background is that I have to propagate lots of pushs to my client and I
want to make sure, nothing gets queued up.
I found only client RPCQueue having a size(), but I have no idea if its the correct items searching for now how to access this.
Thanks for any hint.
Gerry
If you want to know the size of the queue of tasks that have been enqueued using UI.access but not yet run, then you can use VaadinSession.getPendingAccessQueue.
This will, however, not give the full picture since it doesn't cover changes that have been applied to the server-side state (i.e. the UI.access task has already been executed) but not yet sent to the client. Those types of changes are tracked in a couple of different places depending on the type of change and the Vaadin version you're using.
For this kind of use case, it might be good to use the built-in beforeClientResponse functionality to apply your own changes as late as possible instead of applying changes eagerly.
With Vaadin versions up to 8, you do this by overriding the beforeClientResponse method in your component or extension class. You need to use markAsDirty() to ensure that beforeClientResponse will eventually be run for that instance.
Wit Vaadin 10 and newer, there's instead a UI.beforeClientResponse to which you give a callback that will be run once at an appropriate time by the framework.
I have a Spring Boot / Batch app. I want to use an "app data directory" (not the same as a properties file) versus a db based datastore (ie: SQL/Mongo).
The data stored in the app data directory is aggregated from several webservices and stored as XML. Each Step within the Job will fetch data and write locally, then the next Step in the chain will pick up the created Files and process for the next step (and so on).
The problem here, is each Step will only fetch previous app run data. For example, the data at app start time and not directly after the Step execution.
I understand what is happening here, that Spring is checking for any resources at launch and using them as-is before the Step actually is run.
Is there a magic trick to requesting Spring to stop loading specified resources/Files at app launch?
Note: Using Java Config, not XML and the latest Spring/Boot/Batch versions, also tried #StepScope for all reader/writers
Repo: https://github.com/RJPalombo/salesforceobjectreplicator
Thanks in advance!
No, there is no magic :-)
Firstly, your code is very well structured and easy to understand.
The first thing, that pops in my eyes is: Why aren't you using the standard readers and writers from springbatch (FlatFileItemReader/Writer, StaxReader/Writer). There is no need to implement this logic by yourself.
As far as I see, the problem is that you load the whole data in the constructor of the readers.
The whole job-structure (together with step, reader, writer, and processor instances) is created when the spring context is loaded, way before the job actually is executed.
Therefore, the reader just read empty files.
The simplest fix you could make is to implement the "ItemStream" interface for all your readers and writers.
And then reading the data in the open method, instead of the constructur. The open method is called, just before the steps get executed.
But that is only a quick fix and only helps to understand the behaviour of springbatch. The problem with this approach is, that all data is loaded at once, which means, that the memory usage will increase with the amount of data; hence, the memory would blow up when reading lots of data. Something we don't want to have when doing batch processing.
So, I'll recommend that you have a look at the standard readers and writers. Have a look how they work, debug into them. See when the open/close methods are called; check what happens when the read method is called and what it does.
It is not that complicated and having a look at your code, I'm sure that you are able to understand this very fast.
I am using hibernate, spring, jpa.
In a workflow I update an entity; but these updates are not available in another workflow. When I restart the server it works fine.
Is there a way so that when I update an entity; I ask hibernate to remove it from whatever cache it has.. So that when that object is needed by any other workflow a fresh query is made ?
This sounds like you have two separate sessions for the same app, thus, having two 1st level caches. The first level cache is the one that Hibernate uses for itself, in the context of a session. So, if you don't close/clear your session, this will keep growing, possibly conflicting with other 1st level caches (in other threads or in other VMs). It's hard to say if that's the case, as you didn't specify your environment, but you can't change another session's first level cache.
The best solution to avoid this is to use a managed EntityManager (from your application server) to deal with entities. It's then the server's role to deal with this kind of scenario. But it seems that you are doing it the "spring way", so, you'll have to do it manually: either clear the session after you use it, or do a refresh before reading/updating your data. You'll then need some sort of locking (pessimistic/optimistic) to not lose information that might have been changed from another thread.
I have created apps in the past that would have web pages that would call the persistence layer to get some query results or to insert, delete, etc against a db. However, nothing was left running in the background except for the persistence layer. Now I need to develop an app that has an process that is always running in the background, which is waiting for messages to come thru a zeromq messaging system (cannot change this at this point). I am a little lost as to how to setup the object so that it can always be running and yet I can control or query the results from the object.
Is there any tutorial/examples that covers this configuration?
Thanks,
You could use some kind of timer, to start a method every second to look at a specific ressource and process the input taken from that.
If you use Spring than you could have a look at the #Scheduled annotation.
If your input is some kind of java method invokation, than have a look at the java.util.concurrent Package, and concurrent programming at all. -- But be aware of the fact, that there are some restictions one creating own Threads in an EJB environment.