In previous versions of Storm, it seems it was possible to disable automatic back pressure by setting topology.backpressure.enable to false. However, with recent versions of Storm (I am using Storm 2.1.0), this option has been removed (or not fully yet). In the source code of Config, I saw the following snippet of code:
/**
* Whether to enable backpressure in for a certain topology.
*
* #deprecated: In Storm 2.0. Retained for enabling transition from 1.x. Will be removed soon.
*/
#Deprecated
#IsBoolean
public static final String TOPOLOGY_BACKPRESSURE_ENABLE = "topology.backpressure.enable";
Does anyone know how to disable this feature? The purpose is to have a fine-grained control over an experiment and I would like to avoid side effects of this feature.
Any comments/remarks on automatic back pressure on Storm will be greatly appreciated.
You can't disable it outright, but you can make the backpressure checks happen so rarely that it effectively disables the feature. Try setting the topology.backpressure.check.millis setting to a large number.
For more context see the backpressure update code, and the backpressure tracker. You can find more information at the JIRA issue for the work done for the new backpressure design here, where there is also a design document for the feature.
Related
I am working on a Spring application which has an existing mongo collection with a huge amount of data.
I need to create an index on that collection. I will use Mongobee/Mongock like migration framework to create the index.
I want to know that will this index creation affect the duration of Spring application's deployment? What if I set the background property as true for index creation ?
Basically, my desired scenario would be that :-
application's deployment should not affected by the index creation in any way
index creation should happen in background and meanwhile mongo should be able to serve queries on that collection
Please always be sure to include the version that you are running when asking questions like this. There are often important behavioral changes between the versions that are relevant for answers.
Generally speaking, background index creation does what you are looking for. Which is to say that such index builds do not lock the collection and allow the application to continue functioning while the index build is in progress.
That said, the concepts of foreground and background index builds no longer exist. As of 4.2 all index builds are effectively done in the background (ignoring the background argument if provided when issuing the command to create the index). You can read more about that here.
It may also be worth mentioning that you may also choose to build indexes in a rolling manner on a replica set. Clusters in Atlas can choose to use this automatically or you can perform this technique manually otherwise. It is uncommon for this to provide much benefit though, particularly since the new index build method was introduced in version 4.2.
while #user20042973's answer is right and very useful, it only applies to MongoDB.
However, If I am not wrong, you're also worry about Mongock's behaviour and how it may affect your deployment.
What #user20042973 explained above, combined with the use of runner-type: applicationrunner in Mongock, will provide what you are looking for:
Application starts and serves requests, without waiting for Mongock to finish.
MongoDB is available while building the index(for MongoDB version +=4.2)
The deployment of all the instances of your service will start and serve without waiting for the Mongock's lock(not to be confused with MongoDB's lock).
However, it is worth mentioning the following:
You mention the mongock configuration property indexCreation. Well, nothing to do here, it's for the internal Mongock's structure. This is for uses cases where the application doesn't have rights to create the indexes.
If the ChangeUnit fails creating the index and throws an exception, the application is aborted.
If, instead of using runner-type: applicationrunner, you use runner-type: initializingbean, you get the opposite behaviour. Your application won't start until Mongock finishes
runner-type: applicationrunner is the default
For my spring-integration setup, please see this question, which I posted yesterday. The relevant point is that I am using FileSystemPersistentAcceptOnceFileListFilter; the metadataStore is PostgreSQL.
So far, the filter works as expected; filenames and timestamps are properly archived in the metadataStore.
My concern now is that over time, the metadataStore will grow, and that without a flushing policy, it will grow unbounded. The built-in flushing policy for FileSystemPersistentAcceptOnceFileListFilter seems to be rather limited: ie, you can request that on updates to the metadataStore that it be flushed. I fear, however, that that policy may result in missed files for my use-case.
Is there any way to support a different flushing policy for the metadataStore within my Spring app? I suppose one option is to use the Spring scheduler and just periodically flush records in the metadataStore with a timestamp beyond a certain age. But I was really hoping there might be a pre-packaged way to do this in Spring.
I think you have misunderstood the meaning of Flushable - it has nothing to do with expiring store entries; it is for stores that keep data in memory - such as the PropertiesPersistingMetadataStore - by default, that store only writes the entries to disk when the application context is closed normally; flushing on each update persists to disk whenever the store changes.
There is no out of the box mechanism for removing old entries from metadata stores.
I would like to monitor my pushs' to the clients with the famous
UI.access() ... sequence on the server side.
Background is that I have to propagate lots of pushs to my client and I
want to make sure, nothing gets queued up.
I found only client RPCQueue having a size(), but I have no idea if its the correct items searching for now how to access this.
Thanks for any hint.
Gerry
If you want to know the size of the queue of tasks that have been enqueued using UI.access but not yet run, then you can use VaadinSession.getPendingAccessQueue.
This will, however, not give the full picture since it doesn't cover changes that have been applied to the server-side state (i.e. the UI.access task has already been executed) but not yet sent to the client. Those types of changes are tracked in a couple of different places depending on the type of change and the Vaadin version you're using.
For this kind of use case, it might be good to use the built-in beforeClientResponse functionality to apply your own changes as late as possible instead of applying changes eagerly.
With Vaadin versions up to 8, you do this by overriding the beforeClientResponse method in your component or extension class. You need to use markAsDirty() to ensure that beforeClientResponse will eventually be run for that instance.
Wit Vaadin 10 and newer, there's instead a UI.beforeClientResponse to which you give a callback that will be run once at an appropriate time by the framework.
I want to enable the new "Performance Insights" on active RDS.
Can I do that without expecting any downtime?
Thanks
Performance Insights does need you to mention a scheduling rule for modifications [1], which does indicate that it may need to restart the database for changes to take effect.
For Scheduling of Modifications, choose one of the following:
Apply during the next scheduled maintenance window – Wait to apply the Performance Insights modification until the next maintenance window.
Apply immediately – Apply the Performance Insights modification as soon as possible.
However, it does appear to be an instance level setting, which means that if you have a multi instance aurora cluster, then you can modify one instance at a time, and failovers should prevent you from having any downtime.
[1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.Enabling.html
At the top it of the article it states "Enabling and disabling Performance Insights doesn't cause downtime, a reboot, or a failover."
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.Enabling.html
In Websphere MQ series , command level for a queue manager is 701. What does it actually specify ?
WebSphere products use a "[version].[release].[modification].[Fix Pack]" naming convention. For example 7.0.1.6 is the current release specified down to the Fix Pack level.
Fix packs are limited to bug fixes and very limited non-disruptive functionality enhancements.
Modifications can include functionality enhancements but no API changes. For example the Multi-Instance Queue Manager was introduced in 7.0.1.
Releases may introduce significant new function and limited API changes but are highly forward and backward compatible withing the same version.
Versions encapsulate a core set of functionality. Changes at this level may sacrifice some backward compatibility in trade for significant new functionality. For example, WMQ Pub/Sub was moved from Message Broker to base MQ in the V7 release.
Since administrative functionality does not change in Fix Packs but may change at the Modification level, compatibility with administrative tools is based on the queue manager's Command Level.
There is an old but still useful TechNote which described this when the numbering conventions were adopted for WMQ.
It displays the major version number of WMQ - e.g. 530,600,700,701. Despite being 'only' a .0.1 increment, WMQ 7.0.1 gets a new major version number due to a number of internal changes (e.g. multi-instance QMs), although WMQ 6.0.1.x and 6.0.2.x were both CMDLEVEL 600
Command level, although similar to the V.R.M.F, it not exactly the same thing. The Command level is used to allow configuration applications to know what commands (and attributes within those commands) will be understood by the command server.
The first thing any configuration application should do is discover the PLATFORM and CMDLEVEL of the queue manager. Then that application can determine which commands/attributes it would be acceptable to send to that queue manager.
It is possible that CMDLEVEL could be increased in the service stream. Then the V.R.M.F. would not necessarily match the CMDLEVEL. This would happen if some new external attributes were added in the service stream, so queue managers without that patch would not understand them, but queue managers with the patch would. How does an application determine what to send? Well, the CMDLEVEL would determine that and so would have to be upped by the patch.