I did not find a way to update a module count once a stream is deployed. I wonder if it is currently a supported feature?
I find it hard to believe that the only way to do it is by un-deploying and re-deploying the stream as it actually mean downtime in a production environment, when all I want is to scale-up.
Any help is very much appreciated!
Module count can not be modified once the stream is deployed. The obvious reason is not to override the stream partitioning that is established. If you want dynamic scaling, you can specify module.*.count=0 and any new container that matches the deployment criteria would have these modules deployed. Note that this works in case of direct binding when all the modules are co-located on the same container.
Related
I am working on a spring boot application in java. I have a case where I need to pick up some runtime configuration(which can change dynamically without any need of deployment or rebooting the app) and use it in all the classes further ahead. It basically stores all the plug and play configurations I want my app to support.
I have tried listener pattern but it doesn't seem to be the best option since I don't want that config to be listened by a few, rather I want that config to flow through whole of the code.
Is there an existing design pattern or a technique that is a standard for such activity?
Kindly suggest.
In general you need to store the application configuration externally and then listen to changes on that. How the change is notified depends on the runtime environment. For example cloud platforms may have separate decoupled notification systems, desktop application may want to implement a file change listener and so on.
This pattern is generally called Runtime Reconfiguration Pattern (see this link). In conclusion there is no magic that will apply the changes throughout your code, but you will need to listen for the changes and adjust the runtime behavior based on those.
I am trying to shift from environment files in Chef to using Policyfiles. https://docs.chef.io/policy.html. I really like the concept, especially since you can include a policy from policy into another, but I am trying to understand how do a simple attribute change.
For instance, if I want to change a globally-used attribute that may be an error message for a problem that is happening now. ("The system will be down for 10 minutes. Thanks for your patience"). Or perhaps I want to turn off some AB testing with an attribute working as a flag. From what I can tell, the only way I can do this is to change an attribute in the policyfile, and then I need to create a new version of the policy file.
And if the policyfile is an included in many other policyfiles, like in the case of a base policyfile, then I have a lot of work to do for a simple change.
default['production']['maintenance_message'] = 'We will be down for the next 15 minutes!'
default['production']['start_new_feature'] = true
How do I make a simple change to an attribute that affects an entire policy group? Is there a simple way to change an attribute, or do I have to move all my environment properties to a data bag??
OK, I used Chef Support and got an answer: Nope.
This is their response:
"You've called out one of the main reasons we recommend that people use something like Jenkins pipelines to deliver cookbooks to their infrastructure. All that work can be kicked off by a build system recognizing a change in a dependency and initiating new builds for all the downstream consumer jobs.
For what it's worth, I don't really like putting application configurations like that maintenance message example you used in configuration management, I think something like Habitat is a better system for that kind of rapid-change configuration delivery, although you could also go down the route of storing application configuration like that in a system like Hashicorp Vault, Consul, or etcd, and ensuring that whatever apps need to ingest those changes are able to do so without configuration management fighting with the key-value config store.
If that was just an example to illustrate things, ignore the previous comment and refer only to my recommendation to use pipelines to deliver cookbooks, attributes, etc to your infrastructure (and I would generally recommend against data bags these days, but that's mostly a preference thing)."
I want suggestions for my application:
I have Multitenancy in Nifi. For each Process group, I have different Tenants/Users.
For any changes in one Tenant/user like in his custom processor(.nar file will create), we need to copy-paste that .nar file into lib folder and again restart the nifi. But due to this full Nifi server has restarted because of that Each Tenant/User and Processes group get restarted.
So, Please give Some Suggestions So that we can restart only one Tenant/user or process group Or Without Restart Nifi .nar file will reflect?
NiFi does not currently have the kind of warm restart option that you describe, however a lot of the base functionality needed to support it is in the code base and the concept is on the community roadmap.
Some options that might help you today:
Consider segregating the tenants with a high rate of code change into separate development environments. You could possibly leverage the Docker builds to provide flexibility and easy automation. You could then promote the end-of-day versions of your Nars into the 'Production' cluster each night, hopefully without disturbing users.
Consider utilising the NiFi Site-to-Site capability to have linked NiFi environments instead of a single shared one. Processors that change regularly could be called out to and updated in their own schedule
Consider why you are changing processor code so regularly, there may be a better approach than hard coding logic and parameters into the processors - the variable registry, various controller services, flow registry, etc. all provide a very rich featureset.
In spring-cloud-stream, is there a way to change the instance count and instance index of an application without restarting it?
Also, is there any recommended way to automatically populate these values? In the microservices world, this seems like it would prohibitively difficult, since services are starting and stopping all the time.
In spring-cloud-stream, is there a way to change the instance count and instance index of an application without restarting it?
Not in the current version, but open to discuss this in the context of a GitHub issue.
Also, is there any recommended way to automatically populate these values? In the microservices world, this seems like it would prohibitively difficult, since services are starting and stopping all the time.
My recommendation would be to look at http://cloud.spring.io/spring-cloud-dataflow/ which helps with the orchestration of complex microservice topologies (and is designed to work in conjunction with Spring Cloud Stream for streaming scenarios)
I am looking for the best solution for having multiple publish instances. I tried shared nothing and shared datastore configurations.
Is there any advantages or disadvantages of having 2 or more publish instances without cluster setup? In such a configuration how can I start a new publish instance? I mean how I should replicate the data from the author when I start a new publish instance (probably from backup), what are the best practices for this. How to solve reverse replication issues, so while I starting a new instance, other publish instance might get some new user generated data which must be replicated to the new publish instance too. What is your experience in this topic?
Thanks in advance!
Multiple publish instances allow you to use a load balancer & increase the amount of concurrent users your site can serve.
Replcation of content can be done in the standard way, using CRX Package replcation or the Activation tree, once you've set up a replication agent per instance.
You shouldn't run into too much difficulty with content generated while the instance is being set up, as once the instance is up & running, you can just activate it in the normal way. (If you think it will be an issue, you could probably set-up the replication agent first, so that new content is queued while the instance is being set-up, though I couldn't see why you would need to do this).