I am using pull method to pull the messages out of my pubsub subscription as specified https://docs.spring.io/spring-cloud-gcp/docs/current/reference/html/#pull-methods, I see option to set max number of messages but don't see a way to specify the flow control params like max-outstanding-element-count and max-outstanding-request-bytes as in https://docs.spring.io/spring-cloud-gcp/docs/current/reference/html/#publishersubscriber-configuration
By debugging, i see 20MB is set to max-outstanding-request-bytes (screenshot) though it says DEFAULT_FLOW_CONTROL_SETTING should have 100MB https://github.com/googleapis/java-pubsub/blob/main/google-cloud-pubsub/src/main/java/com/google/cloud/pubsub/v1/Subscriber.java#L456
is there a way so that i can specify that value? I need to use PubSubTemplate with pull method!
Spring Boot: 2.3.5.RELEASE
spring-cloud-gcp-starter-pubsub: 2.0.7
Thanks
Related
I am building the application in Mule 4 to cache the response from http request component (for 1 hours). I use the response to validate the jwt. Whenever there is a failure, I need to clear the cache and re-invoke the http request. Component used is : Cache Scope
My Current Code looks like below:-
<ee:object-store-caching-strategy name="Caching_Strategy" doc:name="Caching Strategy" doc:id="50e44473-b55a-4bc3-b53a-18e4229a31a3" keyGenerationExpression="#['token']" >
<os:private-object-store persistent="false" maxEntries="${caching.maxEntries}" entryTtl="${caching.entryTtl}" expirationInterval="${caching.expirationInterval}" />
</ee:object-store-caching-strategy>
I don't have any specific requirement to make it private object store but disabling it does not allow me to configure the maxEntries, entryTTL and Expiration Interval.
It would be more helpful if you could guide me the ways to
clear the cache (could not find any docs mule 4 related to clear the cache)
how can i keep it as non private object store and define those parameters like entryTtl, maxEntries etc.
when do we need to use the private object stores?
I have already tried the exchanges examples for configure the cache ( but there is no mention of clear the cache).
I am using Mule Runtime 4.2.2 and On Prem Mule
I have raised a support ticket with Mulesoft and received the below replies
1) Invalidate Cache is shown only in the palette from the Anypoint Studio Version 7.4.1. Not shown in earlier version of studio.( I was using 7.3.2)
2) They are adding the documentation for the invalidate cache in Mule 4.
3) Usage Note : Cache Scope and the 'Invalidate Cache' processor should share the same Caching_Strategy).
4) In order to explicitly define object store and ttl for a Cache Scope, please ensure object store related modules are added to the project.
This can be done by simply adding a 'Store' processor in your mule flow, this will add all necessary modules to your mule application.
Then, on the Caching Strategy, General > Reference > Object Store, you can select Edit Inline and define your object store with explicit settings such as
- Max entries
- Entry ttl
- Expiration interval
You can remove the unused 'Store' processor as well.
Hope it helps others
There is a palette called invalidate cache in the latest Mule versions. You can simply use it to delete the cache for a cache scope.
You will require a private object store when you want to persist the key used for caching permanently and also to implement time limits like TTL for a cache.
I am currently looking at the options to add real-time tracking of all the messages going over in my application.
I have enabled messagehistory so technically I'll get all the steps happened during the flow.
Now I want to store each message with the history in disk cache using CQEngine as it has great support and is very quick.
Then from the UI I'll show the message and history and display using GraphVIZ so the users can actually see the lifecycle of each message with diagram.
Option # 01
Add the call in the handle() method in each flow to add the message and history asynchrnously in the disk cache.
But this means I'll have to manually do that every time for new flow.
Is there any option 02 I can utilize like adding interceptor etc that
will be called whenever a flow is completed succesfully or
un-sucessfully?
You need to use a WireTap interceptor for channels: https://docs.spring.io/spring-integration/docs/5.2.2.RELEASE/reference/html/core.html#channel-interceptors.
And configure a global channel interceptor pattern for that wire-tap to specify those channels you would like to track. The same doc has info on the matter. Also see annotation configuration on the matter: https://docs.spring.io/spring-integration/docs/5.2.2.RELEASE/reference/html/configuration.html#annotations
I am using the PublishKafka_2_0 processor and need to disable SSL endpoint identification algorithm while connecting to Kafka. From the docs I can do this by setting the following producer property to an empty string:
ssl.endpoint.identification.algorithm=
The documentation of PublishKafka_2_0 mentions that this can be set using dynamic properties. Can someone show how to set this property? A screenshot or link to an example would be great.
I am using NiFi 1.9.1 and Apache Kafka 2.1.1
Thank you
While I strongly discourage disabling endpoint verification, to accomplish this, click the + icon on the top right of the processor properties tab, enter the value ssl.endpoint.identification.algorithm for the dynamic property name, and check the "Empty string" box for the value of the property.
I only took a quick look, but I do not see that property defined in the ProducerConfig class in the org.apache.kafka:kafka-clients-2.0.0.jar dependency. If the predefined string is not present, you may get a validation error in NiFi of "Must be a known configuration parameter for this kafka client".
I am trying to use the rest API to dynamically update and control my Apache NiFi Flow. I am using Postman to explore the REST API but am having trouble deleting properties/descriptors.
My current process is to call a GET to this address - http://localhost:8080/nifi-api/processors/{ID}
I then modify the response as desired and do a PUT with the modified response as the body. If I add a descriptor or change the content of a descriptor it works ok. But if I try to delete a descriptor by removing it from the properties and descriptors area then nothing happens.
I still get a 200 OK response, but it is the same as the original.
I am using NiFi 1.1.2 on Windows.
The PropertyDescriptors are specified by the Processor in question. These are read-only values and describe the properties the Processor currently supports. In you want to remove a property, and it is optional, you should be able to remove the value for it by setting it's entry to null in properties object in your request.
I'm using the new 2.X NEST client. That part is important, because there were a great many breaking changes which will effect potential answers here.
Previously, I used the Glimpse Elasticsearch plugin to see the underlying queries being generated by NEST. However, it would appear that that plugin is no longer compatible with 2.X NEST. As a result, I'm trying to find a workaround to see the JSON query. The problem here is that the old way of accessing response.RequestInformation to get at the request body is gone. It seems to have been replaced with a combination of ApiCall, CallDetails, and DebugInformation. The problem here is that in all of these the request byte array is null unless you add .DisableDirectStreaming() to the ConnectionSettings instance you pass into ElasticClient. The problem there is that I'm handling all that using dependency injection with Ninject, so in something like a controller action, I have no access to the ConnectionSettings instance to make such a change. I suppose I could just add .DisableDirectStreaming() globally, but I have no idea what the potential consequences of that is and the documentation on this is frustratingly sparse.
So, there's a few avenues for an answer here, any of which I'll accept. First, if anyone has manage to get the Glimpse plugin functioning with 2.X, I'd love to know what you did. However, based on the fact that the underlying API has changed dramatically, my assumption is that the plugin is fundamentally broken until someone branches it for 2.X or Elastic comes out with their own version (which is supposedly coming at some undetermined point in the future).
Second, if there's some way to get at the request body without disabling direct streaming, and I simply missed it. I'd appreciate guidance there.
Third, if anyone has any ideas for how I can disable direct streaming at the controller action level, without affecting my Ninject setup or applying it globally, feel free to chime in.
Finally, it would be great if someone from the Elastic team can enlighten me to the effects of disabling direct streaming and what potential problems can arise from that, so I can make a determination about whether it would be wise to apply it globally or not.
With .DisableDirectStreaming() set to true, the request bytes and response bytes are buffered in memory streams to enable them to be available on the response via response.RequestBodyInBytes and response.ResponseBodyInBytes, respectively.
By default, it is set to false so the request type e.g. SearchDescriptor<T>, SearchRequest<T>, etc. is serialized directly to the request stream of the http request and similarly, the response type is deserialized from the response stream. The overhead of setting it to true is therefore keeping the request and response bytes around in memory for the lifetime of the response (and GC kicking in).
With Connection Settings, it's best to have one instance for the lifetime of the application; Serialization settings are cached per connection settings as well as caches for field and property expressions. There's no way currently to keep the request and response bytes around on a per request basis i.e. ad-hoc introspection, but I think this would be a useful addition; I'll add an issue for it :)
I'm not personally overly familiar with the Glimpse integration but I would expect it would require updating to work with NEST 2.x because of some of the changes. Having just given it a brief look, the changes look pretty straightforward. Looks like this can be done without having to set .DisableDirectStreaming() to true, but only grabbing the request bytes before they're written to the request stream.