I am trying out spring cloud dataflow. My specific usecase is to dump the response from a GET request to a log. I am trying to use the httpclient processor for this. But i dont understand how come it is a processor, and not a source. If it a processor, what should be the input source to it. Any example would do great.
It requires an incoming Message to trigger the http request. The message may specify the URL, HTTP Method, etc. using SpEL expressions but these may also be statically configured as well. For example, you can use the time source to trigger a request every second.
Related
I wrote some micro-services using Quarkus that communicate via Artemis. Now I want to add OpenTelemetry for tracing purpose.
What I already tried is to call service B from service A using HTTP/REST. Here the trace id from service A is automatically added to the header of the HTTP request and used in service B. So this works fine. In Jaeger I can see the correlation.
But how can this be achieved using Artemis as messaging system? Do I have to (manually) add the trace id from service A into the message and read it in service B to setup somehow the context (don't know whether this is possible)? Or is there possibly an automatism like for HTTP requests?
I would appreciate any assistance.
I have to mention at this point that I have little experience with tracing so far.
There is no quarkus, quarkiverse extension or smallrye lib that provides integration with Artemis and OpenTelemetry, yet.
Also, OpenTelemetry massaging spec is being worked at the moment, because the correct way to correlate sent, received messages and services is under definition at the OTel spec level.
However, I had exactly the same problem as you and did a manual instrumentation that you can use as inspiration: quarkus-observability-demo-activemq
It will correlate the sent service as parent of receiving end.
I am working with ActiveMQ. I want to put gRPC calls as a message on ActiveMQ. As soon as consumer picks a message the gRPC in the message should execute.
gRPC calls take parameters so I want to understand how to send these parameters in the message body along with the method name/identifier. Is there any another way of achieving this which is followed already?
Example:
someResponse response = someStub.rpcMethodName(parameter); // grpc call
What I need is to store this statement in some format on ActiveMQ without calling the gRPC now. Then in the future when the consumer of ActiveMQ picks up the message containing this statement it should execute this gRPC call.
You can structure the gRPC call data in your message any way you want. You could use XML, JSON, ProtoBuf, etc. It's really up to however you want to do it. If all you really need are simple key/value pairs then you could just use JMS message properties to store the data you need.
I don't think using gRPC is particularly common, but it is common to use messages as a kind of "unit of work" to work with all kinds of remote resources (e.g. databases, REST services, microservices, etc.).
In any event, I would strongly recommend against using JMS ObjectMessage. ObjectMessage objects depend on Java serialization to marshal and unmarshal their object payload. This process is not only slow but generally considered unsafe because a malicious payload can exploit the host system. Lots of CVEs have been created for this. There are a number of other issues with using JMS ObjectMessage not related to security that you should read about as well.
The case for event-driven microservices such as Spring Cloud Stream is their asynchronous nature, which I do agree it makes them more scalable
But I have an issue regarding how to code it in a way where I don't lose certain key features that I have access to using synchronous services
In a servlet-based MS, I make full use of servlet context variables and servlet-based Spring autowiring functions
For e.g., I leverage heavily on HTTP headers to carry metadata between microservices without having to impact the payload. But in Spring Cloud Stream using Kafka, Kafka doesn't support message headers of any kind! I lose that immediately if I use SCS. Putting them into the payload causes all sort of changes in my model classes if I define the attributes clearly. Yes, I can use a simple Hashmap to simulate the HTTP header object but it really seems like reinventing the wheel to me.
On the auto-wiring side: I maintain an audit log record per request, which I implement by declaring a request-scoped Hashmap bean and autowiring it into any methods in the Servlet's call stack that needs to append data to the audit log. Basically it's just a global variable to hold some data within a single request. But in SCS, again, I lose that cos bean scopes that leverage on servlets are not available.
So far, there seems to be a lot of trade-offs that I have to make just to make Spring Cloud Stream work for me.
I thought about an alternative approach where I use SCS just to create an entry point but the Source method would just get the event, use a Processor to construct a HTTP request and send the request along to a HTTP endpoint. But, why go through all that trouble then?
Hoping that some more experienced devs would be able to shed some light on how they leverage on SCS.
#feicipet Thanks for the detailed question. let me try to address some of your concerns in the order you have listed them:
+1
+1
I am not sure why you are referring to it as servlet-based instead of Spring-based? Those are features provided by Spring, but read on. . .
Spring Cloud Stream doesn't use Kafka, the end user does while Spring Cloud Stream provides Kafka binder allowing Spring Cloud Stream to integrate with Kafka. Further more, while Kafka indeed did not support headers prior to version 0.11, Spring Cloud Stream always supported and will continue support headers even with Kafka pre-0.11, embedding them in the Message and then extracting them in the consumer side into the proper Message headers completely transparent to the end user. In other words one would assume that Kafka did support headers by simply using Spring Cloud Stream. With Kafka 0.11+ headers are supported natively and we have adjusted to that with the same level of transparency.
So, you don't need to put anything in the payload. Just create an appropriate Message<payload, headers> and SCSt will take care of the rest regardless of the broker (Kafka, Rabbit, Foo etc.).
Yes you do simply due to the fact that as you eluded earlier SCSt promotes an asynchronous and stateless architecture. However, I do not agree that what you are trying to accomplish is un-accomplishable. Rather it is accomplishable the way you are describing, but there are other way to maintain context and I would be more then glad to discuss it as a separate topic.
I would not call them trade-offs, rather difference in the architecture, that has its benefits, but it is a not one-size-fits-all architecture and therefore its viability should be discussed within the context of a concrete use case.
+1. You don't have to separate it as Source and Processor. You can simply create a custom Source app with exposed REST endpoint and custom processing logic. However we are currently working on enhancements i the framework to ensure that you could do the same with the existing starter apps.
Obviously we have touched on many points here and some of them would probably need to be debated further, but I hope this clears up some of your concerns.
Cheers
I am trying to use Spring Boot 1.5.2.RELEASE + Camel (Spring Boot starter) 2.19.2 to listen to ActiveMQ queue and then post the message to a rest endpoint URL (POST method) as its body. What would be the best possible way to achieve this?
I have gathered pieces of information and am trying to tie it all together but getting a bit confused.
Here is what I have gathered for Camel Rest DSL, I am not too sure if camel below is creating these rest services via this or is it just an already exposed endpoint, in my case it is an already exposed endpoint
rest("/basePath")
post("/someEndpoint").to("direct:restEndpoint")
Using the above is what I have gathered for ActiveMQ which I am not too sure is correct
from("activemq:queue:<queue_name>").to("direct:restEndpoint")
But again, I am not too sure how to listen to the ActiveMQ queue for new messages or is it something that Camel would do by default always? Additionally, I need to pass the message as a post body to my rest endpoint. I also saw some references to camel-http4 and camel-http as well and I am completely confused.
Any assistance would be greatly appreciated.
Some confusion is common when starting to use Camel, but your final solution will look something like:
from("activemq:queue:my-route")
.process(/* change the in/out messages if you need to */)
.to("http4://your-endpoint.com");
Don't try to simply copy/paste this code until it works. My Camel rule of thumb is: always read the component documentation and try playing with it using it in your software. In your case I suggest:
Read ActiveMQ component docs and try reading from ActiveMQ / writing to a Log;
Generate some input from a Timer and send to your Rest endpoint using HTTP4 Component;
Your first routes will take some time for simple things but you will get on flow quickly.
I have requirement something like this:
once the request is received by my service, i need to send it 2-3 third party servers at a time and get the response from all server and return the response.
How can I achieve that?.
My thought : I can create separate threads for different servers and send the request to all servers parallely, but here the issue is, how I will come to know the threads are finished and consolidate the response from all servers and return to caller.
Is there any other way to do in spring boot(micro service)?.
You can leverage asyn feature supported by Spring Framework. Let's see the folllowing example which issue multiple calls in Async style from Spring's official guides:Async Method
Another possible solution for internal communications between Microserives is to use the message queue.
You didn't specify what type of services are you consuming. If they are HTTP, you may want to use some Enterprise Integrations abstractions (most popular are Spring Integration and Apache Camel).
If you don't want to introduce message bus solution into your microservice, you may want to take a look at AsyncRestTemplate