I want to realize the next flow of kafka stream
from(kafka topic) -> transform (here should be http request) -> to (kafka topic)
Is it correct to set http request during tranformaton in kafka stream or its more correct to use standart consumer ?
It is possible but not recommended to do external requests within a transform() because the request would need to be synchronous and thus negatively impacts performance (ie, throughput).
However, if this is no concern for you it's no problem to do external request.
Related
I have a use case where we need to persist Http Response (https://square.github.io/okhttp/4.x/okhttp/okhttp3/-response/) into a Chronicle Queue. Since this response object is coming from OkHttp module, I can't make it Marshallable so that I can write it to a Chronicle Queue.
I only care about the HTTP Status code (Integer), HTTP Header values (Array of String) and ByteStream or byte[]. I like to minimize the number of objects getting created if possible (to reduce GC). If I need to extract data from "Response" object to a custom POJO that is Marshallable, won't that add more objects to be GCed?
This will be a great example as I can think of many applications having similar use cases (of handling the rate of message production/consumption in HTTP applications).
I tried to create a POJO from the OkHttp response object. I am not able to persist the data into the Chronicle Queue as it complaints about java.net classes are not Marshallable.
The concepts of "stream, connection, message, and frame" constitute the main design of http2. And what confuses me is the idea of stream.
At first, the stream idea seems to me only as a virtual description of the flow of frames. But then I find the priority of http2 is aimed at streams instead of messages/requests. And why is that, I think the applications both client and server sides care more about and directly control the requests or messages, not which stream these messages reside in.
Plese refer to "stream prioritization":
https://developers.google.com/web/fundamentals/performance/http2#design_and_technical_goals
A stream in HTTP/2 corresponds to all the frames which make up a request and its corresponding response, so is the natural place to handle priority and flow control. The sentences "the response for this request should have high priority" and "the stream for this request and its response should have high priority" are equivalent.
There is a mention in the document you quote of a stream carrying "one or more messages", but I think that's just sloppy language in that document. If you look at section 8.1 of the spec it says "A client sends an HTTP request on a new stream" and "An HTTP request/response exchange fully consumes a single stream."
There can be other frames in that stream, such as PUSH_PROMISE, but those aren't actual requests and responses; the response data for a server push is sent on a new stream, which can then be given a different priority.
I use "github.com/streadway/amqp" for async processing requests via queue (RabbitMQ).
And I use "github.com/gorilla/rpc" to register my service without workaround, but I have to use ugly solution for conversion amqp.Delivery to http.Request (mux.Server can works with http.Request only).
Can I use more elegant solution for this task?
I can't find JSON RPC router for AMQP.
First, RPC and pub-sub (e.g. AMQP) are two very different beasts; trying to use one to implement the other isn't necessarily wrong or bad, but it's definitely suspicious, and implies that there could be a breakdown somewhere in the design. So I would highly recommend reconsidering the design starting with your business goals and make sure that what you're trying to implement is actually the correct way to achieve the desired functionality.
That said, what you're describing is basically possible, but you want to move your abstraction up a level. Trying to send a http.Request via AMQP is mixing protocols in a way that's only going to lead to more problems. The cleaner way to implement this behavior would be to have an HTTP handler that handles http.Requests (as normal), and a AMQP handler that handles amqp.Deliverys (as normal), and have each of those handlers call a shared business logic handler which deals only in your domain model.
So, your HTTP handler would parse an HTTP request and turn it into a domain object - you don't give any concrete details in the question so I'll invent something like maybe myapp.UserRegistration. Your HTTP handler would pass that to a myapp.UserService which would handle the actual business logic of registering a user, it would return a result, which you would then transform into the appropriate type, marshal to JSON, and send back to the client in an http.Response. myapp.UserService would know nothing about HTTP or AMQP; it operates only on your own domain types.
Your AMQP handler would pick up a message, parse it into the same myapp.UserRegistration type, pass it to the same myapp.UserService handler, and get the same response back - ensuring that the business logic for AMQP and HTTP behaves the same way. Then you'd get your response back, and... well, this is AMQP, so you don't get to send a response to the client. I don't know your setup, maybe you have another queue you can send the response back on, maybe you don't care about the response and can discard it. This is where the difference between RPC and AMQP is most apparent.
This also makes your business logic, HTTP handler, and AMQP handler more testable in isolation because you're separating the protocol logic from the business logic, which can be helpful even when you aren't trying to deal with multiple protocols (i.e. it's not a bad idea even if you're only doing HTTP)
I hope that at least gives you enough info to put you on the right track in your implementation. Good luck!
I have a workflow whose message payload (MasterObj) is being enriched several times. During the 2nd enrichment () an UnknownHostException was thrown by an outbound gateway. My error channel on the enricher is called but the message the error-channel receives is an exception, and the failed msg in that exception is no longer my MasterObj (original payload) but it is now the object gotten from request-payload-expression on the enricher.
The enricher calls an outbound-gateway and business-wise this is optional. I just want to continue my workflow with the payload that I've been enriching. The docs say that the error-channel on the enricher can be used to provide an alternate object (to what the enricher's request-channel would return) but even when I return an object from the enricher's error-channel, it still takes me to the workflow's overall error channel.
How do I trap errors from enricher's + outbound-gateways, and continue processing my workflow with the same payload I've been working on?
Is trying to maintain a single payload object for the entire workflow the right strategy? I need to be able to access it whenever I need.
I was thinking of using a bean scoped to the session where I store the payload but that seems to defeat the purpose of SI, no?
Thanks.
Well, if you worry about your MasterObj in the error-channel flow, don't use that request-payload-expression and let the original payload go to the enricher's sub-flow.
You always can use in that flow a simple <transformer expression="">.
On the other hand, you're right: it isn't good strategy to support single object through the flow. You carry messages via channel and it isn't good to be tied on each step. The Spring Integration purpose is to be able to switch from different MessageChannel types at any time with small effort for their producers and consumers. Also you can switch to the distributed mode when consumers and producers are on different machines.
If you still need to enrich the same object several times, consider to write some custom Java code. You can use a #MessagingGateway on the matter to still have a Spring Integration gain.
And right, scope is not good for integration flow, because you can simply switch there to a different channel type and lose a ThreadLocal context.
From https://en.wikipedia.org/wiki/Server_%28computing%29:
The nature of communication between a client and server is request and response. This is in contrast with peer-to-peer model in which the relationship is on-demand reciprocation.
What is "on-demand reciprocation"?
How is it different from "request and response"? Thanks.
Request-response mostly is an unidirectional flow of payload data, the request is just metadata.
In a p2p system data flows in both directions. For example in bittorrent you have have an asynchronous stream of messages in each direction where messages can be both requests and responses. The asynchronous part is important that data can flow both ways continuously.
If you wanted to compare it to HTTP then it would be like having one pipelined HTTP 1.1 connection open in each direction.