In the Chronicle Queue I have two types of messages written. I wanna read this messages using the same tailer and if it is possible with the same method for example using tailer.readDocument().
Anyone now if it is possible, the message types are from different kind of objects. They haven't relationship.
In my actual reading logic I need to read all the entries of the queue and the order is important, for example:
Queue
MessageA
MessageA
MessageB
I need to read message B only after message A in this example, because of that I am looking for a method that read all the entries independent of message type.
The simplest approach is to write messages using a MethodWriter/MethodReader https://github.com/OpenHFT/Chronicle-Queue#high-level-interface
You start by defining an asynchronous interface, where all methods have:
arguments which are only inputs
no return value or exceptions expected.
A simple asynchronous interface
import net.openhft.chronicle.wire.SelfDescribingMarshallable;
interface MessageListener {
void method1(Message1 message);
void method2(Message2 message);
}
static class Message1 extends SelfDescribingMarshallable {
String text;
public Message1(String text) {
this.text = text;
}
}
static class Message2 extends SelfDescribingMarshallable {
long number;
public Message2(long number) {
this.number = number;
}
}
To write to the queue you can call a proxy that implements this interface.
SingleChronicleQueue queue1 = ChronicleQueue.singleBuilder(path).build();
MessageListener writer1 = queue1.acquireAppender().methodWriter(MessageListener.class);
// call method on the interface to send messages
writer1.method1(new Message1("hello"));
writer1.method2(new Message2(234));
These calls produce messages which can be dumped as follows.
# position: 262568, header: 0
--- !!data #binary
method1: {
text: hello
}
# position: 262597, header: 1
--- !!data #binary
method2: {
number: !int 234
}
To read the messages, you can provide a reader which calls your implementation with the same calls that you made.
// a proxy which print each method called on it
MessageListener processor = ObjectUtils.printAll(MessageListener.class)
// a queue reader which turns messages into method calls.
MethodReader reader1 = queue1.createTailer().methodReader(processor);
assertTrue(reader1.readOne());
assertTrue(reader1.readOne());
assertFalse(reader1.readOne());
Running this example prints:
method1 [!Message1 {
text: hello
}
]
method2 [!Message2 {
number: 234
}
]
Nice #PeterLawrey has a different way to build the processor. I mean in your example you print the objects I want populate the two different types of objects. I don't find a way until now using the methods in the same listener to do it.
Related
Assume a chronicle queue, and a producer that writes 2 types of messages into the queue.
Each type of message is written with a different "WireKey".
// Writes: {key1: TestMessage}
appender.writeDocument(w -> w.write("key1").text("TestMessage"));
// Writes: {key2: AnotherTextMessage}
appender.writeDocument(w -> w.write("key2").text("AnotherTextMessage"));
Question:
How can I write a single-threaded consumer that can read both types of messages and handle them differently?
What I've tried:
// This can read both types of messages, but cannot
// tell which type a message belongs to.
tailer.readDocument(wire -> {
wire.read().text();
});
// This only reads type "key1" messages, skips all "key2" messages.
tailer.readDocument(wire -> {
wire.read("key1").text();
});
// This crashes. (because it advances the read position illegally?)
tailer.readDocument(wire -> {
wire.read("key1").text();
wire.read("key2").text();
});
I was hoping I can do something like wire.readKey() and get the WireKey of a document, then proceed to read the document and handle it dynamically. How can I do this?
Note: I'm aware this can be accomplished using methodReader and methodWriter, and it seems like documentation/demo recommends this approach (?) But I'd prefer not to use that API, and be explicit about reading and writing messages. I assume there has to be a way to accomplish this use case.
Thank you.
You are correct, e.g. MethodReader accomplishes it.
You can do it two ways
// a reused StringBuilder
StringBuilder sb = new StringBuilder();
wire.read(sb); // populate the StringBuilder
or a more convenient method is
String name = wire.readEvent(String.class);
switch(name) {
case "key1":
String text1 = wire.getValueIn().text();
// do something with text1
break;
case "key2":
String text2 = wire.getValueIn().text();
// do something with text1
break;
default:
// log unexpected key
}
For other readers who don't know about MethodReader, the same messages can be accomplished with
interface MyEvents {
void key1(String text1);
void key2(String text2);
}
MyEvents me = wire.methodWriter(MyEvents.class);
me.key1("text1");
me.key2("text2");
MyEvents me2 = new MyEvents() {
public void key1(String text1) {
// handle text1
}
public void key2(String text2) {
// handle text2
}
};
Reader reader = wire.methodReader(me2;
do {
} while(reader.readeOne());
NOTE: The content is the same, so you can mix and match the two options
You can use a Chronicle Queue instead of a Wire to persist this information
I have a service, which receives different structured messages from different message queues. Having #StreamListener conditions we can choose at every message type how that message should be handled. As an example:
We receive two different types of messages, which have different header fields and values e.g.
Incoming from "order" queue:
Order1: { Header: {catalog:groceries} }
Order2: { Header: {catalog:tools} }
Incoming from "shipment" queue:
Shipment1: { Header: {region:Europe} }
Shipment2: { Header: {region:America} }
There is a binding for each queue, and with according #StreamListener I can process the messages by catalog and region differently
e.g.
#StreamListener(target = OrderSink.ORDER_CHANNEL, condition = "headers['catalog'] == 'groceries'")
public void onGroceriesOrder(GroceryOder order){
...
}
So the question is, how to achieve this with the new Spring Cloud Function approach?
At the documentation https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/3.0.2.RELEASE/reference/html/spring-cloud-stream.html#_event_routing it is mentioned:
Also, for SpEL, the root object of the evaluation context is Message so you can do evaluation on individual headers (or message) as well ….routing-expression=headers['type']
Is it possible to add the routing-expression to the binding like (in application.yml)
onGroceriesOrder-in-0:
destination: order
routing-expression: "headers['catalog']==groceries"
?
EDIT after first answer
If the above expression at this location is not possible, what the first answer implies, than my question goes as follows:
As far as I understand, an expression like routing-expression: headers['catalog'] must be set globally, because the result maps to certain (consumer) functions.
How can I control that the 2 different messages on each queue will be forwarted to their own consumer function, e.g.
Order1 --> MyOrderService.onGroceriesOrder()
Order2 --> MyOrderService.onToolsOrder()
Shipment1 --> MyShipmentService.onEuropeShipment()
Shipment2 --> MyShipmentService.onAmericaShipment()
That was easy with #StreamListener, because each method gets their own #StreamListener annotation with different conditions. How can this be achieved with the new routing-expression setting?
?
Aside from the fact that the above is not a valid expression, but I think you meant headers['catalog']==groceries. If so, what would you expect to happen from evaluating it as the only two option could be true/false. Anyway, these are rhetorical but helps to understand the problem and how to fix it.
The expression must result in a value of a function to route TO. So. . .
routing-expression: headers['catalog'] - assumes that the actual value of catalog header is the name of the function to invoke
routing-expression: headers['catalog']==groceries ? 'processGroceries' : 'processOther' - maps value 'groceries' to 'processGroceries' function.
For a specific routing, you can use MessageRoutingCallback strategy:
MessageRoutingCallback
The MessageRoutingCallback is a strategy to assist with determining
the name of the route-to function definition.
public interface MessageRoutingCallback {
FunctionRoutingResult routingResult(Message<?> message);
. . .
}
All you need to do is implement and register it as a bean to be picked
up by the RoutingFunction. For example:
#Bean
public MessageRoutingCallback customRouter() {
return new MessageRoutingCallback() {
#Override
FunctionRoutingResult routingResult(Message<?> message) {
return new FunctionRoutingResult((String) message.getHeaders().get("func_name"));
}
};
}
Spring Cloud Function
Given that i have multiple models, each needed to have their own create/get/get list API.
Do i need to add two different types of messages (single and list) for every model?
For example :
If i have a student type -
message Student{
string name = 1;
}
and a rpc:
rpc CreateStudent(Student) returns (google.protobuf.Empty){
..............
}
If i'd like to add a rpc to create a list of students, or get a list of students
rpc CreateStudends(??????) returns (google.protobuf.Empty){
..............
}
rpc GetAllStudents() returns (??????){
..............
}
Do i need to also define
message StudentList{
repeated Student students = 1;
}
Or is there a way to use a list type directly in the message input/output?
Yes, basically - you would want a different message type per element type, or maybe a single root type with a oneof style content. Raw protobuf does not include a concept of generics or templates.
Some libraries do, but: that's outside of the specification.
You can simply add the stream keyword to your RPCs. No need to define a message field as repeated, stream will send or receive multiple independent messages.
message Student {
string name = 1;
}
with RPCs:
rpc CreateStudent(Student) returns (google.protobuf.Empty) {
..............
}
rpc CreateStudents(stream Student) returns (google.protobuf.Empty) {
..............
}
rpc GetAllStudents() returns (stream Student) {
..............
}
It's good practice to send/stream a response object rather than empty. Otherwise, you only have the gRPC response code to indicate a problem and will need to reference the logs to debug.
I have the following Akka actor:
public class MyActor extends AbstractActor {
protected Logger log = LoggerFactory.getLogger(this.getClass());
#Override
public Receive createReceive() {
return receiveBuilder()
.matchAny(message -> {
String myFullName = self().path().toString();
String myName = self().path().name();
ActorRef reincarnatedMe = context().actorFor(self().path().name());
String reincarnatedFullName = reincarnatedMe.path().toString();
String reincarnatedName = reincarnatedMe.path().name();
log.info("myFullName: {}", myFullName);
log.info("myName: {}", myName);
log.info("reincarnatedFullName: {}", reincarnatedFullName);
log.info("reincarnatedName: {}", reincarnatedName);
}).build();
}
}
At runtime it produces this output:
05:43:14.617 [MySystem-akka.actor.default-dispatcher-4] INFO myapp.actors.MyActor - myFullName: akka://MySystem/user/MyActor
05:43:14.623 [MySystem-akka.actor.default-dispatcher-4] INFO myapp.actors.MyActor - myName: MyActor
05:43:14.623 [MySystem-akka.actor.default-dispatcher-4] INFO myapp.actors.MyActor - reincarnatedFullName: akka://MySystem/user/MyActor/MyActor
05:43:14.623 [MySystem-akka.actor.default-dispatcher-4] INFO myapp.actors.MyActor - reincarnatedName: MyActor
My understanding was that context().actorFor(...) doesn't create a new actor, rather it finds an existing actor that matches the path/string you provide and returns a reference to it.
However, it appears that in my code above, self() becomes the parent of reincarnatedMe as evidenced by myFullName simply being "MySystem/user/MyActor" whereas reincarnatedFullName is "MySystem/user/MyActor/MyActor"...
Am I reading this right? If so, how can I invoke context().actorFor(...) (or any other method for that matter) such that myFullName becomes the same as reincarnatedFullName (so that self() and reincarnatedMe reference the same actor? And if I'm not reading this right, why is myFullName different than reincarnatedFullName?
Update:
public class AnotherActor extends AbstractActor { ... }
// Inside MyActor#createReceive:
ActorSelection anotherActorSel = context().actorSelection("AnotherActor");
anotherActorSel.tell(new SomeMessage(), self());
First, ActorContext.actorFor(String) is deprecated in favor of ActorContext.actorSelection(String). This method returns an ActorSelection, but you can still send a message to an ActorSelection (such as an Identify, which response with an ActorIdentity message automatically).
The documentation for the actorFor method says that, "Absolute URIs like akka://appname/user/actorA are looked up as described for look-ups by actorOf(ActorPath)." I can't find documentation on an actorOf(ActorPath) method, but the other actorOf methods state they create new actors, so I suspect this does the same. The behavior you've found is likely the reason for the deprecation -- or because it was deprecated and the methods used for something else.
Should FIFO queue be synchronized if there is only one reader and one writer?
What do you mean by "synchronized"? If your reader & writer are in separate threads, you want the FIFO to handle the concurrency "correctly", including such details as:
proper use of FIFO API should never cause data structures to be corrupted
proper use of FIFO API should not cause deadlock (although there should be a mechanism for a reader to wait until there is something to read)
the objects read from the FIFO should be the same objects, in the same order, written to the FIFO (there shouldn't be missing objects or rearranged order)
there should be a bounded time (one would hope!) between when the writer puts something into the FIFO, and when it is available to the reader.
In the Java world there's a good book on this, Java Concurrency In Practice. There are multiple ways to implement a FIFO that handles concurrency correctly. The simplest implementations are blocking, more complex ones use non-blocking algorithms based on compare-and-swap instructions found on most processors these days.
Yes, if the reader and writer interact with the FIFO queue from different threads.
Depending on implementation, but most likely. You don't want reader to read partially written data.
Yes, unless its documentation explicitly says otherwise.
(It is possible to implement a specialized FIFO that doesn't need synchronization if there is only one reader and one writer thread, e.g. on Windows using InterlockedXXX functions.)
Try this code for concurrent fifo usage:
public class MyObjectQueue {
private static final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
private static final ReadLock readLock;
private static final WriteLock writeLock;
private static final LinkedList<MyObject> objects;
static {
readLock = lock.readLock();
writeLock = lock.writeLock();
objects = new LinkedList<MyObject>();
}
public static boolean put(MyObject p) {
writeLock.lock();
try {
objects.push(p);
return objects.contains(p);
} finally {
writeLock.unlock();
}
}
public static boolean remove(MyObject p) {
writeLock.lock();
try {
return objects.remove(p);
} finally {
writeLock.unlock();
}
}
public static boolean contains(MyObject p) {
readLock.lock();
try {
return objects.contains(p);
} finally {
readLock.unlock();
}
}
public MyObject get() {
MyObject o = null;
writeLock.lock();
try {
o = objects.getLast();
} catch (NoSuchElementException nse) {
//list is empty
} finally {
writeLock.unlock();
}
return o;
}
}