Indy10, IdTCPClient's IOHandler send additional data - indy10

Trying to send a data to another machine. I use C++Builder XE6 and Indy10.
TMemoryStream *sms = new TMemoryStream();
sms->Write(msgData, msgSize);
Form1->IdTCPClient1->IOHandler->WriteBufferOpen();
Form1->IdTCPClient1->IOHandler->Write(sms, 0, true);
Form1->IdTCPClient1->IOHandler->WriteBufferFlush();
delete sms;
When I checked send data with Wireshark, I sent data that additional data and msgData to the machine.
The additional data is " 00 00 00 20 ", and it is at head of sending data.
Does IdTcpClient usually send additional data like this?

It is sending extra data because you are setting the AWriteByteCount parameter of Write(TStream) to true. 00 00 00 20 is the stream size in network byte order. msgSize is 0x00000020, aka 32. If you do not want the stream size sent, you need to set the AWriteByteCount parameter to false instead:
Form1->IdTCPClient1->IOHandler->Write(sms, 0, false);
Also, you should be using WriteBufferClose() instead of WriteBufferFlush(), and do not forget to call WriteBufferCancel() if Write() raises an exception. WriteBufferClose() sends the buffered data to the socket and then closes the buffer so subsequent writes are not buffered. WriteBufferFlush() sends the buffered data to the socket, but does not close the buffer, thus subsequent writes will be buffered.
Also, you can simplify the overhead a little by replacing TMemoryStream with TIdMemoryBufferStream, so that you do not have to make a separate copy of your message data in memory:
TIdMemoryBufferStream *sms = new TIdMemoryBufferStream(msgData, msgSize);
try
{
Form1->IdTCPClient1->IOHandler->WriteBufferOpen();
try
{
Form1->IdTCPClient1->IOHandler->Write(sms, 0, true); // or false
Form1->IdTCPClient1->IOHandler->WriteBufferClose();
}
catch (const Exception &)
{
Form1->IdTCPClient1->IOHandler->WriteBufferCancel();
throw;
}
}
__finally
{
delete sms;
}
Alternatively, use a RAII approach:
class BufferIOWriting
{
private:
TIdIOHandler *m_IO;
bool m_Finished;
public:
BufferIOWriting(TIdIOHandler *aIO) : m_IO(aIO), m_Finished(false)
{
IO->WriteBufferOpen();
}
~BufferIOWriting()
{
if (m_Finished)
m_IO->WriteBufferClose();
else
m_IO->WriteBufferCancel();
}
void Finished()
{
m_Finished = true;
}
};
{
std::auto_ptr<TIdMemoryBufferStream> sms(new TIdMemoryBufferStream(msgData, msgSize));
BufferIOWriting buffer(Form1->IdTCPClient1->IOHandler);
Form1->IdTCPClient1->IOHandler->Write(sms.get(), 0, true); // or false
buffer.Finished();
}
With that said, I would suggest just getting rid of write buffering altogether:
TIdMemoryBufferStream *sms = new TIdMemoryBufferStream(msgData, msgSize);
try
{
Form1->IdTCPClient1->IOHandler->Write(sms, 0, true); // or false
}
__finally
{
delete sms;
}
Or:
{
std::auto_ptr<TIdMemoryBufferStream> sms(new TIdMemoryBufferStream(msgData, msgSize));
Form1->IdTCPClient1->IOHandler->Write(sms.get(), 0, true); // or false
}
Write buffering is useful when you need to make multiple related Write() calls that should have their data transmitted together in as few TCP frames as possible (in other words, letting the Nagle algorithm do its job better). For instance, if you need to Write() individual fields of your message. Write buffering does not make much sense to use when making a single Write() call, especially of such a small size. Let Write(TStream) handle its own buffering internally for you.

Related

Chronicle Queue: How to read excepts/documents with different WireKey?

Assume a chronicle queue, and a producer that writes 2 types of messages into the queue.
Each type of message is written with a different "WireKey".
// Writes: {key1: TestMessage}
appender.writeDocument(w -> w.write("key1").text("TestMessage"));
// Writes: {key2: AnotherTextMessage}
appender.writeDocument(w -> w.write("key2").text("AnotherTextMessage"));
Question:
How can I write a single-threaded consumer that can read both types of messages and handle them differently?
What I've tried:
// This can read both types of messages, but cannot
// tell which type a message belongs to.
tailer.readDocument(wire -> {
wire.read().text();
});
// This only reads type "key1" messages, skips all "key2" messages.
tailer.readDocument(wire -> {
wire.read("key1").text();
});
// This crashes. (because it advances the read position illegally?)
tailer.readDocument(wire -> {
wire.read("key1").text();
wire.read("key2").text();
});
I was hoping I can do something like wire.readKey() and get the WireKey of a document, then proceed to read the document and handle it dynamically. How can I do this?
Note: I'm aware this can be accomplished using methodReader and methodWriter, and it seems like documentation/demo recommends this approach (?) But I'd prefer not to use that API, and be explicit about reading and writing messages. I assume there has to be a way to accomplish this use case.
Thank you.
You are correct, e.g. MethodReader accomplishes it.
You can do it two ways
// a reused StringBuilder
StringBuilder sb = new StringBuilder();
wire.read(sb); // populate the StringBuilder
or a more convenient method is
String name = wire.readEvent(String.class);
switch(name) {
case "key1":
String text1 = wire.getValueIn().text();
// do something with text1
break;
case "key2":
String text2 = wire.getValueIn().text();
// do something with text1
break;
default:
// log unexpected key
}
For other readers who don't know about MethodReader, the same messages can be accomplished with
interface MyEvents {
void key1(String text1);
void key2(String text2);
}
MyEvents me = wire.methodWriter(MyEvents.class);
me.key1("text1");
me.key2("text2");
MyEvents me2 = new MyEvents() {
public void key1(String text1) {
// handle text1
}
public void key2(String text2) {
// handle text2
}
};
Reader reader = wire.methodReader(me2;
do {
} while(reader.readeOne());
NOTE: The content is the same, so you can mix and match the two options
You can use a Chronicle Queue instead of a Wire to persist this information

Timeout for ConnectEx() in IOCP mode?

In an IOCP Winsock2 client, after ConnectEx() times-out on an unsuccessful connection attempt, the following happens:
An "IO completion" is queued to the associated IO Completion Port.
GetQueuedCompletionStatus() returns FALSE.
WSAGetOverlappedResult() returns WSAETIMEDOUT.
What determines the timeout period between calling ConnectEx() and 1 above? How can I shorten this timeout period?
I know that it is possible to wait for ConnectEx() by passing it a filled-out structure OVERLAPPED.hEvent = WSACreateEvent() and then waiting for this event, e.g. with WaitForSingleObject(Overlapped.hEvent, millisec) to timeout after no connection has been made for the millisec time period. BUT, that solution is outside the scope of this question because it does not refer to the IOCP notification model.
unfortunatelly look like no built-in option for set socket connect timeout. how minimum i not view this and based on this question - How to configure socket connect timeout - nobody not view too.
one possible solution pass event handle to I/O request and if we got ERROR_IO_PENDING - call RegisterWaitForSingleObject for this event. if this call will be successful - our WaitOrTimerCallback callback function will be called - or because I/O will be complete (with any final status) and at this moment event (which we pass both to I/O request and RegisterWaitForSingleObject) will be set or because timeout (dwMilliseconds) expired - in this case we need call CancelIoEx function.
so let say we have class IO_IRP : public OVERLAPPED which have reference counting (we need save pointer to OVERLAPPED used in I/O request for pass it to CancelIoEx. and need be sure that this OVERLAPPED still not used in another new I/O - so yet not free). in this case possible implementation:
class WaitTimeout
{
IO_IRP* _Irp;
HANDLE _hEvent, _WaitHandle, _hObject;
static VOID CALLBACK WaitOrTimerCallback(
__in WaitTimeout* lpParameter,
__in BOOLEAN TimerOrWaitFired
)
{
UnregisterWaitEx(lpParameter->_WaitHandle, NULL);
if (TimerOrWaitFired)
{
// the lpOverlapped unique here (because we hold reference on it) - not used in any another I/O
CancelIoEx(lpParameter->_hObject, lpParameter->_Irp);
}
delete lpParameter;
}
~WaitTimeout()
{
if (_hEvent) CloseHandle(_hEvent);
_Irp->Release();
}
WaitTimeout(IO_IRP* Irp, HANDLE hObject) : _hEvent(0), _Irp(Irp), _hObject(hObject)
{
Irp->AddRef();
}
BOOL Create(PHANDLE phEvent)
{
if (HANDLE hEvent = CreateEvent(NULL, FALSE, FALSE, NULL))
{
*phEvent = hEvent;
_hEvent = hEvent;
return TRUE;
}
return FALSE;
}
public:
static WaitTimeout* Create(PHANDLE phEvent, IO_IRP* Irp, HANDLE hObject)
{
if (WaitTimeout* p = new WaitTimeout(Irp, hObject))
{
if (p->Create(phEvent))
{
return p;
}
delete p;
}
return NULL;
}
void Destroy()
{
delete this;
}
// can not access object after this call
void SetTimeout(ULONG dwMilliseconds)
{
if (RegisterWaitForSingleObject(&_WaitHandle, _hEvent,
(WAITORTIMERCALLBACK)WaitOrTimerCallback, this,
dwMilliseconds, WT_EXECUTEONLYONCE|WT_EXECUTEINWAITTHREAD))
{
// WaitOrTimerCallback will be called
// delete self here
return ;
}
// fail register wait
// just cancel i/o and delete self
CancelIoEx(_hObject, _Irp);
delete this;
}
};
and use something like
if (IO_IRP* Irp = new IO_IRP(...))
{
WaitTimeout* p = 0;
if (dwMilliseconds)
{
if (!(p = WaitTimeout::Create(&Irp->hEvent, Irp, (HANDLE)socket)))
{
err = ERROR_NO_SYSTEM_RESOURCES;
}
}
if (err == NOERROR)
{
DWORD dwBytes;
err = ConnectEx(socket, RemoteAddress, RemoteAddressLength,
lpSendBuffer, dwSendDataLength, &dwBytes, Irp)) ?
NOERROR : WSAGetLastError();
}
if (p)
{
if (err == ERROR_IO_PENDING)
{
p->SetTimeout(dwMilliseconds);
}
else
{
p->Destroy();
}
}
Irp->CheckErrorCode(err);
}
another possible solution set timer via CreateTimerQueueTimer and if timer expired - call CancellIoEx or close I/O handle from here. difference with event solution - if I/O will be completed before timer expired - the WaitOrTimerCallback callback function will be not automatically called. in case event - I/O subsystem set event when I/O complete (after intial pending status) and thanks to that (event in signal state) callback will be called. but in case timer - no way pass it to io request as parameter (I/O accept only event handle). as result we need save pointer to timer object by self and manually free it when I/O complete. so here will be 2 pointer to timer object - one from pool (saved by CreateTimerQueueTimer) and one from our object (socket) class (we need it for dereference object when I/O complete). this require reference counting on object which incapsulate timer too. from another side we can use timer not for single I/O operation but for several I/O (because it not direct bind to some I/O)

Copy data from one Chronicle to another

For the concept of taking back up , i need to copy data from one chronicle queue to another .
Would it be safe to do a directly copy the whole Bytes object from wire of one queue into another ?
something like
documentContext().wire().bytes().read(byte_buffer)
and then wrapping this byte_buffer into byte_store and writing as
documentContext().wire().bytes().write(byte_Store).
The reason i'm doing it is avoid any conversion back and forth into custom objects?
You can, but a simpler approach is to copy directly from one to the other.
ChronicleQueue inQ = SingleChronicleQueueBuilder.binary("in").build();
ExcerptTailer tailer = inQ.createTailer();
ChronicleQueue outQ = SingleChronicleQueueBuilder.binary("out").build();
ExcerptAppender appender = outQ.acquireAppender();
while(true) {
try (DocumentContext inDC = tailer.readingDocument()) {
if (!inDC.isPresent()) {
// not message available
break; // or pause or do something else.
}
try (DocumentContext outDC = appender.writingDocument()) {
outDC.wire().write(inDC.wire().bytes());
}
}
}
}

Not receiving messages after sometime

I am using JNA to access User32 functions (I dont think it has got to do with Java here, more of concept problem). In my application, I have a Java process which communicates with the Canon SDK. To dispatch any messages I am using the below function:
private void peekMessage(WinUser.MSG msg) throws InterruptedException {
int hasMessage = lib.GetMessage(msg, null, 0, 0);
if (hasMessage != 0) {
lib.TranslateMessage(msg);
lib.DispatchMessage(msg);
}
Thread.sleep(1);
}
peekMessage is called in a loop and it all works well. Whenever an Image is taken from camera, I get the event and do the rest.
But I have observed, say after about 15 seconds (sometimes never or sometimes just at start) of no activity with camera, taking picture does not give me any download event. Later the whole application becomes unusable as it doesn't get any events from camera.
What can be the reason for this? Please let me know of any other info needed, I can paste the respective code along.
Edit:
Initialization:
Map<String, Integer> options = new HashMap<String, Integer>();
lib = User32.INSTANCE;
hMod = Kernel32.INSTANCE.GetModuleHandle("");
options.put(Library.OPTION_CALLING_CONVENTION, StdCallLibrary.STDCALL_CONVENTION);
this.EDSDK = (EdSdkLibrary) Native.loadLibrary("EDSDK/dll/EDSDK.dll", EdSdkLibrary.class, options);
private void runNow() throws InterruptedException {
while (!Thread.currentThread().isInterrupted()) {
Task task = queue.poll();
if (task != null) {
int taskResult = task.call();
switch (taskResult) {
case (Task.INITIALIZE_STATE):
break;
case (Task.PROCESS_STATE):
break;
case (Task.TERMINATE_STATE): {
//queue.add(new InitializeTask());
Thread.currentThread().interrupt();
break;
}
default:
;
}
}
getOSEvents();
}
}
WinUser.MSG msg = new WinUser.MSG();
private void getOSEvents() throws InterruptedException {
if (isMac) {
receiveEvents();
} else {
peekMessage(msg);
}
}
Above, whenever I get my camera event, it add's it to the queue and in each loop I check the queue to process any Task. One more important information: This is a process running on cmd and has no window. I just need the events from my camera and nothing else.
The code where I register callback functions:
/**
* Adds handlers.
*/
private void addHandlers() {
EdSdkLibrary.EdsVoid context = new EdSdkLibrary.EdsVoid(new Pointer(0));
int result = EDSDK.EdsSetObjectEventHandler(edsCamera, new NativeLong(EdSdkLibrary.kEdsObjectEvent_All), new ObjectEventHandler(), context).intValue();
//above ObjectEventHandler contains a function "apply" which is set as callback function
context = new EdSdkLibrary.EdsVoid(new Pointer(0));
result = EDSDK.EdsSetCameraStateEventHandler(edsCamera, new NativeLong(EdSdkLibrary.kEdsStateEvent_All), new StateEventHandler(), context).intValue();
//above StateEventHandler contains a function "apply" which is set as callback function
context = new EdSdkLibrary.EdsVoid(new Pointer(0));
result = EDSDK.EdsSetPropertyEventHandler(edsCamera, new NativeLong(EdSdkLibrary.kEdsStateEvent_All), new PropertyEventHandler(), context).intValue();
//above PropertyEventHandler contains a function "apply" which is set as callback function
}
You are getting ALL messages from ALL windows that belong to this thread, that includes all mouse moves, paints etc. if you aren't rapidly calling this function your message queue will overflow and cause the behavior you describe.
The sleep you definitely don't want as GetMessage yields if no message is waiting.
So if there exists a normal message pump(s) (i.e GetMessage/DispatchMessage) loop somewhere else for this threads window(s) then you should let that pump do most of the work, perhaps use wMsgFilterMin, wMsgFilterMax to just get the event message you require; or even better in this case use peekmessage with PM_NOREMOVE (then you will need your sleep
call as peekmessage returns immediately).
Alternatively provide the hWnd of the window that generates the event to reduce the workload.
Use spy++ to look into which windows this thread owns and what messages are being produced.
To take this answer further please provide answers to: what else is this thread doing and what windows does it own; also is this message pump the only one or do you call into the SDK API where it may be pumping messages too?
There is an OpenSource project wrapping EDSDK with JNA and it has a version of your code that is probably working better:
https://github.com/kritzikratzi/edsdk4j/blob/master/src/edsdk/api/CanonCamera.java#L436
Unfortunately this is not platform independent and specifically the way things work on windows. I am currently in the process of trying to get a MacOS version of things working at:
https://github.com/WolfgangFahl/edsdk4j

del_gendisk haging during cleanup of block device after media removal during IO

I have block driver for a hot-pluggable PCI storage device. if the device is removed during IO, I never seem to get a call to release (i.e. mydev_blk_release(struct gendisk *gd, fmode_t mode)), which I think is preventing del_gendisk() from completing, thus hanging the cleanup of the driver. I am ending all requests on the queue once an eject happens, but it still doesn't seem to cause a release. What is the right way to terminate requests and delete the gendisk in the case of vanished media?
This is caused by not ending the request that is broken by the device removal. In my driver, I had the folowing request_fn:
static void mydev_submit_req(struct request_queue *q)
{
struct mydev_info *mydev = q->queuedata;
if (!mydev){
struct request* req;
while ((req = blk_fetch_request(q)) != NULL){
req->cmd_flags |= REQ_QUIET;
__blk_end_request_all(req, -ENODEV);
}
} else {
queue_work(mydev->wq, &mydev->work);
}
}
This will prevent the requests entering the driver's workqueue when the device disappears (signified by the loss of mydev). However this hung because the last request was not actually completed, causing q->rq->elvpriv (now called q->nr_rqs_elvpriv) to remain at 1, which caused blk_drain_queue() to spin forever, which hung blk_cleanup_queue() and prevented the driver being able to remove the device.
The solution looks like this (in the workqueue callback function in my driver, but this depends on how you structure the IO work):
req = blk_fetch_request(q);
while (req) {
// returns -ENODEV if the disk is ejected during transfer
//bytes tells us how many bytes we managed to do
res = mydev_do_req(q, req, &bytes);
if (unlikely(res == -ENODEV)) {
dev_err(&mydev->pdev->dev,
"device ejected during transfer, returning\n");
//end the current request, since we started it
//THIS IS WHAT WAS MISSING
__blk_end_request_all(req, -ENODEV);
break;
//get out - the rest of the queue will be emptied on the next
//submit_req
} else if (!__blk_end_request(req, res, bytes)) {
req = blk_fetch_request(q); //get the next request
}
}

Resources