http_listener cpprestsdk how to handle multiple POST requests - c++11

I have developed a client server application with casablanca cpprestskd.
Every 5 minutes a client send informations from his task manager (processes,cpu usage etc) to server via POST method.
The project should be able to manage about 100 clients.
Every time that server receives a POST request he opens an output file stream ("uploaded.txt") ,extract some initial infos from client (login,password),manage this infos, save all infos in a file with the same name of client (for example: client1.txt, client2.txt) in append mode and finally reply to client with a status code.
This is basically my POST handle code from server side:
void Server::handle_post(http_request request)
{
auto fileBuffer =
std::make_shared<Concurrency::streams::basic_ostream<uint8_t>>();
try
{
auto stream = concurrency::streams::fstream::open_ostream(
U("uploaded.txt"),
std::ios_base::out | std::ios_base::binary).then([request, fileBuffer](pplx::task<Concurrency::streams::basic_ostream<unsigned char>> Previous_task)
{
*fileBuffer = Previous_task.get();
try
{
request.body().read_to_end(fileBuffer->streambuf()).get();
}
catch (const exception&)
{
wcout << L"<exception>" << std::endl;
//return pplx::task_from_result();
}
//Previous_task.get().close();
}).then([=](pplx::task<void> Previous_task)
{
fileBuffer->close();
//Previous_task.get();
}).then([](task<void> previousTask)
{
// This continuation is run because it is value-based.
try
{
// The call to task::get rethrows the exception.
previousTask.get();
}
catch (const exception& e)
{
wcout << e.what() << endl;
}
});
//stream.get().close();
}
catch (const exception& e)
{
wcout << e.what() << endl;
}
ManageClient();
request.reply(status_codes::OK, U("Hello, World!")).then([](pplx::task<void> t) { handle_error(t); });
return;
}
Basically it works but if i try to send info from due clients at the same time sometimes it works sometimes it doen't work.
Obviously the problem if when i open "uploaded.txt" stream file.
Questions:
1)Is CASABLANCA http_listener real multitasking?how many task it's able to handle?
2)I didn't found in documentation ax example similar to mine,the only one who is approaching to mine is "Casalence120" Project but he uses Concurrency::Reader_writer_lock class (it seems a mutex method).
What can i do in order to manage multiple POST?
3)Is it possible to read some client infos before starting to open uploaded.txt?
I could open an output file stream directly with the name of the client.
4)If i lock access via mutex on uploaded.txt file, Server become sequential and i think this is not a good way to use cpprestsdk.
I'm still approaching cpprestskd so any suggestions would be helpful.

Yes, the REST sdk processes every request on a different thread
I confirm there are not many examples using the listener.
The official sample using the listener can be found here:
https://github.com/Microsoft/cpprestsdk/blob/master/Release/samples/CasaLens/casalens.cpp
I see you are working with VS. I would strongly suggest to move to VC++2015 or better VC++2017 because the most recent compiler supports co-routines.
Using co_await dramatically simplify the readability of the code.
Substantially every time you 'co_await' a function, the compiler will refactor the code in a "continuation" avoiding the penalty to freeze the threads executing the function itself. This way, you get rid of the ".then" statements.
The file problem is a different story than the REST sdk. Accessing the file system concurrently is something that you should test in a separate project. You can probably cache the first read and share the content with the other threads instead of accessing the disk every time.

Related

MassTransit - How to fault messages in a batch

I am trying to utilize the MassTransit batching technique to process multiple messages to reduce the individual queries to be database (read and write).
If there is an exception while processing one/more of the messages, then the expectation is to fault only required messages and have the ability to process the rest of the messages.
This is common scenario in my use case ,what I am trying to establish here is a way to perform batch processing that caters for poisoned messages
For example, if I have a batch size of 10 messages, 10 in the queue and 1 persistently fails, I still need a means of ensuring the other 9 can be processed successfully. It is fine if all 10 need to be returned to the queue and subset re-consumed - but the poisoned message needs to be eliminated somehow. Does this requirement discount the use of batching?
I have tried below, however did solve my use case.
catching the exception and raising NotifyFaulted for that specific message.
modified sample-twitch application, to throw an exception to something like below , based on https://github.com/MassTransit/Sample-Twitch/blob/master/src/Sample.Components/BatchConsumers/RoutingSlipBatchEventConsumer.cs
file.
public Task Consume(ConsumeContext<Batch<RoutingSlipCompleted>> context)
{
if (_logger.IsEnabled(LogLevel.Information))
{
_logger.Log(LogLevel.Information, "Routing Slips Completed: {TrackingNumbers}",
string.Join(", ", context.Message.Select(x => x.Message.TrackingNumber)));
}
for (int i = 0; i < context.Message.Length; i++)
{
try
{
if (i % 2 != 0)
throw new System.Exception("business error -message failed");
}
catch (System.Exception ex)
{
context.Message[i].NotifyFaulted(TimeSpan.Zero, "batch routing silp faulted", ex);
}
}
return Task.CompletedTask;
}
I have dig into a few more threads that look similar to the issue ,for reference.
Masstransit error handling for batch consumer
If you want to use batch, and have a message in that batch that cannot be processed, you should catch the exception and do something else with the poison message. You could write it someplace else, publish some type of event, or whatever else. But MassTransit does not allow you to partially complete/fault messages of a batch.

What is the meaning of "Fb::Error: A transaction has already been started"

I have a Ruby application that crashes sometimes with this error message:
Fb::Error: A transaction has already been started)
I'm now wondering what this message means. I searched a little bit and I read that Firebird is not supporting nested transactions. Could the message hint to this? If not, what else could this mean?
This is not a Firebird error message. It is an error message in the driver you're using. Specifically here:
static void fb_connection_transaction_start(struct FbConnection *fb_connection, VALUE opt)
{
char *tpb = 0;
long tpb_len;
if (fb_connection->transact) {
rb_raise(rb_eFbError, "A transaction has been already started");
}
if (!NIL_P(opt)) {
tpb = trans_parseopts(opt, &tpb_len);
} else {
tpb_len = 0;
tpb = NULL;
}
isc_start_transaction(fb_connection->isc_status, &fb_connection->transact, 1, &fb_connection->db, tpb_len, tpb);
xfree(tpb);
fb_error_check(fb_connection->isc_status);
}
Without in-depth familiarity with this driver, I'm guessing the problem is that you're trying to start a transaction on a connection that already has an active transaction.
Firebird itself supports multiple parallel transactions on a single connection, and it supports nested transactions in the form of SQL standard savepoints, but it looks like the driver you're using doesn't support this.
The solution (or workaround) would seem to be to either not start a transaction when you already have an active transaction, or to first commit or rollback an existing transaction before starting a new one.

How to safely write to one file from many verticle instances in vert.x 3.2?

Instead of using a logger or database server I'd like to append information to one file from possibly many verticle instances.
There are versions of methods for writing asynchronously to a file.
Can I assume that vertx handles the synchronisation between the writes so that these dont interfere when using those versions of methods marked as ¨async¨ ?
There seems to be a rule that one can rely on vertx providing all isolation between concurrent processing out of the box. But is that true in case of writing file access?
Could you please include a code snippet into the answer that shows how to open and write to one file from many verticle instances with finest possible granularity, e.g. for logging requests.
I wouldn't recommend writing to a single file with many different "writers". Regarding concurrent logging I would stick to the Single Writer principle.
Create a Verticle which subscribes to the Event Bus and listens for messages to be logged. Lets call this Verticle Logger which listens to system.logger.
EventBus eb = vertx.eventBus();
eb.consumer("system.logger", message -> {
// write to file
});
Verticles which like to log something need to send a message to the Logger Verticle:
eventBus.send("system.logger", "foobar");
Appending to a existing file work something like this (didn't test):
vertx.fileSystem().open("file.log", new OpenOptions(), result -> {
if (result.succeeded()) {
Buffer buff = Buffer.buffer(message); // message from consume
AsyncFile file = result.result();
file.write(buff, buff.length() * i, ar -> {
if (ar.succeeded()) {
System.out.println("done");
} else {
System.err.println("write failed: " + ar.cause());
}
});
} else {
System.err.println("open file failed " + result.cause());
}
});

Problems with map container

Introduction
I'm trying to create a port-forwarding sample with tcp connections, so I need map client identification with his socket. When the client requests port-forwarding, I have to know who owns the sockets.
To do that, I created the following code:
std::map<std::string, tcp::socket> box_map;
std::map<std::string, tcp::socket>::iterator it;
it = box_map.find(id);
if (it != box_map.end())
return;
else{
box_map.insert(std::pair<std::string, tcp::socket>(id,s));
return;
}
Problem
But I got the following error:
error: use of deleted function ‘boost::asio::basic_stream_socket<boost::asio::ip::tcp>::basic_stream_socket(const boost::asio::basic_stream_socket<boost::asio::ip::tcp>&)’
tcp::socket is not copy constructible. So you'd have to construct the new pair in-place by moving your socket using emplace:
box_map.emplace(id, std::move(s));
Alternatively, you could just still use insert and just move into the pair you're constructing:
box_map.insert(std::make_pair(id, std::move(s)));

blk_cleanup_queue() doesn't return on block device deregistration

I'm writing a block device driver for a hot-pluggable PCI memory device on 2.6.43.2-6.fc15 (so LDD3 is out of date with respect to a lot of functions) and I'm having trouble getting the block device de-registration to go smoothly. When the device is removed, I go to tear down the gendisk and request_queue, but it hangs on blk_cleanup_queue(). Presumably there's some queue-related process I have neglected to carry out before that, but I can't see any major consistent differences with other block drivers from that kernel tree that i am using for reference (memstick, cciss, etc). What are the steps I should carry out before going to tidy up the queue and gendisk?
I am implementing .open, .release, .ioctl in the block_ops as well as a mydev_request(struct request_queue *q) attached with blk_init_queue(mydev_request, &mydev->lock), but I'm not sure exactly how to tidy the queue either when requests occur or when de-registering the block device.
This is caused by not ending the requests that you fetch off the queue. To fix it, end the request as follows:
while ((req = blk_fetch_request(q)) != NULL )
{
res = mydev_submit_request_sg(mydev, req);
if (res)
__blk_end_request_all(req, res);
else
__blk_end_request_cur (req, res);
}

Resources