I wonder if I got it all wrong when I have tried to used concurrent_queue. I am trying to process files using threads. The Threads pickup file names from a concurrent_queue and proceed ahead. My problem is that each thread seems to process the same file 4 times as I have 4 threads.
My main goal was to pick 4 different files from the queue and process them independently till the queue gets exhausted.
#include <string>
#include <strstream>
#include <ppl.h>
#include <concurrent_queue.h>
#include <thread>
using namespace std;
using namespace concurrency;
void ProcessQ(concurrent_queue<string> &fileQ,string folder)
{
TxtFileReader reader;
while (!fileQ.empty())
{
string fileName;
if (fileQ.try_pop(fileName))
{
vector<float> fpTemplate(0);
int n = reader.ReadTxtFile(folder+fileName, fpTemplate);
if (n > 0)
{
cout << "Processed file:" << fileName<<endl;
}
else
cout << "Skipping file:" << fileName<<endl;
}
}
}
int main()
{
stringstream fileNameStream;
concurrent_queue<string> fileQ;
for (int first = 1; last<= 100; first++)
{
fileNameStream << first << ".txt";
fileQ.push(fileNameStream.str());
fileNameStream.str(string());
}
string folder = string("E:\\Tests\\Outputs\\txts\\");
// Create threads and exectue
const short nThreads = 4;
thread fileProcessorThreads[nThreads];
for (short i = 0; i<nThreads; i++)
{
fileProcessorThreads[i] = thread(ProcessQ,fileQ,folder);
}
for (auto& th : fileProcessorThreads) {
th.join();
}
return 0;
}
}
The output on console is
Processed 1.txt
Processed 1.txt
Processed 1.txt
Processed 1.txt
Processed 2.txt
Processed 2.txt
Processed 2.txt
Processed 2.txt
What am I doing wrong?
Got it, I have to use std::ref() in order to make a shared reference to the queue.
fileProcessorThreads[i] = thread(ProcessQ,std::ref(fileQ),folder);
The original code was probably sending a copy of the queue to the thread functions.
Related
I am trying since a few days to set up c++ boost to log different data in 2 different files .
What I managed to get is different files , who all share the same content
The code bellow creates in the log folder , 2 files file.txt and file2.txt with the same content:
file.txt
0: [2019-Feb-14 19:39:01.997479] - thread 2
1: [2019-Feb-14 19:39:02.035582] - thread 1
file2.txt
0: [2019-Feb-14 19:39:01.997479] - thread 2
1: [2019-Feb-14 19:39:02.035582] - thread 1
class logger
{
src::logger_mt lg;
public:
logger(std::string filename)
{
// Create a text file sink
typedef sinks::synchronous_sink< sinks::text_multifile_backend > file_sink;
shared_ptr< file_sink > sink(new file_sink);
// Set up how the file names will be generated
sink->locked_backend()->set_file_name_composer(sinks::file::as_file_name_composer(
expr::stream << "logs/" << filename));
// Set the log record formatter
sink->set_formatter
(
expr::format("%1%: [%2%] - %3%")
% expr::attr< unsigned int >("RecordID")
% expr::attr< boost::posix_time::ptime >("TimeStamp")
% expr::smessage
);
// Add it to the core
logging::core::get()->add_sink(sink);
// Add some attributes too
logging::core::get()->add_global_attribute("TimeStamp", attrs::local_clock());
logging::core::get()->add_global_attribute("RecordID", attrs::counter< unsigned int >());
}
void log(std::string message)
{
BOOST_LOG(lg) << message;
}
};
logger logger1("file2.txt");
logger logger2("file.txt");
// This function is executed in a separate thread
int main(int argc, char* argv[])
{
logger2.log("thread 2");
logger1.log("thread 1");
return 0;
}
I know is a basic question , but i read all the examples from the boost distribution libs\log\example and i couldnt find anything similar.
The help is appreciated.
See this answer, in particular its first half about channels and filters.
I'm currently using Omnet++, and veins, and I have this runtime error appearing suddenly, and I am not able to understand it in order to fix it properly.
Error in module (TraCIDemoRSU11p) RSUExampleScenario.rsu[0].appl
(id=8) at event #6180, t=53.956510612297: Array of size 220 indexed by
220.
TRAPPING on the exception above, due to a debug-on-errors=true configuration option. Is your debugger ready?
I am assuming that it might be related to this message I am sending from the RSU to the vehicles with this code, but I am not sure how it's related.
cplusplus {{
#include "veins/modules/messages/WaveShortMessage_m.h"
}}
class WaveShortMessage;
message DelayedFromControllerMessage extends WaveShortMessage {
string vehiclesList [220] ;
}
I am using omnet++ Version: 5.0 and Veins 4.4
Edited, I'm using the array in these places:
1-
void TraCIDemoRSU11p::sendDelayedMessage(std::list<const char *> vehicleList) {
sentDelayedMessage = true;
//vehicleList = {};
t_channel channel = dataOnSch ? type_SCH : type_CCH;
DelayedFromControllerMessage* dsm = prepareDelayedSM("delayed",dataLengthBits, channel, dataPriority, -1,2,vehicleList);
std::list<const char *>::iterator it = vehicleList.begin();
//const char * v;
char* vx = new char [100];
vx[0] = '\0';
for(int i=0; i<vehicleList.size(); i++){
//v =*it;
strcpy(vx,*it);
//vx = *it;
++it;
dsm->setVehiclesList(i, vx);
}
if (sendDelayedEvt->isScheduled()) {
cancelAndDelete(sendDelayedEvt);
}else {
delete sendDelayedEvt;
}
sendDelayedEvt = new cMessage("delayed evt", SEND_DELAYED_EVT); // create event object to use it in timing
simtime_t offSet = dblrand() * (par("beaconInterval").doubleValue());
TimeStart = simTime() + offSet;
scheduleAt(TimeStart, sendDelayedEvt);
sendDelayedSM(dsm);
}
2-
DelayedFromControllerMessage* BaseWaveApplLayer:: prepareDelayedSM(const char * name, int lengthBits, t_channel channel, int priority, int rcvId,int serial,std::list<const char *>vehicleList ) {
DelayedFromControllerMessage* dsm = new DelayedFromControllerMessage(name);
dsm->addBitLength(headerLength);
dsm->addBitLength(lengthBits);
switch (channel) {
case type_SCH: dsm->setChannelNumber(Channels::SCH1); break; //will be rewritten at Mac1609_4 to actual Service Channel. This is just so no controlInfo is needed
case type_CCH: dsm->setChannelNumber(Channels::CCH); break;
}
dsm->setPsid(0);
dsm->setPriority(priority);
dsm->setWsmVersion(1);
dsm->setTimestamp(simTime());
dsm->setSenderAddress(myId);
dsm->setRecipientAddress(rcvId);
dsm->setSenderPos(curPosition);
dsm->setSerial(serial);
std::list<const char *>::iterator it = vehicleList.begin();
const char * v;
for(int i=0; i<vehicleList.size(); i++){
v =*it;
++it;
VLvar1.push_back(v);
dsm->setVehiclesList(i, v);
}
if ((std::string)name == "beacon") {
DBG << "Creating Beacon with Priority " << priority << " at Applayer at " << dsm->getTimestamp() << std::endl;
}
if ((std::string)name == "delayed") {
DBG << "Creating Data with Priority " << priority << " at Applayer at " << dsm->getTimestamp() << std::endl;
}
return dsm;
}
3-
void MyTraCIDemo11p::onDataDelayed(DelayedFromControllerMessage* dsm) {
int x = 0;
std::string vehichleId = mobility->getExternalId();
for (int i=0 ; i < dsm->getVehiclesListArraySize();i++)
{
vehicleList.push_back(std::string(dsm->getVehiclesList(i)));
}
ttry = std::find(vehicleList.begin(), vehicleList.end(), vehichleId);
if (vehichleId == *ttry){
x = 1;
}
if (state == QUEUING && x == 1){
findHost()->bubble("Received ");
state = WAITING;
stateToString(state);
}
}
The message should be sent from the RSU to the vehicles.
Even without seeing the actual code from the application (appl) or from the configuration file you are using, I am guessing you are trying to get the last element (element 220) from the array.
The error message already tells what the problem is. Your array has a size of 220 and you are trying to use the index 220 which is not possible, since array indexes start at 0. Therefore for addressing the last element in your array, you have to use index 221.
I am not sure if this is the reason for your error:
ttry = std::find(vehicleList.begin(), vehicleList.end(), vehichleId);
if (vehichleId == *ttry){
x = 1;
}
But, It is better to write it this way:
if (std::find(vehicleList.begin(), vehicleList.end(), vehichleId) != vehicleList.end()){
x = 1;
}
I don't recommend referencing thefind iterator (i.e., *ttry) if it is not found, it is like referencing *(vehicleList.end()).
it seems to me you have two veichleList variables, an array in the dsm and another one which is a vector. is this correct? if yes, you should make sure that the veichleList.size() is always less or equal 220.
Check this tutorial to learn how to debug your project in omnet++:
https://docs.omnetpp.org/tutorials/tictoc/part2/
So i ended up finding a solution for this issue just now.
it was by doing this:
cplusplus {{
#include "veins/modules/messages/WaveShortMessage_m.h"
}}
class WaveShortMessage;
message DelayedFromControllerMessage extends WaveShortMessage {
string vehiclesList [] ;
}
=====
DelayedFromControllerMessage* BaseWaveApplLayer:: prepareDelayedSM(const char * name, int lengthBits, t_channel channel, int priority, int rcvId,int serial,std::list<const char *>vehicleList ) {
DelayedFromControllerMessage* dsm = new DelayedFromControllerMessage(name);
dsm->addBitLength(headerLength);
dsm->addBitLength(lengthBits);
switch (channel) {
case type_SCH: dsm->setChannelNumber(Channels::SCH1); break; //will be rewritten at Mac1609_4 to actual Service Channel. This is just so no controlInfo is needed
case type_CCH: dsm->setChannelNumber(Channels::CCH); break;
}
dsm->setPsid(0);
dsm->setPriority(priority);
dsm->setWsmVersion(1);
dsm->setTimestamp(simTime());
dsm->setSenderAddress(myId);
dsm->setRecipientAddress(rcvId);
dsm->setSenderPos(curPosition);
dsm->setSerial(serial);
int NS = 0;
std::list<const char *>::iterator itPD = vehicleList.begin();
const char * vPD;
int i0 = 0;
while(itPD != vehicleList.end()){
vPD = *itPD;
++itPD;
++NS;
dsm->setVehiclesListArraySize(NS);
dsm->setVehiclesList(i0, vPD);
++i0;
VLvar1.push_back(vPD);
}
if ((std::string)name == "beacon") {
DBG << "Creating Beacon with Priority " << priority << " at Applayer at " << dsm->getTimestamp() << std::endl;
}
if ((std::string)name == "delayed") {
DBG << "Creating Data with Priority " << priority << " at Applayer at " << dsm->getTimestamp() << std::endl;
}
return dsm;
}
I removed the array size and placed with a manual counter in BaseWaveApplLayer:: prepareDelayedSM using a while loop. I thought about posting the solution to help others when facing a similar problem. :)
I been struggling reading a file that has a book format. The file is broken into pages by a string that looks like this "---------------------------------------". What I'm trying to do is read all the words and keep track of the page number and the word number of every word, the file looks like this
my file
For example if the word "hello" appears in the first page it would looks like this " hello 1,1" because it's the first word on page one if the word would appear in the second page the output will be "hello 2,1"
This is the code I have so far
ifstream inFile;
inFile.open("GreatExpectations.txt");
if(!inFile.is_open()) {
cout << "Error, can't open the file....."<<endl;
return 1;
}
string word;
string separator;
separator = "----------------------------------------";
int pageNum = 0, wordNum = 0;
IndexMap myMap(200000);
string title;
for(int i = 0; i < 2; i++) {
getline(inFile, title);
cout << title <<endl;
}
while(!inFile.eof())
{
inFile >> word;
//cout << word << " ";
wordNum++;
if(word == separator)
pageNum++;
}
If I well understood your question here is my approach to the problem:
#include <iostream>
#include <fstream>
#include <vector>
#include <sstream>
using namespace std;
struct WordInfo {
string word;
int pageNum;
int wordNum;
};
int main() {
ifstream inFile;
inFile.open("GreatExpectations.txt");
if(!inFile.is_open()) {
cout << "Error, can't open the file....."<<endl;
return 1;
}
int pageNum = 1, wordNum = 0;
vector<WordInfo> words; // container for words with informations
// read the file line-by-line
for(string line; getline(inFile, line);) {
// detect the page separator which is a line from hyphens only
if(line.find_first_not_of("-") == string::npos) {
pageNum++;
wordNum = 0;
continue;
}
// process the line word-by-word
stringstream ss(line);
for(string word; getline(ss, word, ' ');) {
wordNum++;
words.push_back({ word, pageNum, wordNum });
}
}
return 0;
}
The WordInfo structure will hold informations from a word as you wanted. It's not the optimal but more simple to read the file line-by-line, so there is two loops: the first reads a line and the second reads the words from that line. If a word is read, it will be pushed into the words vector for later use. That's all.
I have a program that is supposed to take in a paragraph like
Testing#the hash#tag
#program!#when #beginning? a line
or #also a #comma,
and output something like
#the
#tag
#program
#when
#beginning
#also
#comma,
I feel like the logic makes sense, but obviously not because the program never seems to get into the line of input. The problem is almost definitely in the last source file below.
Here is the main source program
#include "HashTagger.h"
#include <string>
#include <iostream>
using namespace hw02;
using namespace std;
int main() {
// Construct an object for extracting the
// hashtags.
HashTagger hashTagger;
// Read the standard input and extract the
// hashtags.
while (true) {
// Read one line from the standard input.
string line;
getline(cin, line);
if (!cin) {
break;
}
// Get all of the hashtags on the line.
hashTagger.getTags(line);
}
// Print the hashtags.
hashTagger.printTags();
// Return the status.
return 0;
}
my header file
#ifndef HASHTAGGER_H
#define HASHTAGGER_H
#include <string>
namespace hw02 {
class HashTagger {
public:
void getTags(std::string line);
void printTags();
private:
std::string hashtags_;
};
}
#endif
and a source file
the test in the source file seems to show that the program only gets to the second line and then stops before grabbing the last 2 hashtags
#include "HashTagger.h"
#include <iostream>
using namespace std;
using namespace hw02;
void HashTagger::getTags(string line) {
// Loop over all characters in a line that can begin a hashtag
int b = 0;
string hashtags_ = "";
for (unsigned int j = 0; j < line.length(); ++j) {
char c = line.at(j);
// if "#" is found assign beginning of capture to b
if (c == '#') {
b = j;
// if the beginning is less than the end space, newline, ".", "?", or "!" found, add substring of the hashtag to hashtags_
}
if (b < j && (c == ' ' || c == '\n' || c == '.' || c == '?' || c == '!' )) {
hashtags_ = hashtags_ + "\n" + line.substr(b, j - b + 1);
b = 0;
//Test// cout << b << "/" << j << "/" << c << "/" << hashtags_ << "/" << endl;
}
}
}
void HashTagger::printTags() {
// print out hashtags_ to the console
cout << hashtags_ << endl;
}
You are redeclaring hashtags_ inside your getTags function. Therefore, all string modifications operate on a local variable instead of the class member variable.
Change the line
string hashtags_ = "";
to
hashtags_ = "";
in order to avoid the redeclaration and operate on the class member variable used for the output later on.
Also, make sure that your input is terminated with two newline characters (\n\n), to avoid breaking out of the main loop too early, or move your check and break statement after the getTags call:
while (true) {
// Read one line from the standard input.
string line;
getline(cin, line);
// Get all of the hashtags on the line.
hashTagger.getTags(line);
if (!cin) {
break;
}
}
I don't see this program having any practical usage, but while experimenting with c++ 11 concurrency and conditional_variables I stumbled across something I don't fully understand.
At first I assumed that using notify_one() would allow the program below to work. However, in actuality the program just froze after printing one. When I switched over to using notify_all() the program did what I wanted it to do (print all natural numbers in order). I am sure this question has been asked in various forms already. But my specific question is where in the doc did I read wrong.
I assume notify_one() should work because of the following statement.
If any threads are waiting on *this, calling notify_one unblocks one of the waiting threads.
Looking below only one of the threads will be blocked at a given time, correct?
class natural_number_printer
{
public:
void run()
{
m_odd_thread = std::thread(
std::bind(&natural_number_printer::print_odd_natural_numbers, this));
m_even_thread = std::thread(
std::bind(&natural_number_printer::print_even_natural_numbers, this));
m_odd_thread.join();
m_even_thread.join();
}
private:
std::mutex m_mutex;
std::condition_variable m_condition;
std::thread m_even_thread;
std::thread m_odd_thread;
private:
void print_odd_natural_numbers()
{
for (unsigned int i = 1; i < 100; ++i) {
if (i % 2 == 1) {
std::cout << i << " ";
m_condition.notify_all();
} else {
std::unique_lock<std::mutex> lock(m_mutex);
m_condition.wait(lock);
}
}
}
void print_even_natural_numbers()
{
for (unsigned int i = 1; i < 100; ++i) {
if (i % 2 == 0) {
std::cout << i << " ";
m_condition.notify_all();
} else {
std::unique_lock<std::mutex> lock(m_mutex);
m_condition.wait(lock);
}
}
}
};
The provided code "works" correctly and gets stuck by design. The cause is described in the documentation
The effects of notify_one()/notify_all() and
wait()/wait_for()/wait_until() take place in a single total order, so
it's impossible for notify_one() to, for example, be delayed and
unblock a thread that started waiting just after the call to
notify_one() was made.
The step-by-step logic is
The print_odd_natural_numbers thread is started
The print_even_natural_numbers thread is started also.
The m_condition.notify_all(); line of print_even_natural_numbers is executed before than the print_odd_natural_numbers thread reaches the m_condition.wait(lock); line.
The m_condition.wait(lock); line of print_odd_natural_numbers is executed and the thread gets stuck.
The m_condition.wait(lock); line of print_even_natural_numbers is executed and the thread gets stuck also.