I'm trying to use multiple threads in an application that reads out of a large file.
I want to read from the file in the main thread and put each line into a deque called countedAddresses.
The I want multiple threads to pick lines out of countedAddresses and pull all of the email addresses out of them and place them in a deque called resolvedAddresses.
At the moment this all works if there is only one resolveAddressThread running, but if I run two I get heap corruption errors.
I launch the threads with:
_beginthread( resolveAddressThread, 0, NULL );
_beginthread( resolveAddressThread, 0, NULL );
Here is the thread code:
unsigned int linesCounted, linesResolved, linesProcessed;
std::deque<std::string> countedAddresses;std::deque<std::string> resolvedAddresses;
HANDLE linesCMutex, linesRMutex, linesRCMutex;
void resolveAddressThread(void* pParams)
{
string fileLine= "";
while(1)
{
WaitForSingleObject(linesCMutex, INFINITE);
WaitForSingleObject(linesRCMutex, INFINITE);
if (linesCounted>linesResolved)
{
if (countedAddresses.size()!=0)
{
fileLine = countedAddresses.front();
countedAddresses.pop_front();
ReleaseMutex(linesRCMutex);
ReleaseMutex(linesCMutex);
}
else
{
ReleaseMutex(linesRCMutex);
ReleaseMutex(linesCMutex);
Sleep(1000);
}
}
else
{ cout<<"All lines in the file have been processed."<<endl;
ReleaseMutex(linesRCMutex);
ReleaseMutex(linesCMutex);
break;}
if (fileLine.length()>1)
{
string::iterator strIter;
string buffer;
int tabCount=0;
bool useFlag =false, keepFlag = false, stopFlag = false;
buffer.clear();
for(strIter=fileLine.begin(); strIter!=fileLine.end(); ++strIter)
{
if (*strIter == '\t')
{
++tabCount;
}
if (tabCount == 10)
{
if ('\"'== *strIter)
{
useFlag=!useFlag;
}
else if (useFlag && *strIter > 0)
{
if (*strIter == '#')
{
buffer+=*strIter;
keepFlag = true;
}
else if ((*strIter ==' '||*strIter ==';')&& keepFlag)
{
if (buffer.at(buffer.length()-1)=='.')
{buffer = buffer.substr(0, buffer.length() - 1);}
WaitForSingleObject(linesRMutex, INFINITE);
resolvedAddresses.push_back(buffer);
ReleaseMutex(linesRMutex);
buffer.clear();
keepFlag = false;
}
else if (*strIter ==' '||*strIter ==';')
{
buffer.clear();
keepFlag = false;
}
else
{
buffer+=*strIter;
}
}
}
if (tabCount ==11 && keepFlag==true)
{
if (buffer.at(buffer.length()-1)=='.')
{buffer = buffer.substr(0, buffer.length() - 1);}
WaitForSingleObject(linesRMutex, INFINITE);
resolvedAddresses.push_back(buffer);
ReleaseMutex(linesRMutex);
buffer.clear();
keepFlag = false;
}
}
WaitForSingleObject(linesRCMutex, INFINITE);
++linesResolved;
ReleaseMutex(linesRCMutex);
fileLine.clear();
}
}
}
Please let me know where I am going wrong.
Thanks,
ANkh
Related
Following code is work really slow, almost 30 second to process 400 entities:
int page = 0;
org.springframework.data.domain.Page<MyEntity> slice = null;
while (true) {
if (slice == null) {
slice = repo.findAll(PageRequest.of(page, 400, Sort.by("date")));
} else {
slice = repo.findAll(slice.nextPageable());
}
if (!slice.hasNext()) {
break;
}
slice.getContent().forEach(v -> v.setApp(SApplication.NAME_XXX));
repo.saveAll(slice.getContent());
LOGGER.info("processed: " + page);
page++;
}
I use following instead, 4-6 sec per 400 entities (gcp lib to work with datastore)
Datastore service = DatastoreOptions.getDefaultInstance().getService();
StructuredQuery.Builder<?> query = Query.newEntityQueryBuilder();
int limit = 400;
query.setKind("ENTITY_KIND").setLimit(limit);
int count = 0;
Cursor cursor = null;
while (true) {
if (cursor != null) {
query.setStartCursor(cursor);
}
QueryResults<?> queryResult = service.run(query.build());
List<Entity> entityList = new ArrayList<>();
while (queryResult.hasNext()) {
Entity loadEntity = (Entity) queryResult.next();
Entity.Builder newEntity = Entity.newBuilder(loadEntity).set("app", SApplication.NAME_XXX.name());
entityList.add(newEntity.build());
}
service.put(entityList.toArray(new Entity[0]));
count += entityList.size();
if (entityList.size() == limit) {
cursor = queryResult.getCursorAfter();
} else {
break;
}
LOGGER.info("Processed: {}", count);
}
Why I can't use spring to do that batch processing?
Full discussion here: https://github.com/spring-cloud/spring-cloud-gcp/issues/1824
First:
you need to use correct lib version: at least 1.2.0.M2
Second:
you need to implement new method in repository interface:
#Query("select * from your_kind")
Slice<TestEntity> findAllSlice(Pageable pageable);
Final code looks like:
LOGGER.info("start");
int page = 0;
Slice<TestEntity> slice = null;
while (true) {
if (slice == null) {
slice = repo.findAllSlice(DatastorePageable.of(page, 400, Sort.by("date")));
} else {
slice = repo.findAllSlice(slice.nextPageable());
}
if (!slice.hasNext()) {
break;
}
slice.getContent().forEach(v -> v.setApp("xx"));
repo.saveAll(slice.getContent());
LOGGER.info("processed: " + page);
page++;
}
LOGGER.info("end");
So, I've been trying to deserialize the IPXDefaultLibraryURLBookmark of the com.apple.Photos defaults (defaults read com.apple.Photos IPXDefaultLibraryURLBookmark) but no luck. Ideally I'd like a programmatic way in c++ to deserialize that value to retrieve the last known location of the photo
bookd0xUsersmateuscbPicturesPhotos Library.photoslibrary 0#˜ì5$r$Éò|åú¨A∫˙æJ file:///Macintosh HDÇ1tA∫‘}•$6465C0A4-1771-3C89-9055-147CEDFBBF2EÅÔ/∆72cd528f2dcfb4b3434986cf3caa02cc946333b8;00000000;00000000;0000000000000020;com.apple.app-sandbox.read-write;00000001;01000004;0000000002980783;/users/mateuscb/pictures/photos library.photoslibrary¥˛ˇˇˇdº‰#‘ î H ( 8 t0 †–Ä®
I know its not a bplist, since the first format specifier denotes bookd.
But I have no clue what that is. I'm somewhat new to OSX development, so this may be something very basic I'm missing.
I want to retrieve the: /users/mateuscb/pictures/photos library.photoslibrary portion so I can find the defaults photoslibrary.
Unless there is another way to retrieve the default photoslibrary path?
I figured out how to retrieve the path to the .photoslibrary. I used CFURLCreateByResolvingBookmarkData to get a CFURLRef from the plist then used CFURLGetFileSystemRepresentation to get the full path as a string.
Help from this sample to retrieve sandbox preferences: https://gist.github.com/glebd/4759724
Here is my full solution:
int main(int argc, const char * argv[]) {
bool success = FALSE;
UInt8 photosUrlString[PATH_MAX];
struct passwd *pwd = getpwuid(getuid());
if (pwd == NULL){
Log("Unable to retrieve current user");
return 0;
}
const char *home = pwd->pw_dir;
if (home == NULL){
Log("Unable to retrieve current user directory");
return 0;
}
CFMutableStringRef preferencesPath = CFStringCreateMutable(NULL, 0);
if (preferencesPath) {
CFStringAppend(preferencesPath, CFStringCreateWithCStringNoCopy(NULL, home, kCFStringEncodingUTF8, NULL));
CFStringAppend(preferencesPath, CFSTR("/Library/Containers/com.apple.Photos/Data/Library/Preferences/com.apple.Photos"));
} else {
Log("Unable to create CFString of user directory");
return 0;
}
CFPropertyListRef photosUrlPrefs = CFPreferencesCopyValue(CFSTR("IPXDefaultLibraryURLBookmark"), preferencesPath, kCFPreferencesCurrentUser, kCFPreferencesCurrentHost);
CFRelease(preferencesPath);
if (photosUrlPrefs) {
CFTypeID prefsType = CFGetTypeID(photosUrlPrefs);
if (CFDataGetTypeID() == prefsType) {
CFDataRef photosUrlData = (CFDataRef) photosUrlPrefs;
CFErrorRef urlResolveError = nil;
CFURLRef photosUrl = CFURLCreateByResolvingBookmarkData(kCFAllocatorDefault, photosUrlData, NULL, NULL, NULL, NULL, &urlResolveError);
if (photosUrl == NULL) {
if(urlResolveError != NULL) {
CFStringRef resolveErrorString = CFErrorCopyDescription(urlResolveError);
if (resolveErrorString != NULL) {
char resolveErrorCString[PATH_MAX];
if (CFStringGetCString((CFStringRef) resolveErrorString, resolveErrorCString, sizeof(resolveErrorCString), kCFStringEncodingUTF8)) {
Log("Error resolving URL: %s", resolveErrorCString);
}
CFRelease(resolveErrorString);
}
} else {
Log("Error resolving URL, no resolveError");
}
} else {
success = CFURLGetFileSystemRepresentation(photosUrl, false, photosUrlString, sizeof(photosUrlString));
CFRelease(photosUrl);
}
} else {
Log("Url plist value is not CFData");
}
if (photosUrlPrefs != NULL) {
CFRelease(photosUrlPrefs);
}
}
if(success) {
Log("path: %s", photosUrlString);
}
return 0;
}
Pstcreation works properly with outlook installed.
Now, I am trying to create a pst file with standalone version of MAPI . But my process is stuck in oSession.LogOff(). Further if a comment that oSession.LogOff() line and subsequently call the CreatePstWithRedemption function to create an other pst, the process gets stuck in oSession.LogonPstStore
private bool CreatePstWithRedemption(EmailJTableArgs objJTablArgs, EmailFilterArgs objFilterArgs,
EmailExportRequestParams emailExportRequestParams)
{
RDOSession oSession = null;
IRDOStore store = null;
RDOFolder fFOlder = null;
RDOFolder childFolder = null;
IRDOItems folderItems = null;
var pstCreationStatus = false;
try
{
oSession = new RDOSession();
store = oSession.LogonPstStore(_fileName, 1, "PST");
var folderName = Path.GetFileNameWithoutExtension(_fileName);
if (store != null)
{
fFOlder = store.IPMRootFolder;
foreach (RDOFolder folder in fFOlder.Folders)
{
folder.Delete();
}
childFolder = fFOlder.Folders.Add(folderName, Type.Missing);
folderItems = childFolder.Items;
var resultOfGetEmails = new ResultGetEmails();
resultOfGetEmails.TotalCount = -1;
do
{
var journalEmails = GetEmailList(objFilterArgs, objJTablArgs, emailExportRequestParams,
resultOfGetEmails);
for (var i = 0; i < journalEmails.Count; i++)
{
IRDOMail mail = null;
try
{
mail = folderItems.Add(rdoItemType.olMailItem);
// populate mail fields
mail.Sent = true;
mail.Save();
}
finally
{
if (mail != null)
Marshal.ReleaseComObject(mail);
}
}
resultOfGetEmails.TotalCount -= BatchSize;
objJTablArgs.PageStartIndex += BatchSize;
} while (resultOfGetEmails.TotalCount > 0);
pstCreationStatus = true;
}
}
finally
{
// Do cleanup
if (oSession != null && oSession.LoggedOn)
{
try
{
oSession.Logoff();
Marshal.ReleaseComObject(oSession);
}
catch
{
}
}
}
return pstCreationStatus;
}
Note: The same thing works well when run in an environment where outlook is installed.
I have figured out how to move my character around the maze using the algorithm I have written, but the count is not figuring correctly. At the end of each row my character moves up and down several times until the count reaches the specified number to exit the loop, then the character moves along the next row down until it reaches the other side and repeats the moving up and down until the count reaches the specified number again. Can anyone help me find why my count keeps getting off? The algorithm and the maze class I am calling from is listed below.
public class P4 {
public static void main(String[] args) {
// Create maze
String fileName = args[3];
Maze maze = new Maze(fileName);
System.out.println("Maze name: " + fileName);
// Get dimensions
int mazeWidth = maze.getWidth();
int mazeHeight = maze.getHeight();
// Print maze size
System.out.println("Maze width: " + mazeWidth);
System.out.println("Maze height: " + mazeHeight);
int r = 0;
int c = 0;
// Move commands
while (true){
for (c = 0; c <= mazeWidth; c++){
if (maze.moveRight()){
maze.isDone();
c++;
}
if (maze.isDone() == true){
System.exit(1);
}
if (maze.moveRight() == false && c != mazeWidth){
maze.moveDown();
maze.moveRight();
maze.moveRight();
maze.moveUp();
c++;
}
}
for (r = 0; r % 2 == 0; r++){
maze.moveDown();
maze.isDone();
if (maze.isDone() == true){
System.exit(1);
}
}
for (c = mazeWidth; c >= 0; c--){
if (maze.moveLeft()){
c--;
maze.isDone();
System.out.println(c);
}
if (maze.isDone() == true){
System.exit(1);
}
if (maze.moveLeft() == false && c != 0){
maze.moveDown();
maze.moveLeft();
maze.moveLeft();
maze.moveUp();
c--;
}
}
for (r = 1; r % 2 != 0; r++){
maze.moveDown();
maze.isDone();
if (maze.isDone() == true){
System.exit(1);
}
}
}
}
}
public class Maze {
// Maze variables
private char mazeData[][];
private int mazeHeight, mazeWidth;
private int finalRow, finalCol;
int currRow;
private int currCol;
private int prevRow = -1;
private int prevCol = -1;
// User interface
private JFrame frame;
private JPanel panel;
private Image java, student, success, donotpass;
private ArrayList<JButton> buttons;
// Maze constructor
public Maze(String fileName) {
// Read maze
readMaze(fileName);
// Graphics setup
setupGraphics();
}
// Get height
public int getHeight() {
return mazeHeight;
}
// Get width
public int getWidth() {
return mazeWidth;
}
// Move right
public boolean moveRight() {
// Legal move?
if (currCol + 1 < mazeWidth) {
// Do not pass?
if (mazeData[currRow][currCol + 1] != 'D')
{
currCol++;
redraw(true);
return true;
}
}
return false;
}
// Move left
public boolean moveLeft() {
// Legal move?
if (currCol - 1 >= 0) {
// Do not pass?
if (mazeData[currRow][currCol - 1] != 'D')
{
currCol--;
redraw(true);
return true;
}
}
return false;
}
// Move up
public boolean moveUp() {
// Legal move?
if (currRow - 1 >= 0) {
// Do not pass?
if (mazeData[currRow - 1][currCol] != 'D')
{
currRow--;
redraw(true);
return true;
}
}
return false;
}
// Move down
public boolean moveDown() {
// Legal move?
if (currRow + 1 < mazeHeight) {
// Do not pass?
if (mazeData[currRow + 1][currCol] != 'D')
{
currRow++;
redraw(true);
return true;
}
}
return false;
}
public boolean isDone() {
// Maze solved?
if ((currRow == finalRow) && (currCol == finalCol))
return true;
else
return false;
}
private void redraw(boolean print) {
// Wait for awhile
try {
Thread.sleep(500);
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
if (print)
System.out.println("Moved to row " + currRow + ", column " + currCol);
// Compute index and remove icon
int index = (prevRow * mazeWidth) + prevCol;
if ((prevRow >= 0) && (prevCol >= 0)) {
buttons.get(index).setIcon(null);
}
// Compute index and add icon
index = (currRow * mazeWidth) + currCol;
if ((currRow == finalRow) && (currCol == finalCol))
buttons.get(index).setIcon(new ImageIcon(success));
else
buttons.get(index).setIcon(new ImageIcon(student));
// Store previous location
prevRow = currRow;
prevCol = currCol;
}
// Set button
private void setButton(JButton button, int row, int col) {
if (mazeData[row][col] == 'S') {
button.setIcon(new ImageIcon(student));
currRow = row;
currCol = col;
} else if (mazeData[row][col] == 'J') {
button.setIcon(new ImageIcon(java));
finalRow = row;
finalCol = col;
} else if (mazeData[row][col] == 'D') {
button.setIcon(new ImageIcon(donotpass));
}
}
// Read maze
private void readMaze(String filename) {
try {
// Open file
Scanner scan = new Scanner(new File(filename));
// Read numbers
mazeHeight = scan.nextInt();
mazeWidth = scan.nextInt();
// Allocate maze
mazeData = new char[mazeHeight][mazeWidth];
// Read maze
for (int row = 0; row < mazeHeight; row++) {
// Read line
String line = scan.next();
for (int col = 0; col < mazeWidth; col++) {
mazeData[row][col] = line.charAt(col);
}
}
// Close file
scan.close();
} catch (IOException e) {
System.out.println("Cannot read maze: " + filename);
System.exit(0);
}
}
// Setup graphics
private void setupGraphics() {
// Create grid
frame = new JFrame();
panel = new JPanel();
panel.setLayout(new GridLayout(mazeHeight, mazeWidth, 0, 0));
frame.add(Box.createRigidArea(new Dimension(0, 5)), BorderLayout.NORTH);
frame.add(panel, BorderLayout.CENTER);
// Look and feel
try {
UIManager.setLookAndFeel(UIManager.getCrossPlatformLookAndFeelClassName());
} catch (Exception e) {
e.printStackTrace();
}
// Configure window
frame.setSize(mazeWidth * 100, mazeHeight * 100);
frame.setTitle("Maze");
frame.setResizable(false);
frame.setLocationRelativeTo(null);
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.setAlwaysOnTop(true);
// Load and scale images
ImageIcon icon0 = new ImageIcon("Java.jpg");
Image image0 = icon0.getImage();
java = image0.getScaledInstance(100, 100, Image.SCALE_DEFAULT);
ImageIcon icon1 = new ImageIcon("Student.jpg");
Image image1 = icon1.getImage();
student = image1.getScaledInstance(100, 100, Image.SCALE_DEFAULT);
ImageIcon icon2 = new ImageIcon("Success.jpg");
Image image2 = icon2.getImage();
success = image2.getScaledInstance(100, 100, Image.SCALE_DEFAULT);
ImageIcon icon3 = new ImageIcon("DoNotPass.jpg");
Image image3 = icon3.getImage();
donotpass = image3.getScaledInstance(100, 100, Image.SCALE_DEFAULT);
// Build panel of buttons
buttons = new ArrayList<JButton>();
for (int row = 0; row < mazeHeight; row++) {
for (int col = 0; col < mazeWidth; col++) {
// Initialize and add button
JButton button = new JButton();
Border border = new LineBorder(Color.darkGray, 4);
button.setOpaque(true);
button.setBackground(Color.gray);
button.setBorder(border);
setButton(button, row, col);
panel.add(button);
buttons.add(button);
}
}
// Show window
redraw(false);
frame.setVisible(true);
}
}
One error I can see in your code is that you're incrementing your c counter more often than you should. You start with it managed by your for loop, which means that it will be incremented (or decremented, for the leftward moving version) at the end of each pass through the loop. However, you also increment it an additional time in two of your if statements. That means that c might increase by two or three on a single pass through the loop, which is probably not what you intend.
Furthermore, the count doesn't necessarily have anything obvious to do with the number of moves you make. The loop code will always increase it by one, even if you're repeatedly trying to move through an impassible wall.
I don't really understand what your algorithm is supposed to be, so I don't have any detailed advice for how to fix your code.
One suggestion I have though is that you probably don't ever want to be calling methods on your Maze class without paying attention to their return values. You have a bunch of places where you call isDone but ignore the return value, which doesn't make any sense. Similarly, you should always be checking the return values from your moveX calls, to see if the move was successful or not. Otherwise you may just blunder around a bunch, without your code having any clue where you are in the maze.
i write readers and writers where the kernel have to syncronize between them and block writer who already read a massage
when i am in the queue waiting I get signal so I do the fallowing
while (i_Allready_Read(myf) == ALLREADY_READ || isExistWriter == false )
//while (!(i_Allready_Read(aliveProc,current->pid)))
{
int i, is_sig = 0;
printk(KERN_INFO "\n\n*****entered set in read ******\n\n" );
if (i_Allready_Read(myf) == ALLREADY_READ )
wait_event_interruptible (readWaitQ1, !i_Allready_Read(myf));
else
wait_event_interruptible (readWaitQ1, isExistWriter);
//printk(KERN_INFO "Read Wakeup %d\n",current->pid);
for (i = 0; i < _NSIG_WORDS && !is_sig; i++)
{
is_sig = current->pending.signal.sig[i] & ~current->blocked.sig[i];
}
if (is_sig)
{
handledClose(myf);
module_put (THIS_MODULE);
return -EINTR;
}
}
return 0;//success
}
inline void handledClose(struct file *myf)//v
{
/*
*if we close the writer other writer
*would be able to enter to permissiones
*/
if (myf == writerpid )
{
isExistWriter = DOESNT_EXIST;
//printk(KERN_INFO "procfs_close : this is pid that just closed %d \n", writerpid);
}
/*
*else its a reader so our numofreaders
*need to decremented
*/
else
{
removeFromArr(myf);
numOfReaders--;
}
}
and my close does nothing ...
what did i do wrong?
Where are you waking up the wait queue?
You should be calling wake_up(readWaitQ1); somewhere. Possible after you set isExistWriter to true.