I am writing an application that maps a file into memory to make some information resilient to failures (crash, power outage, etc). I know that the idea is to flush as infrequently as allowable, but to Do Things Right, and considering the goal, it seems to me that I should essentially flush to disk whenever the data has changed.
All the mapped data fits into a single page. I have a burst usage pattern (nothing happens for a looong time, then all of a sudden you'd modify the information ~20 times in a row). For this reason, I'm hesitant about FlushViewOfFile, since it seems to be synchronous. Flushing at every hit on a burst would seem to be inefficient.
Is there not a way I can tell Windows to flush pages the next time it has an idle cycle, and without having me wait until it does it?
I do not believe that there is a function in Windows for that. FlushViewOfFile is what you have to work with. You're going to have to think of a 'scheduler' for your program that matches your use-case/profile. Something like starting a short timer after each hit, which resets if there is another hit and if it expires flushes the page, and one long timer which if it expires flushes the page despite still being in a burst would probably work nicely for you. In any case, you'll need to profile what the usage will be and have the program act accordingly.
Related
I'm doing this as a personal project, I want to make a visualizer for this data. but the first step is getting the data.
My current plan is to
make my program debug the target process step through it
each step record the EIP from every thread's context within the target process
construct the memory address the instruction uses from the context and store it.
Is there an easier or built in way to do this?
Have a look at Intel PIN for dynamic binary instrumentation / running a hook for every load / store instruction. intel-pin
Instead of actually single-stepping in a debugger (extremely slow), it does binary-to-binary JIT to add calls to your hooks.
https://software.intel.com/sites/landingpage/pintool/docs/81205/Pin/html/index.html
Honestly the best way to do this is probably instrumentation like Peter suggested, depending on your goals. Have you ever ran a script that stepped through code in a debugger? Even automated it's incredibly slow. The only other alternative I see is page faults, which would also be incredibly slow but should still be faster than single step. Basically you make every page not in the currently executing section inaccessible. Any RW access outside of executing code will trigger an exception where you can log details and handle it. Of course this has a lot of flaws -- you can't detect RW in the current page, it's still going to be slow, it can get complicated such as handling page execution transfers, multiple threads, etc. The final possible solution I have would be to have a timer interrupt that checks RW access for each page. This would be incredibly fast and, although it would provide no specific addresses, it would give you an aggregate of pages written to and read from. I'm actually not entirely sure off the top of my head if Windows exposes that information already and I'm also not sure if there's a reliable way to guarantee your timers would get hit before the kernel clears those bits.
We have a single page application, which runs well at the beginning, but slows down sharply as time goes. I am trying to investigate the root cause.
I use Chrome DevTool to record the timeline for initial page loading and a typical user operation. The JS Heap shows that the memory usage is ok: goes up and down periodically (due to Garbage Collection by browser, maybe).
However, when I check the Chrome Task Manager, I found that my page uses 60MB memory initially. But after 1 hour (and some user operations), the memory goes to 160MB. While the JavaScript Memory seems stable. Later I observed that the memory usage never goes down.
I guess maybe there is some memory leak in our JavaScript code? But the JS Heap seems ok. Does Chrome hold those memory and may release in future (when, say, other process needs more memory)?
Here is the Timeline recorded when I am operating:
I googled but cannot find explanations about this. Could anybody help? Thanks.
It is because of an interval that is not cleared. It keeps calling a function too frequently.
What does REALTIME_PRIORITY_CLASS (with THREAD_PRIORITY_TIME_CRITICAL) actually do?
Does it:
Prevent interrupts from firing
Prevent context switching from happening
on the processor (unless the thread sleeps)?
If it does prevents the above from happening:
How come when I run a program on a processor with this flag, I still get inconsistent timing results? Shouldn't the program take the same amount of time every time, if there's nothing interrupting it?
If it does NOT prevent the above from happening:
Why does my system (mouse, keyboard, etc.) lock up if I use it incorrectly? Shouldn't drivers still get some processor time?
It basically tells the system scheduler to only a lot time to your thread till it gives it up(via Sleep or SwitchToThread) or dies. As for timing not being the same, the OS still runs inbetween each run, this can change ram and caching etc. Secondly, most timing is inaccurate, so it will fluctuate(especially system quanta based timing like GetTickCount). The OS many also have thing things going on, like power saving/dynamic freq adjustment, so you best check would be to use RDTSC, though even with that you might notice other stuff running(especially if you can run more than one physical thread).
I put a stopwatch on it. The first time the app loads (no settings file exists) it takes about 190ms to fail to load four settings. The app runs, three bools and a short string are written as settings, and the next time the app loads, it takes 400ms to read the first setting from the IsolatedStorageSettings.ApplicationSettings collection and about 1ms to get the remainder.
Is there anything I can do to ameliorate this load time?
Ues a better Serialization method ;)
XMLSerialization is okay for more complex graphs, but for simple settings, binary serialization would be much better. Also, when you say fail to load, I assume you're doing a check to see if the files exist? If not, I think there may be exceptions being thrown internally which would slow down execution as well.
I have 15 BackgroundWorers that are running all the time, each one of them works for about half a second (making web request) and none of them is ever stopped.
I've noticed that my program takes about 80% of my computer's processing resources and about 15mb of memory (core 2 duo, 4gb ddr2 memory).
It it normal? web requests are not heavy duty, it just sends and awaits server response, and yes, running 15 of them is really not a pro-performance act (speed was needed) but i didn't think that it would be so intense.
I am new to programming, and i hardly ever (just as any new programmer, I assume) care about performance, but this time it is ridiculous, 80% of processing resources usage for a windows forms application with two listboxes and backgroundworkers making web requests isn't relly what expected.
info:
I use exception handling as part of my routine, which i've once read that isn't really good for performance
I have 15 background workers
My code assures none of them is ever idle
List item
windows forms, visual studio, c#.
------[edit - questions in answers]------
What exactly do you mean by "My code assures none of them is ever idle"?
The program remains waiting
while (bgw1.IsBusy || gbw2.IsBusy ... ... ...) { Application.DoWork();}
then when any of them is free, gets put back to work.
Could you give more details about the workload you're putting this under?
I make an HTTP web request object, open it and wait for the server request. It really has only a couple of lines and does no heavy processing, the half second is due to server awaiting.
In what way, and how many exceptions are being thrown?
When the page doesn't exist, there is a system.WebException, when it works it returns "OK", and about 99% of the pages i check don't exist, so i'd say about 300 exceptions per minute (putting it like this makes it sound creepy, i know, but it works)
If you're running in the debugger, then exceptions are much more expensive than they would be when not debugging
I'm not talking about running it in the debugger, I run the executable, the resulting EXE.
while (bgw1.IsBusy || gbw2.IsBusy ... ... ...) { Application.DoWork();}
What's Application.DoWork(); doing? If it's doing something quickly and returning, this loop alone will consume 100% CPU since it never stops doing something. You can put a sleep(.1) or so inside the loop, to only check the worker threads every so often instead of continuously.
This bit concerns me:
My code assures none of them is ever idle
What exactly do you mean by that?
If you're making thousands and thousands of web requests, and if those requests are returning very quickly, then that could eat some CPU.
Taking 15MB of memory isn't unexpected, but the CPU is the more worrying bit. Could you give more details about the workload you're putting this under? What do you mean by "each one of them workds for about half a second"?
What do you mean by "I use exception handling as part of my routine"? In what way, and how many exceptions are being thrown? If you're running in the debugger, then exceptions are much more expensive than they would be when not debugging - if you're throwing and catching a lot of exceptions, that could be responsible for it...
Run the program in the debugger, pause it ten times, and have a look at the stacktraces. Then you will know what is actually doing when it's busy.
From your text I read that you have a Core 2 Duo. Is that a 2 Threads or a 4 Threads?
If you have a 2 Threads you only should use 2 BackGroundworkers simultaneously.
If you have a 4 Threads then use 4 BGW's simultaneously. If you have more BGW's then use frequently the following statement:
System.Threading.Thread.Sleep(1)
Also use Applications.DOevents.
My general advice is: start simple and slowly make your application more complex.
Have a look at: Visual Basic 2010 Parallel Programming techniques.