Aging values in a queue: Best use of Windows timers? - windows

I have an std::set that contains unique values. I have an std::queue that holds the same values
in order to age the values in std::set.
I'd like to use a timer to determine when to pop a value from the queue and then erase the value from the set.
The timer is created/started every time data is added to an empty set/queue.
If data is added to a non-empty set/queue, no change is made to the timer.
The timer would fire every X milliseconds to execute a function.
The function would pop a value from the queue then erase that value from the set.
If the set/queue is now empty the timer would stop.
If the set/queue is not empty, no change is made to the timer.
This program runs in Windows 10.
Does this way make sense? Is there a better/more efficient/simpler way to age the data?
I've read the docs on Using Timer Queues so I see how the queue and the timers are created and destroyed. What I don't see is a recommendation for starting/stopping timers.
Should I be creating a new TimerQueueTimer to wait for X milliseconds once, run the func and then create a new TimerQueueTimer if the set/queue is not empty?
Should I instead create a single TimerQueueTimer to run periodically X milliseconds but delete it once the set/queue is empty?
Is there a 3rd technique I should use instead?
Here's my example code.
using unsignedIntSet = std::set<std::uint32_t>;
using unsignedIntQ = std::queue<std::uint32_t>;
unsignedIntQ agingQ;
unsignedIntSet agingSet;
HANDLE gDoneEvent = NULL;
HANDLE hTimer = NULL;
HANDLE hTimerQueue = NULL;
VOID CALLBACK ageTimer(PVOID lpParam, BOOLEAN TimerOrWaitFired)
{
if (!agingQ.empty())
{
auto c = agingQ.front();
agingSet.erase(c);
agingQ.pop();
if (!agingQ.empty())
{
// rerun CreateTimerQueueTimer() here?
}
}
SetEvent(gDoneEvent);
}
int createTimerForAgingQ()
{
// create timer if it doesn't already exist
if (gDoneEvent == NULL)
{
gDoneEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
if (gDoneEvent == NULL)
{
std::cerr << "CreateEvent() error: " << WSAGetLastError() << std::endl;
return -1;
}
hTimerQueue = CreateTimerQueue();
if (hTimerQueue == NULL)
{
std::cerr << "CreateTimerQueue() error: " << WSAGetLastError() << std::endl;
return -1;
}
if (!CreateTimerQueueTimer(&hTimer, hTimerQueue, (WAITORTIMERCALLBACK)ageTimer, NULL, 500, 0, WT_EXECUTEONLYONCE))
{
std::cerr << "CreateTimerQueueTimer() error: " << WSAGetLastError() << std::endl;
return -1;
}
}
}
void addUnique(unsigned char* buffer, int bufferLen)
{
// hash value
auto h = hash(buffer, bufferLen);
// test insert into set
auto setResult = agingSet.emplace(h);
if (setResult.second)
{
// enqueue into historyQ
agingQ.emplace(h);
if (!gDoneEvent) createTimerForAgingQ();
}
}

Research shows that the CreateTimerQueue/CreateTimerQueueTimer may not be the way to go.
Use of ThreadpoolTimer

Related

How to find out which process sent the request?

I am writing a process access filter through ObRegisterCallbacks.
OB_PREOP_CALLBACK_STATUS PreCallback(PVOID RegistrationContext, POB_PRE_OPERATION_INFORMATION pOperationInformation)
{
UNREFERENCED_PARAMETER(RegistrationContext);
UNREFERENCED_PARAMETER(pOperationInformation);
PEPROCESS OpenedProcess = (PEPROCESS)pOperationInformation->Object,
CurrentProcess = PsGetCurrentProcess();
char szProcName[16] = { 0, };
strcpy_s(szProcName, 16, ((DWORD64)pOperationInformation->Object + iOffset.ImageFileName_off));
UINT64* id = (UINT64*)((DWORD64)pOperationInformation->Object + iOffset.UniqueProcessid_off);
//PEPROCESS ProtectedProcess;
//PsLookupProcessByProcessId(*id, &ProtectedProcess); // Getting the PEPROCESS using the PID
if (!_strnicmp(szProcName, "notepad.exe", 16))
{
if ((pOperationInformation->Operation == OB_OPERATION_HANDLE_CREATE))
{
if ((pOperationInformation->Parameters->CreateHandleInformation.OriginalDesiredAccess & PROCESS_TERMINATE) == PROCESS_TERMINATE)
{
pOperationInformation->Parameters->CreateHandleInformation.DesiredAccess &= ~PROCESS_TERMINATE;
}
if ((pOperationInformation->Parameters->CreateHandleInformation.OriginalDesiredAccess & PROCESS_VM_READ) == PROCESS_VM_READ)
{
pOperationInformation->Parameters->CreateHandleInformation.DesiredAccess &= ~PROCESS_VM_READ;
}
if ((pOperationInformation->Parameters->CreateHandleInformation.OriginalDesiredAccess & PROCESS_VM_OPERATION) == PROCESS_VM_OPERATION)
{
pOperationInformation->Parameters->CreateHandleInformation.DesiredAccess &= ~PROCESS_VM_OPERATION;
}
if ((pOperationInformation->Parameters->CreateHandleInformation.OriginalDesiredAccess & PROCESS_VM_WRITE) == PROCESS_VM_WRITE)
{
pOperationInformation->Parameters->CreateHandleInformation.DesiredAccess &= ~PROCESS_VM_WRITE;
}
}
}
return OB_PREOP_SUCCESS;
}
If you start the driver after starting the program, everything is fine.
If the program is started after the driver starts, the program hangs.
I assume that the program cannot get a HANDLE on itself.
How to find out who sent the request?
How to find out the PID of the process that sent the request?
In this callback, the current context is the operation requester.
Just call PsGetCurrentProcess and PsGetCurrentProcessId to get the PEPROCESS and Id of the current context.

Why does only some of my objects get created using std::async

I have a loop that pushes back calls of std::async that are used to create objects in the pointed function and emplace them back to another vector. All the calls are pushed to the futures function and the results are ready when i used the VS debugger. However of the 507 calls, only 30 objects are actually created and i cant seem to pin point why.I have tried setting the launch policy to both async and defered but get the same result.
void load_sec_p(vector<Security>* secs, map<string, map<string, vector<daySec>>> *psa_timeline,security sec) {
Security tmp = Security(psa_timeline, &sec.tsymb, &sec.gicsInd);
std::lock_guard<std::mutex> lock(s_SecsMutex);
secs->emplace_back(tmp);
}
Above is the function being executed in the async call
below is the loop that pushes back the futures
for (auto& sec : security_list) {
m_SecFutures.emplace_back(std::async(load_sec_p,&async_secs, &psa_timeline, sec));
}
The following pictures show the watch of both variables after the above loop is completed and the entire future vectors is checked for completion.
I have tried creating the objects by just using a regular for loop and appending them synchronously but it simply just takes too long(2 hours and 11 minutes long). If anyone has any advice on alternatives or how to fix my vector problem it would be greatly appreciated.
The code that checks if all the futures is shown below:
bool done = false;
cout << "Waiting...";
do {
done = futures_ready(m_SecFutures);
} while (!done);
The function is
template<class T>
bool futures_ready(std::vector<std::future<T>>& futures) {
std::chrono::milliseconds span(5);
bool finished = false;
int pends = 0;
while (!finished) {
//allowing thread to sleep so futures can process a bit more and also
//so thread doesnt reach max cpu usage
std::this_thread::sleep_for(std::chrono::milliseconds(100));
for (auto& x : futures) {
if (x.wait_for(span) == std::future_status::timeout) {
++pends;
}
}
if (pends == 0) {
finished = true;
}
else {
pends = 0;
}
}
return finished;
}

rocketmq throw exception "[TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while"

version:rocketmq-all-4.1.0-incubating
We send msg 1000 QPS,sync send, but throw exception:-
[TIMEOUT_CLEAN_QUEUE] broker busy, start flow control for a while
There is the related code:
while (true) {
try {
if (!this.brokerController.getSendThreadPoolQueue().isEmpty()) {
final Runnable runnable = this.brokerController.getSendThreadPoolQueue().peek();
if (null == runnable) {
break;
}
final RequestTask rt = castRunnable(runnable);
if (rt == null || rt.isStopRun()) {
break;
}
final long behind = System.currentTimeMillis() - rt.getCreateTimestamp();
if (behind >= this.brokerController.getBrokerConfig().getWaitTimeMillsInSendQueue()) {
if (this.brokerController.getSendThreadPoolQueue().remove(runnable)) {
rt.setStopRun(true);
rt.returnResponse(RemotingSysResponseCode.SYSTEM_BUSY, String.format("[TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while, period in queue: %sms, size of queue: %d", behind, this.brokerController.getSendThreadPoolQueue().size()));
}
} else {
break;
}
} else {
break;
}
} catch (Throwable ignored) {
}
}
}
I find broker the default value of sendMessageThreadPoolNums is 1,
/**
* thread numbers for send message thread pool, since spin lock will be used by default since 4.0.x, the default value is 1.
*/
private int sendMessageThreadPoolNums = 1; //16 + Runtime.getRuntime().availableProcessors() * 4;
private int pullMessageThreadPoolNums = 16 + Runtime.getRuntime().availableProcessors() * 2;
but the previous version isn't 1, and if I configure sendMessageThreadPoolNums = 100, can resolve this question ? It will lead to what is different with default value.
thanks
SHORT ANSWER:
you have two choices:
set sendMessageThreadPoolNums to a small number, say 1, which is the default value after version 4.1.x. And, remain the default value of useReentrantLockWhenPutMessage=false, which is introduced after 4.1.x
sendMessageThreadPoolNums=1
useReentrantLockWhenPutMessage=false
If you need to use a large numbers of threads to process sending message, you'd better use useReentrantLockWhenPutMessage=true
sendMessageThreadPoolNums=128//large thread numbers
useReentrantLockWhenPutMessage=true // indicating that do NOT use spin lock but use ReentrantLock when putting message

alBufferData() sets AL_INVALID_OPERATION when using buffer ID obtained from alSourceUnqueueBuffers()

I am trying to stream audio data from disk using OpenAL's buffer queueing mechanism. I load and enqueue 4 buffers, start the source playing, and check in a regular intervals to refresh the queue. Everything looks like it's going splendidly, up until the first time I try to load data into a recycled buffer I got from alSourceUnqueueBuffers(). In this situation, alBufferData() always sets AL_INVALID_OPERATION, which according to the official v1.1 spec, it doesn't seem like it should be able to do.
I have searched extensively on Google and StackOverflow, and can't seem to find any reason why this would happen. The closest thing I found was someone with a possibly-related issue in an archived forum post, but details are few and responses are null. There was also this SO question with slightly different circumstances, but the only answer's suggestion does not help.
Possibly helpful: I know my context and device are configured correctly, because loading small wav files completely into a single buffer and playing them works fine. Through experimentation, I've also found that queueing 2 buffers, starting the source playing, and immediately loading and enqueueing the other two buffers throws no errors; it's only when I've unqueued a processed buffer that I run into trouble.
The relevant code:
static constexpr int MAX_BUFFER_COUNT = 4;
#define alCall(funcCall) {funcCall; SoundyOutport::CheckError(__FILE__, __LINE__, #funcCall) ? abort() : ((void)0); }
bool SoundyOutport::CheckError(const string &pFile, int pLine, const string &pfunc)
{
ALenum tErrCode = alGetError();
if(tErrCode != 0)
{
auto tMsg = alGetString(tErrCode);
Log::e(ro::TAG) << tMsg << " at " << pFile << "(" << pLine << "):\n"
<< "\tAL call " << pfunc << " failed." << end;
return true;
}
return false;
}
void SoundyOutport::EnqueueBuffer(const float* pData, int pFrames)
{
static int called = 0;
++called;
ALint tState;
alCall(alGetSourcei(mSourceId, AL_SOURCE_TYPE, &tState));
if(tState == AL_STATIC)
{
Stop();
// alCall(alSourcei(mSourceId, AL_BUFFER, NULL));
}
ALuint tBufId = AL_NONE;
int tQueuedBuffers = QueuedUpBuffers();
int tReady = ProcessedBuffers();
if(tQueuedBuffers < MAX_BUFFER_COUNT)
{
tBufId = mBufferIds[tQueuedBuffers];
}
else if(tReady > 0)
{
// the fifth time through, this code gets hit
alCall(alSourceUnqueueBuffers(mSourceId, 1, &tBufId));
// debug code: make sure these values go down by one
tQueuedBuffers = QueuedUpBuffers();
tReady = ProcessedBuffers();
}
else
{
return; // no update needed yet.
}
void* tConverted = convert(pData, pFrames);
// the fifth time through, we get AL_INVALID_OPERATION, and call abort()
alCall(alBufferData(tBufId, mFormat, tConverted, pFrames * mBitdepth/8, mSampleRate));
alCall(alSourceQueueBuffers(mSourceId, 1, &mBufferId));
if(mBitdepth == BITDEPTH_8)
{
delete (uint8_t*)tConverted;
}
else // if(mBitdepth == BITDEPTH_16)
{
delete (uint16_t*)tConverted;
}
}
void SoundyOutport::PlayBufferedStream()
{
if(!StreamingMode() || !QueuedUpBuffers())
{
Log::w(ro::TAG) << "Attempted to play an unbuffered stream" << end;
return;
}
alCall(alSourcei(mSourceId, AL_LOOPING, AL_FALSE)); // never loop streams
alCall(alSourcePlay(mSourceId));
}
int SoundyOutport::QueuedUpBuffers()
{
int tCount = 0;
alCall(alGetSourcei(mSourceId, AL_BUFFERS_QUEUED, &tCount));
return tCount;
}
int SoundyOutport::ProcessedBuffers()
{
int tCount = 0;
alCall(alGetSourcei(mSourceId, AL_BUFFERS_PROCESSED, &tCount));
return tCount;
}
void SoundyOutport::Stop()
{
if(Playing())
{
alCall(alSourceStop(mSourceId));
}
int tBuffers;
alCall(alGetSourcei(mSourceId, AL_BUFFERS_QUEUED, &tBuffers));
if(tBuffers)
{
ALuint tDummy[tBuffers];
alCall(alSourceUnqueueBuffers(mSourceId, tBuffers, tDummy));
}
alCall(alSourcei(mSourceId, AL_BUFFER, AL_NONE));
}
bool SoundyOutport::Playing()
{
ALint tPlaying;
alCall(alGetSourcei(mSourceId, AL_SOURCE_STATE, &tPlaying));
return tPlaying == AL_PLAYING;
}
bool SoundyOutport::StreamingMode()
{
ALint tState;
alCall(alGetSourcei(mSourceId, AL_SOURCE_TYPE, &tState));
return tState == AL_STREAMING;
}
bool SoundyOutport::StaticMode()
{
ALint tState;
alCall(alGetSourcei(mSourceId, AL_SOURCE_TYPE, &tState));
return tState == AL_STATIC;
}
And here's an annotated screen cap of what I see in my debugger when I hit the error:
I've tried a bunch of little tweaks and variations, and the result is always the same. I've wasted too many days trying to fix this. Please help :)
This error occurs when you trying to fill buffer with data, when the buffer is still queued to the source.
Also this code is wrong.
if(tQueuedBuffers < MAX_BUFFER_COUNT)
{
tBufId = mBufferIds[tQueuedBuffers];
}
else if(tReady > 0)
{
// the fifth time through, this code gets hit
alCall(alSourceUnqueueBuffers(mSourceId, 1, &tBufId));
// debug code: make sure these values go down by one
tQueuedBuffers = QueuedUpBuffers();
tReady = ProcessedBuffers();
}
else
{
return; // no update needed yet.
}
You can fill buffer with data only if it unqueued from source. But your first if block gets tBufId that queued to the source. Rewrite code like so
if(tReady > 0)
{
// the fifth time through, this code gets hit
alCall(alSourceUnqueueBuffers(mSourceId, 1, &tBufId));
// debug code: make sure these values go down by one
tQueuedBuffers = QueuedUpBuffers();
tReady = ProcessedBuffers();
}
else
{
return; // no update needed yet.
}

Inputs in SDL (on key pressed)

I would like to know how can I detect the press of a key or release of a key in a while loop in SDL. Now, I know you can get the events with SDL like OnKeyPressed, OnKeyReleased, OnKeyHit, etc, but I want to know how to build functions like 'KeyPressed' that returns a boolean, instead of being an event. Example:
while not KeyHit( KEY_ESC )
{
//Code here
}
I know you have already selected an answer.. but here is some actual code of how I typically do it with one array. :)
first define this somewhere.
bool KEYS[322]; // 322 is the number of SDLK_DOWN events
for(int i = 0; i < 322; i++) { // init them all to false
KEYS[i] = false;
}
SDL_EnableKeyRepeat(0,0); // you can configure this how you want, but it makes it nice for when you want to register a key continuously being held down
Then later, create a keyboard() function which will register keyboard input
void keyboard() {
// message processing loop
SDL_Event event;
while (SDL_PollEvent(&event)) {
// check for messages
switch (event.type) {
// exit if the window is closed
case SDL_QUIT:
game_state = 0; // set game state to done,(do what you want here)
break;
// check for keypresses
case SDL_KEYDOWN:
KEYS[event.key.keysym.sym] = true;
break;
case SDL_KEYUP:
KEYS[event.key.keysym.sym] = false;
break;
default:
break;
}
} // end of message processing
}
Then when you actually want to use the keyboard input i.e. a handleInput() function, it may look something like this:
void handleInput() {
if(KEYS[SDLK_LEFT]) { // move left
if(player->x - player->speed >= 0) {
player->x -= player->speed;
}
}
if(KEYS[SDLK_RIGHT]) { // move right
if(player->x + player->speed <= screen->w) {
player->x += player->speed;
}
}
if(KEYS[SDLK_UP]) { // move up
if(player->y - player->speed >= 0) {
player->y -= player->speed;
}
}
if(KEYS[SDLK_DOWN]) { // move down
if(player->y + player->speed <= screen->h) {
player->y += player->speed;
}
}
if(KEYS[SDLK_s]) { // shoot
if(SDL_GetTicks() - player->lastShot > player->shotDelay) {
shootbeam(player->beam);
}
}
if(KEYS[SDLK_q]) {
if(player->beam == PLAYER_BEAM_CHARGE) {
player->beam = PLAYER_BEAM_NORMAL;
} else {
player->beam = PLAYER_BEAM_CHARGE;
}
}
if(KEYS[SDLK_r]) {
reset();
}
if(KEYS[SDLK_ESCAPE]) {
gamestate = 0;
}
}
And of course you can easily do what you're wanting to do
while(KEYS[SDLK_s]) {
// do something
keyboard(); // don't forget to redetect which keys are being pressed!
}
**Updated version on my website: **
For the sake of not posting a lot of source code, you can view a complete SDL Keyboard class in C++ that supports
Single Key Input
Simultaneous Key Combos (Keys all pressed in any order)
Sequential Key Combonations (Keys all pressed in specific order)
http://kennycason.com/posts/2009-09-20-sdl-simple-space-shooter-game-demo-part-i.html (if you have any problems, let me know)
There is an SDL function for this: SDL_GetKeyboardState
Example to check whether left or right CTRL key is pressed:
const Uint8* state = SDL_GetKeyboardState(nullptr);
if (state[SDL_SCANCODE_LCTRL] || state[SDL_SCANCODE_RCTRL]) {
std::cerr << "ctrl pressed" << std::endl;
}
I had this problem in LuaJIT with FFI, this is how I solved it:
Global:
KEYS = {}
Event code:
ev = ffi.new("SDL_Event[1]")
function event()
while sdl.SDL_PollEvent(ev) ~= 0 do
local e = ev[0]
local etype = e.type
if etype == sdl.SDL_QUIT then
return false -- quit
-- os.exit() -- prevents interactive mode
elseif etype == sdl.SDL_KEYDOWN then
if e.key.keysym.sym == sdl.SDLK_ESCAPE then
return false -- quit
-- os.exit()
end
print("Pressed: ", e.key.keysym.scancode, "\n")
KEYS[tonumber(e.key.keysym.sym)] = true
-- print("Pressed: ", (e.key.keysym.sym == sdl.SDLK_w), "\n");
elseif etype == sdl.SDL_KEYUP then
KEYS[tonumber(e.key.keysym.sym)] = false
elseif etype == sdl.SDL_VIDEORESIZE then
-- print("video resize W:".. e.resize.w .. " H:" .. e.resize.h)
width = e.resize.w
height = e.resize.h
onResize()
end
end
return true -- everything ok
end
Update function:
if KEYS[sdl.SDLK_w] == true then
rot = rot + 1
end
Most time i wasted on this:
KEYS[tonumber(e.key.keysym.sym)] = false
Because FFI is returning a CData object, which was used as the array-key, but it needs the integer.
You should have 2 tables of booleans for keys. One table, in which you set keys true or false based on the SDL keydown/keyup events, and another one, that you initialize with false. When checking keyPressed, you just compare the second table key with the first table key, and if different, if second table key is false, then it was pressed, else it was released. After that, you do secondTable[key] := not secondTable[key]. Works!

Resources