Assume that I'm currently uploading data to a vertex buffer using glBufferSubData or I currently have it mapped via glMapBufferRange in Thread A on Context A. Then I call glBufferSubData or glMapBufferRange on the same buffer in Thread B on Context B via context sharing.
Will the second call block until the upload in Thread A has finished / the buffer has been unmapped? Or do I have to track myself which thread/context currently maps or works on which buffer?
OpenGL ES does not provide any automatic synchronization for bulk data access across contexts (it has weak synchronization for state within a share group, but not data) so any synchronization must be handled externally.
Related
I'm having a bit of trouble understanding the "lifespan" of context.
I'm receiving RPC requests, and storing the request ID using context.WithValue so that it can be used for logging. There is no deadline or timeout for context in my service.
One of the methods called by RPC pulls some data from the database, calls a goroutine to do some processing (not relevant to the client), sends a response, and the method returns. The goroutine could run for ~15 seconds after the method returns.
I log the request ID in the goroutine a few times, and so far it works fine, but is there a situation where context could be garbage collected, and unavailable when I try to use it? Or will go know to keep context around until my goroutine has completed?
This leads me to another question - I'm not using context.WithCancel, does this mean it will remain in memory indefinitely? I'd imagine after a while this could cause some performance issues.
A context is a glorified done channel. Depending on where the context comes from, determines how it should be reclaimed:
If you create a context (context.WithCancel etc.) ensure you reclaim it when the sub-task it represents completes (defer cancelfn() etc. ensure this happens on API return).
If you are using a context from an external source (e.g.a gRPC request) - it's the responsibility of the API framework to close out the context.
go will reclaim any memory during GC as long as there are no active references (function closures etc.)
If the stateful stream application is started with 6 threads on a single node, would the above exception occur?
Is there any process that needs to be followed, if a stateful stream application started on node 1 consuming a particular topic, is made to run on different node?
If the stateful stream application is started on 2 nodes and if the above exception occurs, would the stream application terminate immediately?
If yes, where can this exception be caught in a try-catch block?
If the exception can be caught, and if we add sleep for 10 mins, would the store automatically gets to valid state?
If not, is there a method that can be used to check the store state and wait until it becomes valid?
Follow-up:
If the stateful stream application is started with 6 threads on a single node, would the above exception occur?
It can
Essentially I was wondering if we keep the entire topic consumption on a single node, would it avoid re-building the store from an internal topic if a re-balancing occurs, due to one of the thread going down/terminates?
store is not ready yet: you can wait until the store is ready -- best to register a restore callback (check the docs for details) to get informed when restore is finished and you can retry to query the store.
Sorry, just to be clear on the above, is it StateRestoreCallback OR StateRestoreListener? I assume it is the later one. Also, is it required to override StateRestoreCallback and include logic to restore the store?
InvalidStateStoreException can have different causes, thus, it's hard to answer your question without more context.
If the stateful stream application is started with 6 threads on a single node, would the above exception occur?
It can.
Is there any process that needs to be followed, if a stateful stream application started on node 1 consuming a particular topic, is made to run on different node?
No.
If the stateful stream application is started on 2 nodes and if the above exception occurs, would the stream application terminate immediately?
Depends where the exception it thrown:
Either, the corresponding StreamThread would die, but the application would not terminate automatically. You should register an uncaught exception handler on the KafkaStreams instance and react to an dying thread with custom code (like, terminating the application).
If it is thrown from KafkaStreams using interactive queries, StreamThread would not be affected.
Where can this exception be caught in a try-catch block?
Usually yes, especially if you refer to interactive queries feature.
if we add sleep for 10 mins, would the store automatically gets to valid state?
If you refer to interactive queries feature, sleeping is not a good strategy. There are multiple causes for the exception and you need to react accordingly:
store is not local but on different node: you can figure this out by check the store metadata.
store is not ready yet: you can wait until the store is ready -- best to register a restore listener (check the docs for details) to get informed when restore is finished and you can retry to query the store.
Update
Essentially I was wondering if we keep the entire topic consumption on a single node, would it avoid re-building the store from an internal topic if a re-balancing occurs, due to one of the thread going down/terminates?
Yes (for non-EOS case). Other threads would detect the local store and reuse it.
StateRestoreCallback OR StateRestoreListener
Yes, it's StateRestoreListener. You would implement StateRestoreCallback only if you write a custom state store.
I'm making a little server for a project, I have a log handler class which contains a log implemented as a map and some methods to act on it (add entry, flush to disk, commit etc..)
This object is instantiated in the server Class, and I'm passing the address to the session so each session can add entries to it.
The sessions are async, the log writes will happen in the async_read callback. I'm wondering if this will be an issue and if i need to use locks?
The map format is map<transactionId map<sequenceNum, pair<head, body>>, each session will access a different transactionId, so there should be no clashes as far as i can figure. Also hypothetically, if they were all writing to the same place in memory -- something large enough that the operation would not be atomic; would i need locks? As far as I understand each async method dispatches a thread to handle the operation, which would make me assume yes. At the same time I read that one of the great uses of async functions is the fact that synchronization primitives are not needed. So I'm a bit confused.
First time using ASIO or any type of asynchronous functions altogether, and i'm not a very experienced coder. I hope the question makes sense! The code seems to run fine so far, but i'm curios if it's correct.
Thank you!
Asynchronous handlers will only be invoked in application threads processing the io_service event loop via run(), run_one(), poll(), or poll_one(). The documentation states:
Asynchronous completion handlers will only be called from threads that are currently calling io_service::run().
Hence, for a non-thread safe shared resource:
If the application code only has one thread, then there is neither concurrency nor race conditions. Thus, no additional form of synchronization is required. Boost.Asio refers to this as an implicit strand.
If the application code has multiple threads processing the event-loop and the shared resource is only accessed within handlers, then synchronization needs to occur, as multiple threads may attempt to concurrently access the shared resource. To resolve this, one can either:
Protect the calls to the shared resource via a synchronization primitive, such as a mutex. This question covers using mutexes within handlers.
Use the same strand to wrap() the ReadHandlers. A strand will prevent concurrent invocation of handlers dispatched through it. For more details on the usage of strands, particularly for composed operations, such as async_read(), consider reading this answer.
Rather than posting the entire ReadHandler into the strand, one could limit interacting with the shared resource to a specific set of functions, and these functions are posted as CompletionHandlers to the same strand. This subtle difference between this and the previous solution is the granularity of synchronization.
If the application code has multiple threads and the shared resource is accessed from threads processing the event loop and from threads not processing the event loop, then synchronization primitives, such as a mutex, needs to be used.
Also, even if a shared resource is small enough that writes and reads are always atomic, one should prefer using explicit and proper synchronization. For example, although the write and read may be atomic, without proper memory fencing to guarantee memory visibility, a thread may not observe a chance in memory even though the actual memory has chanced. Boost.Asio's will perform the proper memory barriers to guarantee visibility. For more details, on Boost.Asio and memory barriers, consider reading this answer.
This question looks similar to Should my Scala actors' properties be marked #volatile? but not sure that answer will be the same.
As example, in case when the fork-join dispatcher was configured and actor's state wasn't marked by #volatile, is it guarantied that state of the actor will be propagated through the cache hierarchy from one core (or processor) to another if fork/join worker threads run on different cores (or processors)?
P.S. Is it right that after JSR133 only one write/read operation to/from any volatile variable required to flush cache to main memory and see all preceding non-volatile writings from this thread on other thread that running on other core (or processor)? If yes then it can be answer, because scanning of work queue do some readings and writing from/to volatile variables of FJ task.
No, you shouldn't put volatile on your actor fields. Why?
if an actor makes changes to its internal state while processing a
message, and accesses that state while processing another message
moments later. It is important to realize that with the actor model
you don’t get any guarantee that the same thread will be executing the
same actor for different messages.
It's all here: http://doc.akka.io/docs/akka/2.0/general/jmm.html
Regarding your PS, you need to read/write to the same volatile field to get the happens-before guarantee. Read up on "volatile piggybacking"
I'm working on a DirectShow filter which takes input samples and turns them into modified output samples but where there isn't a one-to-one correspondence between input and output samples so CTransformFilter doesn't seem to be the way to go.
The best way of writing this appears to be writing a filter using CBaseFilter, CBaseInputPin and CBaseOutputPin where samples are received on an input pin and processed by a worker thread which creates and delivers new samples from the output pin. The worker thread copies the input sample data before starting work so that my filter doesn't have to maintain a reference to the input samples outside the input CBaseInputPin::Receive call.
What's the best practice for maintaining thread safety and avoiding deadlocks in this case? Should the input and output pin share the same streaming lock or should they have a streaming lock each for their streaming operations? Do buffer allocation, sample delivery and other output pin operations need to hold the streaming lock(s) and/or the filter lock? Any sample code that does something similar? Any other gotchas to watch out for in this situation?
The DirectShow bases classes contain scary comments for CBaseOutputPin::Deliver and CBaseOutputPin::GetDeliveryBuffer which I don't fully understand (pasted below).
/* Deliver a filled-in sample to the connected input pin. NOTE the object must
have locked itself before calling us otherwise we may get halfway through
executing this method only to find the filter graph has got in and
disconnected us from the input pin. If the filter has no worker threads
then the lock is best applied on Receive(), otherwise it should be done
when the worker thread is ready to deliver. There is a wee snag to worker
threads that this shows up. The worker thread must lock the object when
it is ready to deliver a sample, but it may have to wait until a state
change has completed, but that may never complete because the state change
is waiting for the worker thread to complete. The way to handle this is for
the state change code to grab the critical section, then set an abort event
for the worker thread, then release the critical section and wait for the
worker thread to see the event we set and then signal that it has finished
(with another event). At which point the state change code can complete */
You have a few samples in Windows SDK in \Samples\multimedia\directshow\filters and earlier version of SDK had even more. This would be perhaps the best sample code you can check for locking practices.
The filter and pins normally use shared critical sections to ensure thread safety. For instance, CTransformFilter::m_csFilter protects the state data of the filter and not only section, but pins also use the section. The additional section is also used to serialize streaming requests (pushing through samples, sending EOS notifications).
Your filter might be using state critical section, or you can alternatively use additional synchronization object (section, reader-writer lock, or a mutex) in order to avoid deadlocks with critical section being possibly locked by base classes.
Regular suggestions apply: to avoid deadlocks you should make sure your locking order is designed in a way that if section A can be locked on a thread which already has section B locked, you should only lock B [on other threads] when without existing lock on A, so that no deadlock is possible.
So typically, you have two scenarios which most use cases fall into:
you are reusing state critical section of the filter
you are using a separate critical section which protects your private data, and you don't keep the section locked while doing calls on base class methods and methods of other object such as peer filters