In kfifo.h kfifo_get calls kfifo_is_empty which checks if __kfifo->in == __kfifo_out. Meanwhile kfifo_put does __kfifo->in++ after adding data. Since this is a lockless implementation of circular buffer with 1 reader and writer what prevents the writer from corrupting the data while kfifo_is_empty is reading the value of __kfifo->in?
There is no problem here.
In short, the reader may see in not equal to out only after data has been transferred into the kfifo. This is achived by executing barrier before incrementing in counter at the writer side.
Related
Is there an equivalent of PublishSubject from RxJava in Kotlin Coroutines library?
Channels cannot be a replacement for PublishSubject since they do not publish values to multiple collectors (each value can be collected by a single collector only). Even MutableSharedFlow that supports multiple collectors, still does not allow emitting values without waiting for collectors to finish processing previous values. How can we create a flow with functionality similar to the PublishSubject?
The following code will create a Flow equivalent to the PublishSubject:
fun <T> publishFlow(): MutableSharedFlow<T> {
return MutableSharedFlow(
replay = 0,
extraBufferCapacity = Int.MAX_VALUE
)
}
The main attributes of the PublishSubject are that it does not replay old values to new observers, and still allows to publish new values/events without waiting for the observers to handle them. So this functionality can be achieved with MutableSharedFlow by specifying replay = 0 for preventing new collectors from collecting old values, and extraBufferCapacity = Int.MAX_VALUE to allow publishing new values without waiting for busy collectors to finish collecting previous values.
One can add the following forceEmit function to be called instead of tryEmit, to ensure that the value is actually emitted:
fun <T> MutableSharedFlow<T>.forceEmit(value: T) {
val emitted = tryEmit(value)
check(emitted){ "Failed to emit into shared flow." }
}
Since we have a buffer with MAX_VALUE capacity, this forceEmit function should never fail if we use it with our publishFlow. If the flow will be replaced somehow with a different flow that does not support emitting without suspending, we will get an exception and will know to handle the case where the buffer is full and one cannot emit without suspending.
Notice that having a buffer of MAX_VALUE capacity may cause high consumption of memory if the collection of values by the collectors takes a long time, so it is more suitable for cases where the collectors perform a short synchronous operation (similarly to RxJava observers).
I'm just wondering if there is potential for corruption as a result of writing the same value to a global variable at the same time. My brain is telling me there is nothing wrong with this because its just a location in memory, but I figure I should probably double check this assumption.
I have concurrent processes writing to a global map var linksToVisit map[string]bool. The map is actually tracking what links on a website need to be further crawled.
However it can be the case that concurrent processes may have the same link on their respective pages and therefore each will mark that same link as true concurrently. There's nothing wrong with NOT using locks in this case right? NOTE: I never change the value back to false so either the key exists and it's value is true or it doesn't exist.
I.e.
var linksToVisit = map[string]bool{}
...
// somewhere later a goroutine finds a link and marks it as true
// it is never marked as false anywhere
linksToVisit[someLink] = true
What happens if concurrent processes write to a global variable the
same value?
The results of a data race are undefined.
Run the Go data race detector.
References:
Wikipedia: Race condition
Benign Data Races: What Could Possibly Go Wrong?
The Go Blog: Introducing the Go Race Detector
Go: Data Race Detector
Go 1.8 Release Notes
Concurrent Map Misuse
In Go 1.6, the runtime added lightweight, best-effort detection of
concurrent misuse of maps. This release improves that detector with
support for detecting programs that concurrently write to and iterate
over a map.
As always, if one goroutine is writing to a map, no other goroutine
should be reading (which includes iterating) or writing the map
concurrently. If the runtime detects this condition, it prints a
diagnosis and crashes the program. The best way to find out more about
the problem is to run the program under the race detector, which will
more reliably identify the race and give more detail.
For example,
package main
import "time"
var linksToVisit = map[string]bool{}
func main() {
someLink := "someLink"
go func() {
for {
linksToVisit[someLink] = true
}
}()
go func() {
for {
linksToVisit[someLink] = true
}
}()
time.Sleep(100 * time.Millisecond)
}
Output:
$ go run racer.go
fatal error: concurrent map writes
$
$ go run -race racer.go
==================
WARNING: DATA RACE
Write at 0x00c000078060 by goroutine 6:
runtime.mapassign_faststr()
/home/peter/go/src/runtime/map_faststr.go:190 +0x0
main.main.func2()
/home/peter/gopath/src/racer.go:16 +0x6a
Previous write at 0x00c000078060 by goroutine 5:
runtime.mapassign_faststr()
/home/peter/go/src/runtime/map_faststr.go:190 +0x0
main.main.func1()
/home/peter/gopath/src/racer.go:11 +0x6a
Goroutine 6 (running) created at:
main.main()
/home/peter/gopath/src/racer.go:14 +0x88
Goroutine 5 (running) created at:
main.main()
/home/peter/gopath/src/racer.go:9 +0x5b
==================
fatal error: concurrent map writes
$
It is better to use locks if you are changing the same value concurrently using multiple go routines. Since mutex and locks are used whenever it comes to secure the value from accessing when another function is changing the same just like writing to database table while accessing the same table.
For your question on using maps with different keys it is not preferable in Go as:
The typical use of maps did not require safe access from multiple
goroutines, and in those cases where it did, the map was probably part
of some larger data structure or computation that was already
synchronized. Therefore requiring that all map operations grab a mutex
would slow down most programs and add safety to few.
Map access is unsafe only when updates are occurring. As long as all
goroutines are only reading—looking up elements in the map, including
iterating through it using a for range loop—and not changing the map
by assigning to elements or doing deletions, it is safe for them to
access the map concurrently without synchronization.
So In case of update of maps it is not recommended. For more information Check FAQ on why maps operations not defined atomic.
Also it is noticed that if you realy wants to go for there should be a way to synchronize them.
Maps are not safe for concurrent use: it's not defined what happens
when you read and write to them simultaneously. If you need to read
from and write to a map from concurrently executing goroutines, the
accesses must be mediated by some kind of synchronization mechanism.
One common way to protect maps is with sync.RWMutex.
Concurrent map write is not ok, so you will most likely get a fatal error. So I think a lock should be used
As of Go 1.6, simultaneous map writes will cause a panic. Use a sync.Map to synchronize access.
See the map value assign implementation:
https://github.com/golang/go/blob/fe8a0d12b14108cbe2408b417afcaab722b0727c/src/runtime/hashmap.go#L519
I have a SourceTask which has a simple poll method (completes quite fast). I found that the offsets value got from the context.offsetStorageReader is mostly stale, which means not matching the offsets value returned in the previous poll() method.
At the same time, I can observe from logs that the offsets value only get updated to "fresh" when "commitOffsets successfully" occurred.
My question is: is this designed on purpose? Should I decrease the "OFFSET_COMMIT_INTERVAL_MS_CONFIG" value to assure the offset is committed faster than the SourceTask.poll() method executed?
The comments of org.apache.kafka.connect.runtime.OffsetStorageWriter class says "Offset data should only be read during startup or reconfiguration of a task...", instead of being read in each execution of poll() method.
I had the same misconception about how Kafka Connect SourceTasks work:
Fetching the current offset in poll() only makes sense if you think about Tasks as "one-off" jobs: The Task is started, then poll() is called, which sends the record(s) to Kafka and persists its offset with it; then the Task is killed and the next Task will pick up the offset and continue reading data from the source.
This would not work well with Kafka, because the Connector partitions/offsets are themselves persisted in a Kafka topic - so there is no guarantee on how long the replication of the partition/offset value will take. This is why you receive stale offsets in the poll() method.
In reality, Tasks are started once, and after that their poll() method is called as long as the Task is running. So the Task can store its offset in memory (e.g. a field in the SourceTask deriving class) and access/update it during each poll(). An obvious result of this is that multiple tasks working on the same partition (e.g. same file, same table, ...) can lead to message duplication as their offsets are not in sync within the partition.
FileStreamSourceTask from the official Kafka repository is a good example on how reading offsets can be handled in a Source connector:
// stream is only null if the poll method has not run yet - only in this case the offset will be read!
if (stream == null) {
...
Map<String, Object> offset = context.offsetStorageReader().offset(Collections.singletonMap(FILENAME_FIELD, filename));
Object lastRecordedOffset = offset.get(POSITION_FIELD);
...
streamOffset = (lastRecordedOffset != null) ? (Long) lastRecordedOffset : 0L;
}
What is the recommended way of using libssh2 to implement the “read data from whichever channel has it first” primitive?
E.g. I have a simple two-tab terminal program, where each tab corresponds to a LIBSSH2 channel. I want to wait until ANY of the 2 channels gets data and then print it.
The single-channel examples use libssh2_channel_read() in a non-blocking way like this:
while(not done) {
1. Try reading with libssh2_channel_read()
2. If returned LIBSSH2_ERROR_EAGAIN, wait with select()
}
The trivial way of extending this to two-channel case would be:
while(not done) {
1. Try reading channel 1
2. Try reading channel 2
3. If BOTH channels returned LIBSSH2_ERROR_EAGAIN, wait with select()
}
This leads to a rare bug when a packet with some data for channel 1 arrives just before reading channel 2. Then both calls return LIBSSH2_ERROR_EAGAIN, but as the attempt to read channel 2 actually recv()’d the data for channel 1, select() will now hang.
The workaround I am currently employing involves keeping raw data counters for the socket and using them to determine if any new data was consumed by libssh2, but I get the feeling of making a really complex workaround for a fairly simple problem. Am I missing something?
Is there some kind of libssh2_session_read_any_channel()?
I did experiments with boost::upgrade_to_unique_lock/unique_lock && boost::shared_lock, scenario is:
1 write thread, where it has
boost::unique_lock existing with a
boost::shared_mutex, in the thread, I
write to a global AClass
3 read thread, each one has
boost::shared_lock with the same
boost:;shrared_mutex, they have a
loop to read the global AClass
I observed all the threads are holding locks( 1 unique, 3 shared ) at the same time, and they all
running data access loops.
my concern is AClass is not thread-safe, if I can do read/write at the same time in different threads, the read could crash. Even it's not AClass, we use primitive types, reading them surely will not crash, but the data could be dirty, isn't it ?
boost::shared_lock<boost::shared_mutex>(gmutex);
This is not an "unnamed lock." This creates a temporary shared_lock object which locks gmutex, then that temporary shared_lock object is destroyed, unlocking gmutex. You need to name the object, making it a variable, for example:
boost::shared_lock<boost::shared_mutex> my_awesome_lock(gmutex);
my_awesome_lock will then be destroyed at the end of the block in which it is declared, which is the behavior you want.