shared memory memcpy performance issue - caching

Question

I'm doing some performance tunning on a shared memory based message queue. I found a strange phenomenon that I can't explain: I ran the same code for 3 epochs, the avg running time is getting better for each epoch.
Here's the minimal demo code:
inline uint64_t current_time_nanos() {
static timespec tp;
clock_gettime(CLOCK_REALTIME, &tp);
return tp.tv_nsec + tp.tv_sec * 1000000000LLU;
}
void test() {
static constexpr size_t TOTAL_SIZE = 16 * 1024 * 1024;
static constexpr size_t COUNT = TOTAL_SIZE / sizeof(market_data);
static_assert(TOTAL_SIZE % sizeof(market_data) == 0);
market_data md;
market_data *ptr =
(market_data *)mmap(nullptr, TOTAL_SIZE, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (MAP_FAILED == ptr) {
printf("failed to mmap: %s\n", strerror(errno));
}
pthread_mutex_t mtx;
assert(0 == pthread_mutex_init(&mtx, nullptr));
std::atomic_uint64_t pos{0};
mlock(ptr, TOTAL_SIZE);
// Epoach 1
auto st = current_time_nanos();
for (int i = 0; i < COUNT; i++) {
assert(0 == pthread_mutex_lock(&mtx));
memcpy(&ptr[pos.fetch_add(1, std::memory_order_acq_rel) % COUNT], &md,
sizeof(market_data));
assert(0 == pthread_mutex_unlock(&mtx));
}
auto ed = current_time_nanos();
printf("total used: %lu, avg = %f.\n", ed - st, double(ed - st) / COUNT);
// Epoach 2
pos = 0;
st = current_time_nanos();
for (int i = 0; i < COUNT; i++) {
assert(0 == pthread_mutex_lock(&mtx));
memcpy(&ptr[pos.fetch_add(1, std::memory_order_acq_rel) % COUNT], &md,
sizeof(market_data));
assert(0 == pthread_mutex_unlock(&mtx));
}
ed = current_time_nanos();
printf("total used: %lu, avg = %f.\n", ed - st, double(ed - st) / COUNT);
// Epoach 3
pos = 0;
st = current_time_nanos();
for (int i = 0; i < COUNT; i++) {
assert(0 == pthread_mutex_lock(&mtx));
memcpy(&ptr[pos.fetch_add(1, std::memory_order_acq_rel) % COUNT], &md,
sizeof(market_data));
assert(0 == pthread_mutex_unlock(&mtx));
}
ed = current_time_nanos();
printf("total used: %lu, avg = %f.\n", ed - st, double(ed - st) / COUNT);
}
I've run the code for multiple times, It can be sure that the avg execution time is getting better for each epoch. e.g epoch 3 has the best performance.
I wonder why this happens and How can I do some warmup that I could gain the performance from 3rd epoch without actually do memcpy?
sample result:
total used: 2479219, avg = 75.659760.
total used: 2092045, avg = 63.844147.
total used: 1718318, avg = 52.438904.
Here's the detailed info:
CPU: Intel Xeon 6348 2.6GHZ (Cascade-Lake)
Compiler: G++ 10.2.1 with O3 enabled
I've already use mlock to avoid page fault. it helps a lot. I also try to use _mm_prefetch but there's no performance gain actually.(Or may be I'm not using it correctly)