OMP Settings issue - openmp

Question

1. I checked out oneDNN v1.2
2. Build it using clang
export CC=clang and export CXX=clang++
mkdir -p build && cd build && cmake ..
make -j
sudo make install
3. modified getting_started.cpp to add a print as first line inside main() function
(base) ratan#dattonaxq1648:~/my_work/oneDNN/examples$ git diff
diff --git a/examples/getting_started.cpp b/examples/getting_started.cpp
index 2376013eb..3c43dab91 100644
--- a/examples/getting_started.cpp
+++ b/examples/getting_started.cpp
## -469,6 +469,7 ## void getting_started_tutorial(engine::kind engine_kind) {
/// #snippet getting_started.cpp Main
// [Main]
int main(int argc, char **argv) {
std::cout << "Start of getting_started code\n";
int exit_code = 0;
4. export these env variables for verbose
export KMP_AFFINITY=verbose
export OMP_DISPLAY_ENV=TRUE
export KMP_SETTINGS=TRUE
5. build examples/getting_started.cpp
clang++ -std=c++11 getting_started.cpp -o bin/getting_started_cpp -L/usr/local/lib -ldnnl
6. Run above test
(base) ratan#dattonaxq1648:~/my_work/oneDNN/examples$ ./bin/getting_started_cpp
User settings:
KMP_AFFINITY=verbose
KMP_SETTINGS=TRUE
OMP_DISPLAY_ENV=TRUE
Effective settings:
KMP_ABORT_DELAY=0
KMP_ADAPTIVE_LOCK_PROPS='1,1024'
KMP_ALIGN_ALLOC=64
KMP_ALL_THREADPRIVATE=384
KMP_ATOMIC_MODE=2
KMP_BLOCKTIME=200
KMP_CPUINFO_FILE: value is not defined
KMP_DETERMINISTIC_REDUCTION=false
KMP_DEVICE_THREAD_LIMIT=2147483647
KMP_DISP_NUM_BUFFERS=7
KMP_DUPLICATE_LIB_OK=false
KMP_ENABLE_TASK_THROTTLING=true
KMP_FORCE_REDUCTION: value is not defined
KMP_FOREIGN_THREADS_THREADPRIVATE=true
KMP_FORKJOIN_BARRIER='2,2'
KMP_FORKJOIN_BARRIER_PATTERN='hyper,hyper'
KMP_FORKJOIN_FRAMES=true
KMP_FORKJOIN_FRAMES_MODE=3
KMP_GTID_MODE=3
KMP_HANDLE_SIGNALS=false
KMP_HOT_TEAMS_MAX_LEVEL=1
KMP_HOT_TEAMS_MODE=0
KMP_INIT_AT_FORK=true
KMP_ITT_PREPARE_DELAY=0
KMP_LIBRARY=throughput
KMP_LOCK_KIND=queuing
KMP_MALLOC_POOL_INCR=1M
KMP_NUM_LOCKS_IN_BLOCK=1
KMP_PLAIN_BARRIER='2,2'
KMP_PLAIN_BARRIER_PATTERN='hyper,hyper'
KMP_REDUCTION_BARRIER='1,1'
KMP_REDUCTION_BARRIER_PATTERN='hyper,hyper'
KMP_SCHEDULE='static,balanced;guided,iterative'
KMP_SETTINGS=true
KMP_SPIN_BACKOFF_PARAMS='4096,100'
KMP_STACKOFFSET=64
KMP_STACKPAD=0
KMP_STACKSIZE=8M
KMP_STORAGE_MAP=false
KMP_TASKING=2
KMP_TASKLOOP_MIN_TASKS=0
KMP_TASK_STEALING_CONSTRAINT=1
KMP_TEAMS_THREAD_LIMIT=96
KMP_TOPOLOGY_METHOD=all
KMP_USE_YIELD=1
KMP_VERSION=false
KMP_WARNINGS=true
OMP_AFFINITY_FORMAT='OMP: pid %P tid %i thread %n bound to OS proc set {%A}'
OMP_ALLOCATOR=omp_default_mem_alloc
OMP_CANCELLATION=false
OMP_DEFAULT_DEVICE=0
OMP_DISPLAY_AFFINITY=false
OMP_DISPLAY_ENV=true
OMP_DYNAMIC=false
OMP_MAX_ACTIVE_LEVELS=1
OMP_MAX_TASK_PRIORITY=0
OMP_NESTED: deprecated; max-active-levels-var=1
OMP_NUM_THREADS: value is not defined
OMP_PLACES: value is not defined
OMP_PROC_BIND='false'
OMP_SCHEDULE='static'
OMP_STACKSIZE=8M
OMP_TARGET_OFFLOAD=DEFAULT
OMP_THREAD_LIMIT=2147483647
OMP_TOOL=enabled
OMP_TOOL_LIBRARIES: value is not defined
OMP_WAIT_POLICY=PASSIVE
KMP_AFFINITY='verbose,warnings,respect,granularity=core,none'
OPENMP DISPLAY ENVIRONMENT BEGIN
_OPENMP='201611'
[host] OMP_AFFINITY_FORMAT='OMP: pid %P tid %i thread %n bound to OS proc set {%A}'
[host] OMP_ALLOCATOR='omp_default_mem_alloc'
[host] OMP_CANCELLATION='FALSE'
[host] OMP_DEFAULT_DEVICE='0'
[host] OMP_DISPLAY_AFFINITY='FALSE'
[host] OMP_DISPLAY_ENV='TRUE'
[host] OMP_DYNAMIC='FALSE'
[host] OMP_MAX_ACTIVE_LEVELS='1'
[host] OMP_MAX_TASK_PRIORITY='0'
[host] OMP_NESTED: deprecated; max-active-levels-var=1
[host] OMP_NUM_THREADS: value is not defined
[host] OMP_PLACES: value is not defined
[host] OMP_PROC_BIND='false'
[host] OMP_SCHEDULE='static'
[host] OMP_STACKSIZE='8M'
[host] OMP_TARGET_OFFLOAD=DEFAULT
[host] OMP_THREAD_LIMIT='2147483647'
[host] OMP_TOOL='enabled'
[host] OMP_TOOL_LIBRARIES: value is not defined
[host] OMP_WAIT_POLICY='PASSIVE'
OPENMP DISPLAY ENVIRONMENT END
OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0-95
OMP: Info #156: KMP_AFFINITY: 96 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 2 packages x 24 cores/pkg x 2 threads/core (48 total cores)
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6193 thread 0 bound to OS proc set 0-95
Start of getting_started code
dnnl_verbose,info,DNNL v1.2.0 (commit 75d0b1a7f3586c212e37acebbb8acd221cee7216)
dnnl_verbose,info,cpu,runtime:OpenMP
dnnl_verbose,info,cpu,isa:Intel AVX2
dnnl_verbose,info,gpu,runtime:none
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6194 thread 1 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6195 thread 2 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6196 thread 3 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6197 thread 4 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6198 thread 5 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6199 thread 6 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6200 thread 7 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6201 thread 8 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6202 thread 9 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6203 thread 10 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6205 thread 12 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6204 thread 11 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6206 thread 13 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6207 thread 14 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6208 thread 15 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6209 thread 16 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6210 thread 17 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6211 thread 18 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6212 thread 19 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6213 thread 20 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6214 thread 21 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6215 thread 22 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6217 thread 24 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6216 thread 23 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6218 thread 25 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6219 thread 26 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6220 thread 27 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6221 thread 28 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6222 thread 29 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6223 thread 30 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6224 thread 31 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6225 thread 32 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6226 thread 33 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6227 thread 34 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6228 thread 35 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6229 thread 36 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6230 thread 37 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6231 thread 38 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6233 thread 40 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6232 thread 39 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6235 thread 42 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6234 thread 41 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6236 thread 43 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6237 thread 44 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6238 thread 45 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6239 thread 46 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6240 thread 47 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6241 thread 48 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6242 thread 49 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6243 thread 50 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6244 thread 51 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6246 thread 53 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6245 thread 52 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6248 thread 55 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6247 thread 54 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6249 thread 56 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6250 thread 57 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6251 thread 58 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6252 thread 59 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6254 thread 61 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6253 thread 60 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6255 thread 62 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6256 thread 63 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6257 thread 64 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6258 thread 65 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6259 thread 66 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6260 thread 67 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6261 thread 68 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6262 thread 69 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6263 thread 70 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6264 thread 71 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6265 thread 72 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6266 thread 73 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6267 thread 74 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6268 thread 75 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6269 thread 76 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6270 thread 77 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6271 thread 78 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6272 thread 79 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6273 thread 80 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6274 thread 81 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6275 thread 82 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6276 thread 83 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6277 thread 84 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6278 thread 85 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6279 thread 86 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6280 thread 87 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6281 thread 88 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6282 thread 89 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6283 thread 90 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6285 thread 92 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6284 thread 91 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6286 thread 93 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6288 thread 95 bound to OS proc set 0-95
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6287 thread 94 bound to OS proc set 0-95
dnnl_verbose,exec,cpu,eltwise,jit:avx2,forward_inference,data_f32::blocked:acdb:f0 diff_undef::undef::f0,,alg:eltwise_relu alpha:0 beta:0,1x3x13x13,32.114
Example passed on CPU.
7. If we see that thread 0 is created before 1st line in main() is executed.
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6193 thread 0 bound to OS proc set 0-95
Start of getting_started code
dnnl_verbose,info,DNNL v1.2.0 (commit 75d0b1a7f3586c212e37acebbb8acd221cee7216)
dnnl_verbose,info,cpu,runtime:OpenMP
dnnl_verbose,info,cpu,isa:Intel AVX2
dnnl_verbose,info,gpu,runtime:none
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6194 thread 1 bound to OS proc set 0-95
8. Asked same question mkl-dnn forum and i was suggested to ask here. Tried with other mkl-dnn version (V1.3, v1.2.1, v1.2.2) and it has same behaviour.
Question:
9. Question.
What build setting / or run time env variable to use so that thread bounds happens after entering main()
Expected behavior:
Start of getting_started code
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6193 thread 0 bound to OS proc set 0-95
dnnl_verbose,info,DNNL v1.2.0 (commit 75d0b1a7f3586c212e37acebbb8acd221cee7216)
dnnl_verbose,info,cpu,runtime:OpenMP
dnnl_verbose,info,cpu,isa:Intel AVX2
dnnl_verbose,info,gpu,runtime:none
OMP: Info #249: KMP_AFFINITY: pid 6193 tid 6194 thread 1 bound to OS proc set 0-95