Hybrid Parallel Programming using OpenMPI and OpenMP : OpenMP threads always use 1 out of 4 cores per node - parallel-processing

Question

I have created a home cluster using 3 Raspberry PI boards to learn and experiment with Hybrid Parallel programming.
My setup is as follows:-
1 RPi3 B+ (4 cores ) and 2 RPi4 (4 cores each)and all these nodes are connected over ethernet using a switch. Therefore, in total I have 12 cores in my cluster. RPI3B+ is designated as node1 and rest of the two RPI4 boards as node2 and node 3 respectively.
I have written a simple program using OpenMPI and OpenMP, which sums up two large vectors.
The structure of the code is as follows:-
My intention here is - I would start 3 MPI processes with ranks 0, 1, and 2 and bind them to node1, node2, and node 3 respectively. From the inside of each MPI process, I would start 4 OMP threads which would do their job of summing up the distributed vector chunks and return back the result to rank 0.
Although the code is working and giving the correct result, but my observation is that the OMP threads are not leveraging all the CPU cores present in a node. Using htop, I have noticed that only 1 core on each of the nodes is 100% loaded.
The way I am launching the program is as follows:-
#mpirun --map-by node --display-devel-map --hostfile nodeconfig -np 3 ./addVectorsMPIHybrid 3000000 3000000
The nodeconfig contains just names of the nodes i.e, node1, node2 and node3
Please note: I have verified that 4 omp threads are getting created in the #pragma parallel section.
Can someone please give me pointers - why OMP threads are not running on all the cores?
The code that I am running:
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
#include <assert.h>
#include <omp.h>
int main(int argc, char *argv[])
{
double *v1, *v2, *resulv;
double *v1local, *v2local, *resulvlocal;
double startTime, endTime;
int rank, size;
int local_n;
int n;
int root = 0;
int provided;
if(argc <2)
{
printf("Please provide correct inputs\n");
exit(0);
}
n = atoi(argv[1]);
if(n <=0)
{
printf("Please enter valid size of vector\n");
exit(0);
}
//MPI_Init(NULL, NULL);
MPI_Init_thread(NULL, NULL, MPI_THREAD_FUNNELED, &provided);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
local_n = n/size;
if(rank == 0)
{
v1 = (double *) malloc (sizeof(double) * n);
assert(v1);
v2 = (double *) malloc (sizeof(double) * n);
assert(v2);
resulv = (double *) malloc (sizeof(double) * n);
assert(resulv);
resulv[n] = {0};
for(int i=0; i<n; i++)
v1[i] = v2[i] = (double)i;
}
v1local = (double *) malloc (sizeof(double) * local_n);
assert(v1local);
v2local = (double *) malloc (sizeof(double) * local_n);
assert(v2local);
resulvlocal = (double *) malloc (sizeof(double) * local_n);
assert(resulv);
resulvlocal[local_n] = {0.0};
startTime = MPI_Wtime();
//Scatter v1
MPI_Scatter(v1, local_n,MPI_DOUBLE, v1local, local_n, MPI_DOUBLE, root, MPI_COMM_WORLD);
MPI_Scatter(v2, local_n,MPI_DOUBLE, v2local, local_n, MPI_DOUBLE, root, MPI_COMM_WORLD);
#pragma omp parallel num_threads(4) default(none) shared(resulvlocal, v1local, v2local) firstprivate(local_n)
{
#pragma omp single
{
printf("num_threads = %d\n", omp_get_num_threads());
}
#pragma omp for
for(int i=0; i<local_n; i++)
resulvlocal[i] = v1local[i] + v2local[i];
}
MPI_Gather(resulvlocal, local_n, MPI_DOUBLE, resulv, local_n, MPI_DOUBLE, root, MPI_COMM_WORLD);
endTime = MPI_Wtime();
if(rank == 0)
{
//Display resulv
#if 0
for(int i=0; i<n; i++)
printf("%0.3f ",resulv[i]);
printf("\n");
#endif
printf("Time taken to compute is:%f in sec\n",endTime-startTime);
free(v1);
free(v2);
free(resulv);
}
free(v1local);
free(v2local);
free(resulvlocal);
MPI_Finalize();
return 0;
}