Discrepancy in output of lscpu - performance

Question

I have a question about the performance impact when two boxes of same spec shows different results
Box1:
lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Model name: Intel(R) Xeon(R) CPU E5-2690 v2 # 3.00GHz
Stepping: 0
CPU MHz: 2999.999 <=============
BogoMIPS: 5999.99 <=============
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 25600K
NUMA node0 CPU(s): 0-7
NUMA node1 CPU(s): 8-15
Box2:
lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Model name: Intel(R) Xeon(R) CPU E5-2690 v2 # 3.00GHz
Stepping: 0
CPU MHz: 3000.00 <=============
BogoMIPS: 6000.00 <=============
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 25600K
NUMA node0 CPU(s): 0-7
NUMA node1 CPU(s): 8-15
For the same application running on both nodes I see the load average being 2x to 3x higher(compared to Box2) on Box1
The only difference I see in the output is the numbers being off by a fraction in CPU MHz in lscpu output.
Why do we see such difference for actual CPU and will there be a perf difference because of this?

Answer 1

The only difference is that bogomips calibration randomly got a slightly lower value, e.g. a crystal oscillator might be off by a fraction of a percent, or just a pure timing artifact between the CPU core clock vs. whatever clocks Linux uses to time the bogomips loop.
So that doesn't explain anything, and is unrelated to any significant software performance difference you observe. Obviously we can't tell you anything more without any details.
Possible guesses at an explanation for a big perf difference could include the RAM config, like do they both have all memory controllers populated?
Otherwise almost certainly some software difference. Like one running a debug build, or some difference in how you built the binary, or in the libraries it uses, or the kernel or kernel config. Or running on different data, and your application is sensitive to different data.
Or possibly if your VMware config has one of those VMs mapping its CPU cores to fewer physical cores on the bare metal, e.g. competing via hyperthreading when the kernel in the VM assumes they're not. Or if the guest kernel has wrong info about NUMA.
Of obviously if your VM is sharing the bare metal on one of them with some other workload!
There can be minor differences in inter-core latency between different instances of the same Xeon CPU model, depending on exactly where on the ring bus the enabled cores are. (Except on the top-end models for each core-count, some of the cores on each die are fused off due to defects or just for market segmentation.) But this is a very small effect, and only in inter-core latency.
But this is all kind of off-topic for your question about a .001 MHz difference in measured CPU frequency. We can safely say that's not an explanation. If you do want to ask about that, post a separate question with full details on your application. But probably it's going to be some difference only you can find, some wrong assumption about something being the same. Maybe run some other benchmarks on the machines, especially pre-compiled to rule out compiler differences.