Compiling single vs multiple source files in Intel Fortran - performance

Question

I have been compiling a project with modules and subroutines in different files. Each subroutine written in separate file. The same for the modules. Then, I tested compiling these files separately to object files (-c) and than linking with the optimization flags, and also using cat to merge the entire source code and applying the same procedure to this single source file. What I found is that the executable generated by compiling the single file was about 40% faster than that generated by the multiple files, despite using exactly same flags for both.
I would like to know if anyone knows why it is happening, and if there is any flag on the Intel Fortran compiler that compiles multiple files as they were a single file.

Answer 1

As #chw21 requested, I created a small program showing the problem:
program main
use operators
implicit none
integer :: n
real(8), dimension(:,:), allocatable :: a, b, c
integer :: i,j,k
n = 1000
allocate(a(n,n), b(n,n), c(n,n))
call random_number(a)
call random_number(b)
do j = 1, n
do i = 1, n
do k = 1, n
!c(i,j) = c(i,j) + a(k,i) * b(k,j)
c(i,j) = add(c(i,j), mul(a(k,i), b(k,j)))
enddo
enddo
enddo
write(*,*) sum(c)
end program
with module:
module operators
contains
function add(a,b) result (c)
real(8), intent(in) :: a, b
real(8) :: c
c = a + b
end function
function mul(a,b) result (c)
real(8), intent(in) :: a, b
real(8) :: c
c = a * b
end function
end module
The idea is that these functions should normally get inlined, if the compiler knows that they are so extremely small. I did three tests with -O2:
complete source in a single file
split in two files
split in two files with -ipo (or -flto)
The results for ifort 13.0.0 and gfortran 5.2.0 on different machines are:
Test | 1. | 2. | 3.
---------+-------+-------+-------
ifort | 1.3s | 15.7s | 1.9s
gfortran | 1.1s | 3.7s | 1.1s
Unfortunately, I don't know why there is still a difference between the 1st and 3rd test with ifort ... I guess, a look at the generated code would shed some light on this issue.
Update: The times were measured by executing time ./a.out which resulted in stable times. Due to the standard compilation with ifort -O2, the maximum instruction set should be SSE2 (thus, no FMA), the processor supports upto SSE4a (Opteron 6128). An additional test on a recent Intel processor (upto AVX) showed similar results.
An important thing seems to be the lack of inlining and vectorization of the inner loop, which gets applied during IPO and single-file-compilation (see --opt-report). Additionally, there seem to be some differences concerning vectorization between IPO and single-file-compilation.