Creating hardlinks in docker container takes long the first time - performance

Question

I am creating hard links of all the files in a directory (some 500MB) in a docker container using following command:
cp -al source destination
Interestingly, when I issue the above command the first time (i.e. when there is no hardlink for each file in that directory), it takes quite a long time. This is not the case when I issue the above command again to create hardlinks of same files in other destination. This is also not the case when I do that in my docker host (even if it's the first hardlink that is being created).
My docker container uses overlay2 as the storage driver. Here is my docker info output
Containers: 61
Running: 1
Paused: 0
Stopped: 60
Images: 142
Server Version: 1.13.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: runc oci
Default Runtime: oci
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: fbadd789ddf86a4be9d6905528b7486c61e52612 (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: fbadd789ddf86a4be9d6905528b7486c61e52612-dirty (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 4.14.13-300.fc27.x86_64
Operating System: Fedora 27 (Workstation Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 8
Total Memory: 15.58 GiB
Name: pranavk.local.me
ID: STYG:PZCA:FMHL:FLCU:N7VG:Z463:KYQF:QVHO:KWKP:FHTY:EUBA:QBUK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure), registry.fedoraproject.org (secure), registry.access.redhat.com (secure), docker.io (secure)
Edit:
If I remove all the hardlinks and then issue above cp command again, it's fast again. Phew!
I am not sure what's causing this or from where I should start looking from.