Processes are running slow in Docker container on Ubuntu 18.04 - performance

Question

Processes are taking time when running them in a Docker container on an Ubuntu 18 machine. But the same process with the same Docker version is running fine on an Ubuntu 16 machine.
I have a node application listening on some port. Accepting get requests on the path "/" and "/docker" which simply runs a command "whoami" in the host machine and in a Docker container respectively and returns the result. The same node application with the same Docker container is running on both the machines (Ubuntu 16 and Ubuntu 18).
Firstly, I tried sending 20 concurrent get requests with path "/" to both the machines. And both the machines executed the command in an average of
35-40ms.
Secondly, I tried sending 20 concurrent get requests with path "/docker" to both the machines. Here, the Ubuntu 16 machine took a maximum of 4.3 seconds and an average of 3 seconds. But the Ubuntu 18 machine takes a maximum of 10 seconds and an average of 9 seconds.
I tried the above test multiple times and concluded that when running the process inside Docker, the time taken to execute is almost double in the Ubuntu 18 machine compared to Ubuntu 16.
I checked the following:
I tried monitoring through top and htop while hitting 20 requests. But everything seems the same there.
Also tried monitoring using perf command. But unable to find any unusual difference there. But I am not very used to perf command and so unable to understand clearly.
While these 20 requests were in processing. I run the same Docker command manually with strace. And found random results i.e. sometimes time taken in clock_gettime or futex (FUTEX_WAIT) or sometimes in +++ exited with 0 +++ message on Ubuntu 18, but it took less time on Ubuntu 16.
Below are the different configurations and code snippets I am using and running:
Machine1: Giving better performance.
node v10.16.0
npm 6.9.0
docker 18.09.8
ubuntu 16.04.3 LTS, xenial
Machine2: Giving poor performance.
node v10.16.0
npm 6.9.0
docker 18.09.8
ubuntu 18.04.2 LTS, bionic
Node application code snippet:
// for path "/docker"
var excuteInDocker = function() {
var cmd = "docker";
var args = ["exec", "ubuntu", "whoami"];
return executeCmd(cmd, args);
}
// for path "/"
var execute = function(){
var cmd = 'whoami';
var args = [];
return executeCmd(cmd, args);
}
Output of docker info which are common to both ubuntu 16 and 18:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 2
Server Version: 18.09.8
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.296GiB
Name: myhostname
ID: LLLO:OMTS:PNNM:T3MP:AD2F:UMDG:IIZK:OGBO:3ZLL:YDBX:ONAO:AY5G
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 27
Goroutines: 42
System Time: 2019-07-25T15:25:54.991694211+05:30
EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support
docker info specific to Ubuntu 16:
Kernel Version: 4.4.0-112-generic
Operating System: Ubuntu 16.04.3 LTS
Total Memory: 7.303GiB
ID: FOFI:RW7N:RZSP:HHKH:BKS3:LMWL:TC2J:W7V2:222Y:Q2AU:XMU3:KLU7
docker info specific to Ubuntu 18:
Kernel Version: 4.15.0-1040-aws
Operating System: Ubuntu 18.04.2 LTS
Total Memory: 7.296GiB
ID: LLLO:OMTS:PNNM:T3MP:AD2F:UMDG:IIZK:OGBO:3ZLL:YDBX:ONAO:AY5G
Ubuntu 16 machine Data:
1. Data of time taken in execution
2019-07-25 14:06:42.851 INFO uid: 540ae880-aeb7-11e9-919d-dd32b3cf84d5 time: 475 result: {"success":true,"data":"root"}
2019-07-25 14:06:43.183 INFO uid: 54145e60-aeb7-11e9-919d-dd32b3cf84d5 time: 745 result: {"success":true,"data":"root"}
2019-07-25 14:06:45.711 INFO uid: 540c4810-aeb7-11e9-919d-dd32b3cf84d5 time: 3326 result: {"success":true,"data":"root"}
.
.
.
2019-07-25 14:06:46.835 INFO uid: 541d5f10-aeb7-11e9-919d-dd32b3cf84d5 time: 4338 result: {"success":true,"data":"root"}
Logs of command strace -t docker exec ubuntu whoami:
Result of perf top --sort comm,dso:
Ubuntu 18 machine Data:
1. Data of time taken in execution:
2019-07-25 14:07:32.559 INFO uid: 715a6af0-aeb7-11e9-a5a9-2fffd4e800d1 time: 1008 result: {"success":true,"data":"root"}
2019-07-25 14:07:32.941 INFO uid: 7178c860-aeb7-11e9-a5a9-2fffd4e800d1 time: 1191 result: {"success":true,"data":"root"}
2019-07-25 14:07:40.363 INFO uid: 71767e70-aeb7-11e9-a5a9-2fffd4e800d1 time: 8628 result: {"success":true,"data":"root"}
.
.
.
2019-07-25 14:07:41.970 INFO uid: 718af0d0-aeb7-11e9-a5a9-2fffd4e800d1 time: 10101 result: {"success":true,"data":"root"}
Logs of command strace -t docker exec ubuntu whoami:
result of perf top --sort comm,dso:
So, I need help in debugging what is wrong with Docker on the Ubuntu 18 machine. Or if there is any limitation with Docker on Ubuntu 18 or maybe some machine or Docker configuration issue.

Answer 1

I did not encounter such a problem on my desktop, but mysql container was running very slowly in a way that I could not understand on my ubuntu laptop. This solution solved my problem.