When running docker stats I see:
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
11e22031f0be -- -- / -- -- -- / -- -- / -- --
f55b3bbf6115 0.00% 26.98 MiB / 7.795 GiB 0.34% 0 B / 0 B 6.728 MB / 0 B 2
ff7800652fa1 0.30% 21.61 MiB / 7.795 GiB 0.27% 0 B / 0 B 16.69 MB / 270.2 MB 9
d422fe2d0f1a 0.12% 14.23 MiB / 7.795 GiB 0.18% 0 B / 0 B 7.055 MB / 0 B 5
86df65e7c195 0.04% 6.141 MiB / 7.795 GiB 0.08% 0 B / 0 B 6.192 MB / 0 B 2
1d0c20d5ea48 0.27% 32.51 MiB / 7.795 GiB 0.41% 0 B / 0 B 23.48 MB / 0 B 11
--
Does anyone know what the -- indicates? The container is running and I can view the non-zero per-process statistics using top. What does the -- mean?
The result of docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
11e22031f0be aaaaaaaaaaa "run master" 28 hours ago Up 28 hours agitated_bhaskara
f55b3bbf6115 bbbbbbbbb "run ovn-controller" 28 hours ago Up 28 hours ovn-controller
ff7800652fa1 quay.io/coreos/etcd:v3.0.2 "/usr/local/bin/etcd " 28 hours ago Up 28 hours etcd
d422fe2d0f1a ccccc/ovs "run ovs-vswitchd" 28 hours ago Up 28 hours ovs-vswitchd
86df65e7c195 ddddd/ovs "run ovsdb-server" 28 hours ago Up 28 hours ovsdb-server
1d0c20d5ea48 eeeeeeeeeeeeeeee "abcde minino" 28 hours ago Up 28 hours minion
Related
I’m working on embedded Linux targets (32-bit ARM) and need to determine how much RAM is available for applications once the kernel and core software are launched. Available memory reported by free and /proc/meminfo don’t seem to align with what testing shows is actually usable by applications. Is there a way to correctly calculate how much RAM is truly available without running e.g., stress on each system?
The target system used in my tests below has 256 MB of RAM and does not use swap (CONFIG_SWAP is not set). I’m used the 3.14.79-rt85 kernel in the tests below but have also tried 4.9.39 and see similar results. During boot, the following is reported:
Memory: 183172K/262144K available (5901K kernel code, 377K rwdata, 1876K rodata, 909K init, 453K bss, 78972K reserved)
Once system initialization is complete and the base software is running (e.g., dhcp client, ssh server, etc.), I get the following reported values:
[root#host ~]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 210016 320 7880 0 0 0 0 186 568 0 2 97 0 0
[root#host ~]# free -k
total used free shared buff/cache available
Mem: 249616 31484 209828 68 8304 172996
Swap: 0 0 0
[root#host ~]# cat /proc/meminfo
MemTotal: 249616 kB
MemFree: 209020 kB
MemAvailable: 172568 kB
Buffers: 712 kB
Cached: 4112 kB
SwapCached: 0 kB
Active: 4684 kB
Inactive: 2252 kB
Active(anon): 2120 kB
Inactive(anon): 68 kB
Active(file): 2564 kB
Inactive(file): 2184 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 2120 kB
Mapped: 3256 kB
Shmem: 68 kB
Slab: 13236 kB
SReclaimable: 4260 kB
SUnreclaim: 8976 kB
KernelStack: 864 kB
PageTables: 296 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 124808 kB
Committed_AS: 47944 kB
VmallocTotal: 1810432 kB
VmallocUsed: 3668 kB
VmallocChunk: 1803712 kB
[root#host ~]# sysctl -a | grep '^vm'
vm.admin_reserve_kbytes = 7119
vm.block_dump = 0
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.drop_caches = 3
vm.extfrag_threshold = 500
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 32
vm.max_map_count = 65530
vm.min_free_kbytes = 32768
vm.mmap_min_addr = 4096
vm.nr_pdflush_threads = 0
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.scan_unevictable_pages = 0
vm.stat_interval = 1
vm.swappiness = 60
vm.user_reserve_kbytes = 7119
vm.vfs_cache_pressure = 100
Based on the numbers above, I expected to have ~160 MiB available for future applications. By tweaking sysctl vm.min_free_kbytes I can boost this to nearly 200 MiB since /proc/meminfo appears to take this reserve into account, but for testing I left it set as it is above.
To test how much RAM was actually available, i used the stress tool as follows:
stress --vm 11 --vm-bytes 10M --vm-keep --timeout 5s
At 110 MiB, the system remains responsive and both free and vmstat reflect the increased RAM usage. The lowest reported free/available values are below:
[root#host ~]# free -k
total used free shared buff/cache available
Mem: 249616 146580 93196 68 9840 57124
Swap: 0 0 0
[root#host ~]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
11 0 0 93204 1792 8048 0 0 0 0 240 679 50 0 50 0 0
Here is where things start to break down. After increasing stress’ memory usage to 120 MiB - still well shy of the 168 MiB reported as available - the system freezes for the 5 seconds while stress is running. Continuously running vmstat during the test (or as continuously as possible due to the freeze) shows:
[root#host ~]# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 209664 724 6336 0 0 0 0 237 666 0 1 99 0 0
3 0 0 121916 1024 6724 0 0 289 0 1088 22437 0 45 54 0 0
1 0 0 208120 1328 7128 0 0 1652 0 4431 43519 28 22 50 0 0
Due to the significant increase in interrupts and IO, I’m guessing the kernel is evicting pages containing executable code and then promptly needing to read them back in from flash. My questions are a) is this a correct assessment? and b) why would the kernel be doing this with RAM still available?
Note that if try to use a single worker with stress and claim 160 MiB of memory, the OOM gets activated and kills the test. OOM does not trigger in the scenarios described above.
I was wondering how are floppy disk sectors ordered, I am currently writing a program to access the root directory of a floppy disk (fat12 formated High Density), I can load it with debug at sector 13h but in assembly it is at head 1 track 0 sector 2 why is sector 13h, not at head 0 track 1 sector 1?
That's because the sectors on the other side of the disk comes before the sectors on the second track on the first side.
Sectors 0 through 17 (11h) are found at head 0 track 0. Sectors 18 (12h) through 35 (23h) are found at head 1 track 0.
Logical sectors are numbered from zero up, but the sectors in a track are numbered from 1 to 18 (12h).
sector# head track sector usage
------- ---- ----- ------ --------
0 0h 0 0 1 1h boot
1 1h 0 0 2 2h FAT 1
2 2h 0 0 3 3h |
3 3h 0 0 4 4h v
4 4h 0 0 5 5h
5 5h 0 0 6 6h
6 6h 0 0 7 7h
7 7h 0 0 8 8h
8 8h 0 0 9 9h
9 9h 0 0 10 ah
10 ah 0 0 11 bh FAT 2
11 bh 0 0 12 ch |
12 ch 0 0 13 dh v
13 dh 0 0 14 eh
14 eh 0 0 15 fh
15 fh 0 0 16 10h
16 10h 0 0 17 11h
17 11h 0 0 18 12h
18 12h 1 0 1 1h
19 13h 1 0 2 2h root
20 14h 1 0 3 3h |
21 15h 1 0 4 4h v
...
I writing a data mover in go. Taking data located in one data center and moving it to another data center. Figured go would be perfect for this given the go routines.
I notice if I have one program running 1800 threads the amount of data being transmitted is really low
here's the dstat print out averaged over 30 seconds
---load-avg--- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
1m 5m 15m |usr sys idl wai hiq siq| read writ| recv send| in out | int csw
0.70 3.58 4.42| 10 1 89 0 0 0| 0 156k|7306k 6667k| 0 0 | 11k 6287
0.61 3.28 4.29| 12 2 85 0 0 1| 0 6963B|8822k 8523k| 0 0 | 14k 7531
0.65 3.03 4.18| 12 2 86 0 0 1| 0 1775B|8660k 8514k| 0 0 | 13k 7464
0.67 2.81 4.07| 12 2 86 0 0 1| 0 1638B|8908k 8735k| 0 0 | 13k 7435
0.67 2.60 3.96| 12 2 86 0 0 1| 0 819B|8752k 8385k| 0 0 | 13k 7445
0.47 2.37 3.84| 11 2 86 0 0 1| 0 2185B|8740k 8491k| 0 0 | 13k 7548
0.61 2.22 3.74| 10 2 88 0 0 0| 0 1229B|7122k 6765k| 0 0 | 11k 6228
0.52 2.04 3.63| 3 1 97 0 0 0| 0 546B|1999k 1365k| 0 0 |3117 2033
If I run 9 instances of the program with 200 threads each I see much better performance
---load-avg--- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
1m 5m 15m |usr sys idl wai hiq siq| read writ| recv send| in out | int csw
8.34 9.56 8.78| 53 8 36 0 0 3| 0 410B| 38M 32M| 0 0 | 41k 26k
8.01 9.37 8.74| 74 10 12 0 0 4| 0 137B| 51M 51M| 0 0 | 59k 39k
8.36 9.31 8.74| 75 9 12 0 0 4| 0 1092B| 51M 51M| 0 0 | 59k 39k
6.93 8.89 8.62| 74 10 12 0 0 4| 0 5188B| 50M 49M| 0 0 | 59k 38k
7.09 8.73 8.58| 75 9 12 0 0 4| 0 410B| 51M 50M| 0 0 | 60k 39k
7.40 8.62 8.54| 75 9 12 0 0 4| 0 137B| 52M 49M| 0 0 | 61k 40k
7.96 8.63 8.55| 75 9 12 0 0 4| 0 956B| 51M 51M| 0 0 | 59k 39k
7.46 8.44 8.49| 75 9 12 0 0 4| 0 273B| 51M 50M| 0 0 | 58k 38k
8.08 8.51 8.51| 75 9 12 0 0 4| 0 410B| 51M 51M| 0 0 | 59k 39k
load average is a little high but I'll worry about that later. The network traffic though is almost hitting the network potential.
I'm on Ubuntu 12.04,
8 Gigs Ram,
2.3 GHz processors (says EC2 :P)
Also, I've increased my file descriptors from 1024 to 10240
I thought go was designed for this kind of thing or am I expecting too much of go for this application?
Is there something trivial that I'm missing? Do I need to configure my system to maximizes go's potential?
EDIT
I guess my question wasn't clear enough. Sorry. I'm not asking for magic from go, I know the computers have limitations to what they can handle.
So I'll rephrase. Why is 1 instance with 1800 go routines != 9 instances with 200 threads each? Same amount of go routines significantly less performance for 1 instance compared to 9 instances.
Please note, that goroutines are also limited to your local maschine and that channels are not natively network enabled, i.e. your particular case is probably not biting go's chocolate site.
Also: What did you expect from throwing (suposedly) every transfer into a goroutine? IO-Operations tend to have their bottleneck where the bits hit the metal, i.e. the physical transfer of the data to the medium. Think of it like that: No matter how many Threads or (Goroutines in this case) try to write to Networkcard, you still only have one Networkcard. Most likely hitting it with to many concurrent write calls will only slow things down, since the involved overhead increases
If you think this is not the problem or want to audit your code for optimized performance, go has neat builtin features to do so: profiling go programs (official go blog)
But still the actual bottleneck might well be outside your go program AND/OR in the way it interacts with the os.
Adressing your actual problem without code is pointless guessing. Post some and everyone will try their best to help you.
You will probably have to post your source code to get any real input, but just to be sure, you have increased number of cpus to use?
import "runtime"
func main() {
runtime.GOMAXPROCS(runtime.NumCPU())
}
I am looking into a heap, which has many few allocations and number of entries are much more than 20, which is the default of !heap -stat -h command. For example, if you see below, the numbers don't add up to 100. Is there any way I can get all the entries in that heap?
!heap -stat -h 0000000006eb0000
heap # 0000000006eb0000
group-by: TOTSIZE max-display: 20
size #blocks total ( %) (percent of total busy bytes)
3a00 92e - 2146c00 (1.11)
27da8 c0 - 1de3e00 (1.00)
4fb48 5c - 1ca4de0 (0.95)
3bc78 6e - 19afb90 (0.86)
14c18 127 - 17eafa8 (0.80)
778e8 2b - 1414ef8 (0.67)
6f30 29d - 1229070 (0.61)
13ed8 a5 - cd8138 (0.43)
4c00 2a0 - c78000 (0.42)
10a18 a4 - aa7760 (0.36)
63a18 1a - a1e670 (0.34)
18e18 61 - 96d718 (0.31)
9f688 c - 778e60 (0.25)
20 3551e - 6aa3c0 (0.22)
a0 a776 - 68a9c0 (0.22)
8b7b8 b - 5fe4e8 (0.20)
1e08 2b0 - 50b580 (0.17)
30 168fc - 43af40 (0.14)
a898 60 - 3f3900 (0.13)
18 287ae - 3cb850 (0.13)
-Thanks,
Brajesh
You can increase this total by specifying the group by parameter followed by a number so for example:
!heap -stat -h 07300000 -grp A 0n100
gives output:
0:275> !heap -stat -h 07300000 -grp A 0n100
heap # 07300000 group-by: ALLOCATIONSIZE max-display: 100
size #blocks total ( %) (percent of total busy bytes)
7ecc10 1 - 7ecc10 (41.60)
1fc210 1 - 1fc210 (10.42)
1fb310 1 - 1fb310 (10.40)
17d110 1 - 17d110 (7.81)
2c4e0 2 - 589c0 (1.82)
2b330 1 - 2b330 (0.89)
20420 3 - 60c60 (1.98)
20020 4 - 80080 (2.63)
14320 1 - 14320 (0.41)
10020 1 - 10020 (0.33)
fab8 1 - fab8 (0.32)
eb4c 2 - 1d698 (0.60)
c020 1 - c020 (0.25)
9c60 4c - 2e6c80 (15.23)
82c0 3 - 18840 (0.50)
8020 3 - 18060 (0.49)
6420 1 - 6420 (0.13)
5ea0 1 - 5ea0 (0.12)
517c 1 - 517c (0.10)
4f40 1 - 4f40 (0.10)
4ba4 1 - 4ba4 (0.10)
4750 1 - 4750 (0.09)
4020 2 - 8040 (0.16)
3f78 1 - 3f78 (0.08)
2c38 1 - 2c38 (0.06)
25d8 1 - 25d8 (0.05)
21dc 1 - 21dc (0.04)
2040 1 - 2040 (0.04)
2020 3 - 6060 (0.12)
1de0 1 - 1de0 (0.04)
1da8 10 - 1da80 (0.61)
1b6c 3 - 5244 (0.11)
19f0 1 - 19f0 (0.03)
18e4 2 - 31c8 (0.06)
1890 1 - 1890 (0.03)
183c 2 - 3078 (0.06)
1820 1 - 1820 (0.03)
15e8 1 - 15e8 (0.03)
1560 1 - 1560 (0.03)
151c 2 - 2a38 (0.05)
14b0 1 - 14b0 (0.03)
1384 1 - 1384 (0.03)
1098 1 - 1098 (0.02)
102c 3 - 3084 (0.06)
1020 2 - 2040 (0.04)
101f 1 - 101f (0.02)
101c 1 - 101c (0.02)
Will dump the handles for that heap, grouped by allocation size for a max of 100 rows (0n specifies we are decimal based, without that prefix it becomes a hexidecimal value)
See this link for details of !heap
percentage of memory used used by a process.
normally prstat -J will give the memory of process image and RSS(resident set size) etc.
how do i knowlist of processes with percentage of memory is used by a each process.
i am working on solaris unix.
addintionally ,what are the regular commands that you use for monitoring processes,performences of processes that might be very useful to all!
The top command will give you several memory-consumption numbers. htop is much nicer, and will give you percentages, but it isn't installed by default on most systems.
run
top and then Shift+O this will bring you to the options, press n (this maybe different on your machine) for memory and then hit enter
Example of memory sort.
top - 08:17:29 up 3 days, 8:54, 6 users, load average: 13.98, 14.01, 11.60
Tasks: 654 total, 2 running, 652 sleeping, 0 stopped, 0 zombie
Cpu(s): 14.7%us, 1.5%sy, 0.0%ni, 59.5%id, 23.5%wa, 0.1%hi, 0.8%si, 0.0%st
Mem: 65851896k total, 49049196k used, 16802700k free, 1074664k buffers
Swap: 50331640k total, 0k used, 50331640k free, 32776940k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21635 oracle 15 0 6750m 636m 51m S 1.6 1.0 62:34.53 oracle
21623 oracle 15 0 6686m 572m 53m S 1.1 0.9 61:16.95 oracle
21633 oracle 16 0 6566m 445m 235m S 3.7 0.7 30:22.60 oracle
21615 oracle 16 0 6550m 428m 220m S 3.7 0.7 29:36.74 oracle
16349 oracle RT 0 431m 284m 41m S 0.5 0.4 2:41.08 ocssd.bin
17891 root RT 0 139m 118m 40m S 0.5 0.2 41:08.19 osysmond
18154 root RT 0 182m 98m 43m S 0.0 0.2 10:02.40 ologgerd
12211 root 15 0 1432m 84m 14m S 0.0 0.1 17:57.80 java
Another method on Solaris is to do the following
prstat -s size 1 1
Example prstat output
www004:/# prstat -s size 1 1
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
420 nobody 139M 60M sleep 29 10 1:46:56 0.1% webservd/76
603 nobody 135M 59M sleep 29 10 5:33:18 0.1% webservd/96
339 root 134M 70M sleep 59 0 0:35:38 0.0% java/24
435 iplanet 132M 55M sleep 29 10 1:10:39 0.1% webservd/76
573 nobody 131M 53M sleep 29 10 0:24:32 0.0% webservd/76
588 nobody 130M 53M sleep 29 10 2:40:55 0.1% webservd/86
454 nobody 128M 51M sleep 29 10 0:09:01 0.0% webservd/76
489 iplanet 126M 49M sleep 29 10 0:00:13 0.0% webservd/74
405 root 119M 45M sleep 29 10 0:00:13 0.0% webservd/31
717 root 54M 46M sleep 59 0 2:31:27 0.2% agent/7
Keep in mind this is sorted by Size not RSS, if you need it by RSS use the rss key
www004:/# prstat -s rss 1 1
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
339 root 134M 70M sleep 59 0 0:35:39 0.1% java/24
420 nobody 139M 60M sleep 29 10 1:46:57 0.4% webservd/76
603 nobody 135M 59M sleep 29 10 5:33:19 0.5% webservd/96
435 iplanet 132M 55M sleep 29 10 1:10:39 0.0% webservd/76
573 nobody 131M 53M sleep 29 10 0:24:32 0.0% webservd/76
588 nobody 130M 53M sleep 29 10 2:40:55 0.0% webservd/86
454 nobody 128M 51M sleep 29 10 0:09:01 0.0% webservd/76
489 iplanet 126M 49M sleep 29 10 0:00:13 0.0% webservd/74
I'm not sure if ps is standardized but at least on linux, ps -o %mem gives the percentage of memory used (you would obviously want to add some other columns as well)