Multiple arithmetic operation in awk - shell

I have this awk function to do speed conversion.
function hum(x) {
split( "B/s KB/s MB/s GB/s TB/s", v); s=1
while( x>1000 ){ x/=1000; s++ }
printf( "%0.2f %s" , x , v[s] )
}
hum($1)
It works great if used standalone.
$ awk -f /tmp/test.awk <<< 1000000
1000.00 KB/s
$ awk -f /tmp/test.awk <<< 100000000
100.00 MB/s
Now, I want to convert bytes number to bit, to do that I need to multiple the number/input first by 8. I tried to modify the function a bit.
function hum(x) {
split( "B/s KB/s MB/s GB/s TB/s", v); s=1
while( x>1000 ){ res = x * 8 / 1000; s++ }
printf( "%0.2f %s\n" , res , v[s] )
}
hum($1)
But it hangs when I tried to execute it, I had to Ctrl-C to cancel the operation. Any idea what's wrong ?

Now, I want to convert bytes number to bit, to do that I need to multiple the number/input first by 8.
Then multiply the input value by 8 just once:
function hum(x) {
split("b/s Kb/s Mb/s Gb/s Tb/s", v); s=1
x *= 8
while(x > 1000) {x /= 1000; s++}
printf("%0.2f %s\n", x, v[s])
}
hum($1)

Instead of a loop, let's use log():
$ cat program.awk
function hum2(x, v,p) {
split( "b/s kb/s Mb/s Gb/s Tb/s", v) # remember to B -> b
x*=8 # to bits conversion
p=int(log(x)/log(1000)) # figure v slot
return sprintf("%0.2f %s" , x/1000^p , v[p+1] ) # p+1 due to 1 basedness
}
{
print $1,hum2($1) # input bytes, output bits
}
Take it for a spin:
$ for (( i=1 ; i<=10**14 ; i=i*10 )) ; do echo $i ; done | awk -f program.awk
Output:
1 8.00 b/s
10 80.00 b/s
100 800.00 b/s
1000 8.00 kb/s
10000 80.00 kb/s
100000 800.00 kb/s
1000000 8.00 Mb/s
10000000 80.00 Mb/s
100000000 800.00 Mb/s
1000000000 8.00 Gb/s
10000000000 80.00 Gb/s
100000000000 800.00 Gb/s
1000000000000 8.00 Tb/s
10000000000000 80.00 Tb/s
100000000000000 800.00 Tb/s

I wrote this one a while back that might not be exactly what you need but should be easily adaptable -
a function that takes in # of bytes as mandatory 1st parameter,
with optional 2nd param for calculating powers of 1000 instead of 1024
enter M m or 10 for 1000,
anything else, including blank, defaults to 1024
and auto formats it in the most logical range for human readability,
spanning all the way from kilo(kibi)-bit/s,
to YOTTA(yobi)-bit/s
*** extra credit for anyone who can figure out what's common among that list of numbers in my sample below.
——————————————————————— ———————————————————————
mawk/gawk '
function bytesformat(_, _______, __, ___, ____, _____, ______)
{
_____=__=(____^=___*=((____=___+=___^="")/___)+___+___)
___/=___
sub("^0+","",_)
____=_____-=substr(_____,index(_____,index(_____,!__)))*\
(_______ ~ "^(10|[Mm])$")
_______=length((____)____)
if ((____*__)<(_______*_)) {
do {
____ *= _____
++___
} while ((____*__)<(_______*_))
}
__=_
sub("(...)+$", ",&", __)
gsub("[^#-.][^#-.][^#-.]", "&,", __)
gsub("[,]*$|^[,]+", "", __)
sub("^[.]", "0&", __)
return \
sprintf(" %10.4f %sb/s (%42s byte%s) ",
_==""?+_:_/(_____^___)*_______,
substr("KMGTPEZY",___,___~___),
__==""?+__:__, (_~_)<_?"s":" ")
} { printf("%35s bytes ::: %s\n",
$1,
bytesformat($1, 10)) }'
6841 bytes ::: 54.7280 Kb/s ( 6,841 bytes)
15053 bytes ::: 120.4240 Kb/s ( 15,053 bytes)
23677 bytes ::: 189.4160 Kb/s ( 23,677 bytes)
32839 bytes ::: 262.7120 Kb/s ( 32,839 bytes)
42293 bytes ::: 338.3440 Kb/s ( 42,293 bytes)
52183 bytes ::: 417.4640 Kb/s ( 52,183 bytes)
62233 bytes ::: 497.8640 Kb/s ( 62,233 bytes)
72733 bytes ::: 581.8640 Kb/s ( 72,733 bytes)
83269 bytes ::: 666.1520 Kb/s ( 83,269 bytes)
138641 bytes ::: 1.1091 Mb/s ( 138,641 bytes)
149767 bytes ::: 1.1981 Mb/s ( 149,767 bytes)
162011 bytes ::: 1.2961 Mb/s ( 162,011 bytes)
174221 bytes ::: 1.3938 Mb/s ( 174,221 bytes)
186343 bytes ::: 1.4907 Mb/s ( 186,343 bytes)
199181 bytes ::: 1.5934 Mb/s ( 199,181 bytes)
211559 bytes ::: 1.6925 Mb/s ( 211,559 bytes)
224449 bytes ::: 1.7956 Mb/s ( 224,449 bytes)
237733 bytes ::: 1.9019 Mb/s ( 237,733 bytes)
128260807 bytes ::: 1.0261 Gb/s ( 128,260,807 bytes)
128565049 bytes ::: 1.0285 Gb/s ( 128,565,049 bytes)
128932561 bytes ::: 1.0315 Gb/s ( 128,932,561 bytes)
129304523 bytes ::: 1.0344 Gb/s ( 129,304,523 bytes)
129765859 bytes ::: 1.0381 Gb/s ( 129,765,859 bytes)
130111459 bytes ::: 1.0409 Gb/s ( 130,111,459 bytes)
130533133 bytes ::: 1.0443 Gb/s ( 130,533,133 bytes)
131012801 bytes ::: 1.0481 Gb/s ( 131,012,801 bytes)
131305043 bytes ::: 1.0504 Gb/s ( 131,305,043 bytes)
128004093619 bytes ::: 1.0240 Tb/s ( 128,004,093,619 bytes)
128026268633 bytes ::: 1.0242 Tb/s ( 128,026,268,633 bytes)
128056111093 bytes ::: 1.0244 Tb/s ( 128,056,111,093 bytes)
128071706179 bytes ::: 1.0246 Tb/s ( 128,071,706,179 bytes)
128082430067 bytes ::: 1.0247 Tb/s ( 128,082,430,067 bytes)
128102475287 bytes ::: 1.0248 Tb/s ( 128,102,475,287 bytes)
128115312811 bytes ::: 1.0249 Tb/s ( 128,115,312,811 bytes)
128157555781 bytes ::: 1.0253 Tb/s ( 128,157,555,781 bytes)
128175556181 bytes ::: 1.0254 Tb/s ( 128,175,556,181 bytes)
128004004377827 bytes ::: 1.0240 Pb/s ( 128,004,004,377,827 bytes)
128040044659991 bytes ::: 1.0243 Pb/s ( 128,040,044,659,991 bytes)
128074066014953 bytes ::: 1.0246 Pb/s ( 128,074,066,014,953 bytes)
128127783733093 bytes ::: 1.0250 Pb/s ( 128,127,783,733,093 bytes)
128177777757611 bytes ::: 1.0254 Pb/s ( 128,177,777,757,611 bytes)
128200131001829 bytes ::: 1.0256 Pb/s ( 128,200,131,001,829 bytes)
128221782218423 bytes ::: 1.0258 Pb/s ( 128,221,782,218,423 bytes)
128237784424429 bytes ::: 1.0259 Pb/s ( 128,237,784,424,429 bytes)
128262808216561 bytes ::: 1.0261 Pb/s ( 128,262,808,216,561 bytes)
128055360778053559 bytes ::: 1.0244 Eb/s ( 128,055,360,778,053,559 bytes)
128082834342828077 bytes ::: 1.0247 Eb/s ( 128,082,834,342,828,077 bytes)
128112814740831073 bytes ::: 1.0249 Eb/s ( 128,112,814,740,831,073 bytes)
128172605482718161 bytes ::: 1.0254 Eb/s ( 128,172,605,482,718,161 bytes)
128203333333333399 bytes ::: 1.0256 Eb/s ( 128,203,333,333,333,399 bytes)
128240343634404269 bytes ::: 1.0259 Eb/s ( 128,240,343,634,404,269 bytes)
128272818280928081 bytes ::: 1.0262 Eb/s ( 128,272,818,280,928,081 bytes)
128282816070718271 bytes ::: 1.0263 Eb/s ( 128,282,816,070,718,271 bytes)
128289494449498271 bytes ::: 1.0263 Eb/s ( 128,289,494,449,498,271 bytes)
128030578058078030329 bytes ::: 1.0242 Zb/s ( 128,030,578,058,078,030,329 bytes)
128172161171772727271 bytes ::: 1.0254 Zb/s ( 128,172,161,171,772,727,271 bytes)
128234814212823481421 bytes ::: 1.0259 Zb/s ( 128,234,814,212,823,481,421 bytes)
128282727262616060507 bytes ::: 1.0263 Zb/s ( 128,282,727,262,616,060,507 bytes)
128286164949865319531 bytes ::: 1.0263 Zb/s ( 128,286,164,949,865,319,531 bytes)
128372737272827373721 bytes ::: 1.0270 Zb/s ( 128,372,737,272,827,373,721 bytes)
128393838393839382839 bytes ::: 1.0272 Zb/s ( 128,393,838,393,839,382,839 bytes)
128505500051850550037 bytes ::: 1.0280 Zb/s ( 128,505,500,051,850,550,037 bytes)
128669659758768758857 bytes ::: 1.0294 Zb/s ( 128,669,659,758,768,758,857 bytes)
130000000000093999992023 bytes ::: 1.0400 Yb/s ( 130,000,000,000,093,999,992,023 bytes)
131111111311113111311131 bytes ::: 1.0489 Yb/s ( 131,111,111,311,113,111,311,131 bytes)
131111153353153531553111 bytes ::: 1.0489 Yb/s ( 131,111,153,353,153,531,553,111 bytes)
131111531315333335313531 bytes ::: 1.0489 Yb/s ( 131,111,531,315,333,335,313,531 bytes)
131113133333311333331111 bytes ::: 1.0489 Yb/s ( 131,113,133,333,311,333,331,111 bytes)
131113551355135511111111 bytes ::: 1.0489 Yb/s ( 131,113,551,355,135,511,111,111 bytes)
131131113131113131131111 bytes ::: 1.0490 Yb/s ( 131,131,113,131,113,131,131,111 bytes)
131131331133111331313111 bytes ::: 1.0491 Yb/s ( 131,131,331,133,111,331,313,111 bytes)
131133133131333313331311 bytes ::: 1.0491 Yb/s ( 131,133,133,131,333,313,331,311 bytes)

Any idea what's wrong ?
You have created infinte loop which cause it to hang, 1st code has
while( x>1000 ){ x/=1000; s++ }
here each turn decrease x 1000 times so it will finally be equal or lower 1000. 2nd code has
while( x>1000 ){ res = x * 8 / 1000; s++ }
here each turn x is used to compute res but value of x itself never changes, thus you will get either 0 executions of while body or +infinity executions of while body.
Note that But it hangs when I tried to execute it might be observed also in languages other than GNU AWK which do support while construct when you do not change value of variable on which truthiness of condition depends.

Related

bash process substitution with tee dropping data

I expect to see 1MiB in the wc's character count (which works at the bottom when 'tee' is writing to a file rather than a process substitution).
Is there something I'm missing about using process substitution with tee?
[192.168.20.40 (02b4b472) ~ 18:52:39]# dd if=/dev/zero bs=1M count=1 | tee >(dd bs=1M count=1 | wc | sed 's/^/ /' > /dev/stderr) | wc
0+1 records in
0+1 records out
8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00022373 s, 36.6 MB/s
0 0 8192
0 0 81920
[192.168.20.40 (02b4b472) ~ 18:52:56]# dd if=/dev/zero bs=1M count=1 | tee /tmp/foo | wc
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416881 s, 25.2 MB/s
0 0 1048576
[192.168.20.40 (02b4b472) ~ 18:53:41]#
Using tee -p handles the stdout, but not the process substitution... which still comes up short.
[192.168.20.40 (02b4b472) ~ 19:17:05]# dd if=/dev/zero bs=1M count=1 | tee -p warn >(dd bs=1M count=1 | wc | sed 's/^/ /' > /dev/stderr) | wc
0+1 records in
0+1 records out
8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00084348 s, 9.7 MB/s
0 0 8192
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417175 s, 25.1 MB/s
0 0 1048576
[192.168.20.40 (02b4b472) ~ 19:18:25]#

Bash sed awk, format CPU/Mem info from /proc/cpuinfo and /proc/meminfo [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 2 years ago.
Improve this question
The problem that I'm trying to solve is to produce portable output that I can display on all of the servers in our environment to show basic info at login using generic information on all CentOS / Red Hat systems. I would like to pluck info from /proc/cpuinfo and /proc/meminfo (or free -m -h); "why not just 'yum install some-great-tool'?" is not ideal as all of this information is freely available to us right in /proc. I know that this sort of thing can often be a very simple trick for sed/awk experts (I don't know how to approach this
with my limited sed/awk knowledge).
I would like to extract something like the following on a single line:
<model name>, <cpu MHz> MHz, <cpu cores> cores, <detect "vmx" (Intel-VT) or "svm" (AMD-V support)>
e.g. with the below output, this would look like (with "1300.000" rounded to "1300")
"AMD Athlon(tm) II Neo N36L Dual-Core Processor, 1300 MHz, 2 cores, VMX-Virtualization" (or "SVM-Virtualization" or "No Virtualization")
I would like to also combine this info with that of /proc/meminfo or free -mh, so:
"AMD Athlon(tm) II Neo N36L Dual-Core Processor, 1300 MHz, 2 cores, 4.7 GB Memory (1.8 GB Free), SVM-Virtualization"
I have spent some time searching for methods, but without luck, and maybe this is an interesting generic problem as involves taking the format of tables that a lot of info is held in and extracting as required so has some generic application.
$ free -m -h
total used free shared buff/cache available
Mem: 4.5Gi 1.2Gi 1.8Gi 77Mi 1.6Gi 3.0Gi
Swap: 4.8Gi 0B 4.8Gi
$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 16
model : 6
model name : AMD Athlon(tm) II Neo N36L Dual-Core Processor
stepping : 3
microcode : 0x10000c8
cpu MHz : 1300.000
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate vmmcall npt lbrv svm_lock nrip_save
bugs : tlb_mmatch apic_c1e fxsave_leak sysret_ss_attrs null_seg amd_e400 spectre_v1 spectre_v2
bogomips : 2595.59
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate
$ cat /proc/meminfo
MemTotal: 4771304 kB
MemFree: 1862372 kB
MemAvailable: 3195768 kB
Buffers: 2628 kB
Cached: 1542788 kB
SwapCached: 0 kB
Active: 1534572 kB
Inactive: 909316 kB
Active(anon): 917792 kB
Inactive(anon): 62468 kB
Active(file): 616780 kB
Inactive(file): 846848 kB
Unevictable: 8384 kB
Mlocked: 0 kB
SwapTotal: 5070844 kB
SwapFree: 5070844 kB
Dirty: 20 kB
Writeback: 0 kB
AnonPages: 881304 kB
Mapped: 395420 kB
Shmem: 79776 kB
KReclaimable: 152892 kB
Slab: 295508 kB
SReclaimable: 152892 kB
SUnreclaim: 142616 kB
KernelStack: 9328 kB
PageTables: 45156 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 7456496 kB
Committed_AS: 5260708 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 2864 kB
HardwareCorrupted: 0 kB
AnonHugePages: 417792 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 314944 kB
DirectMap2M: 4796416 kB
DirectMap1G: 0 kB
Using /proc/cpuinfo and free -mh along with awk, search for the strings required, using : as the field delimited, set variables accordingly, splitting the output of free -mh further into an array called arr based on " " as the delimiter. At the end, print the data in the required format using the variables created.
When searching for lines beginning with flag, we search for strings svn or vmx using awk's match function. A match will signified by the RSTART variable not being 0 and so we check this to find the type of virtualisatiion being utilised. As we have set virt to No Virtualisation at the beginning, no matches will print No Virtualisation.
awk -F: '/^model name/ {
mod=$2
}
/^cpu MHz/ {
mhz=$2
}
/^cpu core/ {
core=$2
}
/^flags/ {
virt="No Virtualisation";
match($0,"svm");
if (RSTART!=0)
{
virt="SVM-Virtualisation"
};
match($0,"vmx");
if (RSTART!=0) {
virt="VMX-Virtualisation"
}
}
/^Mem:/ {
split($2,arr," ");
tot=arr[1];
free=arr[2]
}
END {
printf "%s %dMHz %s core(s) %s %sB Memory (%sB Free)\n",mod,mhz,core,virt,tot,free
}' /proc/cpuinfo <(free -mh)
One liner:
awk -F: '/^model name/ { mod=$2 } /^cpu MHz/ { mhz=$2 } /^cpu core/ {core=$2} /^flags/ { virt="No Virtualisation";match($0,"svm");if (RSTART!=0) { virt="SVM-Virtualisation" };match($0,"vmx");if (RSTART!=0) { virt="VMX-Virtualisation" } } /^Mem:/ {split($2,arr," ");tot=arr[1];free=arr[2]} END { printf "%s %dMHz %s core(s) %s %sB Memory (%sB Free)\n",mod,mhz,core,virt,tot,free }' /proc/cpuinfo <(free -mh)

Memory is not even getting freed (back to OS)

I'm running a server written in Go and the RSS is very high and the memory is not even getting freed (back to OS).
I used pprof to check but it seems that there is no memory leak.
I also tried:
GODEBUG=madvdontneed=1 ./memorytest
Please tell me how to use madvdontneed.
OS: CentOS 7 (Linux)
Arch: amd64
Go version: 1.14.2
Code:
package main
import (
"fmt"
"os"
"runtime"
"time"
)
var garr []string
var chn chan int
func main() {
chn = make(chan int, 1)
go Alloc()
<-chn
}
func Alloc() {
for {
arr := make([]string, 100000000)
//copy(garr,arr)
garr = arr
print_heap_info()
time.Sleep(5 * time.Second)
}
}
func print_heap_info() {
var m runtime.MemStats
runtime.ReadMemStats(&m)
fmt.Printf("env:%v, heapsys:%d,heapalloc:%d,heapidel:%d,heapreleased:%d,heapinuse:%d\n",
os.Getenv("GODEBUG"), m.HeapSys, m.HeapAlloc, m.HeapIdle, m.HeapReleased, m.HeapInuse)
}
Output:
env:madvdontneed=1, heapsys:1677262848,heapalloc:1600074304,heapidel:76955648,heapreleased:76914688,heapinuse:1600307200
env:madvdontneed=1, heapsys:3287875584,heapalloc:3200081392,heapidel:87556096,heapreleased:87506944,heapinuse:3200319488
env:madvdontneed=1, heapsys:4898619392,heapalloc:4800086512,heapidel:98295808,heapreleased:98115584,heapinuse:4800323584
env:madvdontneed=1, heapsys:6509101056,heapalloc:6400090640,heapidel:108773376,heapreleased:108724224,heapinuse:6400327680
env:madvdontneed=1, heapsys:6509232128,heapalloc:4800086176,heapidel:1708908544,heapreleased:108724224,heapinuse:4800323584
env:madvdontneed=1, heapsys:6509002752,heapalloc:6400090304,heapidel:108675072,heapreleased:108560384,heapinuse:6400327680
env:madvdontneed=1, heapsys:6509199360,heapalloc:4800086712,heapidel:1708875776,heapreleased:108560384,heapinuse:4800323584
env:madvdontneed=1, heapsys:6509068288,heapalloc:6400090744,heapidel:108740608,heapreleased:108560384,heapinuse:6400327680
env:madvdontneed=1, heapsys:6509199360,heapalloc:4800086712,heapidel:1708875776,heapreleased:108560384,heapinuse:4800323584
env:madvdontneed=1, heapsys:6509068288,heapalloc:6400090840,heapidel:108740608,heapreleased:108462080,heapinuse:6400327680
Optimize your memory allocations to lower the peak.
See also: debug.FreeOSMemory()
FreeOSMemory forces a garbage collection followed by an attempt to return as much memory to the operating system as possible. (Even if this is not called, the runtime gradually returns memory to the operating system in a background task.)
Your formatted output is:
sys: 1599 MB alloc: 1525 MB idel: 73 MB released: 73 MB inuse: 1526 MB
sys: 3135 MB alloc: 3051 MB idel: 83 MB released: 83 MB inuse: 3052 MB
sys: 4671 MB alloc: 4577 MB idel: 93 MB released: 93 MB inuse: 4577 MB
sys: 6207 MB alloc: 6103 MB idel: 103 MB released: 103 MB inuse: 6103 MB
sys: 6207 MB alloc: 4577 MB idel: 1629 MB released: 103 MB inuse: 4577 MB
sys: 6207 MB alloc: 6103 MB idel: 103 MB released: 103 MB inuse: 6103 MB
sys: 6207 MB alloc: 4577 MB idel: 1629 MB released: 103 MB inuse: 4577 MB
sys: 6207 MB alloc: 6103 MB idel: 103 MB released: 103 MB inuse: 6103 MB
sys: 6207 MB alloc: 4577 MB idel: 1629 MB released: 103 MB inuse: 4577 MB
sys: 6207 MB alloc: 6103 MB idel: 103 MB released: 103 MB inuse: 6103 MB
Which is (for the last line):
sys: bytes of heap memory obtained from the OS: 6207 MB
alloc: bytes of allocated heap objects: 6103 MB
idel: bytes in idle (unused) spans: 103 MB
released: bytes of physical memory returned to the OS: 103 MB
inuse: bytes in in-use spans: 6103 MB
It is not a memory leak.
Output for a system with total 8GB RAM (system memory monitor):
command:
GODEBUG=madvdontneed=1 go run .
Output:
env: madvdontneed=1, sys: 1087 MB, alloc: 1024 MB, idel: 63 MB, released: 63 MB, inuse: 1024 MB
env: madvdontneed=1, sys: 2111 MB, alloc: 2048 MB, idel: 63 MB, released: 63 MB, inuse: 2048 MB
env: madvdontneed=1, sys: 3135 MB, alloc: 3072 MB, idel: 63 MB, released: 63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel: 63 MB, released: 63 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released: 63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel: 63 MB, released: 63 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released: 63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel: 63 MB, released: 63 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released: 63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel: 63 MB, released: 63 MB, inuse: 4096 MB
Code:
package main
import (
"fmt"
"os"
"runtime"
"time"
)
func main() {
for i := 0; i < 10; i++ {
a = make([]byte, 1024*meg)
var m runtime.MemStats
runtime.ReadMemStats(&m)
fmt.Printf("env: %v, sys: %4d MB, alloc: %4d MB, idel: %4d MB, released: %4d MB, inuse: %4d MB\n",
os.Getenv("GODEBUG"), m.HeapSys/meg, m.HeapAlloc/meg, m.HeapIdle/meg, m.HeapReleased/meg, m.HeapInuse/meg)
time.Sleep(1 * time.Second)
}
}
var a []byte
const meg = 1024 * 1024
htop:
Output for vmstat 1 -S MB command, running two terminals:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 1161 6369 25 595 0 0 702 452 276 567 6 3 91 1 0
3 0 1160 6300 28 632 0 0 43432 0 6288 14013 13 5 82 1 0
1 0 1160 6243 28 634 0 0 528 0 6342 13008 11 3 85 0 0
1 0 1160 6177 28 634 0 0 56 56 979 2090 4 1 95 0 0
2 0 1160 6110 28 634 0 0 0 0 1061 2664 5 2 94 0 0
0 0 1160 6044 28 634 0 0 12 36 994 2233 4 1 94 0 0
0 0 1160 3991 28 634 0 0 32 188 1074 1787 5 6 89 0 0
2 0 1160 2120 28 634 0 0 0 136 1016 1634 4 5 90 0 0
1 0 1160 973 28 633 0 0 0 4 1077 1660 4 5 92 0 0
2 0 1160 143 25 390 0 0 1780 356 1849 2341 4 9 87 0 0
0 9 1323 102 0 108 0 165 51964 169652 20277 21017 1 20 53 25 0
1 5 1510 99 0 129 2 189 54376 193996 57829 52152 1 16 56 27 0
1 5 1794 99 0 129 1 286 10068 293856 81160 59511 0 16 77 6 0
4 5 2047 101 0 98 5 257 21236 263292 69923 65485 0 23 54 23 0
1 2 1479 2867 0 217 43 23 233508 24380 24023 48536 4 27 47 22 0
2 0 1452 2824 0 232 27 0 43960 0 8168 21085 4 5 90 1 0
0 1 1443 2814 0 233 9 0 10956 0 3341 8468 5 2 93 0 0
3 0 1425 2796 0 231 18 0 19688 0 5780 15490 4 3 92 1 0
1 1 1420 2672 10 337 3 0 121628 1920 3292 7934 5 7 81 7 0
0 0 1394 2646 10 338 25 0 27360 0 7975 21555 3 5 92 0 0
0 1 1359 6856 10 339 1 0 2416 0 1035 2108 3 2 95 0 0
0 0 1353 6847 10 348 4 0 13660 0 1696 3471 4 1 95 0 0
Output for first GODEBUG=madvdontneed=1 go run . command (auto killed):
env: madvdontneed=1, sys: 1087 MB, alloc: 1024 MB, idel: 63 MB, released: 63 MB, inuse: 1024 MB
env: madvdontneed=1, sys: 2111 MB, alloc: 2048 MB, idel: 63 MB, released: 63 MB, inuse: 2048 MB
env: madvdontneed=1, sys: 3135 MB, alloc: 3072 MB, idel: 63 MB, released: 63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel: 63 MB, released: 63 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released: 63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel: 63 MB, released: 63 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released: 63 MB, inuse: 3072 MB
signal: killed
Output for second GODEBUG=madvdontneed=1 go run . command:
env: madvdontneed=1, sys: 1087 MB, alloc: 1024 MB, idel: 63 MB, released: 63 MB, inuse: 1024 MB
env: madvdontneed=1, sys: 2111 MB, alloc: 2048 MB, idel: 63 MB, released: 63 MB, inuse: 2048 MB
env: madvdontneed=1, sys: 3135 MB, alloc: 3072 MB, idel: 63 MB, released: 63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel: 63 MB, released: 63 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released: 63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel: 63 MB, released: 63 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released: 63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel: 63 MB, released: 62 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released: 62 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel: 63 MB, released: 62 MB, inuse: 4096 MB
Code:
package main
import (
"fmt"
"os"
"runtime"
"time"
)
func main() {
for i := 0; i < 10; i++ {
a = make([]byte, 1024*meg)
var m runtime.MemStats
runtime.ReadMemStats(&m)
fmt.Printf("env: %v, sys: %4d MB, alloc: %4d MB, idel: %4d MB, released: %4d MB, inuse: %4d MB\n\n",
os.Getenv("GODEBUG"), m.HeapSys/meg, m.HeapAlloc/meg, m.HeapIdle/meg, m.HeapReleased/meg, m.HeapInuse/meg)
time.Sleep(1 * time.Second)
}
}
var a []byte
const meg = 1024 * 1024
Command:
GODEBUG=gctrace=1 go run .
Output:
gc 1 #0.008s 2%: 0.071+0.67+0.034 ms clock, 0.57+0.88/0.76/0.041+0.27 ms cpu, 4->4->0 MB, 5 MB goal, 8 P
gc 2 #0.014s 3%: 0.069+1.3+0.027 ms clock, 0.55+0.48/0.80/0.73+0.21 ms cpu, 4->4->0 MB, 5 MB goal, 8 P
gc 3 #0.029s 2%: 0.056+0.62+0.040 ms clock, 0.45+0.39/0.75/0.67+0.32 ms cpu, 4->4->0 MB, 5 MB goal, 8 P
gc 4 #0.045s 3%: 0.31+0.97+0.12 ms clock, 2.5+0.91/1.2/0.92+1.0 ms cpu, 4->4->0 MB, 5 MB goal, 8 P
gc 5 #0.060s 2%: 0.056+0.60+0.019 ms clock, 0.45+0.44/0.70/1.1+0.15 ms cpu, 4->4->0 MB, 5 MB goal, 8 P
gc 6 #0.071s 2%: 0.025+1.0+0.018 ms clock, 0.20+0.47/1.1/3.5+0.15 ms cpu, 4->4->0 MB, 5 MB goal, 8 P
gc 7 #0.084s 2%: 0.12+0.88+0.042 ms clock, 0.97+0.77/1.2/0.88+0.33 ms cpu, 4->4->1 MB, 5 MB goal, 8 P
gc 8 #0.093s 2%: 0.039+0.83+0.028 ms clock, 0.31+0.38/0.66/1.2+0.22 ms cpu, 4->4->0 MB, 5 MB goal, 8 P
gc 1 #0.007s 3%: 0.013+1.7+0.005 ms clock, 0.11+0.86/2.0/1.2+0.042 ms cpu, 4->5->4 MB, 5 MB goal, 8 P
gc 1 #0.002s 5%: 0.024+2.0+0.042 ms clock, 0.19+0.25/1.5/1.3+0.34 ms cpu, 4->6->5 MB, 5 MB goal, 8 P
gc 2 #0.017s 3%: 0.014+4.6+0.044 ms clock, 0.11+0.13/3.3/1.9+0.35 ms cpu, 9->10->7 MB, 10 MB goal, 8 P
gc 3 #0.058s 2%: 0.030+6.7+0.036 ms clock, 0.24+0.12/5.3/1.8+0.29 ms cpu, 13->15->10 MB, 15 MB goal, 8 P
gc 4 #0.100s 2%: 0.034+5.6+0.015 ms clock, 0.27+0/7.2/0.65+0.12 ms cpu, 18->18->12 MB, 21 MB goal, 8 P
gc env: gctrace=1, sys: 1087 MB, alloc: 1024 MB, idel: 63 MB, released: 63 MB, inuse: 1024 MB
1 #0.024s 0%: 0.028+0.41+0.016 ms clock, 0.22+0.11/0.14/0.093+0.13 ms cpu, 1024->1024->1024 MB, 1025 MB goal, 8 P
env: gctrace=1, sys: 2111 MB, alloc: 2048 MB, idel: 63 MB, released: 63 MB, inuse: 2048 MB
gc 2 #1.049s 0%: 0.021+0.44+0.005 ms clock, 0.16+0.12/0.15/0.12+0.045 ms cpu, 2048->2048->2048 MB, 2049 MB goal, 8 P
env: gctrace=1, sys: 3135 MB, alloc: 3072 MB, idel: 63 MB, released: 63 MB, inuse: 3072 MB
env: gctrace=1, sys: 4159 MB, alloc: 4096 MB, idel: 63 MB, released: 63 MB, inuse: 4096 MB
gc 3 #3.096s 0%: 0.023+0.56+0.017 ms clock, 0.18+0.13/0.20/0.17+0.13 ms cpu, 4096->4096->2048 MB, 4097 MB goal, 8 P
env: gctrace=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released: 63 MB, inuse: 3072 MB
env: gctrace=1, sys: 4159 MB, alloc: 4096 MB, idel: 63 MB, released: 63 MB, inuse: 4096 MB
gc 4 #5.619s 0%: 0.023+0.31+0.018 ms clock, 0.18+0.18/0.18/0.22+0.15 ms cpu, 4096->4096->2048 MB, 4097 MB goal, 8 P
env: gctrace=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released: 63 MB, inuse: 3072 MB
env: gctrace=1, sys: 4159 MB, alloc: 4096 MB, idel: 63 MB, released: 63 MB, inuse: 4096 MB
gc 5 #8.141s 0%: 0.025+0.26+0.004 ms clock, 0.20+0.16/0.15/0.15+0.033 ms cpu, 4096->4096->2048 MB, 4097 MB goal, 8 P
env: gctrace=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released: 63 MB, inuse: 3072 MB
gc 6 #10.413s 0%: 0.028+0.51+0.013 ms clock, 0.22+0.23/0.28/0.29+0.11 ms cpu, 4096->4096->2048 MB, 4097 MB goal, 8 P
env: gctrace=1, sys: 4159 MB, alloc: 4096 MB, idel: 63 MB, released: 63 MB, inuse: 4096 MB
The format of this line is subject to change. Currently, it is:
gc # ##s #%: #+#+# ms clock, #+#/#/#+# ms cpu, #->#-># MB, # MB goal, # P
where the fields are as follows:
gc # the GC number, incremented at each GC
##s time in seconds since program start
#% percentage of time spent in GC since program start
#+...+# wall-clock/CPU times for the phases of the GC
#->#-># MB heap size at GC start, at GC end, and live heap
# MB goal goal heap size
# P number of processors used
Note:
go version go1.15.5 linux/amd64
See also:
Go 1.13 RSS keeps on increasing, suspected scavenging issue
https://github.com/golang/go/issues/36398
https://github.com/golang/go/issues/39295
https://go.googlesource.com/proposal/+/master/design/14951-soft-heap-limit.md
https://docs.google.com/document/d/1wmjrocXIWTr1JxU-3EQBI6BK6KgtiFArkG47XK73xIQ/edit#
https://blog.cloudflare.com/go-dont-collect-my-garbage/

Is it "growpart" or "resize2fs" for these new c5 instanced?

$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 4G 0 loop /var/tmp
nvme0n1 259:0 0 500G 0 disk
├─nvme0n1p1 259:1 0 1M 0 part
└─nvme0n1p2 259:2 0 300G 0 part /
$ sudo fdisk -l /dev/nvme0n1p2
Disk /dev/nvme0n1p2: 322.1 GB, 322120433152 bytes, 629141471 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Thanks!

Oracle 11gr2 failed check of kernel parameters on hp-ux

I'm installing oracle 11gR2 on 64 bit itanium HP-UX (v 11.31) system ( for HP Operation Manager 9 ).
According with the installation requiremens, I've changed the kernel parameters but when I start the installation process it don't recognize them.
Below the parameters that I've set :
Parameter ( Manual) (on server)
-------------------------------------------------------------
fs_async 0 0
ksi_alloc_max (nproc*8) 10240*8 = 81920
executable_stack 0 0
max_thread_proc 1024 3003
maxdsiz 0x40000000 (1073741824) 2063835136
maxdsiz_64bit 0x80000000 (2147483648) 2147483648
maxfiles 256 (a) 4096
maxssiz 0x8000000 (134217728) 134217728
maxssiz_64bit 0x40000000 (1073741824) 1073741824
maxuprc ((nproc*9)/10) 9216
msgmni (nproc) 10240
msgtql (nproc) 32000
ncsize 35840 95120
nflocks (nproc) 10240
ninode (8*nproc+2048) 83968
nkthread (((nproc*7)/4)+16) 17936
nproc 4096 10240
semmni (nproc) 10240
semmns (semmni*2) 20480
semmnu (nproc-4) 10236
semvmx 32767 65535
shmmax size of memory or 0x40000000 (higher one) 1073741824
shmmni 4096 4096
shmseg 512 1024
vps_ceiling 64 64
if this can help:
[root#HUG30055 /] # swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol2
dev 8388608 0 8388608 0% 0 - 1 /dev/vg00/lvol10
reserve - 742156 -742156
memory 7972944 3011808 4961136 38%
[root#HUG30055 /] # bdf /tmp
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol6 4194304 1773864 2401576 42% /tmp

Resources