Metric to compute and compare RAM and CPU usage in Android - performance

I have two versions of Android that I'm testing on a device, both differ in few services in AOSP. I wanted to take some sort of a bench mark test to see how much the new system services are consuming my RAM and CPU usage. I'm not sure how to begin with. Are there any existing benchmarks that achieve the same? If not how should I proceed in this case.
Any help is appreciated

You could run some of my CPU and memory benchmarks (free, no adds) via
http://www.roylongbottom.org.uk/android%20benchmarks.htm
Also, you could install Android Terminal Emulator from Google Play and use Linux commands for top or vmstat to monitor resources used (at least they work on my tablet).
Below are details from vmstat whilst running one of my benchmarks.
Command vmstat -d 1 -n 60 > /mnt/extsd/vmst3.txt
For 60 samples of one second to txt file on external SD card
via path as identified by the df command
Command cat /mnt/extsd/vmst3.txt to display details
Start this, switch to main display, execute MemSpeed.apk that uses up
to 65 MB over nearly 11 seconds (starts in 16 seconds after a bit of
fiddling that uses some system CPU time). Note free memory reduction
and 99% CPU utilisation for 10+ seconds. Idle time is clearly wrong.
procs memory system cpu
r b free mapped anon slab in cs flt us ni sy id wa ir
0 0 909092 124948 197812 17600 365 554 0 3 0 2 99 0 0
1 0 906052 124704 200544 17600 2176 3162 0 21 0 27 99 0 0
0 0 905968 125700 200660 17596 1175 4000 0 15 0 21 99 0 0
0 0 905968 125716 200672 17596 286 397 0 1 0 2 99 0 0
0 0 905968 125716 200672 17596 1528 1226 0 5 0 6 99 0 0
0 0 905160 124944 201276 17596 1533 4771 0 30 0 31 99 0 0
0 0 905224 124944 201288 17596 349 460 0 3 0 4 99 0 0
0 0 904588 125440 201984 17596 2721 6542 0 32 0 30 99 0 0
0 0 903844 124964 202820 17596 2266 3569 0 25 0 17 99 0 0
3 0 903720 126456 202860 17596 1924 6185 0 37 0 37 99 0 0
1 0 897768 126504 202228 17596 2281 3661 0 29 0 20 99 0 0
0 0 894296 134684 202308 17596 1437 4258 0 18 0 21 99 0 0
0 0 894296 134700 202308 17596 521 631 0 4 0 7 99 0 0
1 0 894296 134700 202308 17596 300 420 0 1 0 3 99 0 0
0 0 894296 134700 202308 17596 320 417 0 0 0 4 99 0 0
2 0 846820 142512 205780 17596 2273 3369 0 38 0 37 99 0 0
0 0 846764 142492 205836 17616 1044 3101 0 23 0 21 99 0 0
1 0 838960 142492 205804 17616 1661 974 0 49 0 8 99 0 0
1 0 838960 142500 205908 17616 540 464 0 99 0 0 99 0 0
1 0 838836 142504 205924 17616 515 444 0 99 0 0 98 0 0
procs memory system cpu
r b free mapped anon slab in cs flt us ni sy id wa ir
1 0 838836 142504 205960 17616 521 442 0 99 0 1 99 0 0
1 0 838712 142504 206140 17616 458 348 0 99 0 0 99 0 0
1 0 838464 142504 206392 17616 479 373 0 99 0 0 99 0 0
1 0 837968 142504 206900 17616 598 646 0 99 0 1 99 0 0
1 0 834992 142504 209900 17616 688 772 0 99 0 2 96 0 0
1 0 823088 142504 221928 17616 500 484 0 98 0 4 97 0 0
1 0 774852 142504 269888 17616 447 366 0 95 0 6 97 0 0
1 0 774852 142504 269900 17616 391 315 0 99 0 1 98 0 0
0 0 774852 142504 269900 17616 401 477 0 51 0 4 99 0 0
0 0 774728 142504 269932 17616 231 240 0 0 0 1 99 0 0
0 0 774728 142504 269932 17616 219 239 0 0 0 0 99 0 0
0 0 774728 142504 269932 17616 214 243 0 0 0 1 99 0 0
0 0 774728 142504 269932 17616 196 222 0 0 0 0 99 0 0
0 0 797028 144488 270676 17616 2621 6371 0 40 1 35 99 0 0
0 0 797092 144560 270768 17612 466 801 0 1 0 7 99 0 0
1 0 796564 146172 271020 17648 1548 1657 0 14 0 7 99 0 0
1 0 796500 144044 271088 17648 1157 2890 0 10 0 12 99 0 0
0 0 796564 144044 271096 17648 367 503 0 2 0 4 99 0 0
0 0 796564 144044 271096 17648 343 438 0 2 0 4 99 0 0
0 0 796564 144044 271096 17648 275 348 0 3 0 3 99 0 0
procs memory system cpu
r b free mapped anon slab in cs flt us ni sy id wa ir
0 0 796564 144044 271096 17648 227 274 0 0 0 1 99 0 0
0 0 796564 144044 271096 17648 300 363 0 2 0 2 99 0 0
0 0 796564 144044 271096 17648 273 344 0 1 0 1 99 0 0
0 0 796564 144044 271096 17648 238 284 0 0 0 2 99 0 0
0 0 796564 144044 271096 17648 311 344 0 3 0 5 99 0 0
1 0 796564 144044 271096 17648 246 289 0 1 0 0 99 0 0
1 0 796564 144044 271096 17648 263 322 0 2 0 2 99 0 0
0 0 796564 144044 271096 17648 275 335 0 2 0 2 99 0 0
0 0 796564 144044 271096 17648 442 460 0 1 0 4 99 0 0
0 0 796564 144044 271096 17648 451 417 0 3 0 7 99 0 0
0 0 796564 144044 271096 17648 284 345 0 2 0 1 99 0 0
0 0 796564 144044 271096 17648 310 375 0 1 0 2 99 0 0
0 0 796564 144044 271096 17648 278 339 0 1 0 2 99 0 0
0 0 796564 144044 271100 17648 282 350 0 2 0 1 99 0 0
0 0 796564 144044 271100 17648 242 284 0 2 0 2 99 0 0
0 0 796564 144044 271100 17648 293 352 0 1 0 2 99 0 0
0 0 796564 144044 271100 17648 343 511 0 8 0 2 99 0 0
0 0 796564 144044 271120 17648 198 221 0 0 0 1 99 0 0
0 0 796564 144044 271120 17648 227 263 0 0 0 0 99 0 0
0 0 796564 144044 271120 17648 230 246 0 1 0 2 99 0 0

Related

AWK Formatting Using First Row as a Header and Iterating by column

I'm struggling trying to format a collectd ploted file si I can later import it to an influx db instance.
This is how the file looks like:
#Date Time [CPU]User% [CPU]Nice% [CPU]Sys% [CPU]Wait% [CPU]Irq% [CPU]Soft% [CPU]Steal% [CPU]Idle% [CPU]Totl% [CPU]Intrpt/sec [CPU]Ctx/sec [CPU]Proc/sec [CPU]ProcQue [CPU]ProcRun [CPU]L-Avg1 [CPU]L-Avg5 [CPU]L-Avg15 [CPU]RunTot [CPU]BlkTot [MEM]Tot [MEM]Used [MEM]Free [MEM]Shared [MEM]Buf [MEM]Cached [MEM]Slab [MEM]Map [MEM]Anon [MEM]Commit [MEM]Locked [MEM]SwapTot [MEM]SwapUsed [MEM]SwapFree [MEM]SwapIn [MEM]SwapOut [MEM]Dirty [MEM]Clean [MEM]Laundry [MEM]Inactive [MEM]PageIn [MEM]PageOut [MEM]PageFaults [MEM]PageMajFaults [MEM]HugeTotal [MEM]HugeFree [MEM]HugeRsvd [MEM]SUnreclaim [SOCK]Used [SOCK]Tcp [SOCK]Orph [SOCK]Tw [SOCK]Alloc [SOCK]Mem [SOCK]Udp [SOCK]Raw [SOCK]Frag [SOCK]FragMem [NET]RxPktTot [NET]TxPktTot [NET]RxKBTot [NET]TxKBTot [NET]RxCmpTot [NET]RxMltTot [NET]TxCmpTot [NET]RxErrsTot [NET]TxErrsTot [DSK]ReadTot [DSK]WriteTot [DSK]OpsTot [DSK]ReadKBTot [DSK]WriteKBTot [DSK]KbTot [DSK]ReadMrgTot [DSK]WriteMrgTot [DSK]MrgTot [INODE]NumDentry [INODE]openFiles [INODE]MaxFile% [INODE]used [NFS]ReadsS [NFS]WritesS [NFS]MetaS [NFS]CommitS [NFS]Udp [NFS]Tcp [NFS]TcpConn [NFS]BadAuth [NFS]BadClient [NFS]ReadsC [NFS]WritesC [NFS]MetaC [NFS]CommitC [NFS]Retrans [NFS]AuthRef [TCP]IpErr [TCP]TcpErr [TCP]UdpErr [TCP]IcmpErr [TCP]Loss [TCP]FTrans [BUD]1Page [BUD]2Pages [BUD]4Pages [BUD]8Pages [BUD]16Pages [BUD]32Pages [BUD]64Pages [BUD]128Pages [BUD]256Pages [BUD]512Pages [BUD]1024Pages
20190228 00:01:00 12 0 3 0 0 1 0 84 16 26957 20219 14 2991 3 0.05 0.18 0.13 1 0 198339428 197144012 1195416 0 817844 34053472 1960600 76668 158641184 201414800 0 17825788 0 17825788 0 0 224 0 0 19111168 3 110 4088 0 0 0 0 94716 2885 44 0 5 1982 1808 0 0 0 0 9739 9767 30385 17320 0 0 0 0 0 0 12 13 3 110 113 0 16 16 635592 7488 0 476716 0 0 0 0 0 0 0 0 0 0 0 8 0 0 22 0 1 0 0 0 0 48963 10707 10980 1226 496 282 142 43 19 6 132
20190228 00:02:00 11 0 3 0 0 1 0 85 15 26062 18226 5 2988 3 0.02 0.14 0.12 2 0 198339428 197138128 1201300 0 817856 34054692 1960244 75468 158636064 201398036 0 17825788 0 17825788 0 0 220 0 0 19111524 0 81 960 0 0 0 0 94420 2867 42 0 7 1973 1842 0 0 0 0 9391 9405 28934 16605 0 0 0 0 0 0 9 9 0 81 81 0 11 11 635446 7232 0 476576 0 0 0 0 0 0 0 0 0 0 0 3 0 0 8 0 1 0 0 0 0 49798 10849 10995 1241 499 282 142 43 19 6 132
20190228 00:03:00 11 0 3 0 0 1 0 85 15 25750 17963 4 2980 0 0.00 0.11 0.10 2 0 198339428 197137468 1201960 0 817856 34056400 1960312 75468 158633880 201397832 0 17825788 0 17825788 0 0 320 0 0 19111712 0 75 668 0 0 0 0 94488 2869 42 0 5 1975 1916 0 0 0 0 9230 9242 28411 16243 0 0 0 0 0 0 9 9 0 75 75 0 10 10 635434 7232 0 476564 0 0 0 0 0 0 0 0 0 0 0 2 0 0 6 0 1 0 0 0 0 50029 10817 10998 1243 501 282 142 43 19 6 132
20190228 00:04:00 11 0 3 0 0 1 0 84 16 25755 17871 10 2981 5 0.08 0.11 0.10 3 0 198339428 197140864 1198564 0 817856 34058072 1960320 75468 158634508 201398088 0 17825788 0 17825788 0 0 232 0 0 19111980 0 79 2740 0 0 0 0 94488 2867 4 0 2 1973 1899 0 0 0 0 9191 9197 28247 16183 0 0 0 0 0 0 9 9 0 79 79 0 10 10 635433 7264 0 476563 0 0 0 0 0 0 0 0 0 0 0 5 0 0 12 0 1 0 0 0 0 49243 10842 10985 1245 501 282 142 43 19 6 132
20190228 00:05:00 12 0 4 0 0 1 0 83 17 26243 18319 76 2985 3 0.06 0.10 0.09 2 0 198339428 197148040 1191388 0 817856 34059808 1961420 75492 158637636 201405208 0 17825788 0 17825788 0 0 252 0 0 19112012 0 85 18686 0 0 0 0 95556 2884 43 0 6 1984 1945 0 0 0 0 9176 9173 28153 16029 0 0 0 0 0 0 10 10 0 85 85 0 12 12 635473 7328 0 476603 0 0 0 0 0 0 0 0 0 0 0 3 0 0 7 0 1 0 0 0 0 47625 10801 10979 1253 505 282 142 43 19 6 132
What I'm trying to do, is to get it in a format that looks like this:
cpu_value,host=mxspacr1,instance=5,type=cpu,type_instance=softirq value=180599 1551128614916131663
cpu_value,host=mxspacr1,instance=2,type=cpu,type_instance=interrupt value=752 1551128614916112943
cpu_value,host=mxspacr1,instance=4,type=cpu,type_instance=softirq value=205697 1551128614916128446
cpu_value,host=mxspacr1,instance=7,type=cpu,type_instance=nice value=19250943 1551128614916111618
cpu_value,host=mxspacr1,instance=2,type=cpu,type_instance=softirq value=160513 1551128614916127690
cpu_value,host=mxspacr1,instance=1,type=cpu,type_instance=softirq value=178677 1551128614916127265
cpu_value,host=mxspacr1,instance=0,type=cpu,type_instance=softirq value=212274 1551128614916126586
cpu_value,host=mxspacr1,instance=6,type=cpu,type_instance=interrupt value=673 1551128614916116661
cpu_value,host=mxspacr1,instance=4,type=cpu,type_instance=interrupt value=701 1551128614916115893
cpu_value,host=mxspacr1,instance=3,type=cpu,type_instance=interrupt value=723 1551128614916115492
cpu_value,host=mxspacr1,instance=1,type=cpu,type_instance=interrupt value=756 1551128614916112550
cpu_value,host=mxspacr1,instance=6,type=cpu,type_instance=nice value=21661921 1551128614916111032
cpu_value,host=mxspacr1,instance=3,type=cpu,type_instance=nice value=18494760 1551128614916098304
cpu_value,host=mxspacr1,instance=0,type=cpu,type_instance=interrupt value=552 1551
What I have managed to do so far is just to convert the date string into EPOCH format.
I was thinking somehow to use the first value "[CPU]" as the measurement, and the "User%" as the type, the host I can take it from the system where the script will run.
I would really appreciate your help, because I really basic knowledge of text editing.
Thanks.
EDIT: this is what would expect to get with the information of the second line using as a header the first row:
cpu_value,host=mxspacr1,type=cpu,type_instance=user% value=0 1551128614916131663
EDIT: This is what I have so far, and I'm stuck here.
awk -v HOSTNAME="$HOSTNAME" 'BEGIN { FS="[][]"; getline; NR==1; f1=$2; f2=$3 } { RS=" "; printf f1"_measurement,host="HOSTNAME",type="f2"value="$3" ", system("date +%s -d \""$1" "$2"\"") }' mxmcaim01-20190228.tab
And this is what I get, but this is only for 1 column, now I don't know how to process the remaining columns such as Nice, Sys, Wait and so on.
CPU_measurement,host=mxmcamon05,type=User% value= 1552014000
CPU_measurement,host=mxmcamon05,type=User% value= 1551960000
CPU_measurement,host=mxmcamon05,type=User% value= 1551343500
CPU_measurement,host=mxmcamon05,type=User% value= 1551997620
CPU_measurement,host=mxmcamon05,type=User% value= 1551985200
CPU_measurement,host=mxmcamon05,type=User% value= 1551938400
CPU_measurement,host=mxmcamon05,type=User% value= 1551949200
CPU_measurement,host=mxmcamon05,type=User% value= 1551938400
CPU_measurement,host=mxmcamon05,type=User% value= 1551938400
CPU_measurement,host=mxmcamon05,type=User% value= 1551945600
CPU_measurement,host=mxmcamon05,type=User% value= 1551938400
Please help.
EDIT. First of all, Thanks for your help.
Taking Advantage from you knowledge in text editing, I was expecting to use this for 3 separate files, but unfortunately and I don't know why the format is different, like this:
#Date Time SlabName ObjInUse ObjInUseB ObjAll ObjAllB SlabInUse SlabInUseB SlabAll SlabAllB SlabChg SlabPct
20190228 00:01:00 nfsd_drc 0 0 0 0 0 0 0 0 0 0
20190228 00:01:00 nfsd4_delegations 0 0 0 0 0 0 0 0 0 0
20190228 00:01:00 nfsd4_stateids 0 0 0 0 0 0 0 0 0 0
20190228 00:01:00 nfsd4_files 0 0 0 0 0 0 0 0 0 0
20190228 00:01:00 nfsd4_stateowners 0 0 0 0 0 0 0 0 0 0
20190228 00:01:00 nfs_direct_cache 0 0 0 0 0 0 0 0 0 0
So I don't how to handle the arrays in a way that I can use nfsd_drc as the type and then Iterate through ObjInUse ObjInUseB ObjAll ObjAllB SlabInUse SlabInUseB SlabAll SlabAllB SlabChg SlabPct and use them like the type_instance and finally the value in this case for ObjInUse will be 0, ObjInUseB = 0, ObjAll = 0, an so one, making something like this:
slab_value,host=mxspacr1,type=nfsd_drc,type_instance=ObjectInUse value=0 1551128614916131663
slab_value,host=mxspacr1,type=nfsd_drc,type_instance=ObjInuseB value=0 1551128614916131663
slab_value,host=mxspacr1,type=nfsd_drc,type_instance=ObjAll value=0 1551128614916112943
slab_value,host=mxspacr1,type=nfsd_drc,type_instance=ObjAllB value=0 1551128614916128446
slab_value,host=mxspacr1,type=nfsd_drc,type_instance=SlabInUse value=0 1551128614916111618
slab_value,host=mxspacr1,type=nfsd_drc,type_instance=SlabInUseB value=0 1551128614916127690
slab_value,host=mxspacr1,type=nfsd_drc,type_instance=SlabAll value=0 1551128614916127265
slab_value,host=mxspacr1,type=nfsd_drc,type_instance=SlabAllB value=0 1551128614916126586
slab_value,host=mxspacr1,type=nfsd_drc,type_instance=SlabChg value=0 1551128614916116661
slab_value,host=mxspacr1,type=nfsd_drc,type_instance=SlabPct value=0 1551128614916115893
slab_value is a hard-coded value.
Thanks.
It is not clear where do instance and type_instance=interrupt come from in your final desired format. Otherwise awk code below should work.
Note: it doesn't strip % from tag values and prints timestamp at end of line in seconds (append extra zeros if you want nanoseconds).
gawk -v HOSTNAME="$HOSTNAME" 'NR==1 {split($0,h,/[ \t\[\]]+/,s); for(i=0;i<length(h);i++){ h[i]=tolower(h[i]); };}; NR>1 { for(j=2;j<NF;j++) {k=2*j; printf("%s_value,host=%s,type=%s,type_instance=%s value=%s %s\n", h[k], HOSTNAME, h[k], h[k+1],$(j+1), mktime(substr($1,1,4)" "substr($1,5,2)" "substr($1,7,2)" "substr($2,1,2)" "substr($2,4,2)" "substr($2,7,2)));}}' mxmcaim01-20190228.tab

Extract column from file with shell [duplicate]

This question already has answers here:
bash: shortest way to get n-th column of output
(8 answers)
Closed 4 years ago.
I would like to extract column number 8 from the following table using shell (ash):
0xd024 2 0 32 20 3 0 1 0 2 1384 1692 -61 27694088
0xd028 0 1 5 11 1 0 46 0 0 301 187 -74 27689154
0xd02c 0 0 35 14 1 0 21 0 0 257 250 -80 27689410
0xd030 1 1 15 13 1 0 38 0 0 176 106 -91 27689666
0xd034 1 1 50 20 1 0 8 0 0 790 283 -71 27689980
0xd038 0 0 0 3 4 0 89 0 0 1633 390 -90 27690291
0xd03c 0 0 8 3 3 0 82 0 0 1837 184 -95 27690603
0xd040 0 0 4 5 1 0 90 0 0 0 148 -97 27690915
0xd064 0 0 36 9 1 0 29 0 0 321 111 -74 27691227
0xd068 0 0 5 14 14 0 40 0 0 8066 2270 -85 27691539
0xd06c 1 1 39 19 1 0 15 0 0 1342 261 -74 27691850
0xd070 0 0 12 11 1 0 53 0 0 203 174 -73 27692162
0xd074 0 0 18 2 1 0 75 0 0 301 277 -94 27692474
How can I do that?
the following command "awk '{print $8}' file" works fine

What image format are MNIST images?

I've unpacked the first image from the MNIST training set and I can access the (28,28) matrix.
[[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 3 18 18 18 126 136
175 26 166 255 247 127 0 0 0 0]
[ 0 0 0 0 0 0 0 0 30 36 94 154 170 253 253 253 253 253
225 172 253 242 195 64 0 0 0 0]
[ 0 0 0 0 0 0 0 49 238 253 253 253 253 253 253 253 253 251
93 82 82 56 39 0 0 0 0 0]
[ 0 0 0 0 0 0 0 18 219 253 253 253 253 253 198 182 247 241
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 80 156 107 253 253 205 11 0 43 154
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 14 1 154 253 90 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 139 253 190 2 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 11 190 253 70 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 35 241 225 160 108 1
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 81 240 253 253 119
25 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 45 186 253 253
150 27 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 93 252
253 187 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 249
253 249 64 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 46 130 183 253
253 207 2 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 39 148 229 253 253 253
250 182 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 24 114 221 253 253 253 253 201
78 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 23 66 213 253 253 253 253 198 81 2
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 18 171 219 253 253 253 253 195 80 9 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 55 172 226 253 253 253 253 244 133 11 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 136 253 253 253 212 135 132 16 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0]]
I want to do some image processing on it like converting to grayscale and then binarizing it (for machine learning), however I'm confused as to what kind of image format I'm dealing with. If this was a (28, 28, 3) it's obvious that it's an RGB image with 3 channels. However, this is a (28, 28) image with each pixel taking on a value in the discrete range [0, 255], which is rather odd. Is this image already in gray scale and do I just have to normalize the pixel values? What exactly does normalization entail? Do I multiply the flattened vector by the scalar 1/(sum of all energy values) ?
Thanks!
The images are 28x28 pixel grey-scale images with 8-bit quantization (hence the range [0-255]). The images were apparently binary black/white images but anti-aliasing during resizing caused them to have additional grey-scale values. See here for additional details.
Normally, you would normalize by dividing all values by 255 (not the sum of all pixel values).

Reading bytes from file into memory

I am using bufio to append byte arrays to a file. When I use cat on it I get:
[176 0 0 0 4 0 0 0 1 0 0 0 57 12 127 47 4 0 0 0 1 0 0 0 215 136 47 0 64 0 0 0 2 0 0 0 57 0 97 0 53 0 52 0 100 0 98 0 99 0 52 0 98 0 100 0 100 0 49 0 54 0 99 0 49 0 97 0 49 0 57 0 56 0 48 0 52 0 55 0 53 0 49 0 100 0 49 0 49 0 51 0 99 0 52 0 52 0 98 0 24 0 0 0 2 0 0 0 83 0 111 0 109 0 101 0 32 0 99 0 108 0 105 0 101 0 110 0 116 0 0 0 36 0 0 0 2 0 0 0 83 0 111 0 109 0 101 0 32 0 111 0 116 0 104 0 101 0 114 0 32 0 112 0 97 0 114 0 116 0 110 0 101 0 114 0][176 0 0 0 4 0 0 0 1 0 0 0 4 162 127 47 4 0 0 0 1 0 0 0 215 136 47 0 64 0 0 0 2 0 0 0 57 0 97 0 53 0 52 0 100 0 98 0 99 0 52 0 98 0 100 0 100 0 49 0 54 0 99 0 49 0 97 0 49 0 57 0 56 0 48 0 52 0 55 0 53 0 49 0 100 0 49 0 49 0 51 0 99 0 52 0 52 0 98 0 24 0 0 0 2 0 0 0 83 0 111 0 109 0 101 0 32 0 99 0 108 0 105 0 101 0 110 0 116 0 0 0 36 0 0 0 2 0 0 0 83 0 111 0 109 0 101 0 32 0 111 0 116 0 104 0 101 0 114 0 32 0 112 0 97 0 114 0 116 0 110 0 101 0 114 0][176 0 0 0 4 0 0 0 1 0 0 0 35 200 127 47 4 0 0 0 1 0 0 0 215 136 47 0 64 0 0 0 2 0 0 0 57 0 97 0 53 0 52 0 100 0 98 0 99 0 52 0 98 0 100 0 100 0 49 0 54 0 99 0 49 0 97 0 49 0 57 0 56 0 48 0 52 0 55 0 53 0 49 0 100 0 49 0 49 0 51 0 99 0 52 0 52 0 98 0 24 0 0 0 2 0 0 0 83 0 111 0 109 0 101 0 32 0 99 0 108 0 105 0 101 0 110 0 116 0 0 0 36 0 0 0 2 0 0 0 83 0 111 0 109 0 101 0 32 0 111 0 116 0 104 0 101 0 114 0 32 0 112 0 97 0 114 0 116 0 110 0 101 0 114 0]
So arrays of bytes. Now I want to read the file into memory, so I use:
content, err := ioutil.ReadFile(filename)
But when I display the content variable I get a completely different output than the contents of the file (and one array instead of many). What's wrong?
You wrote it wrong to the file. Its human readable format, should be binary.
My guess you used something like fmt.Fprintln(file, byte_slice) or variants.
It must be fmt.Fprintf(file, "%s", byte_slice) or similar.
Please, show us code how you write to the file.
So what think is happening is that the content inside of that file is a string. When you pull it out of the file it converts that string to bytes. What you need to do is convert those bytes to a string before you append to the file and then convert them back when you pull from the file
https://play.golang.org/p/PyBXAU9rVW

Determine whether matrix is sparse?

I have a matrix. I want to know it whether sparse or not. Is there any function in matlab to evaluate that property? I tried to used issparse function, but it always returns 0(not sparse). For example, my matrix (27 by 27) is
A=
[ 1 0 0 0 0 1 1 1 0 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0
1 1 0 0 0 0 1 0 1 0 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0
1 1 1 0 0 0 0 1 0 1 0 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0
0 1 1 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0
0 0 1 1 1 0 0 1 0 1 0 0 0 0 1 0 0 0 0 0 0 1 1 0 0 0 0
0 0 0 1 1 1 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 0 0 0
0 0 0 0 1 1 1 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 0 0
250 243 247 245 244 244 244 122 61 144 72 36 18 9 4 2 1 1 0 0 0 0 0 0 0 0 0
151 197 236 118 181 212 106 53 26 13 136 68 34 17 8 4 2 0 1 0 0 0 0 0 0 0 0
24 12 6 3 143 201 234 117 180 90 45 152 76 38 19 9 4 0 0 1 0 0 0 0 0 0 0
18 9 138 69 172 86 165 220 224 112 56 28 128 64 32 16 8 0 0 0 1 0 0 0 0 0 0
27 131 207 103 189 94 47 153 194 239 119 59 29 128 64 32 16 0 0 0 0 1 0 0 0 0 0
44 22 133 204 232 116 58 147 199 237 248 124 62 31 129 64 32 0 0 0 0 0 1 0 0 0 0
238 119 181 90 45 152 76 38 19 135 205 232 116 58 29 128 64 0 0 0 0 0 0 1 0 0 0
48 24 12 6 3 143 201 100 50 25 130 207 233 116 58 29 128 0 0 0 0 0 0 0 1 0 0
168 84 42 21 132 66 33 158 79 39 19 135 205 232 116 58 29 0 0 0 0 0 0 0 0 1 0
235 117 58 29 128 64 32 16 8 4 2 1 142 201 234 117 58 0 0 0 0 0 0 0 0 0 1
0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0
0 1 1 0 0 0 0 1 1 0 0 0 1 1 1 0 0 0 0 0 1 1 0 0 0 0 0
1 1 1 1 1 1 1 1 0 1 1 1 1 1 0 1 1 1 1 0 0 0 0 0 0 0 0
0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 1 0
0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 1
0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 1 0 0
0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 1 0 0 0 1
0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 1 0 0 0 0
0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 1 1 0 0 1 0 0 0 0 1 0
0 0 1 0 0 1 0 0 1 0 0 1 0 0 1 0 1 0 0 0 0 1 1 0 0 0 0]
This is figure for above matrix
This seemingly easy question is quite difficult to answer. There is actually no known standard that determines whether a matrix is sparse or full.
However, the most common measure I know is to measure a matrix's sparsity. This is simply the fraction of the total number of zeroes over the total number of elements. If this exceeds some sensible threshold, then you could say that the matrix is sparse.
If you're given the matrix A, perhaps something like this:
sparsity = (numel(A) - nnz(A)) / numel(A);
numel determines the total number of elements in the matrix A and nnz determines the total number of non-zero elements. Therefore, numel(A) - nnz(A) should give you the total number of zero elements.
So, going with the threshold idea, this is what I was talking about:
is_sparse = sparsity > tol;
tol would be a fraction from [0,1], so something like 0.75 could work. This would mean that if 75% of your matrix consisted of zeroes, this could be a sparse matrix. It's all heuristic though. Choose a threshold that you think makes the most sense.

Resources