My jvm options:
-verbose:gc -Xmx200M -Xmn40M -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseG1GC -XX:NewSize=40m -XX:MaxTenuringThreshold=1 -XX:-UseAdaptiveSizePolicy
The following is gc log:
Desired survivor size 2621440 bytes, new threshold 1 (max 1)
- age 1: 792 bytes, 792 total
, 0.0012861 secs]
[Parallel Time: 0.4 ms, GC Workers: 8]
[GC Worker Start (ms): Min: 120214.2, Avg: 120214.3, Max: 120214.3, Diff: 0.2]
[Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.2, Sum: 1.3]
[Update RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.3]
[Processed Buffers: Min: 0, Avg: 0.6, Max: 1, Diff: 1, Sum: 5]
[Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Object Copy (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.4]
[Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 8]
[GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[GC Worker Total (ms): Min: 0.2, Avg: 0.3, Max: 0.4, Diff: 0.2, Sum: 2.2]
[GC Worker End (ms): Min: 120214.5, Avg: 120214.5, Max: 120214.6, Diff: 0.0]
[Code Root Fixup: 0.0 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.1 ms]
[Other: 0.7 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 0.4 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.2 ms]
[Humongous Register: 0.0 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 0.0 ms]
[Eden: 39.0M(39.0M)->0.0B(39.0M) Survivors: 1024.0K->1024.0K Heap: 39.9M(200.0M)->670.6K(200.0M)]
[Times: user=0.00 sys=0.01, real=0.00 secs]
My question is why heap space(670.6K) is smaller than sum of(0+1024=1024) eden and survivors from the gc log [Eden: 39.0M(39.0M)->0.0B(39.0M) Survivors: 1024.0K->1024.0K Heap: 39.9M(200.0M)->670.6K(200.0M)]
Related
How do you convert a sparse matrix to a dense matrix in Julia? According to this I should be able to use full or Matrix, however full is evidently not standard in the SparseArrays module, and when I try to use Matrix:
I = []
J = []
A = []
for i in 1:3
push!(I, i)
push!(J, i^2)
push!(A, sqrt(i))
end
sarr = sparse(I, J, A, 10, 10)
arr = Matrix(sarr)
I get this error:
Exception has occurred: MethodError
MethodError: no method matching zero(::Type{Any})
It is enough to do collect(sarr) or Matrix(sarr).
Note, however that your code uses untyped containers which is not recommended. Indexes in arrays are Ints so it should be:
I = Int[]
J = Int[]
A = Float64[]
for i in 1:3
push!(I, i)
push!(J, i^2)
push!(A, sqrt(i))
end
sarr = sparse(I, J, A, 10, 10)
Now you can do:
julia> collect(sarr)
10×10 Matrix{Float64}:
1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 1.41421 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.73205 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
We are trying to improve our app performance and are doing a performance testing.
We have a Linux VM with 4 cores and 16 GB memory. The application has more then 100 users and are complaining of slowness.
Here are the performance tuning we have done so far
<heap size="8192m" max-size="8192m"/>
***<jvm-options>
<option value="-XX:+UseG1GC"/>
<option value="-XX:+UseStringDeduplication"/>
<option value="-verbose:gc"/>
<option value="-XX:+PrintGCDetails"/>
<option value="-XX:+PrintGCDateStamps"/>
<option value="-XX:+PrintGCTimeStamps"/>
<option value="-XX:+PrintGCApplicationStoppedTime"/>
<option value="-XX:+UseGCLogFileRotation"/>
<option value="-XX:NumberOfGCLogFiles=5"/>
<option value="-XX:GCLogFileSize=3M"/>
<option value="-XX:-TraceClassUnloading"/>
<option value="-XX:+HeapDumpOnOutOfMemoryError"/>***
we are seeing a 11 second garbage collection time as below
2021-09-14T18:43:27.186+1000: 14806.057: [GC pause (G1 Evacuation Pause) (young), 0.1189389 secs]
[Parallel Time: 71.8 ms, GC Workers: 4]
[GC Worker Start (ms): Min: 14806057.7, Avg: 14806057.7, Max: 14806057.7, Diff: 0.1]
[Ext Root Scanning (ms): Min: 5.4, Avg: 6.9, Max: 8.9, Diff: 3.5, Sum: 27.5]
[Update RS (ms): Min: 7.2, Avg: 9.0, Max: 9.7, Diff: 2.6, Sum: 35.8]
[Processed Buffers: Min: 30, Avg: 33.2, Max: 38, Diff: 8, Sum: 133]
[Scan RS (ms): Min: 0.3, Avg: 0.3, Max: 0.3, Diff: 0.1, Sum: 1.2]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.1]
[Object Copy (ms): Min: 54.7, Avg: 55.3, Max: 56.0, Diff: 1.4, Sum: 221.3]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
[GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.3]
[GC Worker Total (ms): Min: 71.5, Avg: 71.6, Max: 71.6, Diff: 0.1, Sum: 286.2]
[GC Worker End (ms): Min: 14806129.2, Avg: 14806129.3, Max: 14806129.3, Diff: 0.1]
[Code Root Fixup: 0.0 ms]
[Code Root Purge: 0.0 ms]
[String Dedup Fixup: 16.8 ms, GC Workers: 4]
[Queue Fixup (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.1]
[Table Fixup (ms): Min: 16.0, Avg: 16.3, Max: 16.6, Diff: 0.6, Sum: 65.3]
[Clear CT: 0.5 ms]
[Other: 29.8 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 26.7 ms]
[Ref Enq: 0.9 ms]
[Redirty Cards: 0.1 ms]
[Humongous Register: 0.2 ms]
[Humongous Reclaim: 0.1 ms]
[Free CSet: 1.3 ms]
[Eden: 4072.0M(4072.0M)->0.0B(4096.0M) Survivors: 98304.0K->86016.0K Heap: 7267.0M(8192.0M)->3187.9M(8192.0M)]
[Times: user=0.38 sys=0.00, real=0.12 secs]
How do we bring it down? we are planning to add 32 GB memory to the server and have the heap size (min and max at 20 GB)
The total GC time in the log is 0.1189389 secs, which is about 118.9 ms instead of 11 seconds.
The MaxGCPauseMillis parameter in G1 is used to control the maximum pause time. The default value is 200 ms. If you want to reduce the pause time, you can consider setting MaxGCPauseMillis to the desired value.
I am wondering why this matrix multiplication is not working in my Scala program, versus the result I am receiving when using Python. I am using the matrix multiplication algorithm described by this math: Matrix Multiplication where I have two matrices a = n x m and b = m x p. The code that I have written for this algorithm is (each matrix is a 2d array of doubles):
def dot(other: Matrix2D): Matrix2D ={
if (this.shape(1) != other.shape(0)){
throw new IndexOutOfBoundsException("Matrices were not the right shape! [" + this.shape(1) + " != " + other.shape(0) + "]")
}
val n = this.shape(1) //returns the number of columns, shape(0) returns number of rows
var a = matrix.clone()
var b = other.matrix.clone()
var c = Array.ofDim[Double](this.shape(0), other.shape(1))
for(i <- 0 until c.length){
for (j <- 0 until c(0).length){
for (k <- 0 until n){
c(i)(j) += a(i)(k) * b(k)(j)
}
}
}
Matrix2D(c)
}
The Input I put into both the Scala and Python code is:
a = [[1.0 1.0 1.0 1.0 0.0 0.0 0.0]
[1.0 1.0 0.0 1.0 0.0 0.0 0.0 ]
[1.0 1.0 1.0 1.0 1.0 1.0 1.0 ]
[1.0 0.0 0.0 0.0 1.0 1.0 1.0 ]
[1.0 0.0 0.0 0.0 1.0 0.0 1.0 ]
[1.0 0.0 0.0 0.0 0.0 0.0 0.0 ]]
b = [[0.0 0.0 0.0 ]
[0.0 -0.053430398509053074 0.021149859549078387 ]
[0.0 -0.010785871994186721 0.04942555653681449 ]
[0.0 0.04849323245519227 -0.0393881161667335 ]
[0.0 -0.03871752673999099 0.05228579488821056 ]
[0.0 0.07935206375269452 0.06511344235965408 ]
[0.0 -0.02462677123918247 1.723607966539059E-4 ]]
The output I receive from this function is:
[[0.0 -0.015723038048047533 0.031187299919159375]
[0.0 -0.0049371660538608045 -0.018238256617655116]
[0.0 2.84727725473527E-4 0.14875889796367792 ]
[0.0 0.01600776577352106 0.11757159804451854 ]
[0.0 -0.06334429797917346 0.05245815568486446 ]
[0.0 0.0 0.0 ]]
compared to python's numpy.dot algorithm:
[[ 0. -0.01572304 0.0311873 ]
[ 0. -0.00493717 -0.01823826]
[ 0. -0.01572304 0.0311873 ]
[ 0. 0.08912777 0.07801112]
[ 0. 0.00977571 0.01289768]
[ 0. 0.08912777 0.07801112]]
I am wondering why this algorithm doesn't completely fill the output algorithm that I need...I've been messing with the for loops and such and have not been able to figure out whats wrong.
Can you show your Python code?
I tried this in Numpy and get the same as your Scala code:
import numpy as np
a = np.array([[1.0,1.0,1.0,1.0,0.0,0.0,0.0],
[1.0, 1.0, 0.0, 1.0, 0.0,0.0,0.0 ],
[1.0, 1.0, 1.0, 1.0, 1.0,1.0,1.0 ],
[1.0, 0.0, 0.0, 0.0, 1.0 ,1.0,1.0 ],
[1.0, 0.0, 0.0, 0.0, 1.0, 0.0,1.0 ],
[1.0, 0.0, 0.0, 0.0, 0.0, 0.0,0.0 ]])
b=np.array([[0.0 ,0.0 ,0.0 ],
[0.0 ,-0.053430398509053074 ,0.021149859549078387 ],
[0.0 ,-0.010785871994186721, 0.04942555653681449 ],
[0.0 , 0.04849323245519227 ,-0.0393881161667335 ],
[0.0 ,-0.03871752673999099 , 0.05228579488821056 ],
[0.0 , 0.07935206375269452 , 0.06511344235965408 ],
[0.0 ,-0.02462677123918247 ,1.723607966539059E-4 ]])
print a.dot(b)
prints:
[[ 0. -0.01572304 0.0311873 ]
[ 0. -0.00493717 -0.01823826]
[ 0. 0.00028473 0.1487589 ]
[ 0. 0.01600777 0.1175716 ]
[ 0. -0.0633443 0.05245816]
[ 0. 0. 0. ]]
I have a text file with RGB data in the form of:
[Pixel 0,0] [Pixel 1,0] [Pixel 2,0]...
[Pixel 0,1] [Pixel 1,1] [Pixel 2,2]...
...
With an input of:
0.0 0.0 0.0 <-- this would be Pixel 0,0
1.0 0.0 0.0
1.0 0.9 0.0
I can create the flag of Germany in size 3x1 with:
%load the data to myData
Germany = reshape(myData,3,1,3);
image(Germany)
The 1px-wide pattern works good as show in picture, however, the goal is to be able to create multiple patterns, e.g. the Germany flag in 3x3 followed by Romania flag in 3x3 or any other pattern of any length and doing that! is where I can not find the proper way to reshape the matrix.
The input that should create the second example shown in picture is this:
|========= Germany Flag ==========| [ Blue ] [ Yellow ] [ Red ]
Black -> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 0.9 0.0 1.0 0.0 0.0
Red -> 1.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 1.0 1.0 0.9 0.0 1.0 0.0 0.0
Yellow-> 1.0 0.9 0.0 1.0 0.9 0.0 1.0 0.9 0.0 0.0 0.0 1.0 1.0 0.9 0.0 1.0 0.0 0.0
Any help is appreciated
Update: Asked by Marcin, the input files are literal as I explained above.
This is the content of the GermanyRomania.txt file:
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 0.9 0.0 1.0 0.0 0.0
1.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 1.0 1.0 0.9 0.0 1.0 0.0 0.0
1.0 0.9 0.0 1.0 0.9 0.0 1.0 0.9 0.0 0.0 0.0 1.0 1.0 0.9 0.0 1.0 0.0 0.0
With that file I must create the 2nd pattern in picture (German+Romania Flag), there is ALL the RGB info required to do it.
I don't think you can achieve what you want by simply using the reshape function.
We must take into account that Matlab stores matrices in column-major order (you can read more about it here).
Therefore, before we can use the reshape function, we must have the data matrix in the following format:
[Pixel 0,0]
[Pixel 0,1]
...
[Pixel 1,0]
[Pixel 1,1]
...
[Pixel n,n]
Here's a possible solution:
# data stores the input
height = size(data, 1)
width = size(data, 2)
vertical_data_cell = mat2cell(data, height, 3 * ones(1, width / 3))'
vertical_data = cell2mat(vertical_data_cell)
flags = reshape(vertical_data, height, width / 3, 3)
image(flags)
Note that we make the matrix transformation on lines 4 and 5.
And here is the result for the input you provided:
It also works with different heights.
Here's the input for the flags of Germany, Argentina and Portugal.
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.46 0.66 0.85 0.46 0.66 0.85 0.46 0.66 0.85
1.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 1.0 1.0 1.0 0.98 0.75 0.29 1.0 1.0 1.0
1.0 0.9 0.0 1.0 0.9 0.0 1.0 0.9 0.0 0.46 0.66 0.85 0.46 0.66 0.85 0.46 0.66 0.85
0.0 1.0 0.0 0.0 1.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0
0.0 1.0 0.0 1.0 0.9 0.0 1.0 0.9 0.0 1.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0
0.0 1.0 0.0 0.0 1.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0
And this is the result:
I have a 2D array containing numbers. I am attempting to work out the product of multiplying a single number from each sub-array with one from each of the other sub-arrays; I then need to do this for all possible combinations.
The aim is that I input a file of the frequency of individual events, and get an output of the probability of a particular series of these events happening, with one event from each set.
I have fudged together some code with the help of a previous question:
for my $aref ( getCartesian(#freq) ) {
my $p = 1;
foreach my $n (#$aref) {
$p = $p * $n;
}
print "$p\n";
}
sub getCartesian {
my #input = #_;
my #ret = map [$_], #{ shift #input };
for my $a2 (#input) {
#ret = map {
my $v = $_;
map [#$v, $_], #$a2;
}
#ret;
}
return #ret;
}
where #freq is an array of arrays, such as:
#freq = [0.1, 0.2, 0.3,]
[0.4, 0.5, 0.6,]
[0.7, 0.8, 0.9,]; `and ~ 20 more sub arrays`
This works fine for a small test file, but when I give it my required input of 24 sub-arrays with 3 items each, the generation of combinations is clearly far too intensive, with 3^24 possibilities.
I have run it on a machine with 22 GB RAM, and it maxed out after 4 minutes before any output.
My question is, how could I modify the code so that I can print out $p for each combination, without having to hold the whole set of combinations in memory, which kills it. I presume that time would be the only limiting factor for computation then, not resources.
Edit: a method in base Perl without packages would be great. I don't have admin on the HPC facility sadly,
Set::CrossProduct lets you iterate through the Cartesian product so you don't have to store everything in memory:
use List::Util qw(reduce);
use Set::CrossProduct;
my #array = (
[0.1, 0.2, 0.3],
[0.4, 0.5, 0.6],
[0.7, 0.8, 0.9]
);
my $iterator = Set::CrossProduct->new(\#array);
while (my $tuple = $iterator->get) {
say '(', join(', ', #$tuple), '): ', reduce { $a * $b } #$tuple;
}
Outputs:
(0.1, 0.4, 0.7): 0.028
(0.1, 0.4, 0.8): 0.032
(0.1, 0.4, 0.9): 0.036
(0.1, 0.5, 0.7): 0.035
(0.1, 0.5, 0.8): 0.04
(0.1, 0.5, 0.9): 0.045
(0.1, 0.6, 0.7): 0.042
(0.1, 0.6, 0.8): 0.048
(0.1, 0.6, 0.9): 0.054
(0.2, 0.4, 0.7): 0.056
(0.2, 0.4, 0.8): 0.064
(0.2, 0.4, 0.9): 0.072
(0.2, 0.5, 0.7): 0.07
(0.2, 0.5, 0.8): 0.08
(0.2, 0.5, 0.9): 0.09
(0.2, 0.6, 0.7): 0.084
(0.2, 0.6, 0.8): 0.096
(0.2, 0.6, 0.9): 0.108
(0.3, 0.4, 0.7): 0.084
(0.3, 0.4, 0.8): 0.096
(0.3, 0.4, 0.9): 0.108
(0.3, 0.5, 0.7): 0.105
(0.3, 0.5, 0.8): 0.12
(0.3, 0.5, 0.9): 0.135
(0.3, 0.6, 0.7): 0.126
(0.3, 0.6, 0.8): 0.144
(0.3, 0.6, 0.9): 0.162