Ruby profiler results interpretation - ruby

I'm using a ruby-prof gem to profile my code.
And results looks like this:
%self total self wait child calls name
50.56 31.06 23.45 0.00 7.62 234593 Array#each
14.29 6.62 6.62 0.00 0.00 562480 Array#-
13.63 6.32 6.32 0.00 0.00 157816 Array#|
11.20 5.20 5.20 0.00 0.00 6210903 Hash#default
2.44 46.36 1.13 0.00 46.36 78909 Analyzer#process
2.02 46.36 0.94 0.00 46.36 78908 Analyzer#try
1.70 0.79 0.79 0.00 0.00 562481 UnboundMethod#bind
1.53 7.34 0.71 0.00 6.62 562480 Method#call
1.18 0.55 0.55 0.00 0.00 562480 Kernel#instance_variable_defined?
0.76 46.36 0.35 0.00 46.36 6250 Array#combination
0.37 0.17 0.17 0.00 0.00 105763 Array#concat
0.25 25.19 0.12 0.00 25.07 77842 Enumerable#group_by
0.02 46.36 0.01 0.00 46.36 3125 Enumerator#each
0.02 0.01 0.01 0.00 0.00 78908 Array#empty?
...
I'm sure that my code does not tries to access non-existing keys in some of the Hashes.
The question is - what Hash#default might mean?
And here is a piece of code:
class Analyzer
def process(level, hypo_right, hypo_wrong)
if portion = #portions[level]
selections = #selections[level] - hypo_wrong
master_size = selections.size
selections -= hypo_right
new_portion = portion + selections.size - master_size
if new_portion > selections.size || new_portion < 0
return
elsif new_portion == 0
try(level, hypo_right, [], hypo_wrong, selections)
else
selections.combination(new_portion).each do |hypo_right2|
try(level, hypo_right, hypo_right2, hypo_wrong, (selections - hypo_right2))
end
end
else
puts hypo_right.inspect
end
end
def try(level, right, right2, wrong, wrong2)
local_right = right | right2
local_wrong = wrong | wrong2
right2.each { |r| local_wrong.concat(#siblings[r]) }
unless wrong2.empty?
grouped_wrong = local_wrong.group_by{|e| #vars[e] }
wrong2.each do |w|
qid = #vars[w]
if grouped_wrong[qid].size == #limits[qid]
local_right << (#q_hash[qid] - grouped_wrong[qid])[0]
end
end
end
process(level + 1, local_right, local_wrong)
end
def start
process(0, [], [])
end
end
#selections, #portions are Arrays; #q_hash, #siblings, #limits and #vars are Hashes.

Thanks to riffraff, found answer:
require 'ruby-prof'
h = (0..9).inject({}) {|h, x| h[x] = (x+97).chr;h }
a = (0..1000000).collect { rand(100) }
RubyProf.start
g = a.group_by {|x| h[x/10] }
RubyProf::FlatPrinter.new(RubyProf.stop).print(STDOUT)
Thread ID: 17188993880
Total: 1.210938
%self total self wait child calls name
100.00 1.21 1.21 0.00 0.00 1 Array#each
0.00 0.00 0.00 0.00 0.00 10 Hash#default
0.00 1.21 0.00 0.00 1.21 1 Enumerable#group_by
0.00 1.21 0.00 0.00 1.21 1 Object#irb_binding

Related

Why are there extra equations when using extra neurons in GEKKO?

I'm a bit puzzled on why there are extra equations introduced in the optimization problem (using GEKKO) when increasing the amount of neurons in an ANN that e.g. is used within the objective function or in the constraints. I was hoping the find the answer in this paper, but I can't seem to pinpoint the reason.
This is the log of a baseline example I made, using 2 Gekko_NN_SKlearn functions.
----------------------------------------------------------------
APMonitor, Version 1.0.0
APMonitor Optimization Suite
----------------------------------------------------------------
Called files( 55 )
files: overrides.dbs does not exist
Run id : 2022y11m03d15h47m50.642s
COMMAND LINE ARGUMENTS
coldstart: 0
imode : 3
dbs_read : T
dbs_write: T
specs : T
rto selected
Called files( 35 )
READ info FILE FOR VARIABLE DEFINITION: gk_model0.info
SS MODEL INIT 0
Parsing model file gk_model0.apm
Read model file (sec): 1.1181
Initialize constants (sec): 0.
Determine model size (sec): 0.7627999999999999
Allocate memory (sec): 0.0049000000000001265
Parse and store model (sec): 0.7630000000000001
--------- APM Model Size ------------
Each time step contains
Objects : 247
Constants : 0
Variables : 752
Intermediates: 249
Connections : 741
Equations : 745
Residuals : 496
Error checking (sec): 0.2740999999999998
Compile equations (sec): 2.8513
Check for uninitialized intermediates (sec): 0.
------------------------------------------------------
Total Parse Time (sec): 5.7744
SS MODEL INIT 1
SS MODEL INIT 2
SS MODEL INIT 3
SS MODEL INIT 4
Called files( 31 )
READ info FILE FOR PROBLEM DEFINITION: gk_model0.info
Called files( 6 )
Files(6): File Read rto.t0 F
files: rto.t0 does not exist
Called files( 51 )
Read DBS File defaults.dbs
files: defaults.dbs does not exist
Called files( 51 )
Read DBS File gk_model0.dbs
files: gk_model0.dbs does not exist
Called files( 51 )
Read DBS File measurements.dbs
Called files( 51 )
Read DBS File overrides.dbs
files: overrides.dbs does not exist
Number of state variables: 1240
Number of total equations: - 989
Number of slack variables: - 0
---------------------------------------
Degrees of freedom : 251
----------------------------------------------
Steady State Optimization with APOPT Solver
----------------------------------------------
Iter: 1 I: 0 Tm: 2.23 NLPi: 67 Dpth: 0 Lvs: 3 Obj: 6.96E-02 Gap: NaN
--Integer Solution: 1.78E-01 Lowest Leaf: 6.96E-02 Gap: 1.08E-01
Iter: 2 I: 0 Tm: 0.13 NLPi: 6 Dpth: 1 Lvs: 2 Obj: 1.78E-01 Gap: 1.08E-01
Iter: 3 I: 0 Tm: 0.33 NLPi: 5 Dpth: 1 Lvs: 2 Obj: 1.74E-01 Gap: 1.08E-01
Iter: 4 I: 0 Tm: 0.52 NLPi: 11 Dpth: 1 Lvs: 3 Obj: 9.49E-02 Gap: 1.08E-01
--Integer Solution: 1.78E-01 Lowest Leaf: 9.49E-02 Gap: 8.27E-02
Iter: 5 I: 0 Tm: 0.28 NLPi: 5 Dpth: 2 Lvs: 2 Obj: 1.04E+00 Gap: 8.27E-02
--Integer Solution: 1.23E-01 Lowest Leaf: 1.23E-01 Gap: 0.00E+00
Iter: 6 I: 0 Tm: 0.30 NLPi: 5 Dpth: 2 Lvs: 2 Obj: 1.23E-01 Gap: 0.00E+00
Successful solution
---------------------------------------------------
Solver : APOPT (v1.0)
Solution time : 3.842599999999999 sec
Objective : 0.12267384658102941
Successful solution
---------------------------------------------------
Called files( 2 )
Called files( 52 )
WRITE dbs FILE
Called files( 56 )
WRITE json FILE
Timer # 1 10.41/ 1 = 10.41 Total system time
Timer # 2 3.84/ 1 = 3.84 Total solve time
Timer # 3 0.02/ 191 = 0.00 Objective Calc: apm_p
Timer # 4 0.01/ 99 = 0.00 Objective Grad: apm_g
Timer # 5 0.03/ 191 = 0.00 Constraint Calc: apm_c
Timer # 6 0.00/ 0 = 0.00 Sparsity: apm_s
Timer # 7 0.00/ 0 = 0.00 1st Deriv #1: apm_a1
Timer # 8 0.00/ 99 = 0.00 1st Deriv #2: apm_a2
Timer # 9 0.67/ 1 = 0.67 Custom Init: apm_custom_init
Timer # 10 0.01/ 1 = 0.01 Mode: apm_node_res::case 0
Timer # 11 0.00/ 1 = 0.00 Mode: apm_node_res::case 1
Timer # 12 0.02/ 1 = 0.02 Mode: apm_node_res::case 2
Timer # 13 0.00/ 1 = 0.00 Mode: apm_node_res::case 3
Timer # 14 0.35/ 387 = 0.00 Mode: apm_node_res::case 4
Timer # 15 1.51/ 198 = 0.01 Mode: apm_node_res::case 5
Timer # 16 0.00/ 0 = 0.00 Mode: apm_node_res::case 6
Timer # 17 0.01/ 99 = 0.00 Base 1st Deriv: apm_jacobian
Timer # 18 0.00/ 99 = 0.00 Base 1st Deriv: apm_condensed_jacobian
Timer # 19 0.00/ 1 = 0.00 Non-zeros: apm_nnz
Timer # 20 0.00/ 0 = 0.00 Count: Division by zero
Timer # 21 0.00/ 0 = 0.00 Count: Argument of LOG10 negative
Timer # 22 0.00/ 0 = 0.00 Count: Argument of LOG negative
Timer # 23 0.00/ 0 = 0.00 Count: Argument of SQRT negative
Timer # 24 0.00/ 0 = 0.00 Count: Argument of ASIN illegal
Timer # 25 0.00/ 0 = 0.00 Count: Argument of ACOS illegal
Timer # 26 0.00/ 1 = 0.00 Extract sparsity: apm_sparsity
Timer # 27 0.00/ 13 = 0.00 Variable ordering: apm_var_order
Timer # 28 0.00/ 1 = 0.00 Condensed sparsity
Timer # 29 0.00/ 0 = 0.00 Hessian Non-zeros
Timer # 30 0.00/ 1 = 0.00 Differentials
Timer # 31 0.00/ 0 = 0.00 Hessian Calculation
Timer # 32 0.00/ 0 = 0.00 Extract Hessian
Timer # 33 0.00/ 1 = 0.00 Base 1st Deriv: apm_jac_order
Timer # 34 0.02/ 1 = 0.02 Solver Setup
Timer # 35 1.87/ 1 = 1.87 Solver Solution
Timer # 36 0.00/ 202 = 0.00 Number of Variables
Timer # 37 0.00/ 105 = 0.00 Number of Equations
Timer # 38 0.02/ 14 = 0.00 File Read/Write
Timer # 39 0.00/ 0 = 0.00 Dynamic Init A
Timer # 40 0.00/ 0 = 0.00 Dynamic Init B
Timer # 41 0.00/ 0 = 0.00 Dynamic Init C
Timer # 42 1.12/ 1 = 1.12 Init: Read APM File
Timer # 43 0.00/ 1 = 0.00 Init: Parse Constants
Timer # 44 0.76/ 1 = 0.76 Init: Model Sizing
Timer # 45 0.00/ 1 = 0.00 Init: Allocate Memory
Timer # 46 0.76/ 1 = 0.76 Init: Parse Model
Timer # 47 0.27/ 1 = 0.27 Init: Check for Duplicates
Timer # 48 2.85/ 1 = 2.85 Init: Compile Equations
Timer # 49 0.00/ 1 = 0.00 Init: Check Uninitialized
Timer # 50 0.00/ 1257 = 0.00 Evaluate Expression Once
Timer # 51 0.00/ 0 = 0.00 Sensitivity Analysis: LU Factorization
Timer # 52 0.00/ 0 = 0.00 Sensitivity Analysis: Gauss Elimination
Timer # 53 0.00/ 0 = 0.00 Sensitivity Analysis: Total Time
When I change the amount of neurons of 1 function from [25,20,20,10] to [50,40,40,40] I get the following log:
----------------------------------------------------------------
APMonitor, Version 1.0.0
APMonitor Optimization Suite
----------------------------------------------------------------
Called files( 55 )
files: overrides.dbs does not exist
Run id : 2023y02m04d11h08m15.999s
COMMAND LINE ARGUMENTS
coldstart: 0
imode : 3
dbs_read : T
dbs_write: T
specs : T
rto selected
Called files( 35 )
READ info FILE FOR VARIABLE DEFINITION: gk_model0.info
SS MODEL INIT 0
Parsing model file gk_model0.apm
Read model file (sec): 1.4879
Initialize constants (sec): 0.
Determine model size (sec): 0.9460000000000002
Allocate memory (sec): 0.01529999999999987
Parse and store model (sec): 0.7256999999999998
--------- APM Model Size ------------
Each time step contains
Objects : 342
Constants : 0
Variables : 1037
Intermediates: 344
Connections : 1026
Equations : 1030
Residuals : 686
Error checking (sec): 0.2522000000000002
Compile equations (sec): 2.8817999999999997
Check for uninitialized intermediates (sec): 0.
------------------------------------------------------
Total Parse Time (sec): 6.3089
SS MODEL INIT 1
SS MODEL INIT 2
SS MODEL INIT 3
SS MODEL INIT 4
Called files( 31 )
READ info FILE FOR PROBLEM DEFINITION: gk_model0.info
Called files( 6 )
Files(6): File Read rto.t0 F
files: rto.t0 does not exist
Called files( 51 )
Read DBS File defaults.dbs
files: defaults.dbs does not exist
Called files( 51 )
Read DBS File gk_model0.dbs
files: gk_model0.dbs does not exist
Called files( 51 )
Read DBS File measurements.dbs
Called files( 51 )
Read DBS File overrides.dbs
files: overrides.dbs does not exist
Number of state variables: 1715
Number of total equations: - 1369
Number of slack variables: - 0
---------------------------------------
Degrees of freedom : 346
----------------------------------------------
Steady State Optimization with APOPT Solver
----------------------------------------------
Iter: 1 I: 0 Tm: 2.84 NLPi: 53 Dpth: 0 Lvs: 3 Obj: 2.69E-01 Gap: NaN
--Integer Solution: 4.43E-01 Lowest Leaf: 2.69E-01 Gap: 1.74E-01
Iter: 2 I: 0 Tm: 0.14 NLPi: 5 Dpth: 1 Lvs: 2 Obj: 4.43E-01 Gap: 1.74E-01
Iter: 3 I: 0 Tm: 0.55 NLPi: 6 Dpth: 1 Lvs: 2 Obj: 3.79E-01 Gap: 1.74E-01
Iter: 4 I: 0 Tm: 0.96 NLPi: 12 Dpth: 1 Lvs: 3 Obj: 4.17E-01 Gap: 1.74E-01
--Integer Solution: 4.43E-01 Lowest Leaf: 4.17E-01 Gap: 2.62E-02
Iter: 5 I: 0 Tm: 0.56 NLPi: 7 Dpth: 2 Lvs: 2 Obj: 1.18E+00 Gap: 2.62E-02
--Integer Solution: 4.43E-01 Lowest Leaf: 4.17E-01 Gap: 2.62E-02
Iter: 6 I: 0 Tm: 0.36 NLPi: 4 Dpth: 2 Lvs: 1 Obj: 1.04E+00 Gap: 2.62E-02
--Integer Solution: 4.43E-01 Lowest Leaf: 5.39E-01 Gap: -9.53E-02
Iter: 7 I: 0 Tm: 0.36 NLPi: 4 Dpth: 2 Lvs: 1 Obj: 5.39E-01 Gap: -9.53E-02
Successful solution
---------------------------------------------------
Solver : APOPT (v1.0)
Solution time : 5.816599999999999 sec
Objective : 0.4433267264972657
Successful solution
---------------------------------------------------
Called files( 2 )
Called files( 52 )
WRITE dbs FILE
Called files( 56 )
WRITE json FILE
Timer # 1 13.21/ 1 = 13.21 Total system time
Timer # 2 5.82/ 1 = 5.82 Total solve time
Timer # 3 0.02/ 189 = 0.00 Objective Calc: apm_p
Timer # 4 0.03/ 91 = 0.00 Objective Grad: apm_g
Timer # 5 0.03/ 189 = 0.00 Constraint Calc: apm_c
Timer # 6 0.00/ 0 = 0.00 Sparsity: apm_s
Timer # 7 0.00/ 0 = 0.00 1st Deriv #1: apm_a1
Timer # 8 0.00/ 91 = 0.00 1st Deriv #2: apm_a2
Timer # 9 0.90/ 1 = 0.90 Custom Init: apm_custom_init
Timer # 10 0.01/ 1 = 0.01 Mode: apm_node_res::case 0
Timer # 11 0.01/ 1 = 0.01 Mode: apm_node_res::case 1
Timer # 12 0.03/ 1 = 0.03 Mode: apm_node_res::case 2
Timer # 13 0.00/ 1 = 0.00 Mode: apm_node_res::case 3
Timer # 14 0.33/ 383 = 0.00 Mode: apm_node_res::case 4
Timer # 15 2.01/ 182 = 0.01 Mode: apm_node_res::case 5
Timer # 16 0.00/ 0 = 0.00 Mode: apm_node_res::case 6
Timer # 17 0.06/ 91 = 0.00 Base 1st Deriv: apm_jacobian
Timer # 18 0.02/ 91 = 0.00 Base 1st Deriv: apm_condensed_jacobian
Timer # 19 0.00/ 1 = 0.00 Non-zeros: apm_nnz
Timer # 20 0.00/ 0 = 0.00 Count: Division by zero
Timer # 21 0.00/ 0 = 0.00 Count: Argument of LOG10 negative
Timer # 22 0.00/ 0 = 0.00 Count: Argument of LOG negative
Timer # 23 0.00/ 0 = 0.00 Count: Argument of SQRT negative
Timer # 24 0.00/ 0 = 0.00 Count: Argument of ASIN illegal
Timer # 25 0.00/ 0 = 0.00 Count: Argument of ACOS illegal
Timer # 26 0.00/ 1 = 0.00 Extract sparsity: apm_sparsity
Timer # 27 0.00/ 13 = 0.00 Variable ordering: apm_var_order
Timer # 28 0.00/ 1 = 0.00 Condensed sparsity
Timer # 29 0.00/ 0 = 0.00 Hessian Non-zeros
Timer # 30 0.00/ 1 = 0.00 Differentials
Timer # 31 0.00/ 0 = 0.00 Hessian Calculation
Timer # 32 0.00/ 0 = 0.00 Extract Hessian
Timer # 33 0.00/ 1 = 0.00 Base 1st Deriv: apm_jac_order
Timer # 34 0.02/ 1 = 0.02 Solver Setup
Timer # 35 3.26/ 1 = 3.26 Solver Solution
Timer # 36 0.00/ 200 = 0.00 Number of Variables
Timer # 37 0.02/ 97 = 0.00 Number of Equations
Timer # 38 0.03/ 14 = 0.00 File Read/Write
Timer # 39 0.00/ 0 = 0.00 Dynamic Init A
Timer # 40 0.00/ 0 = 0.00 Dynamic Init B
Timer # 41 0.00/ 0 = 0.00 Dynamic Init C
Timer # 42 1.49/ 1 = 1.49 Init: Read APM File
Timer # 43 0.00/ 1 = 0.00 Init: Parse Constants
Timer # 44 0.95/ 1 = 0.95 Init: Model Sizing
Timer # 45 0.02/ 1 = 0.02 Init: Allocate Memory
Timer # 46 0.73/ 1 = 0.73 Init: Parse Model
Timer # 47 0.25/ 1 = 0.25 Init: Check for Duplicates
Timer # 48 2.88/ 1 = 2.88 Init: Compile Equations
Timer # 49 0.00/ 1 = 0.00 Init: Check Uninitialized
Timer # 50 -0.00/ 1732 = -0.00 Evaluate Expression Once
Timer # 51 0.00/ 0 = 0.00 Sensitivity Analysis: LU Factorization
Timer # 52 0.00/ 0 = 0.00 Sensitivity Analysis: Gauss Elimination
Timer # 53 0.00/ 0 = 0.00 Sensitivity Analysis: Total Time
Hence, a significant amount of extra objects, variables, intermediates, connections, equations and residuals are introduced.
Many thank in advance for your replies!
the paper talks about this a little in section 3.1.3.
The prediction functions used in both TensorFlow and Scikit-learn neural networks use linear algebra to relate the layers and neurons of the neural network to one another. Each neuron in an input layer has a specific weight that corresponds to an output neuron in the following layer. Each neuron in an output layer also has a corresponding bias value as shown in Figure 1.
Figure 1
For each neuron connection, there is a bias, weight, and an additional activation function. An additional activation equation, such as the linear rectifier or hyperbolic tangent functions, is generally used to normalize the activation between 0 and 1.
As you increase the number of neurons from [25,20,20,10] to [50,40,40,40], a new weight, bias, and activation function is being introduced for each new neuron connection. These objects are represented by an increase in the number of equations, variables, etc... in the optimization problem.
Hopefully this helps!

Reduce total parse (total system) time in GEKKO

I have an RTO problem that I want to solve for multiple simulated timesteps with some time-depended parameters. However, I'm struggling with the run-time and noticed that the total system time is relatively large compared to the actual solve time. I was therefore trying to reduce the total parse time, as all the equations remain the same - yet "only" the values of some parameters change with time. A simple example below:
#parameters from simulation
demand = 100
#do RTO
from gekko import GEKKO
# first, create the model
m = GEKKO(remote=False)
# declare additional decision variables
m.u = m.Var(lb=5, ub=25)
m.v = m.Var(lb=0, ub=100)
m.w = m.Var(lb=0, ub=50)
m.b = m.Var(lb=0, ub=1, integer=True)
m.demand = m.Param(demand)
# now add the objective and the constraints
m.Minimize((1-0.8)*m.u*m.b+(1-0.9)*m.v+(1-0.7)*m.w)
m.Equation(m.u*m.b >= 10)
m.Equation(m.u*m.b + m.v + m.w == m.demand)
m.options.SOLVER=1
m.options.DIAGLEVEL = 1
m.solve()
then I capture the results, execute them in the simulation and move on to the next timestep. Now I could just, execute all code above again - with updated parameters (let's say the demand is now 110). But this results in the before mentioned long run-time (the RTO problem needs to be build from scratch every time, while only some parameters change). So I thought the following could work:
m.demand.VALUE = 110
m.solve()
While this does work. It doesn't seem to improve the run-time (total parse time is still relatively long). Below are the display outputs of the actual problem.
First time solving the RTO problem.
----------------------------------------------------------------
APMonitor, Version 1.0.0
APMonitor Optimization Suite
----------------------------------------------------------------
Called files( 55 )
files: overrides.dbs does not exist
Run id : 2022y11m03d13h18m21.919s
COMMAND LINE ARGUMENTS
coldstart: 0
imode : 3
dbs_read : T
dbs_write: T
specs : T
rto selected
Called files( 35 )
READ info FILE FOR VARIABLE DEFINITION: gk_model6.info
SS MODEL INIT 0
Parsing model file gk_model6.apm
Read model file (sec): 0.6602
Initialize constants (sec): 0.
Determine model size (sec): 0.4170999999999999
Allocate memory (sec): 0.
Parse and store model (sec): 0.45140000000000025
--------- APM Model Size ------------
Each time step contains
Objects : 247
Constants : 0
Variables : 752
Intermediates: 249
Connections : 741
Equations : 745
Residuals : 496
Error checking (sec): 0.17809999999999993
Compile equations (sec): 1.9933000000000003
Check for uninitialized intermediates (sec): 0.
------------------------------------------------------
Total Parse Time (sec): 3.7062
SS MODEL INIT 1
SS MODEL INIT 2
SS MODEL INIT 3
SS MODEL INIT 4
Called files( 31 )
READ info FILE FOR PROBLEM DEFINITION: gk_model6.info
Called files( 6 )
Files(6): File Read rto.t0 F
files: rto.t0 does not exist
Called files( 51 )
Read DBS File defaults.dbs
files: defaults.dbs does not exist
Called files( 51 )
Read DBS File gk_model6.dbs
files: gk_model6.dbs does not exist
Called files( 51 )
Read DBS File measurements.dbs
Called files( 51 )
Read DBS File overrides.dbs
files: overrides.dbs does not exist
Number of state variables: 1240
Number of total equations: - 989
Number of slack variables: - 0
---------------------------------------
Degrees of freedom : 251
----------------------------------------------
Steady State Optimization with APOPT Solver
----------------------------------------------
Iter: 1 I: 0 Tm: 1.20 NLPi: 45 Dpth: 0 Lvs: 3 Obj: 9.86E-02 Gap: NaN
--Integer Solution: 2.32E-01 Lowest Leaf: 9.86E-02 Gap: 1.34E-01
Iter: 2 I: 0 Tm: 0.06 NLPi: 4 Dpth: 1 Lvs: 2 Obj: 2.32E-01 Gap: 1.34E-01
Iter: 3 I: 0 Tm: 0.23 NLPi: 6 Dpth: 1 Lvs: 2 Obj: 2.16E-01 Gap: 1.34E-01
Iter: 4 I: 0 Tm: 0.44 NLPi: 12 Dpth: 1 Lvs: 3 Obj: 1.60E-01 Gap: 1.34E-01
--Integer Solution: 2.32E-01 Lowest Leaf: 1.60E-01 Gap: 7.21E-02
Iter: 5 I: 0 Tm: 0.20 NLPi: 6 Dpth: 2 Lvs: 2 Obj: 1.01E+00 Gap: 7.21E-02
--Integer Solution: 2.06E-01 Lowest Leaf: 2.06E-01 Gap: 0.00E+00
Iter: 6 I: 0 Tm: 0.20 NLPi: 5 Dpth: 2 Lvs: 2 Obj: 2.06E-01 Gap: 0.00E+00
Successful solution
---------------------------------------------------
Solver : APOPT (v1.0)
Solution time : 2.3522999999999996 sec
Objective : 0.20599966381706797
Successful solution
---------------------------------------------------
Called files( 2 )
Called files( 52 )
WRITE dbs FILE
Called files( 56 )
WRITE json FILE
Timer # 1 6.57/ 1 = 6.57 Total system time
Timer # 2 2.35/ 1 = 2.35 Total solve time
Timer # 3 0.01/ 156 = 0.00 Objective Calc: apm_p
Timer # 4 0.01/ 78 = 0.00 Objective Grad: apm_g
Timer # 5 0.01/ 156 = 0.00 Constraint Calc: apm_c
Timer # 6 0.00/ 0 = 0.00 Sparsity: apm_s
Timer # 7 0.00/ 0 = 0.00 1st Deriv #1: apm_a1
Timer # 8 0.01/ 78 = 0.00 1st Deriv #2: apm_a2
Timer # 9 0.42/ 1 = 0.42 Custom Init: apm_custom_init
Timer # 10 0.00/ 1 = 0.00 Mode: apm_node_res::case 0
Timer # 11 0.00/ 1 = 0.00 Mode: apm_node_res::case 1
Timer # 12 0.02/ 1 = 0.02 Mode: apm_node_res::case 2
Timer # 13 0.00/ 1 = 0.00 Mode: apm_node_res::case 3
Timer # 14 0.17/ 317 = 0.00 Mode: apm_node_res::case 4
Timer # 15 0.72/ 156 = 0.00 Mode: apm_node_res::case 5
Timer # 16 0.00/ 0 = 0.00 Mode: apm_node_res::case 6
Timer # 17 0.01/ 78 = 0.00 Base 1st Deriv: apm_jacobian
Timer # 18 0.00/ 78 = 0.00 Base 1st Deriv: apm_condensed_jacobian
Timer # 19 0.00/ 1 = 0.00 Non-zeros: apm_nnz
Timer # 20 0.00/ 0 = 0.00 Count: Division by zero
Timer # 21 0.00/ 0 = 0.00 Count: Argument of LOG10 negative
Timer # 22 0.00/ 0 = 0.00 Count: Argument of LOG negative
Timer # 23 0.00/ 0 = 0.00 Count: Argument of SQRT negative
Timer # 24 0.00/ 0 = 0.00 Count: Argument of ASIN illegal
Timer # 25 0.00/ 0 = 0.00 Count: Argument of ACOS illegal
Timer # 26 0.00/ 1 = 0.00 Extract sparsity: apm_sparsity
Timer # 27 0.00/ 13 = 0.00 Variable ordering: apm_var_order
Timer # 28 0.00/ 1 = 0.00 Condensed sparsity
Timer # 29 0.00/ 0 = 0.00 Hessian Non-zeros
Timer # 30 0.00/ 1 = 0.00 Differentials
Timer # 31 0.00/ 0 = 0.00 Hessian Calculation
Timer # 32 0.00/ 0 = 0.00 Extract Hessian
Timer # 33 0.00/ 1 = 0.00 Base 1st Deriv: apm_jac_order
Timer # 34 0.01/ 1 = 0.01 Solver Setup
Timer # 35 1.39/ 1 = 1.39 Solver Solution
Timer # 36 0.00/ 167 = 0.00 Number of Variables
Timer # 37 0.01/ 84 = 0.00 Number of Equations
Timer # 38 0.01/ 14 = 0.00 File Read/Write
Timer # 39 0.00/ 0 = 0.00 Dynamic Init A
Timer # 40 0.00/ 0 = 0.00 Dynamic Init B
Timer # 41 0.00/ 0 = 0.00 Dynamic Init C
Timer # 42 0.66/ 1 = 0.66 Init: Read APM File
Timer # 43 0.00/ 1 = 0.00 Init: Parse Constants
Timer # 44 0.42/ 1 = 0.42 Init: Model Sizing
Timer # 45 0.00/ 1 = 0.00 Init: Allocate Memory
Timer # 46 0.45/ 1 = 0.45 Init: Parse Model
Timer # 47 0.18/ 1 = 0.18 Init: Check for Duplicates
Timer # 48 1.99/ 1 = 1.99 Init: Compile Equations
Timer # 49 0.00/ 1 = 0.00 Init: Check Uninitialized
Timer # 50 0.01/ 1257 = 0.00 Evaluate Expression Once
Timer # 51 0.00/ 0 = 0.00 Sensitivity Analysis: LU Factorization
Timer # 52 0.00/ 0 = 0.00 Sensitivity Analysis: Gauss Elimination
Timer # 53 0.00/ 0 = 0.00 Sensitivity Analysis: Total Time
Updating one parameter and only calling m.solve() again - is shown in the simple problem above.
----------------------------------------------------------------
APMonitor, Version 1.0.0
APMonitor Optimization Suite
----------------------------------------------------------------
Called files( 55 )
Called files( 55 )
files: overrides.dbs does not exist
Run id : 2022y11m03d13h18m28.729s
COMMAND LINE ARGUMENTS
coldstart: 0
imode : 3
dbs_read : T
dbs_write: T
specs : T
rto selected
Called files( 35 )
READ info FILE FOR VARIABLE DEFINITION: gk_model6.info
SS MODEL INIT 0
Parsing model file gk_model6.apm
Read model file (sec): 0.6901
Initialize constants (sec): 0.
Determine model size (sec): 0.4546999999999999
Allocate memory (sec): 0.
Parse and store model (sec): 0.2824000000000002
--------- APM Model Size ------------
Each time step contains
Objects : 247
Constants : 0
Variables : 752
Intermediates: 249
Connections : 741
Equations : 745
Residuals : 496
Error checking (sec): 0.16720000000000002
Compile equations (sec): 2.0142999999999995
Check for uninitialized intermediates (sec): 0.
------------------------------------------------------
Total Parse Time (sec): 3.6097
SS MODEL INIT 1
SS MODEL INIT 2
SS MODEL INIT 3
SS MODEL INIT 4
Called files( 31 )
READ info FILE FOR PROBLEM DEFINITION: gk_model6.info
Called files( 6 )
Files(6): File Read rto.t0 T
Called files( 51 )
Read DBS File defaults.dbs
files: defaults.dbs does not exist
Called files( 51 )
Read DBS File gk_model6.dbs
Called files( 51 )
Read DBS File measurements.dbs
Called files( 51 )
Read DBS File overrides.dbs
files: overrides.dbs does not exist
Number of state variables: 1240
Number of total equations: - 989
Number of slack variables: - 0
---------------------------------------
Degrees of freedom : 251
----------------------------------------------
Steady State Optimization with APOPT Solver
----------------------------------------------
Iter: 1 I: 0 Tm: 0.26 NLPi: 7 Dpth: 0 Lvs: 3 Obj: 9.35E-02 Gap: NaN
--Integer Solution: 1.21E-01 Lowest Leaf: 9.35E-02 Gap: 2.71E-02
Iter: 2 I: 0 Tm: 0.22 NLPi: 5 Dpth: 1 Lvs: 2 Obj: 1.21E-01 Gap: 2.71E-02
--Integer Solution: 1.21E-01 Lowest Leaf: 9.35E-02 Gap: 2.71E-02
Iter: 3 I: 0 Tm: 0.32 NLPi: 10 Dpth: 1 Lvs: 1 Obj: 1.03E+00 Gap: 2.71E-02
Iter: 4 I: 0 Tm: 0.25 NLPi: 8 Dpth: 1 Lvs: 1 Obj: 1.20E-01 Gap: 2.71E-02
--Integer Solution: 1.21E-01 Lowest Leaf: 1.86E-01 Gap: -6.58E-02
Iter: 5 I: 0 Tm: 0.37 NLPi: 15 Dpth: 2 Lvs: 1 Obj: 1.86E-01 Gap: -6.58E-02
Successful solution
---------------------------------------------------
Solver : APOPT (v1.0)
Solution time : 1.4365000000000006 sec
Objective : 0.12065435497282542
Successful solution
---------------------------------------------------
Called files( 2 )
Called files( 52 )
WRITE dbs FILE
Called files( 56 )
WRITE json FILE
Timer # 1 5.64/ 1 = 5.64 Total system time
Timer # 2 1.44/ 1 = 1.44 Total solve time
Timer # 3 0.00/ 91 = 0.00 Objective Calc: apm_p
Timer # 4 0.00/ 45 = 0.00 Objective Grad: apm_g
Timer # 5 0.01/ 91 = 0.00 Constraint Calc: apm_c
Timer # 6 0.00/ 0 = 0.00 Sparsity: apm_s
Timer # 7 0.00/ 0 = 0.00 1st Deriv #1: apm_a1
Timer # 8 0.00/ 45 = 0.00 1st Deriv #2: apm_a2
Timer # 9 0.44/ 1 = 0.44 Custom Init: apm_custom_init
Timer # 10 0.00/ 1 = 0.00 Mode: apm_node_res::case 0
Timer # 11 0.00/ 1 = 0.00 Mode: apm_node_res::case 1
Timer # 12 0.01/ 1 = 0.01 Mode: apm_node_res::case 2
Timer # 13 0.00/ 1 = 0.00 Mode: apm_node_res::case 3
Timer # 14 0.10/ 187 = 0.00 Mode: apm_node_res::case 4
Timer # 15 0.46/ 90 = 0.01 Mode: apm_node_res::case 5
Timer # 16 0.00/ 0 = 0.00 Mode: apm_node_res::case 6
Timer # 17 0.00/ 45 = 0.00 Base 1st Deriv: apm_jacobian
Timer # 18 0.00/ 45 = 0.00 Base 1st Deriv: apm_condensed_jacobian
Timer # 19 0.00/ 1 = 0.00 Non-zeros: apm_nnz
Timer # 20 0.00/ 0 = 0.00 Count: Division by zero
Timer # 21 0.00/ 0 = 0.00 Count: Argument of LOG10 negative
Timer # 22 0.00/ 0 = 0.00 Count: Argument of LOG negative
Timer # 23 0.00/ 0 = 0.00 Count: Argument of SQRT negative
Timer # 24 0.00/ 0 = 0.00 Count: Argument of ASIN illegal
Timer # 25 0.00/ 0 = 0.00 Count: Argument of ACOS illegal
Timer # 26 0.00/ 1 = 0.00 Extract sparsity: apm_sparsity
Timer # 27 0.00/ 13 = 0.00 Variable ordering: apm_var_order
Timer # 28 0.00/ 1 = 0.00 Condensed sparsity
Timer # 29 0.00/ 0 = 0.00 Hessian Non-zeros
Timer # 30 0.01/ 1 = 0.01 Differentials
Timer # 31 0.00/ 0 = 0.00 Hessian Calculation
Timer # 32 0.00/ 0 = 0.00 Extract Hessian
Timer # 33 0.00/ 1 = 0.00 Base 1st Deriv: apm_jac_order
Timer # 34 0.01/ 1 = 0.01 Solver Setup
Timer # 35 0.84/ 1 = 0.84 Solver Solution
Timer # 36 0.00/ 102 = 0.00 Number of Variables
Timer # 37 0.00/ 51 = 0.00 Number of Equations
Timer # 38 0.12/ 14 = 0.01 File Read/Write
Timer # 39 0.00/ 0 = 0.00 Dynamic Init A
Timer # 40 0.00/ 0 = 0.00 Dynamic Init B
Timer # 41 0.00/ 0 = 0.00 Dynamic Init C
Timer # 42 0.69/ 1 = 0.69 Init: Read APM File
Timer # 43 0.00/ 1 = 0.00 Init: Parse Constants
Timer # 44 0.45/ 1 = 0.45 Init: Model Sizing
Timer # 45 0.00/ 1 = 0.00 Init: Allocate Memory
Timer # 46 0.28/ 1 = 0.28 Init: Parse Model
Timer # 47 0.17/ 1 = 0.17 Init: Check for Duplicates
Timer # 48 2.01/ 1 = 2.01 Init: Compile Equations
Timer # 49 0.00/ 1 = 0.00 Init: Check Uninitialized
Timer # 50 0.00/ 505 = 0.00 Evaluate Expression Once
Timer # 51 0.00/ 0 = 0.00 Sensitivity Analysis: LU Factorization
Timer # 52 0.00/ 0 = 0.00 Sensitivity Analysis: Gauss Elimination
Timer # 53 0.00/ 0 = 0.00 Sensitivity Analysis: Total Time
Many thanks in advance for your ideas.
Here are a few ideas to improve the compile time speed:
If the problem is a simulation, it can be run with IMODE=7 to compile the model once and run through all the timesteps sequentially. This is often the fastest option for simulation.
If the problem has degrees of freedom (find optimal parameter to minimize objective) then it is also possible to set up the problem as IMODE=6 to compile the model once and it solves all time steps simultaneously.
The model re-compiles every time m.solve() is called. Keeping a compiled version of the model is built into the REPLAY parameter, but it has little documentation and is built for exactly reenacting a sequence from historical data. If you'd like the option to keep a compiled version of the model for the next run, please add it as a feature request in GitHub.
Using the simple model shows that the model re-compile time doesn't change significantly between runs with different values of demand.
from gekko import GEKKO
from numpy import random
import numpy as np
import matplotlib.pyplot as plt
import time
# first, create the model
m = GEKKO(remote=False)
# declare additional decision variables
m.u = m.Var(lb=5, ub=25)
m.v = m.Var(lb=0, ub=100)
m.w = m.Var(lb=0, ub=50)
m.b = m.Var(lb=0, ub=1, integer=True)
m.demand = m.Param()
# now add the objective and the constraints
m.Minimize((1-0.8)*m.u*m.b+(1-0.9)*m.v+(1-0.7)*m.w)
m.Equation(m.u*m.b >= 10)
m.Equation(m.u*m.b + m.v + m.w == m.demand)
m.options.SOLVER=1
m.options.DIAGLEVEL = 0
tt = [] # total time
ts = [] # solve time
for i in range(101):
start = time.time()
m.demand.value = 50+random.rand()*50
m.solve(disp=False)
tt.append(time.time()-start)
ts.append(m.options.SOLVETIME)
plt.figure(figsize=(8,4))
plt.plot(tt,label='Total Time')
plt.plot(ts,label='Solve Time')
plt.plot(np.array(tt)-np.array(ts),label='Compile Time')
plt.ylim([0,0.03]); plt.grid(); plt.ylabel('Time (sec)')
plt.savefig('timing.png',dpi=300)
plt.legend()
plt.show()
The compile and solve times are very fast for this simple problem. If you can post the full RTO problem, we can give more specific suggestions to improve solve time.

I need simple formula for data transformation according to this pattern

I'm looking for an algorithm that has two input values and one output value and follows this pattern:
Input_A: 10 (When INPUT_B is increased from 0 to 1 in very small steps, it should reach the value '1' 100/10=10 times.)
Input_B => Output
0.025 => 0.25
...
0.05 => 0.50
...
0.075 => 0.75
...
0.1 => 1.00
...
0.125 => 0.25
...
0.15 => 0.50
...
0.175 => 0.75
...
0.2 => 1.00
....
0.9 => 1.00
....
0.95 => 0.50
...
Input_A: 20 (When INPUT_B is increased from 0 to 1 in very small steps, it should reach the value '1' 100/20=5 times.)
Input_B => Output
0.025 => 0.50
...
0.05 => 1.00
...
0.075 => 0.50
...
0.1 => 1.00
...
0.125 => 0.50
...
0.15 => 1.00
...
0.175 => 0.50
...
0.2 => 1.00
....
0.9 => 1.00
....
0.9125 => 0.25
...
0.925 => 0.50
...
0.95 => 1.00
...
I think I managed to create an algorithm that follows the first pattern. But I couldn't find one that follows both.
myAlgorithm(Input_A,Input_B) {
return (Input_B && Input_B%0.1 == 0) ? 1 : Input_B%0.1 * Input_A;
}
It seems you need something like this:
A10 = A * 10 //0.175 * 10 = 1.75
AInt = (Int)A10 //integer part = 1
AFrac = A10 - AInt //fractional part = 0.75
Output = AFrac? AFrac: 1.0 ; //extra case of zero fractional part

What does "nil#" mean in profiler results?

Here is very simplified part of code:
srand 0
WIDTH, HEIGHT = 10, 10
array = Array.new(HEIGHT){ [0]*WIDTH }
require "profile"
10000.times do
y, x = rand(HEIGHT), rand(WIDTH)
g = array[y][x] + [-1,+1].sample
array[y][x] = g unless [[y-1,x],[y+1,x],[y,x-1],[y,x+1]].any?{ |y, x|
y>=0 && y<HEIGHT && x>=0 && x<WIDTH && 1 < (array[y][x] - g).abs
}
end
We see (ruby 2.0.0p451 (2014-02-24) [i386-mingw32]):
% cumulative self self total
time seconds seconds calls ms/call ms/call name
39.33 0.86 0.86 47471 0.02 0.05 nil#
If we totally remove the unless thing:
53.48 0.48 0.48 10000 0.05 0.09 nil#
Looks like we get this nil# operation:
on every comparision or boolean operation?
on every Array object creation or block envocation?
Would be nice to get a deep answer.

Ruby CSV in Array

I read a CSV file. This file looks like this
1.00 cm; 2.00cm ; 3.00 cm; ... ; 100 cm
2.00 cm; 4.00 cm; 6.00 cm; ... ; 100 cm
4.00 cm; 8.00 cm; 12.00 cm; ... ; 100cm
8.00 cm; 16.00 cm; 24.00 cm; ... ; 100cm
I have already written the following code
CSV.foreach("/Users/testUser/Entwicklung/coverrechner/CoverPaperDE.csv", col_sep: ';') do |row|
puts row[0]
end
This produces the following output:
1.00 cm
2.00 cm
4.00 cm
8.00 cm
Example:
My matrix is constructed
1.1 1.2 1.3 1.4
2.1 2.2 2.3 2.4
3.1 3.2 3.3 3.4
4.1 4.2 4.3 4.4
I want the following output
1.1 2.1 3.1 4.1 1.2 2.2 3.2 4.2 ... 4.4
How does it work?
Do you want the first element of the first row and the third element of the last row?
first = last = nil
CSV.foreach("/Users/testUser/Entwicklung/coverrechner/CoverPaperDE.csv", col_sep: ';') do |row|
first ||= row[0]
last = row[3]
end
puts "first is #{first} and last is #{last}"
In addition, this does give the exact expected output:
csv_str = "1.00 cm; 2.00cm ; 3.00 cm; ... ; 100 cm\n2.00 cm; 4.00 cm; 6.00 cm; ... ; 100 cm\n4.00 cm; 8.00 cm; 12.00 cm; ... ; 100cm\n8.00 cm; 16.00 cm; 24.00 cm; ... ; 100cm"
CSV.parse(csv_str, col_sep: ';') do |row|
puts row[0]
end
output:
1.00 cm
2.00 cm
4.00 cm
8.00 cm
(CSV.parse is the same as your method, just parsing a string instead of using a file, used for the case of a sscce)

Resources