I have a time series dataset of accelerometry values where there are many sub-seconds of measurements but the actual number of sub-seconds recorded per second is variable.
So I would be starting with something that looks like this:
Date time
Dec sec
Acc X
1
.00
0.5
1
.25
0.5
1
.50
0.6
1
.75
0.5
2
.00
0.6
2
.40
0.5
2
.80
0.5
3
.00
0.5
3
.50
0.5
4
.00
0.6
4
.25
0.5
4
.50
0.5
4
.75
0.5
And trying to convert it to wide format where each row is a second, and the columns are the decimal seconds corresponding to each second.
sub1
sub2
sub3
sub4
.5
.5
.6
.5
.6
.5
.5
NaN
.5
.5
NaN
NaN
.6
.5
.5
.5
In code this would look like:
%Preallocate some space
Dpts_observations = NaN(13,3);
%These are the "seconds" number
Dpts_observations(:,1)=[1 1 1 1...
2 2 2...
3 3...
4 4 4 4];
%These are the "decimal seconds"
Dpts_observations(:,2) = [0.00 0.25 0.50 0.75...
0.00 0.33 0.66...
0.00 0.50 ...
0.00 0.25 0.50 0.75]
%Here's actual acceleration values
Dpts_observations(:,3) = [0.5 0.5 0.5 0.5...
0.6 0.5 0.5...
0.4 0.5...
0.5 0.5 0.6 0.4]
%I have this in a separate file but I have summary data that helps me
determine the row indexes corresponding to sub-seconds that belong to the same second and I use them to manually extract from long form to wide form.
%Create table to hold indexing information
Seconds = [1 2 3 4];
Obs_per_sec = [4 3 2 4];
Start_index = [1 5 8 10];
End_index = [4 7 9 13];
Dpts_attributes = table(Seconds, Obs_per_sec, Start_index, End_index);
%Preallocate new array
Acc_X = NaN(4,4);
%Loop through seconds
for i=1:max(size(Dpts_attributes))
Acc_X(i, 1:Dpts_attributes.Obs_per_sec(i))=Dpts_observations(Dpts_attributes.Start_index(i):Dpts_attributes.End_index(i),3);
end
Now this is working but its very slow. In reality, I have a huge data set consisting of millions of seconds and I'm hoping there might be a better solution than the one I currently have going. My data is all numeric to try to make everything as fast a possible.
Thank you!
I am trying to speed up filling in a matrix for a dynamic programming problem in Julia (v0.6.0), and I can't seem to get much extra speed from using pmap. This is related to this question I posted almost a year ago: Filling a matrix using parallel processing in Julia. I was able to speed up serial processing with some great help then, and I'm now trying to get extra speed from parallel processing tools in Julia.
For the serial processing case, I was using a 3-dimensional matrix (essentially a set of equally-sized matrices, indexed by the 1st-dimension) and iterating over the 1st-dimension. I wanted to give pmap a try, though, to more efficiently iterate over the set of matrices.
Here is the code setup. To use pmap with the v_iter function below, I converted the three dimensional matrix into a dictionary object, with the dictionary keys equal to the index values in the 1st dimension (v_dict in the code below, with gcc equal to the 1st-dimension size). The v_iter function takes other dictionary objects (E_opt_dict and gridpoint_m_dict below) as additional inputs:
function v_iter(a,b,c)
diff_v = 1
while diff_v>convcrit
diff_v = -Inf
#These lines efficiently multiply the value function by the Markov transition matrix, using the A_mul_B function
exp_v = zeros(Float64,gkpc,1)
A_mul_B!(exp_v,a[1:gkpc,:],Zprob[1,:])
for j=2:gz
temp=Array{Float64}(gkpc,1)
A_mul_B!(temp,a[(j-1)*gkpc+1:(j-1)*gkpc+gkpc,:],Zprob[j,:])
exp_v=hcat(exp_v,temp)
end
#This tries to find the optimal value of v
for h=1:gm
for j=1:gz
oldv = a[h,j]
newv = (1-tau)*b[h,j]+beta*exp_v[c[h,j],j]
a[h,j] = newv
diff_v = max(diff_v, oldv-newv, newv-oldv)
end
end
end
end
gz = 9
gp = 13
gk = 17
gcc = 5
gm = gk * gp * gcc * gz
gkpc = gk * gp * gcc
gkp = gk*gp
beta = ((1+0.015)^(-1))
tau = 0.35
Zprob = [0.43 0.38 0.15 0.03 0.00 0.00 0.00 0.00 0.00; 0.05 0.47 0.35 0.11 0.02 0.00 0.00 0.00 0.00; 0.01 0.10 0.50 0.30 0.08 0.01 0.00 0.00 0.00; 0.00 0.02 0.15 0.51 0.26 0.06 0.01 0.00 0.00; 0.00 0.00 0.03 0.21 0.52 0.21 0.03 0.00 0.00 ; 0.00 0.00 0.01 0.06 0.26 0.51 0.15 0.02 0.00 ; 0.00 0.00 0.00 0.01 0.08 0.30 0.50 0.10 0.01 ; 0.00 0.00 0.00 0.00 0.02 0.11 0.35 0.47 0.05; 0.00 0.00 0.00 0.00 0.00 0.03 0.15 0.38 0.43]
convcrit = 0.001 # chosen convergence criterion
E_opt = Array{Float64}(gcc,gm,gz)
fill!(E_opt,10.0)
gridpoint_m = Array{Int64}(gcc,gm,gz)
fill!(gridpoint_m,fld(gkp,2))
v_dict=Dict(i => zeros(Float64,gm,gz) for i=1:gcc)
E_opt_dict=Dict(i => E_opt[i,:,:] for i=1:gcc)
gridpoint_m_dict=Dict(i => gridpoint_m[i,:,:] for i=1:gcc)
For parallel processing, I executed the following two commands:
wp = CachingPool(workers())
addprocs(3)
pmap(wp,v_iter,values(v_dict),values(E_opt_dict),values(gridpoint_m_dict))
...which produced this performance:
135.626417 seconds (3.29 G allocations: 57.152 GiB, 3.74% gc time)
I then tried to serial process instead:
for i=1:gcc
v_iter(v_dict[i],E_opt_dict[i],gridpoint_m_dict[i])
end
...and received better performance.
128.263852 seconds (3.29 G allocations: 57.101 GiB, 4.53% gc time)
This also gives me about the same performance as running v_iter on the original 3-dimensional objects:
v=zeros(Float64,gcc,gm,gz)
for i=1:gcc
v_iter(v[i,:,:],E_opt[i,:,:],gridpoint_m[i,:,:])
end
I know that parallel processing involves setup time, but when I increase the value of gcc, I still get about equal processing time for serial and parallel. This seems like a good candidate for parallel processing, since there is no need for messaging between the workers! But I can't seem to make it work efficiently.
You create the CachingPool before adding the worker processes. Hence your caching pool passed to pmap tells it to use just a single worker.
You can simply check it by running wp.workers you will see something like Set([1]).
Hence it should be:
addprocs(3)
wp = CachingPool(workers())
You could also consider running Julia -p command line parameter e.g. julia -p 3 and then you can skip the addprocs(3) command.
On top of that your for and pmap loops are not equivalent. The Julia Dict object is a hashmap and similar to other languages does not offer anything like element order. Hence in your for loop you are guaranteed to get the same matching i-th element while with the values the ordering of values does not need to match the original ordering (and you can have different order for each of those three variables in the pmap loop).
Since the keys for your Dicts are just numbers from 1 up to gcc you should simply use arrays instead. You can use generators very similar to Python. For an example instead of
v_dict=Dict(i => zeros(Float64,gm,gz) for i=1:gcc)
use
v_dict_a = [zeros(Float64,gm,gz) for i=1:gcc]
Hope that helps.
Based on #Przemyslaw Szufeul's helpful advice, I've placed below the code that properly executes parallel processing. After running it once, I achieved substantial improvement in running time:
77.728264 seconds (181.20 k allocations: 12.548 MiB)
In addition to reordering the wp command and using the generator Przemyslaw recommended, I also recast v_iter as an anonymous function, in order to avoid having to sprinkle #everywhere around the code to feed functions and data to the workers.
I also added return a to the v_iter function, and set v_a below equal to the output of pmap, since you cannot pass by reference to a remote object.
addprocs(3)
v_iter = function(a,b,c)
diff_v = 1
while diff_v>convcrit
diff_v = -Inf
#These lines efficiently multiply the value function by the Markov transition matrix, using the A_mul_B function
exp_v = zeros(Float64,gkpc,1)
A_mul_B!(exp_v,a[1:gkpc,:],Zprob[1,:])
for j=2:gz
temp=Array{Float64}(gkpc,1)
A_mul_B!(temp,a[(j-1)*gkpc+1:(j-1)*gkpc+gkpc,:],Zprob[j,:])
exp_v=hcat(exp_v,temp)
end
#This tries to find the optimal value of v
for h=1:gm
for j=1:gz
oldv = a[h,j]
newv = (1-tau)*b[h,j]+beta*exp_v[c[h,j],j]
a[h,j] = newv
diff_v = max(diff_v, oldv-newv, newv-oldv)
end
end
end
return a
end
gz = 9
gp = 13
gk = 17
gcc = 5
gm = gk * gp * gcc * gz
gkpc = gk * gp * gcc
gkp =gk*gp
beta = ((1+0.015)^(-1))
tau = 0.35
Zprob = [0.43 0.38 0.15 0.03 0.00 0.00 0.00 0.00 0.00; 0.05 0.47 0.35 0.11 0.02 0.00 0.00 0.00 0.00; 0.01 0.10 0.50 0.30 0.08 0.01 0.00 0.00 0.00; 0.00 0.02 0.15 0.51 0.26 0.06 0.01 0.00 0.00; 0.00 0.00 0.03 0.21 0.52 0.21 0.03 0.00 0.00 ; 0.00 0.00 0.01 0.06 0.26 0.51 0.15 0.02 0.00 ; 0.00 0.00 0.00 0.01 0.08 0.30 0.50 0.10 0.01 ; 0.00 0.00 0.00 0.00 0.02 0.11 0.35 0.47 0.05; 0.00 0.00 0.00 0.00 0.00 0.03 0.15 0.38 0.43]
convcrit = 0.001 # chosen convergence criterion
E_opt = Array{Float64}(gcc,gm,gz)
fill!(E_opt,10.0)
gridpoint_m = Array{Int64}(gcc,gm,gz)
fill!(gridpoint_m,fld(gkp,2))
v_a=[zeros(Float64,gm,gz) for i=1:gcc]
E_opt_a=[E_opt[i,:,:] for i=1:gcc]
gridpoint_m_a=[gridpoint_m[i,:,:] for i=1:gcc]
wp = CachingPool(workers())
v_a = pmap(wp,v_iter,v_a,E_opt_a,gridpoint_m_a)
I'm trying to make an animation of Jupiter, the sun and an asteroid at the stable Lagrange point L5 as they orbit around their center of mass. I want to do this animation using gnuplot.
I have written a programme which finds their positions at time t/AU. The data I get is below, has columns, time, x position, y position, and has rows, planet, sun, asteroid. I have looked at other solutions to animating in gnuplot but they do not seem to work for me. Please help me understand what I need to type into the gnuplot command line to get an animation of this data please.
Thank you.
0 0 5.19481
0 0 -0.00519481
0 4.50634 2.6
0.01 0.0275397 5.19473
0.01 -2.75397e-05 -0.00519473
0.01 4.52006 2.57607
0.02 0.0550786 5.19451
0.02 -5.50786e-05 -0.00519451
0.02 4.53365 2.55208
0.03 0.082616 5.19415
0.03 -8.2616e-05 -0.00519415
0.03 4.54712 2.52801
0.04 0.110151 5.19364
0.04 -0.000110151 -0.00519364
0.04 4.56046 2.50386
0.05 0.137683 5.19298
0.05 -0.000137683 -0.00519298
0.05 4.57367 2.47965
0.06 0.165211 5.19218
0.06 -0.000165211 -0.00519218
0.06 4.58675 2.45537
etc...
This is just a draft:
stats 'test.txt' u 2:3
set xr [STATS_min_x:STATS_max_x]
set yr [STATS_min_y:STATS_max_y]
do for [i=0:STATS_blocks-1] {
plot 'test.txt' index i u 2:3 w p pt 7 title sprintf("time: %f",i*0.01)
pause 1
}
you can directly create an animated gif:
stats 'test.txt' u 2:3
set xr [STATS_min_x:STATS_max_x]
set yr [STATS_min_y:STATS_max_y]
set term gif animate
set output 'test.gif'
do for [i=0:STATS_blocks-1] {
plot 'test.txt' index i u 2:3 w p pt 7 title sprintf("time: %f",i*0.01)
}
Now this is quite basic, but can be tuned to make really high quality images.
I have a file with a 4x4 score matrix and I'd like to plot the upper triangular with one color palette and the lower triangular with a different one, overlaying the score value (MWE at the bottom).
The original file looks like this
0.00 0.65 0.65 0.25
0.25 0.00 0.75 0.25
0.50 0.60 0.00 0.25
0.75 0.25 0.10 0.00
First, I created two separate files and used multiplot to have 2 different palettes.
FILE1 (upper triangular)
0.00 0.65 0.65 0.25
nan 0.00 0.75 0.25
nan nan 0.00 0.25
nan nan nan 0.00
FILE2 (lower triangular)
0.00 nan nan nan
0.25 0.00 nan nan
0.50 0.60 0.00 nan
0.75 0.25 0.10 0.00
Second, I plot the score values with
using 1:2:( sprintf('%.2f', $3 ) )
However, the 'nan' isn't interpreted as blank/empty and skipped but written onto the plot.
Any idea how to skip the nans and make gnuplot plot empty labels from individual entries of the data files?
The ternary operator in the following fashion do not seem to do the job
using 1:2:( $3 == 'nan' ? 1/0 : sprintf('%.2f', $3 ))
Thanks.
set multiplot
set autoscale fix
unset key
set datafile missing "nan"
set cbrange [0:1]
unset colorbox
set palette defined (0 "white", 0.1 "#9ecae1", 1.0 "#3182bd")
plot FILE1 matrix with image, \
FILE1 matrix using 1:2:( sprintf('%.2f', $3) ) with labels font ',16'
set palette defined (0 "white", 0.1 "#a1d99b", 1.0 "#31a354")
plot FILE2 matrix with image, \
FILE2 matrix using 1:2:( sprintf('%.2f', $3) ) with labels font ',16'
unset multiplot
You don't need to use multiplot and two separate files (I also couldn't get this working with the labels).
Just define a single palette, which contains as negative values one palette and as positive values the other palette. Based on the x and y-value from the single file you show first, you can now distinguish if the color value should be taken from the negative or from the positive palette part:
set autoscale fix
set cbrange [-1:1]
unset colorbox
unset key
set palette defined (-1.0 "#31a354", -0.1 "#a1d99b", 0 "white", 0.1 "#9ecae1", 1.0 "#3182bd")
plot 'FILE' matrix using 1:2:($1<$2 ? -$3 : $3) with image,\
'' matrix using 1:2:(sprintf('%.2f', $3)) with labels font ',16'
I'm making a chart but I would like to use lines rather than points.
Using the style of lines, all the points are connected and the graph has a network appearance, which I don't want.
set grid
set ticslevel 0.1
set samples 51, 51
set isosamples 20, 20
set border 1+2+4+8
unset key
splot 'matrix.dat' matrix
part of data to matrix plot
0.261 0.665 0.225 0.382 0.255 0.574 0.356
0.338 0.845 0.0363 0.167 0.727 0.0805 0.764
0.225 0.196 0.107 0.153 0.347 0.338 0.168
0.157 0.443 0.0671 0.135 0.312 0.408 0.362
0.151 0.281 0.0572 0.103 0.309 0.49 0.242
0.12 0.336 0.0604 0.173 0.19 0.395 0.153
0.119 0.173 0.0336 0.145 0.156 0.219 0.177
0.123 0.0452 0.0165 0.149 0.0932 0.0663 0.133
0.123 0.0741 0.00373 0.136 0.0346 0.485 0.131
0.111 0.241 0.0124 0.105 0.0127 1.01 0.122
0.096 0.475 0.0194 0.0569 0.0284 1.67 0.102
0.0777 0.773 0.0175 0.00929 0.0375 2.42 0.0831
0.059 1.11 0.0123 0.0322 0.0408 3.23 0.0635
0.0438 1.48 6.44E-4 0.0659 0.0265 4.07 0.0445
0.0349 1.92 0.0192 0.078 0.00585 4.92 0.0254
0.0392 2.42 0.0446 0.0632 0.0306 5.73 0.00774
0.0518 2.97 0.0745 0.031 0.0729 6.46 0.00716
This cannot be done automatically. You must determine the rows and columns of your matrix. First, to get the number of rows, use
stats 'matrix.dat' using 1 nooutput
rows = STATS_records
For the number of columns, use then
stats 'matrix.dat' matrix nooutput
cols = STATS_records/rows
And now plot every line
unset key
splot for [i=0:cols-1] 'matrix.dat' matrix every ::i::i lt 1 with lines
Result (with 4.6.4) is:
I think Christoph's solution is just what you need, but to make the point clear, by providing the matrix and using splot matrix alone will just generate a mesh.
So you will need to specify the lines with complete X, Y and Z vectors and then plot them using splot with lines/linespoints. I'm adding an example below in case it may be helpful for anyone else.
You arrange your data file as follows:
10 1 0.261 2 0.665 3 0.225 4 0.382 5 0.255 6 0.574 7 0.356
20 1 0.338 2 0.845 3 0.0363 4 0.167 5 0.727 6 0.0805 7 0.764
30 1 0.225 2 0.196 3 0.107 4 0.153 5 0.347 6 0.338 7 0.168
40 1 0.157 2 0.443 3 0.0671 4 0.135 5 0.312 6 0.408 7 0.362
And then plot as follows:
set grid
set ticslevel 0.1
#set samples 51, 51
#set isosamples 20, 20
#set border 1+2+4+8
unset key
splot 'matrix.dat' using 1:2:3 with linespoints, \
'matrix.dat' using 1:4:5 with linespoints, \
'matrix.dat' using 1:6:7 with linespoints, \
'matrix.dat' using 1:8:9 with linespoints, \
'matrix.dat' using 1:10:11 with linespoints, \
'matrix.dat' using 1:12:13 with linespoints, \
'matrix.dat' using 1:14:15 with linespoints
With the resultant plot