Please find below a minimum example of how I iterate through time in xarray.
ds = xr.Dataset({'time': pd.date_range(start='1/1/2018', periods=8)})
for ii, date in enumerate(ds.time):
nd = date.data
nd is a numpy.ndarray but of size = 1; no shape: shape = () and 0-dimension: dims = 0.
I can access the element through nd[()] (it took me a while, thx Clive), but I wonder if it is something we should expect or if it is a bug.
If there is a better way to enumerate through my date, please let me know or point me out where to find it.
The nd array as a 0d array is a feature; explained here: https://stackoverflow.com/a/49621796/3064736.
There is a small bug given a recent pandas change such that nd.item() returns an int rather than a date on the most recent versions of xarray & pandas. That's being tracked here: https://github.com/pydata/xarray/pull/4292.
Generally we would want nd=data.item()
Related
In Stata, after running the xthtaylor command, the command
matrix regtab = r(table)
yields an empty matrix. I think this is because of the multilevel of the output of this command
Being new to Stata, I haven't found how to fix this. The purpose here is to extract the coeffecient and standard errors to add them to another output (as is done in the accepted solution of How do I create a table wth both plain and robust standard errors?)
To expand on Nick's point: matrix regtab = r(table) gives you an empty matrix, because xthtaylor doesn't put anything into r(table).
To see this run the following example:
clear all // empties r(table) and everything else
webuse psidextract
* the example regression from `help xthtaylor`
xthtaylor lwage wks south smsa ms exp exp2 occ ind union fem blk ed, endog(exp exp2 occ ind union ed) constant(fem blk ed)
return list doesn't have anything in r(table), but ereturn list will show you that you have access to the coefficients through e(b) and the variance-covariance matrix through e(V).
You can assign these to their own matrices as follows:
matrix betas = e(b)
matrix varcovar = e(V)
Then you can use matrix commands (see help matrix) to manipulate these matrices.
As you discovered, ereturn display creates r(table) which appears quite convenient for your use. It's worth taking a look at help return for more information about the differences between the contents of return list and ereturn list.
Can anyone kindly help to adjust the remaining code as I'm confused with that about the Google Page Rank Algorithm using PySpark. Thanks a lot.
I have done some parts:
def computeContribs(neighbors, rank):
for neighbor in neighbors:
yield (neighbor, rank/len(neighbors))
rdd = sc.textFile('network.txt').persist()
linksRDD = rdd.map(lambda x:tuple(x.split(" "))).map(lambda x:(x[0],[x[1]])).reduceByKey(lambda x, y: x+y).collect()
linksRDD2 = sc.parallelize (linksRDD)
ranksRDD = linksRDD2.map(lambda x:(x[0],1.0)).collect()
but I'm confused with the calculation of the contribution of each page's outgoing link.
contribs =
update each page's page rank by summing up all incoming link's contribution
ranksRDD =
notebook link: https://colab.research.google.com/drive/1g5E-tqGN8u8cioUSqPNhCM5SIdxO0cLB
Thanks
The spark graphframes package has two pagerank implementations. You can use those out of box implementations than writing yours.
https://graphframes.github.io/graphframes/docs/_site/user-guide.html#pagerank
Yesterday, I asked about From a List representation of a Map, to a real Map in Scala
After few really smart response, I know that it is necessary change my mind to work in Scala and I am sure that stackoverflow is the way.
In this occasion, I want to decompress a sequence of number stored using delta compression.
I think that my implementation is really simple, but I am sure that you guys are going to find other more functional way.
def deltaDecompression(compressed : Seq[Long]) = {
var previous = 0L
compressed.map(current => {
previous += current
previous
})
}
assert(deltaDecompression(Seq(100,1,2,3)) == Seq(100,101,103,106))
So like in my previous question, the question is: Is possible to implement this function using a more functional way?
Example input data and expected output in the last line of the code, as an assertion.
compressed.scanLeft(0l) { _ + _ }.drop(1)
I'm trying to speed up my code using parfor. The purpose of the code is to slide a 3D square window on a 3D image and for each block of mxmxm apply a function.
I wrote this code:
function [ o_image ] = SlidingWindow( i_image, i_padSize, i_fun, i_options )
%SLIDINGWINDOW Summary of this function goes here
% Detailed explanation goes here
o_image = zeros(size(i_image,1),size(i_image,2),size(i_image,3));
i_image = padarray(i_image,i_padSize,'symmetric');
i_padSize = num2cell(i_padSize);
[m,n,p] = deal(i_padSize{:});
[row,col,depth] = size(i_image);
windowShape = i_options.windowShape;
mask = i_options.mask;
parfor (i = m+1:row-m,i_options.cores)
temp = i_image(i-m:i+m,:,:);
for j = n+1:col-n
for h = p+1:depth-p
ii = i-m;
jj = j-n;
hh = h-p;
temp = temp(:,j-n:j+n, h-p:h+p);
o_image(ii,jj,hh) = parfeval(i_fun, temp, windowShape, mask);
end
end
end
end
I get one warning and one error that I don't understand how to solve.
The warning says:
the entire array or structure 'i_image' is a broadcast variable.
The error says:
the PARFOR loop can not run due to the way variable 'o_image' is used.
I don't understand how to fix these two things. Any help is greatly appreciated!
As far as I understand, parfeval takes care of running your function on the available number of workers, which is why it doesn't need to be surrounded by parfor. Assuming you already have an active parpool, changing the external parfor into for eliminates both problems.
Unfortunately, I can't support my answer with a benchmark or suggest a more fitting solution because your inputs are unknown.
It seems to me that the code can be optimized in other ways, mainly by vectorization. I would suggest you looked into the following resources:
This question, for additional info on parfeval.
Examples on how to use bsxfun and permute and benchmarks thereof: ex1, ex2, ex3.
P.S.: The 2nd part of (i = m+1:row-m,i_options.cores) seems out of place...
Here is my problem.
I'm using sympy and a complex matrix P (all elements of P are complex valued).
I wanna extract the real/imaginary part of the first row.
So, I use the following sequence:
import sympy as sp
P = sp.Matrix([ [a+sp.I*b,c-sp.I*d], [c-sp.I*d,a+sp.I*b] ])
Row = P.row(0)
Row.as_mutable()
Re_row = sp.re(Row)
Im_row = sp.im(Row)
But the code returns me the following error:
"AttributeError: ImmutableMatrix has no attribute as_coefficient."
The error occurs during the operation sp.re(Row) and sp.im(Row)...
Sympy tells me that Row is an Immutable matrix but I specify that I want a mutable one...
So I'm in a dead end, and I don't have the solution...
Could someone plz help me ?
thank you very much !
Most SymPy functions won't work if you just pass a Matrix to them directly. You need to use the methods of the Matrix, or if there is not such method (as is the case here), use applyfunc
In [34]: Row.applyfunc(re)
Out[34]: [re(a) - im(b) re(c) + im(d)]
In [35]: Row.applyfunc(im)
Out[35]: [re(b) + im(a) -re(d) + im(c)]
(I've defined a, b, c, and d as just ordinary symbols here, if you set them as real the answer will come out much simpler).