I want to try adpat quantile regression to caret package. I wrote below code to adapt but I get the error as below:
library(quantreg)
quantregression <- list(type='Regression', library='quantreg',loop=NULL)
param <- data.frame(parameter = c("tau"), class = c("numeric"), label = c("tau"))
quantregression$parameters <- param
quantGrid <- function(x, y, len=NULL, search="grid"){
if(search == "grid"){
out <- expand.grid(tau=c(.05, .25, .5, .75, .95))
}
out
}
quantregression$grid <- quantGrid
quantregFit <- function(x, y, wts, param, lev, last, weights, classProbs, ...){
dat <- if(is.data.frame(x)) x else as.data.frame(x)
dat$.outcome <- y
if(!is.null(wts))
{
if (param$tau)
out <- quantreg::rq(.outcome ~ ., data = dat, weights = wts,tau=tau, ...)
else
out <- quantreg::rq(.outcome ~0 + ., data = dat, weights = wts,tau=tau, ...)
} else
{
if (param$tau)
out <- quantreg::rq(.outcome ~ ., data = dat,tau=tau, ...)
else
out <- quantreg::rq(.outcome ~0 + ., data = dat,tau=tau, ...)
}
out
}
quantregression$fit <- quantregFit
quantPred <- function(modelFit, newdata, preProc=NULL, submodels=NULL){
if(!is.data.frame(newdata)) newdata <- as.data.frame(newdata)
quantreg::predict.rq(modelFit, newdata)
}
quantregression$predict <- quantPred
quantProb <- function(){
return(NULL)
}
quantregression$prob <- quantProb
I get the error as below:
Something is wrong; all the RMSE metric values are missing:
RMSE Rsquared MAE
Min. : NA Min. : NA Min. : NA
1st Qu.: NA 1st Qu.: NA 1st Qu.: NA
Median : NA Median : NA Median : NA
Mean :NaN Mean :NaN Mean :NaN
3rd Qu.: NA 3rd Qu.: NA 3rd Qu.: NA
Max. : NA Max. : NA Max. : NA
NA's :5 NA's :5 NA's :5
Error: Stopping
In addition: There were 50 or more warnings (use warnings() to see the first 50)
warnings()
Warning messages:
1: model fit failed for Fold01.Rep1: tau=0.05 Error in unique(tau) : object 'tau' not found
I want to adopt the quantile regression using "quantreg" package for caret to obtain cross validation results for quantile regression. I wrote above code but I could not find the error in the code.
I am trying to shift the Pandas dataframe column data by group of first index. Here is the demo code:
In [8]: df = mul_df(5,4,3)
In [9]: df
Out[9]:
COL000 COL001 COL002
STK_ID RPT_Date
A0000 B000 -0.5505 0.7445 -0.3645
B001 0.9129 -1.0473 -0.5478
B002 0.8016 0.0292 0.9002
B003 2.0744 -0.2942 -0.7117
A0001 B000 0.7064 0.9636 0.2805
B001 0.4763 0.2741 -1.2437
B002 1.1563 0.0525 -0.7603
B003 -0.4334 0.2510 -0.0105
A0002 B000 -0.6443 0.1723 0.2657
B001 1.0719 0.0538 -0.0641
B002 0.6787 -0.3386 0.6757
B003 -0.3940 -1.2927 0.3892
A0003 B000 -0.5862 -0.6320 0.6196
B001 -0.1129 -0.9774 0.7112
B002 0.6303 -1.2849 -0.4777
B003 0.5046 -0.4717 -0.2133
A0004 B000 1.6420 -0.9441 1.7167
B001 0.1487 0.1239 0.6848
B002 0.6139 -1.9085 -1.9508
B003 0.3408 -1.3891 0.6739
In [10]: grp = df.groupby(level=df.index.names[0])
In [11]: grp.shift(1)
Out[11]:
COL000 COL001 COL002
STK_ID RPT_Date
A0000 B000 NaN NaN NaN
B001 -0.5505 0.7445 -0.3645
B002 0.9129 -1.0473 -0.5478
B003 0.8016 0.0292 0.9002
A0001 B000 NaN NaN NaN
B001 0.7064 0.9636 0.2805
B002 0.4763 0.2741 -1.2437
B003 1.1563 0.0525 -0.7603
A0002 B000 NaN NaN NaN
B001 -0.6443 0.1723 0.2657
B002 1.0719 0.0538 -0.0641
B003 0.6787 -0.3386 0.6757
A0003 B000 NaN NaN NaN
B001 -0.5862 -0.6320 0.6196
B002 -0.1129 -0.9774 0.7112
B003 0.6303 -1.2849 -0.4777
A0004 B000 NaN NaN NaN
B001 1.6420 -0.9441 1.7167
B002 0.1487 0.1239 0.6848
B003 0.6139 -1.9085 -1.9508
The mul_df() code is attached here : How to speed up Pandas multilevel dataframe sum?
Now I want to grp.shift(1) for a big dataframe.
In [1]: df = mul_df(5000,30,400)
In [2]: grp = df.groupby(level=df.index.names[0])
In [3]: timeit grp.shift(1)
1 loops, best of 3: 5.23 s per loop
5.23s is too slow. How to speed it up ?
(My computer configuration is: Pentium Dual-Core T4200#2.00GHZ, 3.00GB RAM, WindowXP, Python 2.7.4, Numpy 1.7.1, Pandas 0.11.0, numexpr 2.0.1 , Anaconda 1.5.0 (32-bit))
How about shift the total DataFrame object and then set the first row of every group to NaN?
dfs = df.shift(1)
dfs.iloc[df.groupby(level=0).size().cumsum()[:-1]] = np.nan
the problem is that the shift operation is not cython optimized, so it involves callback to python. Compare this with:
In [84]: %timeit grp.shift(1)
1 loops, best of 3: 1.77 s per loop
In [85]: %timeit grp.sum()
1 loops, best of 3: 202 ms per loop
added an issue for this: https://github.com/pydata/pandas/issues/4095
similar question and added answer with that works for shift in either direction and magnitude: pandas: setting last N rows of multi-index to Nan for speeding up groupby with shift
Code (including test setup) is:
#
# the function to use in apply
#
def replace_shift_overlap(grp,col,N,value):
if (N > 0):
grp[col][:N] = value
else:
grp[col][N:] = value
return grp
length = 5
groups = 3
rng1 = pd.date_range('1/1/1990', periods=length, freq='D')
frames = []
for x in xrange(0,groups):
tmpdf = pd.DataFrame({'date':rng1,'category':int(10000000*abs(np.random.randn())),'colA':np.random.randn(length),'colB':np.random.randn(length)})
frames.append(tmpdf)
df = pd.concat(frames)
df.sort(columns=['category','date'],inplace=True)
df.set_index(['category','date'],inplace=True,drop=True)
shiftBy=-1
df['tmpShift'] = df['colB'].shift(shiftBy)
#
# the apply
#
df = df.groupby(level=0).apply(replace_shift_overlap,'tmpShift',shiftBy,np.nan)
# Yay this is so much faster.
df['newColumn'] = df['tmpShift'] / df['colA']
df.drop('tmpShift',1,inplace=True)
EDIT: Note that the initial sort really eats into the effectiveness of this. So in some cases the original answer is more effective.
try this:
import numpy as np
import pandas as pd
df = pd.DataFrame({'A': [10, 20, 15, 30, 45,43,67,22,12,14,54],
'B': [13, 23, 18, 33, 48, 1,7, 56,66,45,32],
'C': [17, 27, 22, 37, 52,77,34,21,22,90,8],
'D': ['a','a','a','a','b','b','b','c','c','c','c']
})
df
#> A B C D
#> 0 10 13 17 a
#> 1 20 23 27 a
#> 2 15 18 22 a
#> 3 30 33 37 a
#> 4 45 48 52 b
#> 5 43 1 77 b
#> 6 67 7 34 b
#> 7 22 56 21 c
#> 8 12 66 22 c
#> 9 14 45 90 c
#> 10 54 32 8 c
def groupby_shift(df, col, groupcol, shift_n, fill_na = np.nan):
'''df: dataframe
col: column need to be shifted
groupcol: group variable
shift_n: how much need to shift
fill_na: how to fill nan value, default is np.nan
'''
rowno = list(df.groupby(groupcol).size().cumsum())
lagged_col = df[col].shift(shift_n)
na_rows = [i for i in range(shift_n)]
for i in rowno:
if i == rowno[len(rowno)-1]:
continue
else:
new = [i + j for j in range(shift_n)]
na_rows.extend(new)
na_rows = list(set(na_rows))
na_rows = [i for i in na_rows if i <= len(lagged_col) - 1]
lagged_col.iloc[na_rows] = fill_na
return lagged_col
df['A_lag_1'] = groupby_shift(df, 'A', 'D', 1)
df
#> A B C D A_lag_1
#> 0 10 13 17 a NaN
#> 1 20 23 27 a 10.0
#> 2 15 18 22 a 20.0
#> 3 30 33 37 a 15.0
#> 4 45 48 52 b NaN
#> 5 43 1 77 b 45.0
#> 6 67 7 34 b 43.0
#> 7 22 56 21 c NaN
#> 8 12 66 22 c 22.0
#> 9 14 45 90 c 12.0
#> 10 54 32 8 c 14.0
I have this data
y x1 x2 pre
1 16 1 1 14
2 15 1 1 13
3 14 1 2 14
4 13 1 2 13
5 12 2 1 12
6 11 2 1 12
7 11 2 2 13
8 13 2 2 13
9 10 3 1 10
10 11 3 1 11
11 11 3 2 11
12 9 3 2 10
And I fitted the following model
lm(y ~ x1 + x2 + x1*x2)
My design matrix is
[,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,] 1 14 1 0 1 1 0
[2,] 1 13 1 0 1 1 0
[3,] 1 14 1 0 0 0 0
[4,] 1 13 1 0 0 0 0
[5,] 1 12 0 1 1 0 1
[6,] 1 12 0 1 1 0 1
[7,] 1 13 0 1 0 0 0
[8,] 1 13 0 1 0 0 0
[9,] 1 10 0 0 1 0 0
[10,] 1 11 0 0 1 0 0
[11,] 1 11 0 0 0 0 0
[12,] 1 10 0 0 0 0 0
I'm trying to use this design to reproduce the following table:
Source DF Squares Mean Square F Value Pr > F
Model 6 44.79166667 7.46527778 12.98 0.0064
Error 5 2.87500000 0.57500000
Corrected Total 11 47.66666667
Source DF Type III SS Mean Square F Value Pr > F
pre 1 3.12500000 3.12500000 5.43 0.0671
x1 2 4.58064516 2.29032258 3.98 0.0923
x2 1 3.01785714 3.01785714 5.25 0.0706
x1*x2 2 1.25000000 0.62500000 1.09 0.4055
The first part is fine
XtX <- t(x) %*% x
XtXinv <- solve(XtX)
betahat <- XtXinv %*% t(x) %*% y
H <- x %*% XtXinv %*% t(x)
IH <- (diag(1,12) - H)
yhat <- H %*% y
e <- IH %*% y
ybar <- mean(y)
MSS <- t(betahat) %*% t(x) %*% y - length(y)*(ybar^2)
ESS <- t(e) %*% e
TSS <- MSS + ESS
dfM <- sum(diag(H)) - 1
dfE <- sum(diag(IH))
dfT <- dfM + dfE
MSM <- MSS/dfM
MSE <- ESS/dfE
Ftest <- MSM / MSE
pr <- 1 - pf(Ftest, dfM, dfE)
The contrast coefficient matrix for 'pre' seems correct.
L <- matrix(c(0,1,0,0,0,0,0), 1, 7, byrow=T)
Lb <- L %*% betahat
LXtXinvLt <- round(L %*% XtXinv %*% t(L), digits=4)
SSpre <- t(Lb) %*% solve(LXtXinvLt) %*% (Lb)
MSpre <- SSpre / 1
Fpre <- MSpre / MSE
PRpre <- 1 - pf(Fpre, 1, 12-7)
But I can't understand how to define the contrast coefficient matrix for x1, x2, and x1*x2. What's the problem with the rest of my code? Below an example for how I think I should calculate for x1
L <- matrix(c(0,0,1,1,0,0,0), 1, 7, byrow=T)
Lb <- L %*% betahat
LXtXinvLt <- round(L %*% XtXinv %*% t(L), digits=4)
SSX1 <- t(Lb) %*% solve(LXtXinvLt) %*% (Lb)
MSX1 <- SSX1 / 1
FX1 <- MSX1 / MSE
PRX1 <- 1 - pf(FX1, 1, 12-7)
Thanks!
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
Generate a list of lists (or print, I don't mind) a Pascal's Triangle of size N with the least lines of code possible!
Here goes my attempt (118 characters in python 2.6 using a trick):
c,z,k=locals,[0],'_[1]'
p=lambda n:[len(c()[k])and map(sum,zip(z+c()[k][-1],c()[k][-1]+z))or[1]for _ in range(n)]
Explanation:
the first element of the list comprehension (when the length is 0) is [1]
the next elements are obtained the following way:
take the previous list and make two lists, one padded with a 0 at the beginning and the other at the end.
e.g. for the 2nd step, we take [1] and make [0,1] and [1,0]
sum the two new lists element by element
e.g. we make a new list [(0,1),(1,0)] and map with sum.
repeat n times and that's all.
usage (with pretty printing, actually out of the code-golf xD):
result = p(10)
lines = [" ".join(map(str, x)) for x in result]
for i in lines:
print i.center(max(map(len, lines)))
output:
1
1 1
1 2 1
1 3 3 1
1 4 6 4 1
1 5 10 10 5 1
1 6 15 20 15 6 1
1 7 21 35 35 21 7 1
1 8 28 56 70 56 28 8 1
1 9 36 84 126 126 84 36 9 1
K (Wikipedia), 15 characters:
p:{x{+':x,0}\1}
Example output:
p 10
(1
1 1
1 2 1
1 3 3 1
1 4 6 4 1
1 5 10 10 5 1
1 6 15 20 15 6 1
1 7 21 35 35 21 7 1
1 8 28 56 70 56 28 8 1
1 9 36 84 126 126 84 36 9 1
1 10 45 120 210 252 210 120 45 10 1)
It's also easily explained:
p:{x {+':x,0} \ 1}
^ ^------^ ^ ^
A B C D
p is a function taking an implicit parameter x.
p unfolds (C) an anonymous function (B) x times (A) starting at 1 (D).
The anonymous function simply takes a list x, appends 0 and returns a result by adding (+) each adjacent pair (':) of values: so e.g. starting with (1 2 1), it'll produce (1 2 1 0), add pairs (1 1+2 2+1 1+0), giving (1 3 3 1).
Update: Adapted to K4, which shaves off another two characters. For reference, here's the original K3 version:
p:{x{+':0,x,0}\1}
J, another language in the APL family, 9 characters:
p=:!/~#i.
This uses J's builtin "combinations" verb.
Output:
p 10
1 1 1 1 1 1 1 1 1 1
0 1 2 3 4 5 6 7 8 9
0 0 1 3 6 10 15 21 28 36
0 0 0 1 4 10 20 35 56 84
0 0 0 0 1 5 15 35 70 126
0 0 0 0 0 1 6 21 56 126
0 0 0 0 0 0 1 7 28 84
0 0 0 0 0 0 0 1 8 36
0 0 0 0 0 0 0 0 1 9
0 0 0 0 0 0 0 0 0 1
Haskell, 58 characters:
r 0=[1]
r(n+1)=zipWith(+)(0:r n)$r n++[0]
p n=map r[0..n]
Output:
*Main> p 5
[[1],[1,1],[1,2,1],[1,3,3,1],[1,4,6,4,1],[1,5,10,10,5,1]]
More readable:
-- # row 0 is just [1]
row 0 = [1]
-- # row (n+1) is calculated from the previous row
row (n+1) = zipWith (+) ([0] ++ row n) (row n ++ [0])
-- # use that for a list of the first n+1 rows
pascal n = map row [0..n]
69C in C:
f(int*t){int*l=t+*t,*p=t,r=*t,j=0;for(*t=1;l<t+r*r;j=*p++)*l++=j+*p;}
Use it like so:
int main()
{
#define N 10
int i, j;
int t[N*N] = {N};
f(t);
for (i = 0; i < N; i++)
{
for (j = 0; j <= i; j++)
printf("%d ", t[i*N + j]);
putchar('\n');
}
return 0;
}
F#: 81 chars
let f=bigint.Factorial
let p x=[for n in 0I..x->[for k in 0I..n->f n/f k/f(n-k)]]
Explanation: I'm too lazy to be as clever as the Haskell and K programmers, so I took the straight forward route: each element in Pascal's triangle can be uniquely identified using a row n and col k, where the value of each element is n!/(k! (n-k)!.
Python: 75 characters
def G(n):R=[[1]];exec"R+=[map(sum,zip(R[-1]+[0],[0]+R[-1]))];"*~-n;return R
Shorter prolog version (112 instead of 164):
n([X],[X]).
n([H,I|T],[A|B]):-n([I|T],B),A is H+I.
p(0,[[1]]):-!.
p(N,[R,S|T]):-O is N-1,p(O,[S|T]),n([0|S],R).
another stab (python):
def pascals_triangle(n):
x=[[1]]
for i in range(n-1):
x.append(list(map(sum,zip([0]+x[-1],x[-1]+[0]))))
return x
Haskell, 164C with formatting:
i l=zipWith(+)(0:l)$l++[0]
fp=map (concatMap$(' ':).show)f$iterate i[1]
c n l=if(length l<n)then c n$' ':l++" "else l
cl l=map(c(length$last l))l
pt n=cl$take n fp
Without formatting, 52C:
i l=zipWith(+)(0:l)$l++[0]
pt n=take n$iterate i[1]
A more readable form of it:
iterateStep row = zipWith (+) (0:row) (row++[0])
pascalsTriangle n = take n $ iterate iterateStep [1]
-- For the formatted version, we reduce the number of rows at the final step:
formatRow r = concatMap (\l -> ' ':(show l)) r
formattedLines = map formatRow $ iterate iterateStep [1]
centerTo width line =
if length line < width
then centerTo width (" " ++ line ++ " ")
else line
centerLines lines = map (centerTo (length $ last lines)) lines
pascalsTriangle n = centerLines $ take n formattedLines
And perl, 111C, no centering:
$n=<>;$p=' 1 ';for(1..$n){print"$p\n";$x=" ";while($p=~s/^(?= ?\d)(\d* ?)(\d* ?)/$2/){$x.=($1+$2)." ";}$p=$x;}
Scheme — compressed version of 100 characters
(define(P h)(define(l i r)(if(> i h)'()(cons r(l(1+ i)(map +(cons 0 r)(append r '(0))))))(l 1 '(1)))
This is it in a more readable form (269 characters):
(define (pascal height)
(define (next-row row)
(map +
(cons 0 row)
(append row '(0))))
(define (iter i row)
(if (> i height)
'()
(cons row
(iter (1+ i)
(next-row row)))))
(iter 1 '(1)))
VBA/VB6 (392 chars w/ formatting)
Public Function PascalsTriangle(ByVal pRows As Integer)
Dim iRow As Integer
Dim iCol As Integer
Dim lValue As Long
Dim sLine As String
For iRow = 1 To pRows
sLine = ""
For iCol = 1 To iRow
If iCol = 1 Then
lValue = 1
Else
lValue = lValue * (iRow - iCol + 1) / (iCol - 1)
End If
sLine = sLine & " " & lValue
Next
Debug.Print sLine
Next
End Function
PHP 100 characters
$v[]=1;while($a<34){echo join(" ",$v)."\n";$a++;for($k=0;$k<=$a;$k++)$t[$k]=$v[$k-1]+$v[$k];$v=$t;}
Ruby, 83c:
def p(n);n>0?(m=p(n-1);k=m.last;m+[([0]+k).zip(k+[0]).map{|x|x[0]+x[1]}]):[[1]];end
test:
irb(main):001:0> def p(n);n>0?(m=p(n-1);k=m.last;m+[([0]+k).zip(k+[0]).map{|x|x[0]+x[1]}]):[[1]];end
=> nil
irb(main):002:0> p(5)
=> [[1], [1, 1], [1, 2, 1], [1, 3, 3, 1], [1, 4, 6, 4, 1], [1, 5, 10, 10, 5, 1]]
irb(main):003:0>
Another python solution, that could be much shorter if the builtin functions had shorter names... 106 characters.
from itertools import*
r=range
p=lambda n:[[len(list(combinations(r(i),j)))for j in r(i+1)]for i in r(n)]
Another try, in prolog (I'm practising xD), not too short, just 164c:
s([],[],[]).
s([H|T],[J|U],[K|V]):-s(T,U,V),K is H+J.
l([1],0).
l(P,N):-M is N-1,l(A,M),append(A,[0],B),s(B,[0|A],P).
p([],-1).
p([H|T],N):-M is N-1,l(H,N),p(T,M).
explanation:
s = sum lists element by element
l = the Nth row of the triangle
p = the whole triangle of size N
VBA, 122 chars:
Sub p(n)
For r = 1 To n
l = "1"
v = 1
For c = 1 To r - 1
v = v / c * (r - c)
l = l & " " & v
Next
Debug.Print l
Next
End Sub
I wrote this C++ version a few years ago:
#include <iostream>
int main(int,char**a){for(int b=0,c=0,d=0,e=0,f=0,g=0,h=0,i=0;b<atoi(a[1]);(d|f|h)>1?e*=d>1?--d:1,g*=f>1?--f:1,i*=h>1?--h:1:((std::cout<<(i*g?e/(i*g):1)<<" "?d=b+=c++==b?c=0,std::cout<<std::endl?1:0:0,h=d-(f=c):0),e=d,g=f,i=h));}
The following is just a Scala function returning a List[List[Int]]. No pretty printing or anything. Any suggested improvements? (I know it's inefficient, but that's not the main challenge now, is it?). 145 C.
def p(n: Int)={def h(n:Int):List[Int]=n match{case 1=>1::Nil;case _=>(0::h(n-1) zipAll(h(n-1),0,0)).map{n=>n._1+n._2}};(1 to n).toList.map(h(_))}
Or perhaps:
def pascal(n: Int) = {
def helper(n: Int): List[Int] = n match {
case 1 => 1 :: List()
case _ => (0 :: helper(n-1) zipAll (helper(n-1),0,0)).map{ n => n._1 + n._2 }
}
(1 to n).toList.map(helper(_))
}
(I'm a Scala noob, so please be nice to me :D )
a Perl version (139 chars w/o shebang)
#p = (1,1);
while ($#p < 20) {
#q =();
$z = 0;
push #p, 0;
foreach (#p) {
push #q, $_+$z;
$z = $_
}
#p = #q;
print "#p\n";
}
output starts from 1 2 1
PHP, 115 chars
$t[][]=1;
for($i=1;$i<$n;++$i){
$t[$i][0]=1;
for($j=1;$j<$i;++$j)$t[$i][$j]=$t[$i-1][$j-1]+$t[$i-1][$j];
$t[$i][$i]=1;}
If you don't care whether print_r() displays the output array in the correct order, you can shave it to 113 chars like
$t[][]=1;
for($i=1;$i<$n;++$i){
$t[$i][0]=$t[$i][$i]=1;
for($j=1;$j<$i;++$j)$t[$i][$j]=$t[$i-1][$j-1]+$t[$i-1][$j];}
Perl, 63 characters:
for(0..9){push#z,1;say"#z";#z=(1,map{$z[$_-1]+$z[$_]}(1..$#z))}
My attempt in C++ (378c). Not anywhere near as good as the rest of the posts.. but I'm proud of myself for coming up with a solution on my own =)
int* pt(int n)
{
int s=n*(n+1)/2;
int* t=new int[s];
for(int i=0;i<n;++i)
for(int j=0;j<=i;++j)
t[i*n+j] = (!j || j==i) ? 1 : t[(i-1)*n+(j-1)] + t[(i-1)*n+j];
return t;
}
int main()
{
int n,*t;
std::cin>>n;
t=pt(n);
for(int i=0;i<n;++i)
{
for(int j=0;j<=i;j++)
std::cout<<t[i*n+j]<<' ';
std::cout<<"\n";
}
}
Old thread, but I wrote this in response to a challenge on another forum today:
def pascals_triangle(n):
x=[[1]]
for i in range(n-1):
x.append([sum(i) for i in zip([0]+x[-1],x[-1]+[0])])
return x
for x in pascals_triangle(5):
print('{0:^16}'.format(x))
[1]
[1, 1]
[1, 2, 1]
[1, 3, 3, 1]
[1, 4, 6, 4, 1]