I feel like my run time is extremely slow for my data set, this is the code:
library(caret)
library(data.table)
knnImputeValues <- preProcess(mainData[trainingRows, imputeColumns], method = c("zv", "knnImpute"))
knnTransformed <- predict(knnImputeValues, mainData[ 1:1000, imputeColumns])
the PreProcess into knnImputeValues run's fairly quickly, however the predict function takes a tremendous amount of time. When I calculated it on a subset of the data this was the result:
testtime <- system.time(knnTransformed <- predict(knnImputeValues, mainData[ 1:15000, imputeColumns
testtime
user 969.78
system 38.70
elapsed 1010.72
Additionally, it should be noted that caret preprocess uses "RANN".
Now my full dataset is:
str(mainData[ , imputeColumns])
'data.frame': 1809032 obs. of 16 variables:
$ V1: int 3 5 5 4 4 4 3 4 3 3 ...
$ V2: Factor w/ 3 levels "1000000","1500000",..: 1 1 3 1 1 1 1 3 1 1 ...
$ V3: Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2 2 2 2 ...
$ V4: int 2 5 5 12 4 5 11 8 7 8 ...
$ V5: int 2 0 0 2 0 0 1 3 2 8 ...
$ V6: int 648 489 489 472 472 472 497 642 696 696 ...
$ V7: Factor w/ 4 levels "","N","U","Y": 4 1 1 1 1 1 1 1 1 1 ...
$ V8: int 0 0 0 0 0 0 0 1 1 1 ...
$ V9: num 0 0 0 0 0 ...
$ V10: Factor w/ 56 levels "1","2","3","4",..: 45 19 19 19 19 19 19 46 46 46 ...
$ V11: Factor w/ 2 levels "0","1": 2 2 2 2 2 2 2 2 2 2 ...
$ V12: num 2 5 5 12 4 5 11 8 7 8 ...
$ V13: num 2 0 0 2 0 0 1 3 2 8 ...
$ V14: Factor w/ 4 levels "1","2","3","4": 2 2 2 2 2 2 2 2 3 3 ...
$ V15: Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 2 2 2 ...
$ V16: num 657 756 756 756 756 ...
So is there something I'm doing wrong, or is this typical for how long it will take to run this? If you back of the envelop extrapolate (which I know isn't entire accurate) you'd get what 33 days?
Also it looks like system time is very low and user time is very high, is that normal?
My computer is a laptop, with a Intel(R) Core(TM) i5-6300U CPU # 2.40Ghz processor.
Additionally would this improve the runtime of the predict function?
cl <- makeCluster(4)
registerDoParallel()
I tried it, and it didn't seem to make a difference other than all the processors looked more active in my task manager.
FOCUSED QUESTION: I'm using Caret package to do KNN Imputation on 1.8 Million Rows, the way I'm currently doing it will take over a month to run, how do I write this in such a way that I could do it in a much faster amount of time(if possible)?
Thank you for any help provided. And the answer might very well be "that's how long it takes don't bother" I just want to rule out any possible mistakes.
You can speed this up via the imputation package and use of canopies which can be installed from Github:
Sys.setenv("PKG_CXXFLAGS"="-std=c++0x")
devtools::install_github("alexwhitworth/imputation")
Canopies use a cheap distance metric--in this case distance from the data mean vector--to get approximate neighbors. In general, we wish to keep the canopies each sized < 100k so for 1.8M rows, we'll use 20 canopies:
library("imputation")
to_impute <- mainData[trainingRows, imputeColumns] ## OP undefined
imputed <- kNN_impute(to_impute, k= 10, q= 2, verbose= TRUE,
parallel= TRUE, n_canopies= 20)
NOTE:
The imputation package requires numeric data inputs. You have several factor variables in your str output. They will cause this to fail.
You'll also get some mean vector imputation if you have fulling missing rows.
# note this example data is too small for canopies to be useful
# meant solely to illustrate
set.seed(2143L)
x1 <- matrix(rnorm(1000), 100, 10)
x1[sample(1:1000, size= 50, replace= FALSE)] <- NA
x_imp <- kNN_impute(x1, k=5, q=2, n_canopies= 10)
sum(is.na(x_imp[[1]])) # 0
# with fully missing rows
x2 <- x1; x2[5,] <- NA
x_imp <- kNN_impute(x2, k=5, q=2, n_canopies= 10)
[1] "Computing canopies kNN solution provided within canopies"
[1] "Canopies complete... calculating kNN."
row(s) 1 are entirely missing.
These row(s)' values will be imputed to column means.
Warning message:
In FUN(X[[i]], ...) :
Rows with entirely missing values imputed to column means.
I need generate a matrix and fill with numbers and inactive cells, but that the sum of each columns or rows are equal. I know the magic box and sudoku, but is different. Can you help me please? What kind algorithm I need use for generate this matrix?
E.g
X = 0 = block inactive
Matrix ( 4x4 )
0 8 4 X | 12
2 0 8 2 | 12
10 1 X 1 | 12
0 3 X 9 | 12
____________|
12 12 12 12
Other example:
Matrix ( 5x5 )
0 2 2 3 5 | 12
2 4 0 5 1 | 12
8 2 0 2 0 | 12
0 4 2 0 6 | 12
2 0 8 2 0 | 12
______________|
12 12 12 12 12
The result can be any other number, it is not always 12. Just as in Example I was easier to do for me. It's not be symmetrical.
Note: This is not magic box, also is not sudoku.
Conclusion:
1) I need build this box and fill with number and block inactive.
2) Always matrix is square(3x3, 4x4, 5x5, NxN, ...)
3) When I fill of space is not block, I can use number one, two or three digits.
4) The sum of all sides must be equal.
5) In the above example, X is block. Block mean not use for player.
6) you can inactive block can be 0, however does not affect the sum.
7) There is also no restriction on how many blocks or inactive will have no
8) To fill cells with numbers, this can be repeated if you want. There is not restriction.
9) The matrix is always a square and may be of different dimensions. (2)
Thanks guys for your help. And sorry that the problem is incomplete and for my english is too bad, but that's all.
In terms of agorithms, I would approach it as a system of linear equations. You can put the box as a matrix of variables:
x11 x12 x13 x14
x21 x22 x23 x24
x31 x32 x33 x34
x41 x42 x43 x44
Then you would make the equations as:
row1 = row2 (x11 + x12 + x13 + x14 = x21 + x22 + x23 + x24)
row1 = row3 (...)
row1 = row4
row1 = col1
row1 = col2
row1 = col3
row1 = col4
For N = 4, you would have 16 variables and 7 equations, so you would have a solution with a number of degrees of freedom (at least 9, as pointed out by #JamesMcLeod, and exactly 9, as stated by #Chris), so you could generate every possible matrix satisfying the restrictions just giving values to every free parameter. In the resulting matrix, you could mark every cell with 0 as an inactive cell.
To do this however you would need a library or software package with the ability to solve systems of linear equations with degrees of freedom (several math software packages can do this, but right now only Maple comes to my mind).
PD: I've just read that numbers must have one, two or three digits (and be positive, too?). To address this, you could just "take care" when choosing the values for the free parameters once the system of equations is solved, or you could add inequalities to the problem like:
x11 < 1000
x11 >= 0 (if values must be positive)
x12 < 1000
(...)
But then it would be a linear programming problem. You may approach it like this too.
PD2: You can also make simple cases with diagonal matrices:
7 X X X
X 7 X X
X X 7 X
X X X 7
But I guess you already knew that...
Edit: Thanks James McLeod and Chris for your corrections.
do you fill the matrix with random numbers? You need a function that has an argument as 1 dimension vector which will verify if the sum of the row's elements is 12, then you can still use this function for columns(with a loop) into your main.
Based on this code and this one I'm familiar with implementing imfilter function.But as you know these kind of codes( using sequential for loops) would be very slow in matlab specially for high resolution images and they are more efficient in other programming languages. In matlab it's better to vectorize your code as much as possible.
Can anyone suggest me a way for vectorizing imfilter implementation?
Note: I know that I can use edit('imfilter') to study the code that developers have used for implementing imfilter function but it's pretty hard for me. I don't understand much of the codes. I'm pretty new to matlab.
Note: I know that some part of the codes introduced as an example could be vectorized very easily fore example padding section in this code could be implemented more easily.
But I'm thinking of a away for vectorizing the main part of the code (part of applying the filter). I mean the parts that are shown in the pictures:
I don't know how to vectorize these parts?
Oh, I forgot to tell that I have written the accepted answer for this question. Is there any way if I also don't want to use conv2 function?
There is a function just for you... it's called im2col. See http://www.mathworks.com/help/images/ref/im2col.html for a description. It allows you to turn "blocks" of the image into "columns" - if you are looking for 3x3 blocks to filter, each column will be 9 elements long. After that, the filter operation can be very simple. Here is an example:
n = 20; m = 30
myImg = rand(n, m)*255;
myImCol = im2col(myImg, [3 3], 'sliding');
myFilter = [1 2 1 2 4 2 1 2 1]';
myFilter = myFilter / sum(myFilter(:)); % to normalize
filteredImage = reshape( myImCol' * myFilter, n-2, m-2);
Didn't use conv2, and didn't use any explicit loops. This does, however, create an intermediate matrix which is a good deal bigger than the image (in this case, almost 9x). That could be a problem in its own right.
Disclaimer: I usually test Matlab code before posting, but could not connect to the license server. Let me know if you run into issues!
edit some further clarifications for you
1) Why reshape with n-2 and m-2? Well - the im2col function only returns "complete" columns for the blocks that it can create. When I create 3x3 blocks, the first one I can make is centered on (2,2), and the last one on (end-1, end-1). Thus the result is a bit smaller than the original image- it's "like padding". This is in fact the exact opposite of what happens when you use conv2 - in that case things get expanded. If you want to avoid that you could first expand your image with
paddedIm = zeros(n+2, m+2);
paddedIm(2:end-1, 2:end-1) = myImg;
and run the filter on the padded image.
2) The difference between 'sliding' and 'distinct' is best explained with an example:
>> M = magic(4)
M =
16 2 3 13
5 11 10 8
9 7 6 12
4 14 15 1
>> im2col(M,[2 2], 'distinct')
ans =
16 9 3 6
5 4 10 15
2 7 13 12
11 14 8 1
xx-- --xx ---- ----
xx-- --xx ---- ----
---- ---- xx-- --xx
---- ---- xx-- --xx
>> im2col(M,[2 2], 'sliding')
ans =
16 5 9 2 11 7 3 10 6
5 9 4 11 7 14 10 6 15
2 11 7 3 10 6 13 8 12
11 7 14 10 6 15 8 12 1
xx-- ---- ---- -xx-
xx-- xx-- ---- -xx- ... etc ...
---- xx-- xx-- ----
---- ---- xx-- ----
As you can see, the 'distinct' option returns non-overlapping blocks: the 'sliding' option returns "all blocks that fit" even though some will overlap.
3) The implementation of conv2 is likely some lower level code for speed - you may know about .mex files which allow you to write your own C code that can be linked with Matlab and gives you a big speed advantage? This is likely to be something like that. They do claim on their website that they use a "straightforward implementation" - so the speed is most likely just a matter of implementing in a fast manner (not "efficient Matlab").
The two inner loops can be vectorized by-
orignalFlip=flipud(fliplr(orignal(i-1:i+1,j-1:j+1)));
temp=orignalFlip.*filter;
but what the problem with 'conv2' ? seems that exactly what you need...
However, you should not doing 4 nested-loops in matlab.
There are a limited number of players and a limited number of tennis courts. At each round, there can be at most as many matches as there are courts.
Nobody plays 2 rounds without a break. Everyone plays a match against everyone else.
Produce the schedule that takes as few rounds as possible. (Because of the rule that there must a break between rounds for everyone, there can be a round without matches.)
The output for 5 players and 2 courts could be:
| 1 2 3 4 5
-|-------------------
2| 1 -
3| 5 3 -
4| 7 9 1 -
5| 3 7 9 5 -
In this output the columns and rows are the player-numbers, and the numbers inside the matrix are the round numbers these two players compete.
The problem is to find an algorithm which can do this for larger instances in a feasible time. We were asked to do this in Prolog, but (pseudo-) code in any language would be useful.
My first try was a greedy algorithm, but that gives results with too many rounds.
Then I suggested an iterative deepening depth-first search, which a friend of mine implemented, but that still took too much time on instances as small as 7 players.
(This is from an old exam question. No one I spoke to had any solution.)
Preface
In Prolog, CLP(FD) constraints are the right choice for solving such scheduling tasks.
See clpfd for more information.
In this case, I suggest using the powerful global_cardinality/2 constraint to restrict the number of occurrences of each round, depending on the number of available courts. We can use iterative deepening to find the minimal number of admissible rounds.
Freely available Prolog systems suffice to solve the task satisfactorily. Commercial-grade systems will run dozens of times faster.
Variant 1: Solution with SWI-Prolog
:- use_module(library(clpfd)).
tennis(N, Courts, Rows) :-
length(Rows, N),
maplist(same_length(Rows), Rows),
transpose(Rows, Rows),
Rows = [[_|First]|_],
chain(First, #<),
length(_, MaxRounds),
numlist(1, MaxRounds, Rounds),
pairs_keys_values(Pairs, Rounds, Counts),
Counts ins 0..Courts,
foldl(triangle, Rows, Vss, Dss, 0, _),
append(Vss, Vs),
global_cardinality(Vs, Pairs),
maplist(breaks, Dss),
labeling([ff], Vs).
triangle(Row, Vs, Ds, N0, N) :-
length(Prefix, N0),
append(Prefix, [-|Vs], Row),
append(Prefix, Vs, Ds),
N #= N0 + 1.
breaks([]).
breaks([P|Ps]) :- maplist(breaks_(P), Ps), breaks(Ps).
breaks_(P0, P) :- abs(P0-P) #> 1.
Sample query: 5 players on 2 courts:
?- time(tennis(5, 2, Rows)), maplist(writeln, Rows).
% 827,838 inferences, 0.257 CPU in 0.270 seconds (95% CPU, 3223518 Lips)
[-,1,3,5,7]
[1,-,5,7,9]
[3,5,-,9,1]
[5,7,9,-,3]
[7,9,1,3,-]
The specified task, 6 players on 2 courts, solved well within the time limit of 1 minute:
?- time(tennis(6, 2, Rows)),
maplist(format("~t~w~3+~t~w~3+~t~w~3+~t~w~3+~t~w~3+~t~w~3+\n"), Rows).
% 6,675,665 inferences, 0.970 CPU in 0.977 seconds (99% CPU, 6884940 Lips)
- 1 3 5 7 10
1 - 6 9 11 3
3 6 - 11 9 1
5 9 11 - 2 7
7 11 9 2 - 5
10 3 1 7 5 -
Further example: 7 players on 5 courts:
?- time(tennis(7, 5, Rows)),
maplist(format("~t~w~3+~t~w~3+~t~w~3+~t~w~3+~t~w~3+~t~w~3+~t~w~3+\n"), Rows).
% 125,581,090 inferences, 17.476 CPU in 18.208 seconds (96% CPU, 7185927 Lips)
- 1 3 5 7 9 11
1 - 5 3 11 13 9
3 5 - 9 1 7 13
5 3 9 - 13 11 7
7 11 1 13 - 5 3
9 13 7 11 5 - 1
11 9 13 7 3 1 -
Variant 2: Solution with SICStus Prolog
With the following additional definitions for compatibility, the same program also runs in SICStus Prolog:
:- use_module(library(lists)).
:- use_module(library(between)).
:- op(700, xfx, ins).
Vs ins D :- maplist(in_(D), Vs).
in_(D, V) :- V in D.
chain([], _).
chain([L|Ls], Pred) :-
chain_(Ls, L, Pred).
chain_([], _, _).
chain_([L|Ls], Prev, Pred) :-
call(Pred, Prev, L),
chain_(Ls, L, Pred).
pairs_keys_values(Ps, Ks, Vs) :- keys_and_values(Ps, Ks, Vs).
foldl(Pred, Ls1, Ls2, Ls3, S0, S) :-
foldl_(Ls1, Ls2, Ls3, Pred, S0, S).
foldl_([], [], [], _, S, S).
foldl_([L1|Ls1], [L2|Ls2], [L3|Ls3], Pred, S0, S) :-
call(Pred, L1, L2, L3, S0, S1),
foldl_(Ls1, Ls2, Ls3, Pred, S1, S).
time(Goal) :-
statistics(runtime, [T0|_]),
call(Goal),
statistics(runtime, [T1|_]),
T #= T1 - T0,
format("% Runtime: ~Dms\n", [T]).
Major difference: SICStus, being a commercial-grade Prolog that ships with a serious CLP(FD) system, is much faster than SWI-Prolog in this use case and others like it.
The specified task, 6 players on 2 courts:
?- time(tennis(6, 2, Rows)),
maplist(format("~t~w~3+~t~w~3+~t~w~3+~t~w~3+~t~w~3+~t~w~3+\n"), Rows).
% Runtime: 34ms (!)
- 1 3 5 7 10
1 - 6 11 9 3
3 6 - 9 11 1
5 11 9 - 2 7
7 9 11 2 - 5
10 3 1 7 5 -
The larger example:
| ?- time(tennis(7, 5, Rows)),
maplist(format("~t~w~3+~t~w~3+~t~w~3+~t~w~3+~t~w~3+~t~w~3+~t~w~3+\n"), Rows).
% Runtime: 884ms
- 1 3 5 7 9 11
1 - 5 3 9 7 13
3 5 - 1 11 13 7
5 3 1 - 13 11 9
7 9 11 13 - 3 1
9 7 13 11 3 - 5
11 13 7 9 1 5 -
Closing remarks
In both systems, global_cardinality/3 allows you to specify options that alter the propagation strength of the global cardinality constraint, enabling weaker and potentially more efficient filtering. Choosing the right options for a specific example may have an even larger impact than the choice of Prolog system.
This is very similar to the Traveling Tournament Problem, which is about scheduling football teams. In TTP, they can find the optimal solution up to only 8 teams. Anyone who breaks an ongoing record of 10 or more teams, has it a lot easier to get published in a research journal.
It is NP hard and the trick is to use meta-heuristics, such as tabu search, simulated annealing, ... instead of brute force or branch and bound.
Take a look my implementation with Drools Planner (open source, java).
Here are the constraints, it should be straightforward to replace that with constraints such as Nobody plays 2 rounds without a break.
Each player must play at least n - 1 matches where n is the number of players. So the minimum of rounds is 2(n - 1) - 1, since every player needs to rest a match. The minimum is also bound by (n(n-1))/2 total matches divided by number of courts. Using the smallest of these two gives you the length of the optimal solution. Then it's a matter of coming up with a good lower estimating formula ((number of matches+rests remaining)/courts) and run A* search.
As Geoffrey said, I believe the problem is NP Hard, but meta-heuristics such as A* is very applicable.
Python Solution:
import itertools
def subsets(items, count = None):
if count is None:
count = len(items)
for idx in range(count + 1):
for group in itertools.combinations(items, idx):
yield frozenset(group)
def to_players(games):
return [game[0] for game in games] + [game[1] for game in games]
def rounds(games, court_count):
for round in subsets(games, court_count):
players = to_players(round)
if len(set(players)) == len(players):
yield round
def is_canonical(player_count, games_played):
played = [0] * player_count
for players in games_played:
for player in players:
played[player] += 1
return sorted(played) == played
def solve(court_count, player_count):
courts = range(court_count)
players = range(player_count)
games = list( itertools.combinations(players, 2) )
possible_rounds = list( rounds(games, court_count) )
rounds_last = {}
rounds_all = {}
choices_last = {}
choices_all = {}
def update(target, choices, name, value, choice):
try:
current = target[name]
except KeyError:
target[name] = value
choices[name] = choice
else:
if current > value:
target[name] = value
choices[name] = choice
def solution(games_played, players, score, choice, last_players):
games_played = frozenset(games_played)
players = frozenset(players)
choice = (choice, last_players)
update(rounds_last.setdefault(games_played, {}),
choices_last.setdefault(games_played, {}),
players, score, choice)
update(rounds_all, choices_all, games_played, score, choice)
solution( [], [], 0, None, None)
for games_played in subsets(games):
if is_canonical(player_count, games_played):
try:
best = rounds_all[games_played]
except KeyError:
pass
else:
for next_round in possible_rounds:
next_games_played = games_played.union(next_round)
solution(
next_games_played, to_players(next_round), best + 2,
next_round, [])
for last_players, score in rounds_last[games_played].items():
for next_round in possible_rounds:
if not last_players.intersection( to_players(next_round) ):
next_games_played = games_played.union(next_round)
solution( next_games_played, to_players(next_round), score + 1,
next_round, last_players)
all_games = frozenset(games)
print rounds_all[ all_games ]
round, prev = choices_all[ frozenset(games) ]
while all_games:
print "X ", list(round)
all_games = all_games - round
if not all_games:
break
round, prev = choices_last[all_games][ frozenset(prev) ]
solve(2, 6)
Output:
11
X [(1, 2), (0, 3)]
X [(4, 5)]
X [(1, 3), (0, 2)]
X []
X [(0, 5), (1, 4)]
X [(2, 3)]
X [(1, 5), (0, 4)]
X []
X [(2, 5), (3, 4)]
X [(0, 1)]
X [(2, 4), (3, 5)]
This means it will take 11 rounds. The list shows the games to be played in the rounds in reverse order. (Although I think the same schedule works forwards and backwords.)
I'll come back and explain why I have the chance.
Gets incorrect answers for one court, five players.
Some thoughts, perhaps a solution...
Expanding the problem to X players and Y courts, I think we can safely say that when given the choice, we must select the players with the fewest completed matches, otherwise we run the risk of ending up with one player left who can only play every other week and we end up with many empty weeks in between. Picture the case with 20 players and 3 courts. We can see that during round 1 players 1-6 meet, then in round 2 players 7-12 meet, and in round 3 we could re-use players 1-6 leaving players 13-20 until later. Therefor, I think our solution cannot be greedy and must balance the players.
With that assumption, here is a first attempt at a solution:
1. Create master-list of all matches ([12][13][14][15][16][23][24]...[45][56].)
2. While (master-list > 0) {
3. Create sub-list containing only eligible players (eliminate all players who played the previous round.)
4. While (available-courts > 0) {
5. Select match from sub-list where player1.games_remaining plus player2.games_remaining is maximized.
6. Place selected match in next available court, and
7. decrement available-courts.
8. Remove selected match from master-list.
9. Remove all matches that contain either player1 or player2 from sub-list.
10. } Next available-court
11. Print schedule for ++Round.
12. } Next master-list
I can't prove that this will produce a schedule with the fewest rounds, but it should be close. The step that may cause problems is #5 (select match that maximizes player's games remaining.) I can imagine that there might be a case where it's better to pick a match that almost maximizes 'games_remaining' in order to leave more options in the following round.
The output from this algorithm would look something like:
Round Court1 Court2
1 [12] [34]
2 [56] --
3 [13] [24]
4 -- --
5 [15] [26]
6 -- --
7 [35] [46]
. . .
Close inspection will show that in round 5, if the match on Court2 had been [23] then match [46] could have been played during round 6. That, however, doesn't guarantee that there won't be a similar issue in a later round.
I'm working on another solution, but that will have to wait for later.
I don't know if this matters, the "5 Players and 2 Courts" example data is missing three other matches: [1,3], [2,4] and [3,5]. Based on the instruction: "Everyone plays a match against everyone else."