Related
I have a file which, when I import into Mathematica, looks like this:
{{1,1,n1},{1,2,n2},{1,3,n3},{2,1,n4},{2,2,n5},{2,3,n6}} where n1...n6 are some numbers that I want to import as a matrix that looks like :
The first number in each block specifies the row and the second the column, but they are not a part of the matrix. Only the third number in each block is part of the matrix. How can I do that?
If
data = {{1, 1, n1}, {1, 2, n2}, {1, 3, n3}, {2, 1, n4}, {2, 2, n5}, {2, 3, n6}};
you can simply do
mat = Partition[data[[All, 3]], 3, 3]
There are a couple of interpretations of this question that I can think of.
If your data is in a regular format and you wish to read it in a memory efficient manner I recommend looking closely at ReadList and related functionality as I already directed you toward and the Partition function that the other answer illustrates.
I shall instead focus on the idea that the data is not in an entirely regular form in that the given row and column indexes are necessary to describe the positions of the data in the array. For that the most natural method is to use SparseArray at it accepts data in the form of position and value Rule pairs:
data = {{1, 1, n1}, {1, 2, n2}, {1, 3, n3}, {2, 1, n4}, {2, 2, n5}, {2, 3, n6}};
array = SparseArray[{#, #2} -> #3 & ### data];
array // MatrixForm
The function Normal can be used to convert the SparseArray into a regular list-of-lists array as needed:
Normal # array
{{n1, n2, n3}, {n4, n5, n6}}
Also there is a StackExchange site dedicated to Mathematica that I encourage you to explore.
I have a Mathematica file called myUsefulFunctions.m containing, for example, a function called mySuperUsefulFunction. Suppose I call mySuperUsefulFunction in a notebook and get the following error:
Part::pspec: Part specification #1 is neither an integer nor a list of integers. >>
Is there a way to find the line in myUsefulFunctions.m where this error occurred?
A light-weight debug function
In addition to other suggestions, here is a function which helped me a few times:
ClearAll[debug];
SetAttributes[debug, HoldAll];
debug[code_] :=
Internal`InheritedBlock[{Message},
Module[{inMessage},
Unprotect[Message];
Message[args___] /; ! MatchQ[First[Hold[args]], _$Off] :=
Block[{inMessage = True},
Print[{
Shallow /# Replace[#, HoldForm[f_[___]] :> HoldForm[f], 1],
Style[Map[Short, Last[#], {2}], Red]
} &#Drop[Drop[Stack[_], -7], 4]
];
Message[args];
Throw[$Failed, Message];
] /; ! TrueQ[inMessage];
Protect[Message];
];
Catch[StackComplete[code], Message]]
This basically redefines Message temporarily to walk up the execution stack and print the names of called functions in an easy to understand form, plus the final call which resulted in an error message, and an error message itself. After that, we exit the execution via exception, to not generated confusing chains of error messages.
Examples of use
Here is how this works on an example from #Mr.Wizard's answer:
In[211]:= debug[myFunc2[Range#10,#1]]
During evaluation of In[211]:=
{{myFunc2,Pick,myFunc1,Part},{1,2,3,4,5,6,7,8,9,10}[[#1]]}
During evaluation of In[211]:= Part::pspec: Part specification #1 is neither
an integer nor a list of integers. >>
Out[211]= $Failed
(in the notebook the problematic function call is painted red). This allows one to quickly see the chain of function calls which lead to the problem.
Here is another example: we construct a custom gatherBy function which gathers elements in a list according to another list of "marks", which is supposed to be the same length as the original:
listSplit[x_, lengths_] :=
MapThread[Take[x, {##}] &, {Most[#], Rest[#] - 1}] &#
Accumulate[Prepend[lengths, 1]];
gatherBy[lst_, flst_] :=
listSplit[lst[[Ordering[flst]]], (Sort#Tally[flst])[[All, 2]]];
For example:
In[212]:= gatherBy[Range[10],{1,1,2,3,2,4,5,5,4,1}]
Out[212]= {{1,2,10},{3,5},{4},{6,9},{7,8}}
Because I intentionally left all type-checking out, calls with arguments of wrong types will result in chains of nasty error messages:
In[213]:= gatherBy[Range[10],Range[15]]//Short
During evaluation of In[206]:= Part::partw: Part 11 of {1,2,3,4,5,6,7,8,9,10} does not exist. >>
(* 4 more messages here *)
Out[213]//Short= {{1,2,3,4,5,6,7,8,9,10},<<14>>}
Using debug, we can see what's wrong pretty quickly:
In[214]:= debug[gatherBy[Range[10],Range[15]]]
During evaluation of In[214]:=
{{gatherBy,listSplit,Part},
{1,2,3,4,5,6,7,8,9,10}[[Ordering[{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}]]]}
During evaluation of In[214]:= Part::partw: Part 11 of {1,2,3,4,5,6,7,8,9,10} does not exist. >>
Out[214]= $Failed
Calling gatherBy[Range[10], a] with some symbolic a is another example where wrapping debug around helps.
Applicability
The other suggested methods are more systematic and probably more generally recommended, but this one is easy to apply and leads to results which are often easier to understand (e.g. compared to the output of Trace, which is not always easy to read). I did not use it as often as to guarantee that it always works, however.
Apart from the debugger in the Workbench there's also a debugger built-in in Mathematica. You can find it in the Evaluation menu. It is not well documented and rather difficult/unconventional to get it to work. Here is a step-by-step instruction how to use it:
Assuming you have the debugger switched on in the Evaluation menu your window bar will indicate it's a debug session and you will have a few debugger palettes.
Now select a number of lines you want to act as breakpoints and click on the "break at selection" text. Breakpoints will be marked by a red outline.
and run the code by pressing Shift-return and be prepared for a slight disappointment: it doesn't work. It appears you cannot define breakpoints on the line level. It must be at function level. Also, MMA is rather picky about the functions you can use. The Print function apparently doesn't work neither do assignments. However, the Integrate in this example does, but you have to select its head and both brackets and make that a breakpoint. If you have done that and you then execute the block of code you get this:
The breakpoint is highlighted green, some additional options in the control palette have come available to control further program flow, and there are expressions in the stack window. The rest is more or less similar to a standard debugger. Note that you can nest breakpoints like the Cos in the Integrate. For a language that can have deeply nested structures this is essential.
Another option would be David Bailey's debugger. He offers the free debugger DebugTrace on his website. I haven't tried it myself but I know David as a very capable Mathematica expert so I trust it must be good.
I don't know of a way to find the line in the file, which I assume was read without error.
You can however use Trace and related functions to see where in the evaluation chain the error occurs.
Example:
myFunc1[x_, y_] := x[[y]]
myFunc2[a_List, n_] := Pick[a, a, myFunc1[a, n]]
myFunc2[Range#10, #1]
During evaluation of In[4]:= Part::pspec: Part specification #1 is neither an integer nor a list of integers. >>
With Trace:
myFunc2[Range#10, #1] // Trace // Column
{Range[10], {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}}
myFunc2[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, #1]
Pick[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, myFunc1[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, #1]]
{myFunc1[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, #1], {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}[[#1]], {Message[Part::pspec, #1], {MakeBoxes[Part::pspec: Part specification #1 is neither an integer nor a list of integers. >>, StandardForm], RowBox[{RowBox[{Part, ::, "pspec"}], : , "\!\(\*StyleBox[\"\\\"Part specification \\\"\", \"MT\"]\)\!\(\*StyleBox[\!\(#1\), \"MT\"]\)\!\(\*StyleBox[\"\\\" is neither an integer nor a list of integers.\\\"\", \"MT\"]\) \!\(\*ButtonBox[\">>\", ButtonStyle->\"Link\", ButtonFrame->None, ButtonData:>\"paclet:ref/message/General/pspec\", ButtonNote -> \"Part::pspec\"]\)"}]}, Null}, {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}[[#1]]}
You can see that just before Message[Part::pspec, #1] is called, which results in a long mess of formatting, we had:
{myFunc1[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, #1], {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}[[#1]]
This shows that myFunc1[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, #1] is called, and this causes evaluation of {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}[[#1]] which is clearly in error.
Please see this question and its answers for a more convenient use of Trace:
https://stackoverflow.com/q/5459735/618728
You can use the WolframWorkbench and the debugger there:
http://www.wolfram.com/broadcast/screencasts/workbench/debugging/
you can then set a break point and step through the code.
Example:
list:={ Plus[1,1], Times[2,3] }
When looking at list, I get
{2,6}
I want to keep them unevaluated (as above) so that list returns
{ Plus[1,1], Times[2,3] }
Later I want to evaluate the functions in list sequence to get
{2,6}
The number of unevaluated functions in list is not known beforehand. Besides Plus, user defined functions like f[x_] may be stored in list
I hope the example is clear.
What is the best way to do this?
The best way is to store them in Hold, not List, like so:
In[255]:= f[x_] := x^2;
lh = Hold[Plus[1, 1], Times[2, 3], f[2]]
Out[256]= Hold[1 + 1, 2 3, f[2]]
In this way, you have full control over them. At some point, you may call ReleaseHold to evaluate them:
In[258]:= ReleaseHold#lh
Out[258]= Sequence[2, 6, 4]
If you want the results in a list rather than Sequence, you may use just List##lh instead. If you need to evaluate a specific one, simply use Part to extract it:
In[261]:= lh[[2]]
Out[261]= 6
If you insist on your construction, here is a way:
In[263]:= l:={Plus[1,1],Times[2,3],f[2]};
Hold[l]/.OwnValues[l]
Out[264]= Hold[{1+1,2 3,f[2]}]
EDIT
In case you have some functions/symbols with UpValues which can evaluate even inside Hold, you may want to use HoldComplete in place of Hold.
EDIT2
As pointed by #Mr.Wizard in another answer, sometimes you may find it more convenient to have Hold wrapped around individual items in your sequence. My comment here is that the usefulness of both forms is amplified once we realize that it is very easy to transform one into another and back. The following function will split the sequence inside Hold into a list of held items:
splitHeldSequence[Hold[seq___], f_: Hold] := List ## Map[f, Hold[seq]]
for example,
In[274]:= splitHeldSequence[Hold[1 + 1, 2 + 2]]
Out[274]= {Hold[1 + 1], Hold[2 + 2]}
grouping them back into a single Hold is even easier - just Apply Join:
In[275]:= Join ## {Hold[1 + 1], Hold[2 + 2]}
Out[275]= Hold[1 + 1, 2 + 2]
The two different forms are useful in diferrent circumstances. You can easily use things such as Union, Select, Cases on a list of held items without thinking much about evaluation. Once finished, you can combine them back into a single Hold, for example, to feed as unevaluated sequence of arguments to some function.
EDIT 3
Per request of #ndroock1, here is a specific example. The setup:
l = {1, 1, 1, 2, 4, 8, 3, 9, 27}
S[n_] := Module[{}, l[[n]] = l[[n]] + 1; l]
Z[n_] := Module[{}, l[[n]] = 0; l]
placing functions in Hold:
In[43]:= held = Hold[Z[1], S[1]]
Out[43]= Hold[Z[1], S[1]]
Here is how the exec function may look:
exec[n_] := MapAt[Evaluate, held, n]
Now,
In[46]:= {exec[1], exec[2]}
Out[46]= {Hold[{0, 1, 1, 2, 4, 8, 3, 9, 27}, S[1]], Hold[Z[1], {1, 1, 1, 2, 4, 8, 3, 9, 27}]}
Note that the original variable held remains unchanged, since we operate on the copy. Note also that the original setup contains mutable state (l), which is not very idiomatic in Mathematica. In particular, the order of evaluations matter:
In[61]:= Reverse[{exec[2], exec[1]}]
Out[61]= {Hold[{0, 1, 1, 2, 4, 8, 3, 9, 27}, S[1]], Hold[Z[1], {2, 1, 1, 2, 4, 8, 3, 9, 27}]}
Whether or not this is desired depends on the specific needs, I just wanted to point this out. Also, while the exec above is implemented according to the requested spec, it implicitly depends on a global variable l, which I consider a bad practice.
An alternative way to store functions suggested by #Mr.Wizard can be achieved e.g. like
In[63]:= listOfHeld = splitHeldSequence[held]
Out[63]= {Hold[Z1], Hold[S1]}
and here
In[64]:= execAlt[n_] := MapAt[ReleaseHold, listOfHeld, n]
In[70]:= l = {1, 1, 1, 2, 4, 8, 3, 9, 27} ;
{execAlt[1], execAlt[2]}
Out[71]= {{{0, 1, 1, 2, 4, 8, 3, 9, 27}, Hold[S[1]]}, {Hold[Z[1]], {1, 1, 1, 2, 4, 8, 3, 9, 27}}}
The same comments about mutability and dependence on a global variable go here as well. This last form is also more suited to query the function type:
getType[n_, lh_] := lh[[n]] /. {Hold[_Z] :> zType, Hold[_S] :> sType, _ :> unknownType}
for example:
In[172]:= getType[#, listOfHeld] & /# {1, 2}
Out[172]= {zType, sType}
The first thing that spings to mind is to not use List but rather use something like this:
SetAttributes[lst, HoldAll];
heldL=lst[Plus[1, 1], Times[2, 3]]
There will surely be lots of more erudite suggestions though!
You can also use Hold on every element that you want held:
a = {Hold[2 + 2], Hold[2*3]}
You can use HoldForm on either the elements or the list, if you want the appearance of the list without Hold visible:
b = {HoldForm[2 + 2], HoldForm[2*3]}
c = HoldForm#{2 + 2, 2*3}
{2 + 2, 2 * 3}
And you can recover the evaluated form with ReleaseHold:
a // ReleaseHold
b // ReleaseHold
c // ReleaseHold
Out[8]= {4, 6}
Out[9]= {4, 6}
Out[10]= {4, 6}
The form Hold[2+2, 2*3] or that of a and b above are good because you can easily add terms with e.g. Append. For b type is it logically:
Append[b, HoldForm[8/4]]
For Hold[2+2, 2*3]:
Hold[2+2, 2*3] ~Join~ Hold[8/4]
Another way:
lh = Function[u, Hold#u, {HoldAll, Listable}];
k = lh#{2 + 2, Sin[Pi]}
(*
->{Hold[2 + 2], Hold[Sin[\[Pi]]]}
*)
ReleaseHold#First#k
(*
-> 4
*)
I am trying to prepare the best tools for efficient Data Analysis in Mathematica.
I have a approximately 300 Columns & 100 000 Rows.
What would be the best tricks to :
"Remove", "Extract" or simply "Consider" parts of the data structure, for plotting for e.g.
One of the trickiest examples I could think of is :
Given a data structure,
Extract Column 1 to 3, 6 to 9 as well as the last One for every lines where the value in Column 2 is equal to x and the value in column 8 is different than y
I also welcome any general advice on data manipulation.
For a generic manipulation of data in a table with named columns, I refer you to this solution of mine, for a similar question. For any particular case, it might be easier to write a function for Select manually. However, for many columns, and many different queries, chances to mess up indexes are high. Here is the modified solution from the mentioned post, which provides a more friendly syntax:
Clear[getIds];
getIds[table : {colNames_List, rows__List}] := {rows}[[All, 1]];
ClearAll[select, where];
SetAttributes[where, HoldAll];
select[cnames_List, from[table : {colNames_List, rows__List}], where[condition_]] :=
With[{colRules = Dispatch[ Thread[colNames -> Thread[Slot[Range[Length[colNames]]]]]],
indexRules = Dispatch[Thread[colNames -> Range[Length[colNames]]]]},
With[{selF = Apply[Function, Hold[condition] /. colRules]},
Select[{rows}, selF ## # &][[All, cnames /. indexRules]]]];
What happens here is that the function used in Select gets generated automatically from your specifications. For example (using #Yoda's example):
rows = Array[#1 #2 &, {5, 15}];
We need to define the column names (must be strings or symbols without values):
In[425]:=
colnames = "c" <> ToString[#] & /# Range[15]
Out[425]= {"c1", "c2", "c3", "c4", "c5", "c6", "c7", "c8", "c9", "c10", "c11", "c12",
"c13", "c14", "c15"}
(in practice, usually names are more descriptive, of course). Here is the table then:
table = Prepend[rows, colnames];
Here is the select statement you need (I picked x = 4 and y=2):
select[{"c1", "c2", "c3", "c6", "c7", "c8", "c9", "c15"}, from[table],
where["c2" == 4 && "c8" != 2]]
{{2, 4, 6, 12, 14, 16, 18, 30}}
Now, for a single query, this may look like a complicated way to do this. But you can do many different queries, such as
In[468]:= select[{"c1", "c2", "c3"}, from[table], where[EvenQ["c2"] && "c10" > 10]]
Out[468]= {{2, 4, 6}, {3, 6, 9}, {4, 8, 12}, {5, 10, 15}}
and similar.
Of course, if there are specific correlations in your data, you might find a particular special-purpose algorithm which will be faster. The function above can be extended in many ways, to simplify common queries (include "all", etc), or to auto-compile the generated pure function (if possible).
EDIT
On a philosophical note, I am sure that many Mathematica users (myself included) found themselves from time to time writing similar code again and again. The fact that Mathematica has a concise syntax makes it often very easy to write for any particular case. However, as long as one works in some specific domain (like, for example, data manipulations in a table), the cost of repeating yourself will be high for many operations. What my example illustrates in a very simple setting is a one possible way out - create a Domain-Specific Language (DSL). For that, one generally needs to define a syntax/grammar for it, and write a compiler from it to Mathematica (to generate Mathematica code automatically). Now, the example above is a very primitive realization of this idea, but my point is that Mathematica is generally very well suited for DSL creation, which I think is a very powerful technique.
data = RandomInteger[{1, 20}, {40, 20}]
x = 5;
y = 8;
Select[data, (#[[2]] == x && #[[8]] != y &)][[All, {1, 2, 3, 6, 7, 8, 9, -1}]]
==> {{5, 5, 1, 4, 18, 6, 3, 5}, {10, 5, 15, 3, 15, 14, 2, 5}, {18, 5, 6, 7, 7, 19, 14, 6}}
Some useful commands to get pieces of matrices and list are Span (;;), Drop, Take, Select, Cases and more. See tutorial/GettingAndSettingPiecesOfMatrices and guide/PartsOfMatrices,
Part ([[...]]) in combination with ;; can be quite powerful. a[[All, 1;;-1;;2]], for instance, means take all rows and all odd columns (-1 having the usual meaning of counting from the end).
Select can be used to pick elements from a list (and remember a matrix is a list of lists), based on a logical function. It's twin brother is Cases which does selection based on a pattern. The function I used here is a 'pure' function, where # refers to the argument on which this function is applied (the elements of the list in this case). Since the elements are lists themselves (the rows of the matrix) I can refer to the columns by using the Part ([[..]]) function.
To pull out columns (or rows) you can do it by part indexing
data = Array[#1 #2 &, {5, 15}];
data[[All, Flatten#{Range#3, Range ## {6, 9}, -1}]]
MatrixForm#%
The last line is just to view it pretty.
As Sjoerd mentioned in his comment (and in the explanation in his answer), indexing a single range can be easily done with the Span (;;) command. If you are joining multiple disjoint ranges, using Flatten to combine the separate ranges created with Range is easier than entering them by hand.
I read:
Extract Column 1 to 3, 6 to 9 as well as the last One for every lines where the value in Column 2 is equal to x and the value in column 8 is different than y
as meaning that we want:
elements 1-3 and 6-9 from each row
AND
the last element from rows wherein [[2]] == x && [[8]] != y.
This is what I hacked together:
a = RandomInteger[5, {20, 10}]; (*define the array*)
x = 4; y = 0; (*define the test values*)
Join ## Range ### {1 ;; 3, 6 ;; 9}; (*define the column ranges*)
#2 == x && #8 != y & ### a; (*test the rows*)
Append[%%, #] & /# % /. {True -> -1, False :> Sequence[]}; (*complete the ranges according to the test*)
MapThread[Part, {a, %}] // TableForm (*extract and display*)
Does mathematica have something like "select any" that gets any element of a list that satisfies a criterion?
If you just want to return after the first matching element, use the optional third argument to Select, which is the maximum number of results to return. So you can just do
Any[list_List, crit_, default_:"no match"] :=
With[{maybeMatch = Select[list, crit, 1]},
If[maybeMatch =!= {},
First[maybeMatch],
default]
Mathematica lacks a great way to signal failure to find an answer, since it lacks multiple return values, or the equivalent of Haskell's Maybe type. My solution is to have a user-specifiable default value, so you can make sure you pass in something that's easily distinguishable from a valid answer.
Well, the downside of Eric's answer is that it does execute OddQ on all elements of the list. My call is relatively costly, and it feels wrong to compute it too often. Also, the element of randomness is clearly unneeded, the first one is fine with me.
So, how about
SelectAny[list_List, criterion_] :=
Catch[Scan[ If[criterion[#], Throw[#, "result"]] &, list];
Throw["No such element"], "result"]
And then
SelectAny[{1, 2, 3, 4, 5}, OddQ]
returns 1.
I still wish something were built into Mathematica. Using home-brew functions kind of enlarges your program without bringing much direct benefit.
The Select function provides this built-in, via its third argument which indicates the maximum number of items to select:
In[1]:= Select[{1, 2, 3, 4, 5}, OddQ, 1]
Out[1]= {1}
When none match:
In[2]:= Select[{2, 4}, OddQ, 1]
Out[2]= {}
Edit: Oops, missed that nes1983 already stated this.
You can do it relatively easily with Scan and Return
fstp[p_, l_List] := Scan[ p## && Return## &, l ]
So
In[2]:= OddQ ~fstp~ Range[1,5]
Out[2]= 1
In[3]:= EvenQ ~fstp~ Range[1,5]
Out[3]= 2
I really wish Mathematica could have some options to make expressions evaluated lazily. In a lazy language such as Haskell, you can just define it as normal
fstp p = head . filter p
There's "Select", that gets all the elements that satisfy a condition. So
In[43]:= Select[ {1, 2, 3, 4, 5}, OddQ ]
Out[43]= {1, 3, 5}
Or do you mean that you want to randomly select a single matching element? I don't know of anything built-in, but you can define it pretty quickly:
Any[lst_, q_] :=
Select[ lst, q] // (Part[#, 1 + Random[Integer, Length[#] - 1]]) &
Which you could use the same way::
In[51]:= Any[ {1, 2, 3, 4, 5}, OddQ ]
Out[51]= 3