I frequently import spreadsheets into Mathematica for analysis and am having trouble coding a simple way to select non-sequential elements for the analysis. For example, If I import a spreadsheet with 20 columns and 100 rows, I commonly will need to drop selected rows/columns.
In this example I need all rows and columns 2,4,7-17. It seems logical the following code should work but results in error:
v[[ All, {2,4,7;;17} ]]
Instead I have been using:
v[[ All, {2,4,7,8,9,10,11,12,13,14,15,16,17} ]]
Is it possible to use Span (;;) to select a block of columns (7-17) while also selecting rows 2 and 4 in my example?
the x ;; y syntax is a special argument to Part, not a general syntax that can be used to build lists. So you could say v[[ All, 7;;17 ]], but not v[[All, {7;;17}]] -- the latter is neither a list of integers nor a special syntax recognizable by Part.
But it is pretty easy to solve your problem. You can try:
v[[All, {2,4}~Join~Table[x,{x,7,17}] ]]
for example, or
Join[v[[All, {2, 4}]], v[[All, 7 ;; 17]], 2]
Good luck!
This is a known limitation of Part and Span. See my own closely related question:
Part and Span: is there a reason this *should* not work?
Your solution is the most common work-around. If you find it too inconvenient to build lists of indices you can try to make it easier with a custom Part function. For example:
SetAttributes[part, HoldFirst]
part[x_, parts__] := x[[##]] & ## Flatten /# ({parts} /. Span -> Range)
Use:
a = Range#24 ~Partition~ 4;
part[a, {1 ;; 3, 6}, {1, 3 ;; 4}]
{{1, 3, 4}, {5, 7, 8}, {9, 11, 12}, {21, 23, 24}}
This makes no attempt to handle negative index Spans which would be considerably more complicated, but perhaps it is useful at least to give you some ideas.
another approach..
pys[{all___}] :=
Flatten[(Switch[Head[#], Span,
Range ## (# /. Span -> List), __, #]) & /# {all}]
list = Range[100];
list[[pys[{1, 3 ;; 12 ;; 2, 19, -3 ;; -1}] ]]
{1, 3, 5, 7, 9, 11, 19, 98, 99, 100}
This notably does not handle open ends {1,3;;} or mixed +/- spans { 5;;-5 }
Related
In general, I was trying to compute the norm of the difference between every set two elements in a list which looks something like
X = {{1,2,3,4,5},{6,7,8,9,10},{11,12,13,14,15}}
therefore evaluating
Norm[X[[1]]-X[[2]]]
Norm[X[[1]]-X[[3]]]
Norm[X[[2]]-X[[3]]]
Now, applying Outer[] is one possible way how to do this
Outer[Norm[X[[#1]] - X[[#2]]] &, {1,2,3}, {1,2,3}]
but unfortunately it results in a quite slow code if I increase the number of elements in X and the length of each element.
Is there any possible way to construct a Map[] operation? Something like
MapThread[Norm[X[[#1]] - X[[#2]]] &,{{1,2,3},{1,2,3}}]
does not work give the desired "currying" which I was looking for.
I'm using Mathematica Version 11.2.0.0, so I don't have access to Curry[].
Would be happy about any advice!
mX = {{1, 2, 3, 4, 5}, {6, 7, 8, 9, 10}, {11, 12, 13, 14, 15}}
Apply[Norm#*Subtract, #] & /# Subsets[mX, {2}]
Two equivalent approaches:
(Norm#*Subtract) ### Subsets[mX, {2}]
Apply[Norm#*Subtract, Subsets[mX, {2}], {1}]
Is there any way to find the lowest modulus of a list of integers? I'm not sure how to say it correctly, so I'm going to clarify with an example.
I'd like to input a list (mod x) and output the "same" list, modulus y (< x). For example, the list {0, 4, 6, 10, 12, 16, 18, 22} (mod 24) is essentially the same as {0, 4} (mod 6).
Thank you for all your help.
You are looking for a set of arithmetic sequences. We'll consider your example
ee = {0, 4, 6, 10, 12, 16, 18, 22};
which has two such sequences, and an example with four of them.
ff = {0, 3, 7, 11, 17, 20, 24, 28, 34, 37, 41, 45};
In this second one we start with {0,3,7,11} and then increase by 17. So what is the general way to get from the nth term to the n+1th? If the set has k sequences (k=2 for ee and 4 for ff) then add the modulus to the n-k+1th term. What is the modulus? It is the difference between the nth and n-kth terms.
Putting this together and assuming we know k (we don't in general, but we'll get to that) we have a recurrence of the form f(n+1)=f(n-k+1) + (f(n)-f(n-k)). So we need to find a recurrence (if one exists), check that it is of the correct form, and post-process if so.
Here is code to do all this. Note that it in effect solves for k.
findArithmeticSequences[ll : {_Integer ..}] := With[
{rec = FindLinearRecurrence[ll]},
{Take[ll, Length[rec] - 1], ll[[Length[rec]]]} /;
ListQ[rec] &&
(rec === {1, 1, -1} || MatchQ[rec, {1, 0 .., 1, -1}])
]
(Afficionados of pure functions might prefer the variant below. Failure cases are handled a bit differently, for no compelling reason.)
findArithmeticSequences2[ll : {_Integer ..}] :=
If[ListQ[#] &&
(# === {1, 1, -1} || MatchQ[#, {1, 0 .., 1, -1}]), {Take[ll,
Length[#] - 1], ll[[Length[#]]]}, $Failed] &[
FindLinearRecurrence[ll]]
Tests:
In[115]:= findArithmeticSequences[ee]
Out[115]= {{0, 4}, 6}
In[116]:= findArithmeticSequences[ff]
Out[116]= {{0, 3, 7, 11}, 17}
Note that one can "almost" do such problems by polynomial factorization (if the input has no partial sequences at the end). For example, the polynomial
In[117]:= poly = Plus ## (x^ee)
Out[117]= 1 + x^4 + x^6 + x^10 + x^12 + x^16 + x^18 + x^22
factors into
(1+x^4)*(1+x^6+x^12+x^18)
which contains the needed information in a way that is easy to see. Unfortunately for this particular purpose, Factor will factor beyond this point, and obscure the information in so doing.
I keep wondering if there might be a signal processing way to go about this sort of thing, e.g. via DFTs. But I've not come up with anything.
Daniel Lichtblau
Wow, thank you Daniel for this! It works nearly the way I want it to. Your method is just a bit "too restrictive". It doesn't return anything useful if 'FindLinearRecurrence' doesn't find any recurrence. I've modified your method a bit, so it suits my needs better. I hope you don't mind. Here's my code.
findArithmeticSequences[ll_List] := Module[{rec = FindLinearRecurrence[ll]},
If[! MatchQ[rec, {1, 0 ..., 1, -1}], Return[ll],
Return[{ll[[Length[rec]]], Take[ll, Length[rec] - 1]}];
];
];
I had a feeling it'd have to involve recurrence, I just don't have enough experience with Mathematica to implement it. Thank you again for your time!
Mod is listable, and you can remove duplicate elements by DeleteDuplicates. So
DeleteDuplicates[Mod[{0, 4, 6, 10, 12, 16, 18, 22}, 6]]
(*
-> {0,4}
*)
I am trying to prepare the best tools for efficient Data Analysis in Mathematica.
I have a approximately 300 Columns & 100 000 Rows.
What would be the best tricks to :
"Remove", "Extract" or simply "Consider" parts of the data structure, for plotting for e.g.
One of the trickiest examples I could think of is :
Given a data structure,
Extract Column 1 to 3, 6 to 9 as well as the last One for every lines where the value in Column 2 is equal to x and the value in column 8 is different than y
I also welcome any general advice on data manipulation.
For a generic manipulation of data in a table with named columns, I refer you to this solution of mine, for a similar question. For any particular case, it might be easier to write a function for Select manually. However, for many columns, and many different queries, chances to mess up indexes are high. Here is the modified solution from the mentioned post, which provides a more friendly syntax:
Clear[getIds];
getIds[table : {colNames_List, rows__List}] := {rows}[[All, 1]];
ClearAll[select, where];
SetAttributes[where, HoldAll];
select[cnames_List, from[table : {colNames_List, rows__List}], where[condition_]] :=
With[{colRules = Dispatch[ Thread[colNames -> Thread[Slot[Range[Length[colNames]]]]]],
indexRules = Dispatch[Thread[colNames -> Range[Length[colNames]]]]},
With[{selF = Apply[Function, Hold[condition] /. colRules]},
Select[{rows}, selF ## # &][[All, cnames /. indexRules]]]];
What happens here is that the function used in Select gets generated automatically from your specifications. For example (using #Yoda's example):
rows = Array[#1 #2 &, {5, 15}];
We need to define the column names (must be strings or symbols without values):
In[425]:=
colnames = "c" <> ToString[#] & /# Range[15]
Out[425]= {"c1", "c2", "c3", "c4", "c5", "c6", "c7", "c8", "c9", "c10", "c11", "c12",
"c13", "c14", "c15"}
(in practice, usually names are more descriptive, of course). Here is the table then:
table = Prepend[rows, colnames];
Here is the select statement you need (I picked x = 4 and y=2):
select[{"c1", "c2", "c3", "c6", "c7", "c8", "c9", "c15"}, from[table],
where["c2" == 4 && "c8" != 2]]
{{2, 4, 6, 12, 14, 16, 18, 30}}
Now, for a single query, this may look like a complicated way to do this. But you can do many different queries, such as
In[468]:= select[{"c1", "c2", "c3"}, from[table], where[EvenQ["c2"] && "c10" > 10]]
Out[468]= {{2, 4, 6}, {3, 6, 9}, {4, 8, 12}, {5, 10, 15}}
and similar.
Of course, if there are specific correlations in your data, you might find a particular special-purpose algorithm which will be faster. The function above can be extended in many ways, to simplify common queries (include "all", etc), or to auto-compile the generated pure function (if possible).
EDIT
On a philosophical note, I am sure that many Mathematica users (myself included) found themselves from time to time writing similar code again and again. The fact that Mathematica has a concise syntax makes it often very easy to write for any particular case. However, as long as one works in some specific domain (like, for example, data manipulations in a table), the cost of repeating yourself will be high for many operations. What my example illustrates in a very simple setting is a one possible way out - create a Domain-Specific Language (DSL). For that, one generally needs to define a syntax/grammar for it, and write a compiler from it to Mathematica (to generate Mathematica code automatically). Now, the example above is a very primitive realization of this idea, but my point is that Mathematica is generally very well suited for DSL creation, which I think is a very powerful technique.
data = RandomInteger[{1, 20}, {40, 20}]
x = 5;
y = 8;
Select[data, (#[[2]] == x && #[[8]] != y &)][[All, {1, 2, 3, 6, 7, 8, 9, -1}]]
==> {{5, 5, 1, 4, 18, 6, 3, 5}, {10, 5, 15, 3, 15, 14, 2, 5}, {18, 5, 6, 7, 7, 19, 14, 6}}
Some useful commands to get pieces of matrices and list are Span (;;), Drop, Take, Select, Cases and more. See tutorial/GettingAndSettingPiecesOfMatrices and guide/PartsOfMatrices,
Part ([[...]]) in combination with ;; can be quite powerful. a[[All, 1;;-1;;2]], for instance, means take all rows and all odd columns (-1 having the usual meaning of counting from the end).
Select can be used to pick elements from a list (and remember a matrix is a list of lists), based on a logical function. It's twin brother is Cases which does selection based on a pattern. The function I used here is a 'pure' function, where # refers to the argument on which this function is applied (the elements of the list in this case). Since the elements are lists themselves (the rows of the matrix) I can refer to the columns by using the Part ([[..]]) function.
To pull out columns (or rows) you can do it by part indexing
data = Array[#1 #2 &, {5, 15}];
data[[All, Flatten#{Range#3, Range ## {6, 9}, -1}]]
MatrixForm#%
The last line is just to view it pretty.
As Sjoerd mentioned in his comment (and in the explanation in his answer), indexing a single range can be easily done with the Span (;;) command. If you are joining multiple disjoint ranges, using Flatten to combine the separate ranges created with Range is easier than entering them by hand.
I read:
Extract Column 1 to 3, 6 to 9 as well as the last One for every lines where the value in Column 2 is equal to x and the value in column 8 is different than y
as meaning that we want:
elements 1-3 and 6-9 from each row
AND
the last element from rows wherein [[2]] == x && [[8]] != y.
This is what I hacked together:
a = RandomInteger[5, {20, 10}]; (*define the array*)
x = 4; y = 0; (*define the test values*)
Join ## Range ### {1 ;; 3, 6 ;; 9}; (*define the column ranges*)
#2 == x && #8 != y & ### a; (*test the rows*)
Append[%%, #] & /# % /. {True -> -1, False :> Sequence[]}; (*complete the ranges according to the test*)
MapThread[Part, {a, %}] // TableForm (*extract and display*)
How to get the oddly indexed elements in a list? I am thinking of Select, but did not find anything returning an element's position, especially considering there are repetitive elements in the list.
Also in general, how to select those elements whose indices satisfy some certain conditions?
Here's a few more in addition to #belisarius's answer, which don't require computing Length[lis]:
Take[lis, {1, -1, 2}]
lis[[1 ;; -1 ;; 2]]
You can often use -1 to represent the "last" position.
There are a lot of ways, here are some of them:
In[2]:= a = Range[10];le = Length#a;
In[3]:= Table[a[[i]], {i, 1, le, 2}]
In[5]:= Pick[a, Table[Mod[i, 2], {i, 1, le}], 1]
In[6]:= a[[1 ;; le ;; 2]]
In general, with Pick[] (as an example) you can model any conceivable index mask.
For some reason the terse form of Span has been omitted from the answers.
Range[20][[;;;;2]]
{1, 3, 5, 7, 9, 11, 13, 15, 17, 19}
Quoting the documentation:
;;;;k
from the beginning to the end in steps of k.
Does mathematica have something like "select any" that gets any element of a list that satisfies a criterion?
If you just want to return after the first matching element, use the optional third argument to Select, which is the maximum number of results to return. So you can just do
Any[list_List, crit_, default_:"no match"] :=
With[{maybeMatch = Select[list, crit, 1]},
If[maybeMatch =!= {},
First[maybeMatch],
default]
Mathematica lacks a great way to signal failure to find an answer, since it lacks multiple return values, or the equivalent of Haskell's Maybe type. My solution is to have a user-specifiable default value, so you can make sure you pass in something that's easily distinguishable from a valid answer.
Well, the downside of Eric's answer is that it does execute OddQ on all elements of the list. My call is relatively costly, and it feels wrong to compute it too often. Also, the element of randomness is clearly unneeded, the first one is fine with me.
So, how about
SelectAny[list_List, criterion_] :=
Catch[Scan[ If[criterion[#], Throw[#, "result"]] &, list];
Throw["No such element"], "result"]
And then
SelectAny[{1, 2, 3, 4, 5}, OddQ]
returns 1.
I still wish something were built into Mathematica. Using home-brew functions kind of enlarges your program without bringing much direct benefit.
The Select function provides this built-in, via its third argument which indicates the maximum number of items to select:
In[1]:= Select[{1, 2, 3, 4, 5}, OddQ, 1]
Out[1]= {1}
When none match:
In[2]:= Select[{2, 4}, OddQ, 1]
Out[2]= {}
Edit: Oops, missed that nes1983 already stated this.
You can do it relatively easily with Scan and Return
fstp[p_, l_List] := Scan[ p## && Return## &, l ]
So
In[2]:= OddQ ~fstp~ Range[1,5]
Out[2]= 1
In[3]:= EvenQ ~fstp~ Range[1,5]
Out[3]= 2
I really wish Mathematica could have some options to make expressions evaluated lazily. In a lazy language such as Haskell, you can just define it as normal
fstp p = head . filter p
There's "Select", that gets all the elements that satisfy a condition. So
In[43]:= Select[ {1, 2, 3, 4, 5}, OddQ ]
Out[43]= {1, 3, 5}
Or do you mean that you want to randomly select a single matching element? I don't know of anything built-in, but you can define it pretty quickly:
Any[lst_, q_] :=
Select[ lst, q] // (Part[#, 1 + Random[Integer, Length[#] - 1]]) &
Which you could use the same way::
In[51]:= Any[ {1, 2, 3, 4, 5}, OddQ ]
Out[51]= 3