I am looking to find the mode (most occurrent) value of one column, given another column.
I know how to do it but the calculation takes multiple minutes and can make Excel unresponsive. Therefore I am looking for a better way.
Suppose my data looks like this
group | Level
1 D
1 A
1.1 B
1 C
1 A
1 E
Then I want the output to look like this:
group | LevelMode
1 A
1.1 B
Assuming the data is in the upper left corner of the worksheet, I am now using this formula, which I drag along the C column:
=COUNTIFS(A:A;A2;B:B;B2)/COUNTIF(A:A;A2)
This gives me the ratio of values within the group. Afterwards I remove all that are not above 0.5 as well as duplicates, giving me the mode for each group if it exists.
This formula does exactly what I want, but for about 50000 lines it simply takes too many resources. I expected to find this in the pivot table options, or after a quick search online, but I have not found any way to achieve my goal.
I have decent hardware, a fairly recent version of Excel and would prefer to do this without macro's. However, if macros are required to achieve this then so be it.
Try this:
Step 1.
Create the Pivot Table and place in RowLabels "Group" and under it "Level". Now, place "Level" as count in "Values" field.
Step 2.
See the picture below. Click in the black arrow in the right side of "Level" (you can't see it in the picture but it is supposed to be inside the red circle).
Choose "Values Filters", then "Top 10" and then set the number to 1 (top 1). Done!!
Attention: you need to do step 2 for "Level". If you do the same to "Group" it will not work.
Edit: this is the result and how the Pivot Table should look like.
Now to shape the output to the desired format:
Under PivotTable tools > Design, set the Report Layout to Tabular form
Disable totals and subtotals
Assuming default settings you can now easily copy the list of Groups and LevelModes
Related
I have a table like this：
a
b
c
1
2
abc
2
3
4.00
note c2 is text while c3 is a number.
When I do
=QUERY(A1:C,"select *")
The result is like
a
b
c
1
2
2
3
4.00
The "text" in C2 has been missed. You can see the live sheet here:
https://docs.google.com/spreadsheets/d/1UOiP1JILUwgyYUsmy5RzQrpGj7opvPEXE46B3xfvHoQ/edit?usp=sharing
How to deal with this issue?
QUERY is very useful, but it has a main limitation: only can handle one kind of data per column. The other data is left as blank. There are usually ways to try to overcome this from inside the QUERY, but I've found them unfruitful. What you can do is just to use:
={A:C}
You can work with filters by its own, but as a step-by-step to adapt the main features of query: If you need to add conditions, use LAMBDA INDEX and FILTER
For example, to check where A is not null:
=LAMBDA(quer,FILTER(quer,INDEX(quer,,1)<>""))({A:C}) --> with INDEX(quer,,1), I've accesed the first column
Where B is more than one cell and less than other:
=LAMBDA(quer,FILTER(quer,INDEX(quer,,2)>D1,INDEX(quer,,2)<D2))({A:C})
For sorting and limiting an amount of items, use SORTN. For example, you want to sort by 3rd column and limit to 5 higher values in that column:
=LAMBDA(quer,SORTN(FILTER(quer,INDEX(quer,,1)<>""),5,1,3,0))({A:C})
Or, to limit to 5 elements without sorting use ARRAY_CONSTRAIN:
=ARRAY_CONSTRAIN(LAMBDA(quer,FILTER(quer,INDEX(quer,,1)<>""))({A:C}),5)
There are other options, you can use REGEXMATCH and other options, and emulate QUERYs functions without missing data. Let me know!
shenkwen,
If you are comfortable with adding an Google App Script in your sheet to give you a custom function, I have a QUERY replacement function that supports all standard SQL SELECT syntax. I don't analyze the column data to try and force to one type based on which is the most common data in the column - so this is not an issue.
The custom function code - is one file and is at:
https://github.com/demmings/gsSQL/tree/main/dist
After you save, you have a new function from your sheet. In your example, the syntax would be
=gsSQL("select a,b,c from testTable", {{"testTable", "F150:H152", 60, true}})
If your data is on a separate tab called 'testTable'(or whatever you want), the second parameter is not required.
I have typed in your example data into my test sheet (see line 150)
https://docs.google.com/spreadsheets/d/1Zmyk7a7u0xvICrxen-c0CdpssrLTkHwYx6XL00Tb1ws/edit?usp=sharing
I have a bunch of statistical data that I have chopped up and pieced together in power query. In looks like there is category missing from the Database. To fill in the gaps I am trying to take the grand totals which are correct (red), subtract from that what I know to be correct, with the remaining numbers giving me my answer (orange).
The correct data starts with I1, I2, I3, I4, so possibly a grand total of these, by state.
At the moment this is filled by the following formula in excel;
=E53-SUMIFS($E$5:$E$44,$B$5:$B$44,B45,$C$5:$C$44,C45,$D$5:$D$44,D45)
Any help with how the heck I can do this in power query. I realise I cant use the same formula but any ideas would be much appreciated. I can change the text in red to total if that helps in some way?
Thanks
Here's one potential way. If you start with your spreadsheet set up similar to this:
I only used a subset of your StateIDs from your example and generated my own Values for this example. And the figures in the Available column would be from your red section.
Then add the table from the spreadsheet to Power Query (in Excel, you would click on the table and then Data > From Table/Range > Select My table has headers and click OK).
In Power Query:
You'll probably have to change the TimeID type to date if you want to use the dates for anything, because it will probably come in as date-time type--I won't use the dates here though, so you could skip changing the type (otherwise, right-click the TimeID column > Change Type > Date)
Then use Group By to aggregate values and set the stage for the calculation you want (select the StateID > Group By > and setup the groupings like below and click OK)
You should see something like this:
Then add a new column with your calculation (Add Column > Custom Column > Set it up like below and click OK)
You should see something like this:
I know there must be a simple answer to this, but I can't find it.
I have added a couple of textboxes to a Matrix in a BIDS/SSRS report. I've given these textboxes values such as:
=Fields!WEEK1USAGE.Value
It works (after a fashion); when I run the report (either on the Preview tab, or on the Report Server site) I see the first corresponding data value on the report - but only one.
I would think that once a value has been assigned via expressions such as "=Fields!WEEK1USAGE.Value", each value would display (rows would automatically be added).
There must be some property on the Matrix or the textbox that specified this, but I can't see what it might be.
Here is how my report looks (very minimalistic, so far) in the Layout pane:
...and after running, on the Preview tab:
Obviously, I want the report to display as many rows as necessary, not just one. The textboxes do have a "RepeatWith" property, but there description doesn't sound interesting/useful/promising.
I don't see any property on the Matrix control that looks right, either.
I thought maybe the designer was only showing one row of values, and ran the report on the server, too, but there also it just shows the two values.
So what do I need to do to get all the data for a provided field?
Matrices are for display of grouped data and summary information, usually in a horizontally expanding pivot table type of format. Is a matrix really what you are after? Looking at your expression you have =Fields!Week1Usage.Value but in a matrix what I expect to see would be at least =Sum(Fields!Week1Usage.Value) or even better just =Sum(Fields!Usage.Value). Then you would have ProactDescription as your row group and the week as your column group and it would all just work out everything for you, grouping and summing by Proact vertically and expanding the weeks out horizontally.
What seems to be happening is that you have no grouping on rows or columns and no aggregation so it is falling back to the default display which is effectively the First function - it displays the first row of data and as far as the matrix is concerned it has done its job because there is no grouping.
Without knowing your problem or data, I'll make up a scenario that might be what you are doing and discuss how the matrix does the heavy lifting to solve that problem. Let's say you have usage data for multiple Proacts. Each time one is used you record the usage amount and the date and time it is used. It could be used multiple times per day but certainly multiple times in a week. So you might be able to get the times each Proact is used from a table like so:
SELECT ProactDescription, TimeUsed, Usage
FROM ProactUsage
ORDER BY ProactDescription, TimeUsed
In your report you want to show the total weekly usage for each Proact over multiple weeks. Something like this:
Proact Week1 Week2 Week3 ...
Description Usage Usage Usage ...
--------------------------------------------
Anise, Fennel 1 CT 20.00 22.50 16.35 ...
St John's Wort 15.20 33.90 28.25 ...
...
and so on. Using a dataset based on the SQL above we create a matrix and in the row group properties we group on =Fields!ProactDescription.Value and in the column group properties we group on a week expression like =DateDiff(DateInterval.Week, Fields!TimeUsed.Value, Today) and then in the intersection of the row and column we put =Sum(Fields!Usage.Value). To display the header of the column nicely put an expression like
="Week " & DateDiff(DateInterval.Week, Fields!TimeUsed.Value, Today)
The matrix automatically does all the summing by week and product and expands the weeks horizontally for as many as you are reporting. For bonus points you can also put totaling at the end of the columns and the rows to show the total use of that Proact for the period (row total) and total use of all Proacts in that week (column total).
I'm testing Kibana 4 for a project.
I have created an index from my database table which is composed by 3 fields:
Date
User
Action
I would like to display my index as a simple table (3 column, N rows) in my dashboard.
I tried to use "Data table" visualization but I can't find a way to display my results without any Metrics (Count, Sum etc...)
Maybe is pretty simple and I missed something... is there a way to do this?
Regards,
On the Discover tab, create a view that has just the fields you want and then save that as a search.
On the Dashboard tab, click on Edit then hit the + Create new button to add a widget, but if you look at the top, there's a Searches tab. Select that and add your saved search in.
[Elastic 7.x / 2019 Update]
I was a bit confused when I read #Alcanzar's answer so I am sharing a little more noob-friendly step-by-step how-to here :
STEP 1 : Create the Index Pattern
STEP 2 : Go to the Dashboard view, and create a view on your index
Select each column you want to include/add in your view by clicking "add" on it (The confusing part is that until you do that, you will have a "scrambled" view listing everything in a jumbled way.)
STEP 3 : Go to the Dashboard view, and create a view on your index
The trick is to select the specific columns you want to include... and voila !
Don't forget to save your view, this will help a lot in the process.
In Kibana 7.5.0 you can do it as follows:
Go to Discover section
Select fields you are interested in
Click on Save to save your discover search so you can use it in visualizations and dashboards
Click on Dashboard and create a new dashboard
Click on Add and select the panel
There is no step 6
The accepted solution has its pros (if, for simplicity, you see your index as a table, this is the only way to deal with rows naturally) but also cons (it allows the user to see too much information, by expanding the records that appear in the table; users cannot get an export of the values).
So if you plan to build tables to use in reports seen by users which should not see everthing and may want to get exports of the data, I recommend a different (hacky) approach using Table visualizations:
Say you have three columns A, B and C:
If there are no duplicates considering the combined values of A and B, you can use these two vales as aggregation fields, and then set a Max or Top hit Metric for C.
If even A, B and C have duplicates, then you can use the three of them as aggregation fields and add a Metric count, that will give you the number of repeated rows. This solution makes somehow sense, because instead of repeating the same row 'n' times you just tells you should have repeated 'n' times that row.
If A and B have duplicates but A, B and C are unique, then there is, afaik, no elegant solution. You have to use the three of them as aggregation fields, but then you would have a dummy metric at the end (e.g. count, always equal to 1).
Why? why do we have to go through all of this? that is another question...
I'm testing Kibana 4 for a project.
I have created an index from my database table which is composed by 3 fields:
Date
User
Action
I would like to display my index as a simple table (3 column, N rows) in my dashboard.
I tried to use "Data table" visualization but I can't find a way to display my results without any Metrics (Count, Sum etc...)
Maybe is pretty simple and I missed something... is there a way to do this?
Regards,
On the Discover tab, create a view that has just the fields you want and then save that as a search.
On the Dashboard tab, click on Edit then hit the + Create new button to add a widget, but if you look at the top, there's a Searches tab. Select that and add your saved search in.
[Elastic 7.x / 2019 Update]
I was a bit confused when I read #Alcanzar's answer so I am sharing a little more noob-friendly step-by-step how-to here :
STEP 1 : Create the Index Pattern
STEP 2 : Go to the Dashboard view, and create a view on your index
Select each column you want to include/add in your view by clicking "add" on it (The confusing part is that until you do that, you will have a "scrambled" view listing everything in a jumbled way.)
STEP 3 : Go to the Dashboard view, and create a view on your index
The trick is to select the specific columns you want to include... and voila !
Don't forget to save your view, this will help a lot in the process.
In Kibana 7.5.0 you can do it as follows:
Go to Discover section
Select fields you are interested in
Click on Save to save your discover search so you can use it in visualizations and dashboards
Click on Dashboard and create a new dashboard
Click on Add and select the panel
There is no step 6
The accepted solution has its pros (if, for simplicity, you see your index as a table, this is the only way to deal with rows naturally) but also cons (it allows the user to see too much information, by expanding the records that appear in the table; users cannot get an export of the values).
So if you plan to build tables to use in reports seen by users which should not see everthing and may want to get exports of the data, I recommend a different (hacky) approach using Table visualizations:
Say you have three columns A, B and C:
If there are no duplicates considering the combined values of A and B, you can use these two vales as aggregation fields, and then set a Max or Top hit Metric for C.
If even A, B and C have duplicates, then you can use the three of them as aggregation fields and add a Metric count, that will give you the number of repeated rows. This solution makes somehow sense, because instead of repeating the same row 'n' times you just tells you should have repeated 'n' times that row.
If A and B have duplicates but A, B and C are unique, then there is, afaik, no elegant solution. You have to use the three of them as aggregation fields, but then you would have a dummy metric at the end (e.g. count, always equal to 1).
Why? why do we have to go through all of this? that is another question...