I've been struggling with this query for a few hours and it seems that it should be fairly straightforward, but for some reason I'm finding it quite difficult. My customer wants to see the top 2 most recent "key" for each clientType.
(Really it's the top N - but more than "first" or "last")
While some translation is done before the step I'm asking about, I have the data looking like this as the output of the query (note: due to proprietary reasons I cannot provide the actual steps above this, nor is this real data but I think it should translate OK). I've starred the records that should end up in the output.
out? clientType key _time
---- ---------- ------------- -----------------
* Mobile asfk129458715 2020-10-13 12:10Z
* Online askg259750505 2020-10-12 11:59Z
* Email dh8iwwih33e99 2020-10-12 11:58Z
Online schf38hrnf98u 2020-10-12 11:00Z
* Online vn8n34rf9v83j 2020-10-12 11:56Z
* Mobile sjvn98h3idv9d 2020-10-12 11:56Z
* Email 92hnfi928rdh9 2020-10-12 11:55Z
* Fax jkcni983iiff4 2020-10-09 06:54Z
Now, I've been able to get this working on a smaller scale, say 1 day. but some clientTypes are not very frequent and we need to see the most recent of those as well. The output of this is to be used for certain audit purposes, and What I've found is that when I extend the search to multiple days (returning > 10,000 events), the output is erratic, and I see results that are out of order, not the most recent, or otherwise askew.
What I want to see is this - only the top 2 for each client type, sorted by time descending within each group
clientType key _time
---------- ------------- --------------
Mobile asfk129458715 2020-10-13 12:10Z
Mobile sjvn98h3idv9d 2020-10-12 11:56Z
Online askg259750505 2020-10-12 11:59Z
Online vn8n34rf9v83j 2020-10-12 11:56Z
Email dh8iwwih33e99 2020-10-12 11:58Z
Email 92hnfi928rdh9 2020-10-12 11:55Z
Fax jkcni983iiff4 2020-10-09 06:54Z
for some "what I tried", I've tried using some query code in various orders mostly revolving around stats list(key), sort 0 -_time etc, with various "by" clauses. The output of this query will also go through some additional translation to be used in our audit system, which takes a list of keys, each wrapped in single quotes and comma-delimited.
I've used stats delim="','" and mvcombine with some success at this point in the query to get results that finally look like this. I wanted to include this as part of the question so it's clear what the end state needs to look like, in case something needs to change somewhere in the grouping and sorting section to make this easier.
clientType keys
---------- -------------
Mobile 'asfk129458715','sjvn98h3idv9d'
Online 'askg259750505','vn8n34rf9v83j'
Email 'dh8iwwih33e99','92hnfi928rdh9'
Fax 'jkcni983iiff4'
To get the two (or 'N') most recent events by a certain field, first sort by time then use the dedup command to select the first N results.
<your query>
| sort - _time
| dedup 2 clientType
While #RichG's dedup option may work, here's one that uses stats and mvindex:
index=ndx sourcetype=srctp clientType=* key=*
| eval comb=_time+" | "+key
| stats values(comb) as comb by clientType
Using mvindex in its range form, instead of selecting merely the last item
| eval mostrecents=mvindex(comb,-1-N,-1)
(Of course - sub-in for N whatever you need)
| fields - comb
| mvexpand mostrecents
| rex field=mostrecent "(?<timemost>\d+)\s\|\s(?<keymost>.+)"
| table clientType timemost keymost
| eval timemost=strftime(timemost,"%c")
Related
I am a beginner trying to achieve a simple operation in SSRS using Visual Studio 2019. I have a query which returns a table as follows
ID | Name | Married
1 | Jack | Y
2 | Jack | N
The number of records might vary depending on the number of results. On the report, I want to display only the field 'Married' once. The value of the field will be determined using an OR operation, i.e. if the field 'Married' is 'Y' for any one record, I want to display a 'Y' on the report.
Assuming the Values are either Y or N, you should be able to use something like
=MAX(Fields!Married.Value)
If you report is grouped by, for example, Name then this will give you the MAX value within each group which is probably what you want.
If this does not help, edit your question and show
Your report design
Row Group panel plus details of grouping
A larger sample of data
Expected results from that sample data
I have a couple of joined Athena tables in Quicksight. The data looks something like this:
Ans_Count | ID | Alias
10 | 1 | A
10 | 1 | B
10 | 1 | C
20 | 2 | D
20 | 2 | E
20 | 2 | F
I want to create a calculated field such that it sums the Ans_Count column based on distinct IDs only. i.e., in the example above the result should be 30.
How do I do that?? Thanks!
Are you looking for the sum before or after applying a filter?
Sumif(Ans_Count,ID) may be what your looking for.
If you need to always return the result of the sum, regardless of the filter on the visual, look at the sumOver() function.
You can use distinctCountOver at PRE_AGG level to count unique number of values for a given partition. You could use that count to drive the sumIf condition as well.
Example : distinctCountOver(operand, [partition fields], PRE_AGG)
More details about what will be visual's group by specification and an example where there duplicate IDs will help give a specific solution.
It might even be as simple as minOver(Ans_Count, [ID], PRE_AGG) and using SUM aggregation on top of it in the visual.
If you want another column with the values repeated, use sumOver(Ans_Count, [ID], PRE_AGG). Or, if you want to aggregate via QuickSight, you would use sumOver(sum(Ans_Count), [ID]).
I agree with the above suggestions to use sumOver(sum(Ans_Count), [ID]).
I have yet to understand the use cases for pre_agg, so if anyone has concrete examples please share them!
Another suggestion would be to do a sumover + partition by in your table (if possible) before uploading the dataset, then checking if the results matche with Quicksight's aggregations. I find Quicksight can be tricky with calculated fields, aggregations, and nested ifs so I've been doing calculations in SQL where possible before bringing it in to quicksight to have a better grasp of what the outputs should look like. This obviously is an extra step, but can help in understanding how quicksight pulls off calcs and brings up figures (as the documentation doesn't always give much), and spotting things that don't look right (I've had a few) before you share your analysis with a wider group.
I am new to OBIEE and would like to create an analysis where I can place one next to the other 2 columns with figures from same dimension but with different data.
To better explain it: let's say that in Dim1 we have Invoices and Payments as members. We also have other dims as Date, Invoice Number and so on. This would be the current output:
Date | Dim1 | Invoice Number | Amount
10/01/17 Invoice 1234 -450
10/02/17 Payment 1234 450
So, what I want is, instead of creating 2 reports, one for the Invoices and the other one for Payments, a single report with the following output:
Invoice Date | Invoice | Payment date | Payment | Invoice Number | Amount inv | Amount paid
10/01/17 Invoice 10/02/17 Payment 1234 -450 450
Is this kind of output achievable inside OBIEE?
Thanks!
You are not trying to "filter on same dimension" but you are trying to convert rows into columns.
While it is possible to cheat your way around this it is definitely not something which is suggested! You are facing an analytical system - not Excel.
If this is an actual requirement and not simply a "I wish to see it this way" then the best approach is to store the data properly.
Second-best approach is to model it in the RPD with different logical table sources.
Last and the option NOT to go for right away is what you are asking for: Doing it in the front-end.
Apart from that: It's "analyses" that you are working with in OBI. If you have a "report" then you are in BI Publisher which is a completely different tool.
I want to accomplish something easy to understand (and maybe easy to do but I can't find a way...).
I have a table which represents the date when a client has bought something.
Let's have this example:
=============================================
Purchase_id | Purchase_date | Client_id
=============================================
1 | 2016/03/02 | 1
---------------------------------------------
2 | 2016/03/02 | 2
---------------------------------------------
3 | 2016/03/11 | 3
---------------------------------------------
I want to create a single number card which will be the average of purchase realised by day.
So for this example, the result would be:
Result = 3 purchases / 2 different days = 1.5
I managed doing it by grouping in my query by Purchase_date and my new column is the number of rows.
It gives me the following query:
==================================
Purchase_date | Number of rows
==================================
2016/03/02 | 2
----------------------------------
2016/03/11 | 1
----------------------------------
Then I put the field Number of rows in a single number card, selecting "Average".
I have to precise that I am using Direct Query with SQL Server.
But the problem is that I want to have a filter on the Client_id. And once I do the grouping, I lose this column.
Is there a way to have this Client_id as a parameter?
Maybe even the fact of grouping is not the right solution here.
Thank you in advance.
You can create a measure to calculate this average.
From Power BI's docs:
The calculated results of measures are always changing in response to
your interaction with your reports, allowing for fast and dynamic
ad-hoc data exploration
This means filtering client_id's will change the measure accordingly.
Here is an easy way of defining this measure:
Result = DISTINCTCOUNT(tableName[Purchase_date])/DISTINCTCOUNT(tableName[Purchase_id])
I'm very new to Cascading and Hadoop both, so be gentle... :-D
I think I'm finding myself way over-engineering something. Basically my situation is that I have a pipe delimited file with 9 fields. I want to compute some aggregated statistics over those 9 fields using different groupings. The result should be 10 fields of which only 6 are either counts or sums. So far I'm up to 4 Unique pipes, 4 CountBy pipes, 1 SumBy, 1 GroupBy, 1 Every, 2 Each, 5 CoGroups and a couple others. I'm needing to add another small piece of functionality and the only way I can see to do it is to add in 2 Filters, 2 more CoGroups and 2 more Each pipes. This all seems like way overkill just to compute a few aggregated statistics. So I'm thinking I'm really misunderstanding something.
My input file looks like this:
storeID | invoiceID | groupID | customerID | transaction date | quantity | price | item type | customer type
Item type is either "I", "S" or "G" for inventory, service or group items, customers belong to groups. The rest should be self-explanatory
The result I want is:
project ID | storeID | year | month | unique invoices | unique groups | unique customers | customer visits | inventory type sales | service type sales |
project ID is a constant, customer visits is how many days during the month the customer came in and bought something
The setup that I'm using right now uses a TextDelimited Tap as my source to read the file and passes the records to an Each pipe which uses a DateParser to parse the transaction date and adds in year, month and day fields. So far so good. This is where it gets out of control.
I'm splitting the stream from there up into 5 separate streams to process each of the aggregated fields that I want. Then I'm joining all the results together in 5 CoGroup pipes, sending the result through Insert (to insert the project ID) and writing through a TextDelimited sink Tap.
Is there an easier way than splitting into 5 streams like that? The first four streams do almost the exact same thing just on different fields. For example, the first stream uses a Unique pipe to just get unique invoiceID's then uses a CountBy to count the number of records with the same storeID, year and month. That gives me the number of unique invoices created for each store by year and month. Then there is a stream that does the same thing with groupID and another that does it with customerID.
Any ideas for simplifying this? There must be an easier way.