I have a sheet of non-table groups and would like to view them as a single table.
Each group consists of 4 or 5 rows and 2+ columns with 1 or more blank rows/columns between.
Overall, the groups are organized into rows and columns on the sheet. There shouldn't be more than 3 groups in a row, but some rows may have blank spaces for future groups.
New groups and group columns are added regularly so existing groups can be relocated on the sheet anytime.
The group names are a unique combination of letters and a number. Unfortunately, they are not prefixed "Group" or numbered consecutively like in my example. Giving a generic example that conveys all criteria is harder than it looks 😅.
This is a shared document so I'd like to avoid making structural changes that would affect other users.
I've experimented with some of the transform options, but I'm new to PQ and didn't make much progress.
This answer to a similar question is a step in the right direction, but it looks like I'll need additional steps since my starting data isn't quite as consistent.
Thank you for your time.
Before Example
After Example
In this production data the Total Quantity processed through the machines is in table WeekProd and the number of parts scrapped is stored in table WeekScrap. There can be more than one scrap reason and quantity for each production line and they are summed using a measure.
The two tables are filtered in common by the calendar week number, the machine lookup and the shift lookup. Using 2 visuals shows that this filtering is working as expected:
However, when I place the Sum Scrap Qty measure onto the WeekProd visual it show the scrap qty on every row, although the total figure of 34 ends up correct.
How can I stop this from happening?
I'm implementing a Data Mart following the Kimball methodology and I have a challenge with applying deltas from multiple source tables against a single target dimension.
Here's an example of the incoming source data:
STG_APPLICATION
APP_ID, APP_NAME, APP_START_DATE, CDC_HASH, ...
1, FOOBAR, 20/10/2018, MD5_XXX
STG_APPLICATION_STATUS
APP_ID, STATUS_CODE, STATUS_DESC, CDC_HASH, ...
1, SUBMITTED, "APP WAS SUBMITTED", MD5_YYY
Each of these tables (there are several others) represent a normalised version of the source data i.e. a single application can have one or more statuses associated with it.
Now then, because we only get a full alpha for these tables we have to do a snapshot merge, i.e. apply a full outer join on the current day set of records against the previous day set of records for each individual table. This is computed by comparing the CDC_HASH (a concat of all source columns). The result of this comparison is stored in a delta table as follows:
STG_APPLICATION_DELTA
APP_ID, APP_NAME, APP_START_DATE, CDC_HASH, CDC_STATUS ...
STG_APPLICATION_STATUS
APP_ID, STATUS_CODE, STATUS_DESC, CDC_HASH, CDC_STATUS...
1, AWARDED, "APP WAS AWARDED", MD5_YYY, NEW
So in this example, the first table, STG_APPLICATION did not generate a delta record as the attributes pertaining to that table did not change between daily loads. However, the associated table, STG_APPLICATION_STATUS, did calculate a delta, i.e. one or more fields have changed since the last load. This is highlighted by the CDC_STATUS which identifies it as a new record to insert.
The problem now of course is how to correctly handle this situation when loading the target dimension? For example:
DIM_APPLICATION
ID, APPLICATION_ID, APP_NAME, APP_START_DATE, APP_STATUS_CODE, FROM_DATE, TO_DATE
1, 1, FOOBAR, 20/10/2018, SUBMITTED, 20/10/2018, 12/04/2019
2, 1, NULL, NULL, NULL, AWARDED, 13/04/2019, 99/99/9999
This shows the first record - based on these two staging tables being joined - and the second record which is meant to reflect an updated version of the record. However, as previously illustrated, my Delta tables are only partially populated, and therefore I am unable to correctly update the dimension as shown here.
Logically, I understand that I need to be able to include all fields that are used by the dimension as part of my delta calculation, so that I have a copy of a full record when updating the dimension, but I'm not sure of the best way to implement this in my staging area. As shown already, I currently only have independent staging tables, each of which calculate their delta separately.
Please can somebody advise on the best way to handle this? I'm scrutinized Kimball's books on this but to no avail. And I've equally found no suitable answer on any online forums. This is a common problem so I'm sure there exists a suitable architectural pattern to resolve this.
You will need to either compare on joined records or lookup the current dimension values.
If the amount of (unchanged) data is not excessive, you could join the full snapshots of STG_APPLICATION and STG_APPLICATION_STATUS together on APP_ID until they resemble the dimension record column-wise and store those in a separate table with their CDC hash to use as previous day. You then take the deltas at this level and send the (complete) changed records as updates to the dimension.
If the amount of records in the daily update makes it impractical to join the full tables, you can take the deltas and full outer join them as you do now. Then you look up the current dimension record for this APP_ID and fill in all empty fields in the delta record. The completed record is then sent as an update to the dimension.
This solution requires less storage but seems more fragile, especially if multiple changes are possible within a day. If there are many changes, performance may also suffer. For a handful of changes in millions of records, it should be more efficient.
The information is correct but its in a "staircase" with repeating columns and rows and a lot of excess empty cells.
(im new to ssrs hopefully this is really basic)
hours are stored in sub-projects, i am trying to get the sum for each project within each department
So i Added two parent groups one for projects(row) and one for departments(col), and hid the hours row since i only want the sum
inputs
_______________________
|_________|__[dept]____|
|[proj]___|__sum(hours)|
[hidden] |_________|__[hours]___|
Here's what i got
__________________________________
|_____|__D1__|__D1__|__D2__|__D2__|
|__P1_|__10__|______|______|______|
|__P2_|______|__76__|______|______|
|__P1_|______|______|__32__|______|
|__P2_|______|______|______|__48__|
Here's what i want
_____________________
|_____|__D1__|__D2__|
|__P1_|__10__|__32__|
|__P2_|__76__|__48__|
I'm not sure what you have now exactly, but you just need a single row group for Project, remove any other row groups.
I have two files, A and B. The records in both files share the same format and the first n characters of a record is its unique identifier. The record is of fixed length format and consists of m fields (field1, field2, field3, ...fieldm). File B contains new records and records in file A that have changed. How can I use cloverETL to determine which fields have changed in a record that appears in both files?
Also, how can I gather metrics on the frequency of changes for individual fiels. For example, I would like to know how many records had changes in fieldm.
This is typical example of Slowly Changing Dimension problem. Solution with CloverETL is described on theirs blog: Building Data Warehouse with CloverETL: Slowly Changing Dimension Type 1 and Building Data Warehouse with CloverETL: Slowly Changing Dimension Type 2.