Context:
I have a data model in Power pivot with three tables, tTasks, tCaseworks and tCaseworkStatus. I am attempting to create two calculated columns in tCaseworks which from the two data tables. All three tables are linked through the common field casework_id (see illustration below).
The data model is regularly updated with new data. The way I am doing this is as follows:
All three tables are sourced from three corresponding tables in my Excel workbook.
A VBA script deletes all records in the three Excel tables and then refreshes the data model (sidenote: because the data model demands lookup tables to not be empty the VBA code adds one row per table before refreshing).
New data is then added to the excel tables and the data model is refreshed.
This process works perfectly.
Problem:
The problem arises when I am adding calculated columns to tCaseworks and then attempting to update the data as described above. I have added two calculated columns; has_task and status_now. I am using the following DAX code:
has_task:
has_task =
IF (
CONTAINS (
RELATEDTABLE ( tTasks );
tTasks[casework_id]; tCaseworks[casework_id]
);
"Yes";
"No"
)
status_now:
status_now =
VAR TableX = RELATEDTABLE(tCaseworkStatus)
VAR ResultX = IF(
CONTAINS(TableX;tCaseworkStatus[casework_status_code];"Completed");"Completed";
IF(CONTAINS(TableX;tCaseworkStatus[casework_status_code];"Dismissed");"Dismissed";
IF(CONTAINS(TableX;tCaseworkStatus[casework_status_code];"Begun");"Begun";
IF(CONTAINS(TableX;tCaseworkStatus[casework_status_code];"Created");"Created";
"Find no status"))))
RETURN
ResultX
Both of these calculated columns work as expected as long as I do not delete the data in the model (I do have one hickup with both columns as described in this separated problem, but I think that is unrelated).
When the data has been deleted and I refresh the model I get the following error message:
"We cannot get the data from the data model. This is the error message we got: A circular dependency was discovered: 'tCaseworks'[status_now],'tCaseworks'[status_now],'tCaseworks'[has_task],'tCaseworks'[has_task],'tCaseworks'[status_now]."
Question:
What is creating this dependency and how can I avoid it?
My attempted solutions:
The problem only arise when there are two of these calculated columns. Any one of these two works perfectly without the other upon refreshing. I know that calculated columns are prone to circular problems, but unfortunately I need to use columns and not measures. I suspect that perhaps my choice in formula is creating the problem, most likely the contains-function. However, I don't know about any alternative ways of building the formulas I need. Any suggestions?
Edit:
I originally only posted a portion of my data model as I wanted the question to be as concise as possible but I guess it might have been confusing. The whole model concerns five objects from a case handling system: Claims, Cases, Caseworks, Tasks and Action Points. These objects are hierarchical, one claim can have one or more cases, but one case can only have one claim. Similarly, a case can have several caseworks, a casework can have several tasks, a task can have several action points. Additionally, the latter four can have a status attribute which is changed regularly.
I attempted to organize my data model in such a way that I had a lookup table for each object with unique values. I have many attributes for each object in my data that I did not include in the example above, and my goal was to add useful attributes through calculated columns in these tables. The data tables with the changes were intented to provide insight to the lookup tables.
I think your relationship model is a bit unusual. DAX works best when using something like dimensional fact model
I would consider the tCaseworkStatus a fact table since its like a log of the changes to your data. tTasks is a dimension, since it just add an extra dimension to your data.
The tCaseworks is not necessary since it doesn't hold any actual data (only calculated data).
if you want your current model to work, it might fix your problem if you just delete the relationship between tTasks and tCaseworks, and add a new between tTasks and tCaseworksStatus
edit.
it just occurred to me that the reason you have it like this, is that you may have a many-to-many relationship between tTasks and tCaseworksStatus. if that is the case you might have to create a proper many-to-many table. which is kind of what your tCaseworks is, but you cant have a relationship to the same key like you currently have.
edit2.
the solution seemed to be that somehow the Relatedtable function in conjunction with the relationship model was causing the error. using Lookupvalue instead seems to to have fixed the issue.
Related
Context:
I am creating a dashboard in Excel based on the data model I am building in Power Pivot. The source data in the data model is based on various other excel tables I am regularly receiving and copy-pasting into my workbook (their incoming structure is out of my control). My goal is to perform all data processing within Power Pivot/DAX rather than manipulating the data in the worksheets before loading into the model.
Problem:
In my model, I have a table (tabCases) which includes status updates on all cases from a management system. This table has a column named case-ID (not unique). I need to create a lookup-table with unique case-id's where I can create new columns with various KPIs for each case.
How can I do this in Power Pivot?
I found two suggestions in this article but none of them work for me (opt. 1 because it requires a manual creation of the unique ID list and opt. 2 because I don't have a database access).
In my mind there should be something really simple I could do, such as i.e.:
Add new table to data model
Set first column to be equal to DISTINCT(tabCases[caseID])
Is there such a way?
A Linkback Table might help you. Please see the link below:
https://www.sqlbi.com/articles/linkback-tables-in-powerpivot-for-excel-2013/
Thanks
I'm trying to export a large number of records from my database, but I need relationship data in order to build the export correctly. Ideally I would be able to use cursor() to get a Lazy Collection, but that won't load the relationships. I can't load the relationship within a loop, because that will create N+1 queries, and this could be hundreds of thousands of additional queries, which is unacceptable.
Here's what "works" (but runs out of memory):
Record::with('projects')->get()->map(function ($record) {
dd($record); // Shows the `projects` relationship
});
But when I use cursor()...
Record::with('projects')->cursor()->map(function ($record) {
dd($record); // Does NOT show the `projects` relationship
});
Is there a way to get a lazy collection that includes a record's relationship? I have looked in the documentation and it's not clear. Other suggestions have been to use chunk() which is unfortunately not a possibility in this situation.
EDIT: I shouldn't say chunk isn't a possibility, but it's a very expensive re-write. Currently, the data is structured with a lot of variability. So in order to construct the CSV for export, I need (for example) a header for the file. I currently grab that header by looping through all the records (the fields are stored in a JSONB field) and building out an array based on the fields present on those records.
I am also normalizing the data against those headers. So if one record has the field "address-1" but another record doesn't have that, the one that doesn't have it instead shows a blank value in the appropriate column. Otherwise, when inserting the row into the CSV, it doesn't respect the header.
These operations currently grab the entire data set and use a LazyCollection to map the header and normalize the records, and then feed it into the CSV one at a time. It would be ideal if I could grab relationships in a LazyCollection as well rather than having to rewrite the workflow.
according to this doc
cursor work in db stage, while loading relations come after method 'get' or 'first' ...
so: the code in cursor will work in db row represented as Model instance before the overall result, means that this code will run into db, without loading the relation, again db row (iterate through your database records...)
if you can't use chunk... then i think that you can use mySql to manage your data using raw-expressions
I'd like to create a many-to-many relationship between two things: Notes and Labels. However, I'd like to define the labels themselves in code rather than having them in a database table.
Aside from a notes table to represent the Note model, I expect to have a "pivot" table (labels_notes) with two columns: note_id and label.
So, my question is: How would eager loading, getter, setter and "get notes by label" methods on the Note model work?
Background: The primary reason for wanting the Labels in code rather than as content of a table is that they are a small, fixed set of values; users will not be allowed to modify them. Further, there may need to be special logic in the code around certain labels. I considered storing them in a JSON column on notes, but am concerned about the performance impact when searching for Notes by Label.
The solution I opted for was to use a traditional Eloquent model for Labels (including a dedicated database table), but inject the desired values into it via the migration, and use a string primary key. That way we're able to use Eloquent in it's intended manner rather than fighting against it.
Using a string primary key means we can write logic based around specific Labels without worrying about arbitrary numeric IDs (i.e., "id=news" vs "id=12112"). Note that doing this also requires adding public $incrementing = false; in the Label model class.
Injecting the necessary Labels via migration lets us avoid having an additional setup task when deploying, and also avoids coupling our code with an external process.
Ok, so I have 2 entities: Course and Industry
The industry entity is just a reference table which lists all available Industries that can be tagged to a course, to categorizing them. I put in a many to zero or one relationship (a course can choose to have an industry or not, while an industry can be tagged with many courses).
I know I've played around with the diagrams a bit, adding and removing associations in the past.
Now here is the odd part: The column mappings for Course has 2 similar columns, IndustryId and Industry_Id
I suspect it's from a past association, but thought EF would have taken care of that.
Here is the problem:
In my view that creates the course, the IndustryId is the property which needs to be populated. When I create new courses, I see the IndustryId in the database populated.
However, when I access Industry's properties through Course (Course.Industry.Description) nothing is populated. It can't seem to get the Industry entity.
I see the IndustryId populated in the db, so I tried to populate the Industry_Id column. That fixed it.
Weird enough, the property declared in the model is IndustryId, so that column is populated in the db. But when I try to get Industry entities through Course, it needs the Industry_Id, which I don't quite know where it is from.
Anyone have any ideas?
It sounds like in your updating from the database, you changed the column name on your tables from Industry_Id to IndustryId. The next time you updated from the database, EF5 (which can't determine that this is the same column, as it matches on names) dropped the mapping for Industry_Id, and added a new column called IndustryId.
However, you had already created the foreign-key mapping in your EDMX file based on the Industry_Id column - which is why you get the issue around needing it when loading related records.
In general, when using Database-First, whenever you rename a column in the database, you need to update your EF5 model and update / correct any such discrepancies.
I am using Spring + Hibernate, and I will have a HTML from that has like 100+ fields and I must store all these values to database in a single table.
They are all used in one big massive calculation.
How should I handle this, I thought about creating an Entity with 100 fields and setters, getters, but is there a nicer solution for it?
EDIT:
Everytime when someone submits form, a new row will be added, so eventually there will be tens of thousands of rows.
I believe its not about an HTML but about the data modeling.
Think about your data, who are the consumers of it, how and in which business flows you're going to query the data.
In general an entity with 100 fields is not a good idea because it should be mapped to one single table with 100 columns. Its just not maintainable.
Maybe all the data should be normalized and you can store pieces of it in different tables in db with foreign keys?
Hope this helps or at least will give you some direction to think about
I think you could use a Map in this case, because:
You only want to store the fields as key-value elements.
It is more flexible to add/remove fields in the future.
So, instead of having a table with 100 fields you will end with a table with 2 fields (3 if you want to include the form identifier or something like that) and 100 rows.
If many of the form fields are empty (sparse data) you could also save some storage space (it depends on the database you are using).