I have two chart tables both with different data sources. I want one table to act as the filter to the other table.
Here is the problem...
I tried a custom query for my data source which used the email parameter to filter the data source.
The problem is every time a user changes a filter on any page a query is executed in BigQuery, slowing the results and exponentially increasing my BigQuery monthly charges.
I tried blending the two tables.
The problem is the blended data feature only allows for 10 dimensions to be added to the resulting blended data source and is very slow.
I tried creating a control filter using a custom field on the "location" column on each table sharing the same "Field Id".
The problem is that the results table returns all the stores until you click on a location in the control list. And I cannot let a user see other locations.
Here is a link to a data studio sample report you can clearly see what I am trying to do.
https://datastudio.google.com/reporting/dd33be45-ab13-4881-8a3b-cabafa8c0dbb
Thanks
One solution which i can recommend to over come your first challenge, i.e. High cost. You can customize cost by using GCP-Memorystore, depending on frequency of data that is getting updated.
Moreover, Bigquery also cashes data for a query if you are not using Wild cards on tables and Time partitioned tables. So try to customize your solution over analysis cost if it is feasible over your solution. Bigquery Partition and Clusting may also help you in reducing BQ analysis cost.
Related
Here's the situation:
I have a large table consisting of transactional data (credit card transaction records), and I am creating a dashboard application to visualize the data.
For example, I want to display the count of transactions for each customer, or the breakdown of transaction types for a customer, etc.
Obviously, I can't query the table on the fly because it takes too long, and also I don't want to create lots of different aggregated tables manually (like customers_table, terminals_table, etc.) because then I have to implement the update method too.
I did some research and I think what I need is a cube, which has all the necessary columns as its dimensions (like customer_id, transaction_type, city, time, etc.), with pre-computed aggregations, so then I can query the cube and get whatever aggregated data that I need instantly (NOT ON THE FLY).
So I have two questions:
First of all, am I right about how cubes work, or did I misunderstand something?
Secondly, can I create this cube directly using PL/SQL, or should I use ODI or other tools?
i will explain my use case to understand which DB extract utility to use.
I need to extract data from SQL Server tables with varying frequency each day. Each extract query is a complex SQL statement, involving 5-10 tables in joins etc with multiple causes. Have around 20-30 such statements overall.
All these extract queries might be required to run multiple times a day with varying frequencies each day. It depends on how many times we receive data from source system or other cases.
We are planning to use Kafka to publish a message to let Nifi workflow know whenever a RDBMS table is updated and flow needs to be triggered (i can't just trigger Nifi flow based on "incremental" column value, there might only be all row update scenarios and we might not create new rows in tables).
How should i go about designing my Nifi. There are ExecuteSQL/GenerateTableFetch/ExecuteSQLRecord/QueryDatabaseTable all sorts of components available. Which one is going to fit my requirement best?
Thanks!
I am suggesting that you use ExecuteSQL. You can set query from attribute or compose it using attribute. Easiest way is to create json and then parse that json and create attributes. Check this example, here is sql created from file you can adjust it to create it from kafka link
I am developing a report application in Power BI desktop version. I successfully created a dataset using a query and applying the filters on result data. But Now i have to get data from database in real time with user filters i.e. dataset would be created on the basis of some inputs given by users. We need this as database size is quite huge and we can not load the data then apply filters and create reports.
Same can easily be done in Dot Net application but we have to achieve this on Power Bi.
Please suggest if this can be done.
I would use the Query Parameters feature for this. You add them in the Edit Queries window, from Home / Manage Parameters, then you can use them in Calculated columns or replacing a "hard coded" filter.
There's a detailed write up in a recent blog post:
https://powerbi.microsoft.com/de-de/blog/deep-dive-into-query-parameters-and-power-bi-templates/
I'm an ETL developer that's currently being tasked with developing a type 2 SCD from existing historical data in a relational database. I'm perfectly capable of creating a type 2 SCD that's responsible for tracking future changes to the data, but I'm completely useless when it comes to the task at hand.
The relational model is in our ODS . Based on that relational model, I'm supposed to build flat records in our DW dimension. There are multiple attributes which need to be monitored for changes, each in specific related tables in the relational model. Historical changes must be kept on a daily basis, and if multiple changes to the same attribute occur on the same day, only the last subsists.
How can I tackle this? I'm lost. Thanks in advance.
P.S. we're talking tables with 20-30 million rows and multiple attributes that may change at any given time and therefore must result in a new record in the SCD.
This will indeed be painful. I'm assuming from your question that the tables containing the attribute values are currently varying independently (or you wouldn't need to ask the question).
If you have a table 'Table1' containing 'Key', 'Attribute1' and 'Effective From','Effective To' columns, then you can 'explode' that table into a virtual table in the form 'Key','Attribute1','Date', projecting out one row for every date where that attribute was current.
(Note that you probably don't want to do this as a ranged join against your date dimension, because this will be a Triangular Join (ie perform really badly), you probably need to explode the rows in an ETL tool/programmatically)
If you perform this process across multiple tables, you will have a set of tables giving you the full day-by-day snapshot of each attribute for every day that you care about. It's then fairly easy to join those tables based on 'FK' and 'Date' to give you the complete daily snapshot across all of the attribute values.
Then, of course, you need to run this though another process to collapse rows with the same Key, contiguous dates and all the same attribute values, ie 'unexplode' the rows, back into 'effective from','effective to' form. Note again, that this is fundamentally a row-by-row operation (or at very least a windowing function), and a set-based approach will perform very badly. Personally I'd just stream it all though some .net/java code to achieve this.
Given data volumes this will take a while, but should be achievable.
I was wondering if there were any nice widgets in wxwidgets, with search ability. I mean searching for data in large tables like, data in wxgrid.
Thanks in advance
It would be slow and inefficient in several ways to store all of a large dataset in wxGrid, and then to search through wxGrid.
It would be better to keep the dataset in a database and use the database engine to search through it. wxGrid need only store the data that is visible in the GUI.
Here is some high level pseudo code of how this works.
Load data into store. The store should probably be a database, but it might be a vector or any STL container. Even a text file could be made to work!
Set current viewable row to a sensible value. I am assuming your data is arranged in rows
Load rows including and around current into wxGrid.
User enters search term
Send search request to data store, which returns row containing target
Set current row to row containing target
Load rows including and around current into wxGrid.