Is it possible to create an ODI interface that creates multiple flat files based off unique records in a SQL table? There is plenty of information out there regarding how to interface multiple flat files into one SQL datastore - but is it possible the other way around?
Not sure we can tell the interface to create dynamic models based off when certain SQL data primary key data value change for example on a given table? So 600 records with 600 different data points at the PK column corresponding to 600 different rows creates 600 flat files?
Just typing this makes me think the answer is a FIRM NO.
I think you can do it.
Minimal steps with minimal informations:
Create a Package;
Create a loop in the package (For i = 0; i
Inside the loop you will setup the name of the file that will be generate (variable);
Still inside the loop, setup the filter that is used into the interface (variable);
Still inside the loop, run the interface.
Additional setup.
For the interface you will need two models:
for the source
for the target
In the model of the target (Technology File) you will need to setup the name of the file. Instead a static name, you will put a project/global variable. That variable you will setup during the loop. So, each time you will loop, you will have a different name.
I tried to be clear and short in the same time. I can expand my explication and give you an example.
If I didn't understood well, please tell me.
Related
I'm looking to move to sqitch, but my team likes their migrations as multiple files. That is, for example, if we create a foreign key and create an index in the same jira ticket, we like PROJ-123-create-fk.sql and PROJ-123-create-index.sql.
I'd like to keep a single row in the sqitch.plan file, so that each jira ticket corresponds to a single row.
Basically, short of adding lines to sqitch.plan, can I do this? Is there a way to "include" other sql files from a master one? Something like
PROJ-123-main.sql
\include PROJ-123-create-fk.sql
\include PROJ-123-create-index.sql
Thanks so much!
The \ir psql directive solved this for me.
in PROJ-123-deploy.sql
\ir PROJ-123-create-fk.sql
\ir PROJ-123-create-index.sql
If the fk and index sql files are in the same directory, they will be executed.
Since you want to store it as a single line in the plan file, it doesn't quite match, but I will explain the method we use below. Maybe there are parts that you want to use in it.
There is a similar situation in my team, but we use sqitch tags. Each version of our application corresponds to a subtask. And each task corresponds to a tag. We create sql files as many as the number of subtasks. Then we combine them in a tag that we created with the name of the main task. In our CI/CD pipeline that we use for the database, we also provide the transition between versions with tags. I wanted to add this method here in case anyone prefers to use a similar structure.
Simple example;
Let's have v2.0 of our application installed and a new table and an index are required for v2.1
We create two subtasks named create table and create index under the main task named v2.1
We create two sql files;
app_v2.1_table_create.sql to create a table, app_v2.1_index_create.sql to create an index.
After that, we create a sqitch tag called v2.1. Notice it has the same name as the main task.
Context:
I am creating a dashboard in Excel based on the data model I am building in Power Pivot. The source data in the data model is based on various other excel tables I am regularly receiving and copy-pasting into my workbook (their incoming structure is out of my control). My goal is to perform all data processing within Power Pivot/DAX rather than manipulating the data in the worksheets before loading into the model.
Problem:
In my model, I have a table (tabCases) which includes status updates on all cases from a management system. This table has a column named case-ID (not unique). I need to create a lookup-table with unique case-id's where I can create new columns with various KPIs for each case.
How can I do this in Power Pivot?
I found two suggestions in this article but none of them work for me (opt. 1 because it requires a manual creation of the unique ID list and opt. 2 because I don't have a database access).
In my mind there should be something really simple I could do, such as i.e.:
Add new table to data model
Set first column to be equal to DISTINCT(tabCases[caseID])
Is there such a way?
A Linkback Table might help you. Please see the link below:
https://www.sqlbi.com/articles/linkback-tables-in-powerpivot-for-excel-2013/
Thanks
Case:
I have csv data extract to be loaded with a great COPY INTO and also have a csv file with table description.
I want to create table from this description file before loading data.
Q:Is it possible to create a table by using description file?
I looked at 'CREATE TABLE mytable FROM LOADER myloader();' example but it does not seem support additional attributes to define columns.
One solution I know is to create a Python function to parse file and generate ddl statement but was curios if it can be done easier way.
Idea for enhancement: COPY INTO would be expanded to support creating tables at the same time by adding description file as another parameter.
Unfortunately, this is not currently possible.
The problem from my experience with implementing this kind of convenience feature is that it would be difficult to define a syntax that:
is backward compatible so that it does not break existing code
convenient for a majority of users
simple enough to implement
I've just figured out a big mistake I had while creating the dynamodb structure.
I've created 11 tables, whereas one of them is the table mostly refereed to and the others are complementary tables.
For example, I have a table where I hold names (together with other info) called "Names" and another table called "NamesMappings" holding all these names added to the "Names" table so that each time a user wants to add a name to the "Names" table he first tries to put the name in "NamesMappings" and only if it succeed (therefore this name doesn't exist) he can add the name into the "Names" table. This procedure helps if the name is not unique and is not the primary key in the "Names" table and with this technique I don't have to search inside the "Names" table if the name exists, but instead I can try to add it to the "NamesMappings" table and only if it succeed I know this is a unique name.
First of all, I would like to ask you if this is a common approach or there is a better one?
Next, I figured out that with this design I soon reached to 11 tables each has 5 provisioned capacity of read and write which leads to overall 55 provisioned read and write under the free-tier. Then I understood why I get all these payments each month, because as the number of tables is getting bigger, and I leave the provisioned capacity as default (both read/write capacity are 5) I get more and more provisioned capacity.
So, what should be my conclusion from this understanding? Should I try to reduce the number of tables even if it takes more effort to preform scanning and querying inside the table? Or should I split the table same as I do but reduce the capacity of these mappings tables used only for indication if an item exists or not in another table?
If I understand your problem correctly you're missing the whole concept of NoSQL Databases.
Your Names table should have a Hash key (which is similar to a Primary key) that has a uniformly generated identifier (an UUID is a great candidate). This would automatically make this Table queryable by this unique identifier. You said, however, that you don't know the ID but you only know the Name instead. This leads me to think you could create a Global Secondary Index (GSI) on the Name attribute inside the Names table so you can also query by Name. Up to this point, your table structure should look like this:
id | name
Both of them are independently queryable, which gives you a lot of flexibility already.
Now, let's say you want to add the NameMapping attribute (which I don't know how it looks like), you can simply add it under the Names table, getting rid of the NamesMappings table, greatly reducing the number of WCUs and RCUs across your account. Your table structure should now look like this:
id | name | mappings
where mappings is, let's say, a JSON object.
Since you can only query on top level attributes in DynamoDB, you can now perform a query against the name attribute which has a GSI configured. If the query returns nothing, then name is unique. But let's say you still need some data inside the mappings object, then you could query by name and, in your code, you could apply a map/filter/reduce operation on the mappings attribute and decide what to do next.
Remember that duplication is just OK in a NoSQL world. This may look scary if you come from a purely SQL background, but data should be stored in such a way in NoSQL databases that you should be able to fetch all the needed information in one go, therefore avoiding "joins" (joins are still possible in a NoSQL database, but since there are no strong relationships between entities, you need to perform these joins manually on the code level). To give you some real context, imagine you have a Orders table where you keep track of the ordered Products and the Store that the Order belongs to: you'd save both the Products and the Store objects (and not their IDs, as it would happen in the SQL way) inside the Order object, so if you want to query for a given OrderId in the future, you wouldn't need to make extra calls (aka "joins") to the Product/Store tables to fetch the information, since everything would already be stored inside the Order object.
I've tried to create a usercomponent for talend which will act like an input. I have very strange column names in the database export (coming from SAP R3 / ECC) I'm trying to read, however I've created an additional component which can create schema xml-s. These xml-s contain the dBcolumn as well as the column I'd like to use as the output from component.
I can generate these xml-s for different languages, so I can use english/german/italian names 10,20 or 50 character long for the columns instead of their 5-10 character abbreviation.
My problem is that I'm not able to see the Db Column in my user components schema. I didn't find anything related to this. I have my components located under "Business/SAP".
I needed to set the xml like this:
After that I'm able to see the Db Columns: