New to Qlik Sense.
I would like to perform incremental insert, update and delete. Through research i managed to write this script
//This fetches deleted records
SELECT `sale_detail_auto_id`
FROM `iprocure_ods.deleted_records` as dr
INNER JOIN `iprocure_ods.saledetail` sd ON sd.sale_detail_auto_id = dr.identifier AND dr.type = 2
WHERE dr.delete_date > TIMESTAMP('$(vSaleTransactionsRunTime)');
//This fetches new and updated records
[sale_transactions]:
SELECT *
FROM `iprocure_edw.sale_transactions`
WHERE `server_update_date` > TIMESTAMP('$(vSaleTransactionsRunTime)');
Concatenate([sale_transactions])
LOAD *
FROM [lib://qlikPath/saletransactions.qvd] (qvd) Where Not Exists(`sale_detail_auto_id`);
//This part updates runtime dates
MaxUpdateDate:
LOAD Timestamp(MAX(`server_update_date`), '$(TimestampFormat)') As maxServerUpdateDate
FROM [lib://qlikPath/saletransactions.qvd] (qvd);
Let vSaleTransactionsRunTime = peek('maxServerUpdateDate', 0, MaxUpdateDate);
DROP Table MaxUpdateDate;
New and update records works fine. The problem is with the deleted records are replaced with empty column except sale_detail_auto_id column.
How can i fetch data from saletransactions.qvd that are not in deleted records?
In first SELECT you select sale_detail_auto_id fields which is also exists under the same field name in new and updated records, so then you see deleted ids together with new ones. You need to rename that column to avoid conflict.
Please use AS, for example:
sale_detail_auto_id` AS `deleted_sale_detail_auto_id`
and then in EXISTS use that field:
Where Not Exists(deleted_sale_detail_auto_id, sale_detail_auto_id);
UPDATED:
Additionally I think it doesn't make sense to store deleted ids in data model so you can name that table:
[TEMP_deleted_ids]
SELECT sale_detail_auto_id` AS `deleted_sale_detail_auto_id`
and then in the end of the script remove it:
DROP Table [TEMP_deleted_ids];
Related
I am trying this query to insert some records from a table to another one,when recods are not already exsiting in the target table, but I am getting the following error, what is the best query to UPSERT in memsql from another table?
Query:
INSERT INTO ema.device_set
(segment_0, segment_1, segment_2, segment_3, segment_4, last_updated)
SELECT tmp.segment_0, tmp.segment_1, tmp.segment_2, tmp.segment_3, tmp.segment_4, tmp.last_updated
FROM ema.tmp_device_set tmp
WHERE NOT EXISTS (
SELECT *
FROM ema.device_set tab
WHERE tmp.segment_0 = tab.segment_0 and tmp.segment_1 = tab.segment_1 and tmp.segment_2 = tab.segment_2 and tmp.segment_3 = tab.segment_3 and tmp.segment_4 = tab.segment_4
);
error:
Partition has no master instance or Leaf Error: The database will be available to query in 2 seconds after recovery from disk is finished.
That error message means your nodes are down or recovering from disk. It has nothing to do with the specific UPSERT you are trying to do.
Check to make sure your query is not in any violations of the MemSQL INSERT...SELECT rules shown at the following link.
https://docs.memsql.com/docs/insert
I have Documents table and Signs table. Document record can be related with many records in Signs table.
Now, I want to get all records of Documents table when document ID appears in Signs table.
Here I get all documents:
var documents = (from c in context.documents select c);
Here I get all my signs and save into List:
var myDocuments = (from s in context.signs where s.UserId== id select s.ID).ToList();
This list contains collection on document ID.
And here, I'm trying to get all documents that exists in myDocuments list:
documents.Where(item => myDocuments.Contains(item.ID));
But, when I do .ToList() allways return all records (in database only exists one compatible record)
What is wrong in LinQ statement?
The problem is that this statement doesn't modify the contents of documents, it merely returns the results (which you're not doing anything with):
documents.Where(item => myDocuments.Contains(item.ID));
documents is still the full list.
Change this line to something like:
var matchingIDDocs = documents.Where(item => myDocuments.Contains(item.ID));
And then use matchingIDDocs in place of "documents" later in your code.
Let me start with structure first:
[main_node]->field_reference_to_sub_node->[sub_node]->field_ref_to_sub_sub_node->[sub_sub_node]
[sub_sub_node]->field_type = ['wrong_type', 'right_type']
How to efficiently query all [sub_sub_node] ids with right_type, referenced by main_node (which is current opened node)?
Doing node_load on foreach seems a bit of overkill for this. Anybody has some better solutions? Greatly appreciated!
If you want to directly query the table of the fields:
$query = db_select('node', 'n')->fields('n_sub_subnode', array('nid'));
$query->innerJoin('table_for_field_reference_to_sub_node', 'subnode', "n.nid = subnode.entity_id AND subnode.entity_type='node'");
$query->innerJoin('node', 'n_subnode', 'subnode.subnode_target_id = n_subnode.nid');
$query->innerJoin('table_for_field_ref_to_sub_sub_node', 'sub_subnode', "n_subnode.nid = sub_subnode.entity_id AND sub_subnode.entity_type='node'");
$query->innerJoin('node', 'n_sub_subnode', 'sub_subnode.sub_subnode_target_id = n_sub_subnode.nid');
$query->innerJoin('table_for_field_type', 'field_type', "n_sub_subnode.nid = field_type.entity_id AND field_type.entity_type='node'");
$query->condition('n.nid', 'your_main_node_nid');
$query->condition('field_type.field_type_value', 'right_type');
Here is the explanation of each line:
$query = db_select('node', 'n')->fields('n_sub_subnode', array('nid'));
We start by querying the base node table, with the alias 'n'. This is the table used for the 'main_node'. The node ids which will be returned will be however from another alias (n_sub_subnode), you will see it below.
$query->innerJoin('table_for_field_reference_to_sub_node', 'subnode', "n.nid = subnode.entity_id AND subnode.entity_type='node'");
The first join is with the table of the field_reference_to_sub_node field, so you have to replace this with the actual name of the table. This is how we will get the references to the subnodes.
$query->innerJoin('node', 'n_subnode', 'subnode.subnode_target_id = n_subnode.nid');
A join back to the node table for the subnodes. You have to replace the 'subnode_target_id' with the actual field for the target id from the field_reference_to_sub_node table. The main purpose of this join is to make sure there are valid nodes in the subnode field.
$query->innerJoin('table_for_field_ref_to_sub_sub_node', 'sub_subnode', "n_subnode.nid = sub_subnode.entity_id AND sub_subnode.entity_type='node'");
The join to the table that contains references to the sub_sub_node, so you have to replace the 'table_for_field_ref_to_sub_sub_node' with the actual name of the table. This is how we get the references to the sub_sub_nodes.
$query->innerJoin('node', 'n_sub_subnode', 'sub_subnode.sub_subnode_target_id = n_sub_subnode.nid');
The join back to the node table for the sub_sub_nodes, to make sure we have valid references. You have to replace the 'sub_subnode_target_id' with the actual field for the target id from the 'field_ref_to_sub_sub_node' table.
$query->innerJoin('table_for_field_type', 'field_type', "n_sub_subnode.nid = field_type.entity_id AND field_type.entity_type='node'");
And we can now finally join the table with the field_type information. You have to replace the 'table_for_field_type' with the actual name of the table.
$query->condition('n.nid', 'your_main_node_nid');
You can put now a condition for the main node id if you want.
$query->condition('field_type.field_type_value', 'right_type');
And the condition for the field type. You have to replace the 'field_type_value' with the actual name of the table field for the value.
Of course, if you are really sure that you always have valid references, you can skip the joins to the node table and directly join the field tables using the target id and the entity_id fields (basically the target_id from on field table has to be the entity_id for the next one).
I really hope I do not have typos, so please check the queries carefully.
I have a dataset (called dataSet below) with a single table and some records in it. One of the columns is called Message and contains an error message. If any records have a value in this field I want to copy it into an error dataset (errorDataSet below) and then remove it from the original dataset. I managed to get this far with LINQ:
DataSet errorDataSet = dataSet.Copy();
//find all records that have a Message column value
var query = from row in errorDataSet.Tables[0].AsEnumerable()
where !String.IsNullOrEmpty(row.Field<string>("Message"))
select row;
DataSet tempErrorDataSet = errorDataSet.Clone();
foreach (var row in query)
{
tempErrorDataSet.Tables[0].Clear();
tempErrorDataSet.Tables[0].ImportRow(row);
utility.WriteError(connectorName, row["Message"].ToString(), tempErrorDataSet);
//remove the error row from the good data
dataSet.Tables[0].Rows.Remove(row);
}
The bottom line throws an exception or I get errors regarding modifying a collection etc. I'm sure there is a simple way of doing this in LINQ.
Note: The reason I have tempErrorDataSet is that I convert it to XML and pass it into a stored proc - it only takes a record at a time in that format, hence I clear it each time.
Your query is enumerating (indirectly) through the rows in the table. As you should already know, you cannot modify a collection (remove something in this case) as you enumerate over it. Throw the contents into a list beforehand. That way you're not enumerating through the actual table but a copy of some rows.
var query = (from row in errorDataSet.Tables[0].AsEnumerable()
where !String.IsNullOrEmpty(row.Field<string>("Message"))
select row).ToList();
I need to insert to two tables in a single query. Is this possible to do in LINQ?
At present I am using insertonsubmit() 2 times.
If your tables have a primary key/foreign key relationship to each other, then you also have two objects which you can link to each other:
InternetStoreDataContext db = new InternetStoreDataContext();
Category c = new Category();
c.name = "Accessories";
Product p = new Product();
p.name = "USB Mouse";
c.Products.Add(p);
//and finally
db.Categories.Add(c);
db.SubmitChanges();
That adds your object and all linked objects when submitting the changes.
Note that for that to work, you must have a primary key in both tables. Otherwise LINQ doesn't offer you the linking possibility.
Here are good examples of using LINQ to SQL: http://weblogs.asp.net/scottgu/archive/2007/05/19/using-linq-to-sql-part-1.aspx
The database submit doesn't happen until you call SubmitChanges. There is no tangible cost associated with multiple calls to InsertOnSubmit - so why not just do that?
This will still result in two TSQL INSERT commands - it simply isn't possible to insert into two tables in a single regular INSERT command.