SSIS For Loop - Wait for file to carry on - for-loop

I am creating an SSIS package that imports 10 different CSV files, however the availability of each file varies - so i want to get the file check to loop until all files are there and then start to import process.
I check this through a T-SQL script
IF OBJECT_ID('tblFileCheck') IS NOT NULL
DROP TABLE tbFileCheck;
CREATE TABLE tblFileCheck (
id int IDENTITY(1,1)
,subdirectory nvarchar(512)
,depth int
,isfile bit);
INSERT tblFileCheck
EXEC xp_dirtree '\\reports\Reports\CSV', 10, 1
BEGIN
IF EXISTS (
SELECT COUNT(id)
FROM tblFileCheck
HAVING count(id) > 9
)
BEGIN
PRINT 'Success - Import Latest File'
END
ELSE
BEGIN
RAISERROR ('Looping back to start - insufficient files to run', 16, 1 );
END
END
However I cant get the loop to work, I have created a variable varWaitForData (int32, val = 0), created a SQL task editor with result set = single row and set the result set option to the parameter.
Set the InitExpression #varWaitForData = 0 AND EvalExpression #varWaitForData == 0.
But i keep getting the error [Execute SQL Task] Error: An error occurred while assigning a value to variable "varWaitForData": "Exception from HRESULT: 0xC0015005".

I would use a different approach:
set up a variable called fileCount as int
script task: Get file count
System.IO.DirectoryInfo dir = new System.IO.DirectoryInfo("\\reports\Reports\CSV");
Dts.Variables["User::fileCount"].Value = dir.GetFiles().Length;
Now set up a foreach file enumerator
and put a constraint on the path that fileCount > 9

Related

Check if a file exists faster than OPEN DATASET

I'm searching for a way to check if a file exists before using the OPEN DATASET command to open it. The OPEN DATASET command takes up to 30 seconds to trigger an exception, which is too slow for my liking.
This is the code:
TRY.
OPEN DATASET lv_file FOR OUTPUT IN TEXT MODE
ENCODING DEFAULT
WITH SMART LINEFEED.
CONCATENATE ` ` lv_resultdata INTO lv_resultdata.
TRANSFER lv_resultdata TO lv_file.
CLOSE DATASET lv_file.
CATCH cx_sy_file_access_error.
MESSAGE 'Placeholder-message. File cannot be reached'.
EXIT.
ENDTRY.
Try this:
DATA: filepath TYPE epsf-epsdirnam VALUE '/tmp'.
CALL FUNCTION 'EPS_GET_DIRECTORY_LISTING'
EXPORTING
dir_name = filepath
file_mask = 'somefile.txt'
EXCEPTIONS
invalid_eps_subdir = 1
sapgparam_failed = 2
build_directory_failed = 3
no_authorization = 4
read_directory_failed = 5
too_many_read_errors = 6
empty_directory_list = 7
OTHERS = 8.
CHECK sy-subrc = 0.
" writing dataset
It can also be used for remote servers.

DB2 iSeries doesn't lock on select for update

I'm migrating a legacy application using DB2 iSeries on AS400 that has a specific behavior that I have to reproduce using .NET and DB2.Data.DB2.iSeries client for .NET.
What I'm describing works for me with DB2 non AS400 but in AS400 DB2 it worlks for the legacy application i'm replacing - but not with my application.
The behavior in the original application:
Begin Transaction
ExecuteReader () => Select col1 from table1 where col1 = 1 for update.
The row is now locked. anyone else who tries to run Select for update should fail.
Close the Reader opened in line 2.
The row is now unlocked. - anyone else who tried to run select for update should succeed.
Close transaction and live happily ever after.
In my .NET code I have two problems:
Step 2 - only checks if the row is already locked - but doesn't actually lock it. so another user can and does run select for update - WRONG BEHAVIOUR
Once that works - I need the lock to get unlocked when the reader is closed (step 4)
Here's my code:
var cb = new IBM.Data.DB2.iSeries.iDB2ConnectionStringBuilder();
cb.DataSource = "10.0.0.1";
cb.UserID = "User";
cb.Password = "Password";
using (var con = new IBM.Data.DB2.iSeries.iDB2Connection(cb.ToString()))
{
con.Open();
var t = con.BeginTransaction(System.Data.IsolationLevel.ReadUncommitted);
using (var c = con.CreateCommand())
{
c.Transaction = t;
c.CommandText = "select col1 from table1 where col1=1 FOR UPDATE";
using (var r = c.ExecuteReader())
{
while (r.Read()) {
MessageBox.Show(con.JobName + "The Row Should Be Locked");
}
}
MessageBox.Show(con.JobName + "The Row Should Be unlocked");
}
}
When you run this code twice - you'll see both processes reach the "This row should be locked" which is the problem I'm describing.
The desired result would be that the first process will reach the "This row should be locked" and that the second process will fail with resource busy error.
Then when the first process reaches the second message box - "the row should be unlocked" the second process( after running again ) will reach the "This row should be locked" message.
Any help would be greatly appreciated
The documentation says:
When the UPDATE clause is used, FETCH operations referencing the cursor acquire an exclusive row lock.
This implies a cursor is being used, and the lock occurs when the fetch statement is executed. I don't see a cursor, or a fetch in your code.
Now, whether .NET handles this as a cursor, I don't know, but the DB2 UDB documentation does not have this notation.
Isolation Level allows this behavior. Reading rows that are locked.
ReadUncommitted
A dirty read is possible, meaning that no shared locks are issued and no exclusive locks are honored.
After much investigations we created a work around in the form of a stored procedure that performs the lock for us.
The stored procedure looks like this:
CREATE PROCEDURE lib.Select_For_Update (IN SQL CHARACTER (5000) )
MODIFIES SQL DATA CONCURRENT ACCESS RESOLUTION WAIT FOR OUTCOME
DYNAMIC RESULT SETS 1 OLD SAVEPOINT LEVEL COMMIT ON RETURN
NO DISALLOW DEBUG MODE SET OPTION COMMIT = *CHG BEGIN
DECLARE X CURSOR WITH RETURN TO CLIENT FOR SS ;
PREPARE SS FROM SQL ;
OPEN X ;
END
Then we call it using:
var cb = new IBM.Data.DB2.iSeries.iDB2ConnectionStringBuilder();
cb.DataSource = "10.0.0.1";
cb.UserID = "User";
cb.Password = "Password";
using (var con = new IBM.Data.DB2.iSeries.iDB2Connection(cb.ToString()))
{
con.Open();
var t = con.BeginTransaction(System.Data.IsolationLevel.ReadUncommitted);
using (var c = con.CreateCommand())
{
c.Transaction = t;
c.CommandType = CommandType.StoredProcedure;
c.AddParameter("sql","select col1 from table1 where col1=1 FOR UPDATE");
c.CommandText = "lib.Select_For_Update"
using (var r = c.ExecuteReader())
{
while (r.Read()) {
MessageBox.Show(con.JobName + "The Row Should Be Locked");
}
}
MessageBox.Show(con.JobName + "The Row Should Be unlocked");
}
}
We don't like it - but it works.

Airflow retain the same database connection?

I'm using Airflow for some ETL things and in some stages, I would like to use temporary tables (mostly to keep the code and data objects self-contained and to avoid to use a lot of metadata tables).
Using the Postgres connection in Airflow and the "PostgresOperator" the behaviour that I found was: For each execution of a PostgresOperator we have a new connection (or session, you name it) in the database. In other words: We lose all temporary objects of the previous component of the DAG.
To emulate a simple example, I use this code (do not run, just see the objects):
import os
from airflow import DAG
from airflow.operators.postgres_operator import PostgresOperator
default_args = {
'owner': 'airflow'
,'depends_on_past': False
,'start_date': datetime(2018, 6, 13)
,'retries': 3
,'retry_delay': timedelta(minutes=5)
}
dag = DAG(
'refresh_views'
, default_args=default_args)
# Create database workflow
drop_exist_temporary_view = "DROP TABLE IF EXISTS temporary_table_to_be_used;"
create_temporary_view = """
CREATE TEMPORARY TABLE temporary_table_to_be_used AS
SELECT relname AS views
,CASE WHEN relispopulated = 'true' THEN 1 ELSE 0 END AS relispopulated
,CAST(reltuples AS INT) AS reltuples
FROM pg_class
WHERE relname = 'some_view'
ORDER BY reltuples ASC;"""
use_temporary_view = """
DO $$
DECLARE
is_correct integer := (SELECT relispopulated FROM temporary_table_to_be_used WHERE views LIKE '%<<some_name>>%');
BEGIN
start_time := clock_timestamp();
IF is_materialized = 0 THEN
EXECUTE 'REFRESH MATERIALIZED VIEW ' || view_to_refresh || ' WITH DATA;';
ELSE
EXECUTE 'REFRESH MATERIALIZED VIEW CONCURRENTLY ' || view_to_refresh || ' WITH DATA;';
END IF;
END;
$$ LANGUAGE plpgsql;
"""
# Objects to be executed
drop_exist_temporary_view = PostgresOperator(
task_id='drop_exist_temporary_view',
sql=drop_exist_temporary_view,
postgres_conn_id='dwh_staging',
dag=dag)
create_temporary_view = PostgresOperator(
task_id='create_temporary_view',
sql=create_temporary_view,
postgres_conn_id='dwh_staging',
dag=dag)
use_temporary_view = PostgresOperator(
task_id='use_temporary_view',
sql=use_temporary_view,
postgres_conn_id='dwh_staging',
dag=dag)
# Data workflow
drop_exist_temporary_view >> create_temporary_view >> use_temporary_view
At the end of execution, I receive the following message:
[2018-06-14 15:26:44,807] {base_task_runner.py:95} INFO - Subtask: psycopg2.ProgrammingError: relation "temporary_table_to_be_used" does not exist
Someone knows if Airflow has some way to retain the same connection to the database? I think it can save a lot of work in creating/maintaining several objects in the database.
You can retain the connection to the database by building a custom Operator which leverages the PostgresHook to retain a connection to the db while you perform some set of sql operations.
You may find some examples in contrib on incubator-airflow or in Airflow-Plugins.
Another option is to persist this temporary data to XCOMs. This will give you the ability to keep the metadata used with the task in which it was created. This may help troubleshooting down the road.

How to get Oracle exception in SQLcl script when using util.execute?

I try to write a batch file using Oracles SQLcl. In this file, i want to insert a new table row with util.execute. This just returns true / false, which is a boolean return of success/failure.
My question is, how i get the error message of the exception which is thrown, so that i can find out, what the problem is with my insert-statement.
What i do:
First of all, i connect to my database server and start my script:
me#pc:/myproject$ /sqlcl/bin/sql schemaname/pw#server.com:1521/sid
SQLcl: Release 17.3.0 Production [...]
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit [...]
SQL>
SQL> #mybatchscript.js path/image.jpg
My mybatchscript.js looks like this:
script
var tabName = "MY_TABLE_NAME";
var HashMap = Java.type("java.util.HashMap");
var bindmap = new HashMap();
var filePath="&1";
print("\nreading file: "+ filePath);
var blob=conn.createBlob();
var bstream=blob.setBinaryStream(1);
java.nio.file.Files.copy(java.nio.file.FileSystems.getDefault().getPath(filePath),bstream);
bstream.flush();
bindmap.put("content",blob); // has content
bindmap.put("size",blob.length()); // is 341989
// the follow command fails
var doInsert = util.execute("insert into "
+ tabName
+ " (id, main_id, file_name, file_type,"
+ " file_size, file_content, table_name)"
+ " values("
+ " SEQ_MY_TABLE_NAME.nextval, 1,"
+ " 'testname', 'image/jpeg', :size, :content,"
+ " 'my_table_name')"
,bindmap);
sqlcl.setStmt(
"show errors \n"
);
sqlcl.run();
if(!doInsert) {
print("insert failed");
print(doInsert);
exit;
}
/
The console output is like:
reading file: path/image.jpg
insert failed
false
The script is working until the util.execute insert-statement. It returns false, so the insert-statement failed. But it doesn't tell me, why. I have no idea, how i get access to the error message or the exception which is thrown inside the util.execute?
I also tried to turn on SERVEROUTPUT or ERRORLOGGING, but it has the same output as above and the error log table is empty:
SQL> set errorlogging on
SQL> show errorlogging
errorlogging is ON TABLE SPERRORLOG
SQL> set serveroutput on
SQL> show serveroutput
serveroutput ON SIZE UNLIMITED FORMAT WORD_WRAPPED
My knowledge source were these slides where my script is also based on, i didn't find information about the error / exception handling for the util functions in general?
There's basically 2 ways.
1- When using util.execute ( or any util.XYZ functions ) the last error message is retrieved with the following. I also just updated the scripting README with this : https://github.com/oracle/oracle-db-tools/blob/master/sqlcl/README.md
var msg = util.getLastException()
2- When using sqlcl.run()
There's an example I wrote here:
https://github.com/oracle/oracle-db-tools/blob/master/sqlcl/examples/audio.js
The example is a tad silly in that it makes noises on success/failure but you'll see the code that gets the error. Check the ctx.getProperty("sqldev.last.err.message" That will get the last sqlerr message.
if ( ctx.getProperty("sqldev.last.err.message") ) {
//
// FAILED !
//
play("chew_roar.wav");
} else {
//
// Success !!
//
play("R2.wav");
}

run time error '13' type mismatch

I have my query on VB6 which was:
Set Db = DBEngine.OpenDatabase(App.Path & "\sample4nC4.mdb")
Set rs = Db.OpenRecordset("select *from tbl_student;")
Do Until rs.EOF
With ListView1
.ListItems.Add , , rs.Fields("stud_ID")
.ListItems(ListView.ListItems.Count).SubItems(1) = rs.Fields("stud_fname")
.ListItems(ListView1.ListItems.Count).SubItems(2) = rs.Fields("stud_lname")
.ListItems(ListView1.ListItems.Count).SubItems(3) = rs.Fields("stud_address")
.ListItems(ListView1.ListItems.Count).SubItems(4) = rs.Fields("stud_age")
End With
rs.MoveNext
Loop
When I execute this query, there was an error on line 2 says:
Run Time Error '13' Type Mismatch
I really don't get it because when I check the table name, it was correct and yet I cant access the table. Can anybody answer my problem?
Do you have references to ADO and DAO in your project?
If so, look at this Microsoft support article: https://support.microsoft.com/en-us/kb/181542
Do these
Replace your query from 'select *from tbl_student;'
to
'select stud_fname, stud_lname, stud_address, stud_age from tbl_student'
This includes (1. space between '' and 'from' 2. remove ';' 3. specify the field names explicitly)*
Put a breakpoint in the first line your program and step into, if it still fails check which line it's failing.

Resources