"How to write sql task and vary something based on a variable in foreach loop container"
I have created a table(Dummy) with 10 columns. These columns will exactly replicate the input files. In a folder i have five excel files with different names. Each file contain say 100 line items. While loading all these files in my table, i need to capture each of the filenames.
So, i created a column in my table(Dummy) as "File_name". I created foreachloop in SSIS, and declared the variable by referencing the folder and path name. How do I write an update query in SQL task to pick this variable and set as file_name for all the rows loaded?
What's your WHERE Clause in Update Statement(Is there a column in Dummy table that Identifies each set of data from different files),unless you have column or set of columns that identifies each set of data Update does not make sense here.
Other alternative is to have your INSERT(Data flow task) and Update SQL task(Where file_name IS NULL or empty) in same For each loop container and parameterize your Excel file connection as well;
Here is the link to Pass variable to Excel Connection : SSIS with variable excel connection manager
Related
I'm trying read a file from a PL/SQL procedure but I am geting ORA-00942 Table or view does not exist error.
Caused by: Error : 942, Position : 21, Sql =
SELECT UBIC_ID FROM LIST_UBICS
, Error Msg = ORA-00942: table or view does not exist
I have a file with an id per line. This file is called list_ubics.csv. I have a File model and a datastore pointing to the file called LIST_UBIC with the UBIC_ID Field.
I created a Task in a new Procedure with this SQL:
SELECT UBIC_ID From LIST_UBICS
LIST_UBICS is my datastore I don't have any table with these name.
I want read these file and make some processing for each line but I don't see any way in the docs to read a text file that works for me.
How can I read this file?
Thanks in advance for any help.
An ODI Procedure written in PL/SQL (Oracle Technology) will be pushed down on the database. The database executing doesn't know about the File Datastore and can not execute SQL statement against it.
If the goal is to load the file with ODI it can be done using an interface (11g) or a mapping (12c) with LKM File to SQL. That will copy the content of the file into a table in the database and any SQL statement can then be executed against it.
Alternatively, it is possible to create a directory in the database, land the file there and create an external table on top of it. Queries can be used on external table but not DML operations. More information here : https://oracle-base.com/articles/9i/external-tables-9i
I just found out last week that there is one more solution and probably the best!
The solution is written here-part-1 and here-part-2 and if you follow the steps exactly, it will work (I implemented it).
Anyway, I will summarize the main idea and steps.
We can use a variable into a package. There is a code (see code at the end) that reads a column from the given file. Making a for statement in the package, will help us read every row, by changing the value of "CRFILE_FIRST_ROW" variable from the code below, with a sequential number starting with 1 (
So, everything is easy as above. Besides the "CRFILE_FIRST_ROW", there are more variable that can be changed, like: CRFILE_FORMAT=D (format:decimal), CRFILE_SEP_FIELD=0x0009 (hexadecimal fileformat) and so on.
Also, as you can see in the original posts (above links), you can generate your view code; you don't need to copy paste from below.
View code:
select TES.C1 C1
from location_of_file/objects_to_import.txt TES
/*$$SNPS_START_KEYSNP$CRDWG_TABLESNP$CRTABLE_NAME=TESTSNP$CRLOAD_FILE=location_of_file/objects_to_import.txtSNP$CRFILE_FORMAT=DSNP$CRFILE_SEP_FIELD=0x0009SNP$CRFILE_SEP_LINE=0x000ASNP$CRFILE_FIRST_ROW=#UTILS.IMPORT_OBJ_READ_INCRSNP$CRFILE_ENC_FIELD=SNP$CRFILE_DEC_SEP=SNP$CRSNP$CRDWG_COLSNP$CRCOL_NAME=C1SNP$CRTYPE_NAME=STRINGSNP$CRORDER=1SNP$CRLENGTH=50SNP$CRPRECISION=50SNP$CRACTION_ON_ERROR=NULLSNP$CR$$SNPS_END_KEY*/
Dears,
I am trying to delete multiple columns values in an oracle table, I am running the below script but it seems that it is not working (although it did in SQL server).
DELETE a.mobile_num ,
a.price_list,
a.cust_segment,
a.classification,
a.region,
a.district,
a.localty,
a.dsl_install_dt,
a.dsl_oper_status,
a.fl_install_dt,
a.fl_status,
a.oper_status_cd
From mkt_wl_history_2 a;
Thanks,
You cannot delete column values in Oracle, you can delete entire rows. If I understand correctly you are trying to set these values to NULL.
In this case you can use the following statement.
UPDATE mkt_wl_history_2
SET price_list=NULL,
cust_segment=NULL,
classification=NULL,
region=NULL,
district=NULL,
localty=NULL,
dsl_install_dt=NULL,
dsl_oper_status=NULL,
fl_install_dt=NULL,
fl_status=NULL,
oper_status_cd=NULL
Also, I assume you have to do this for all rows in your table. If not then please apply an appropriate where condition.
Use queries similar to this to set all values of a column to the default value:
UPDATE mtk_wl_history_2 SET mobile_num = DEFAULT
I want to use an external table to load a csv file as it's very convenient, but the problem is how do i make sure i don't load the same file twice in a row? i can't validate the data loaded because it can be the same information as before; i need to find a way to make sure the user doesnt load the same file as 2h ago for example.
I thought about uploading the file with a different name each time and issuing an alter table command to change the name of the file in the definition of the external table, but it sounds kinda risky.
I also thought about marking each row in the file with a sequence to help differentiate files, but i doubt the client would accept it as they would need to manually do this (the file is exported from somewhere).
Is there any better way to make sure i don't load the same file in the external table except changing the file's name and executing an alter on the table?
Thank you
when you bring the data from external table to your database you can use MERGE command instead of insert. it let you don't worry about duplicate data
see the blog about The Oracle Merge Command
What's more, we can wrap up the whole transformation process into this
one Oracle MERGE command, referencing the external table and the table
function in the one command as the source for the MERGED Oracle data.
alter session enable parallel dml;
merge /*+ parallel(contract_dim,10) append */
into contract_dim d
using TABLE(trx.go(
CURSOR(select /*+ parallel(contracts_file,10) full (contracts_file) */ *
from contracts_file ))) f
on d.contract_id = f.contract_id
when matched then
update set desc = f.desc,
init_val_loc_curr = f.init_val_loc_curr,
init_val_adj_amt = f.init_val_adj_amt
when not matched then
insert values ( f.contract_id,
f.desc,
f.init_val_loc_curr,
f.init_val_adj_amt);
So there we have it - our complex ETL function all contained within a
single Oracle MERGE statement. No separate SQL*Loader phase, no
staging tables, and all piped through and loaded in parallel
I can only think of a solution somewhat like this:
Have a timestamp encoded in the datafile name (like: YYYYMMDDHHMISS-file.csv), where YYYYMMDDHHMISS is the timestamp.
Create a table with the fields timestamp (as above).
Create a shell scripts that:
extracts the timestamp from the datafilename.
calls an sqlscript with the timestamp as the parameter, and return 0 if that timestamp does not exist, <>0 if the timestamp already exist, and in that case exit the script with the error: File: YYYYMMDDHHMISS-file.csv already loaded.
copy the YYYYMMDDDHHMISS-file.csv to input-file.csv.
run the sql loader script that loads the input-file.csv file
when succes: run a second sql script with parameter timestamp that inserts the record in the database to indicate that the file is loaded and move the original file to a backup folder.
when failure: report the failure of the load script.
I have a requirement to insert bulk data into an Oracle database from a CSV file. Now table columns specs match those of the CSV file's header with the exception of three additional fields in database:
A Primary Key field (for which a simple SEQUENCE.NEXTVAL is called)
A field for the name of the CSV file
A field for the last modified date+time of the file
The following stack question address an extra column issue, but the solution is pretty easy because it used Oracle sysdate which is internally available. I need to pass a parameter from either batch script/shell script.
Insert actual date time in a row with SQL*loader
Can PARFILE help here somehow?
My other alternative would be to do the whole task in two steps by writing a small java code:
Use SQL Loader for bulk upload leaving out data for the filename and
modified time
And then run a separate update statement to populate the newly
created rows
But I'm looking for something which will get the job done in one shot. Any advice??
I'm affraid it's not possible with sqlldr alone.
There is no tools for this in sqlldr.
You'd need some sort of script or a program to dynamically create a .ctl file for each load.
Here is a bash script to help you get started:
#!/bin/bash -xv
readonly MY_FILENAME=$1
readonly DB_BUF_TABLE=$2
readonly SQLLDR_CTL="LOAD DATA
CHARACTERSET UTF8
APPEND INTO TABLE $DB_BUF_TABLE
FIELDS TERMINATED BY ';'(
filename \"$MY_FILE_NAME\",
col_foo,
col_bar
)"
echo "$SQLLDR_CTL" > "loader.ctl"
sqlldr control=loader.ctl parfile=loader.par data="$MY_FILENAME"
sqlldrReturnValue=$?
You'd needsome locking with this.. or path separation for concurrent loads to be sure sqlldr starts with proper ctl file
If i have a CSV file that is in the following format
"fd!","sdf","dsfds","dsfd"
"fd!","asdf","dsfds","dsfd"
"fd","sdf","rdsfds","dsfd"
"fdd!","sdf","dsfds","fdsfd"
"fd!","sdf","dsfds","dsfd"
"fd","sdf","tdsfds","dsfd"
"fd!","sdf","dsfds","dsfd"
Is it possible to exclude any row where the first column has an exclamation mark at the end of the string.
i.e. it should only load the following rows
"fd","sdf","rdsfds","dsfd"
"fd","sdf","tdsfds","dsfd"
Thanks
According to the Loading Records Based on a Condition section of the SQL*Loader Control File Reference (11g):
"You can choose to load or discard a logical record by using the WHEN clause to test a condition in the record."
So you'd need something like this:
LOAD DATA ... INSERT INTO TABLE mytable WHEN mycol1 NOT LIKE '%!'
(mycol1.. ,mycol2 ..)
But the LIKE operator is not available! You only have = and !=
Maybe you could try an External Table instead.
I'd stick a CONSTRAINT on the table, and just let them be rejected. Maybe delete them after load. Or a unix "grep -v" to clear them out the file.