I am working on Oracle Data Integrator 11g
I have to create an ODI package, where I need to process an incoming file. The file name is not a constant string, as it has a timestamp entry appended to it, something like this: FILTER_DATA_011413.TXT
Due to the MMDDYY, I can't hardcode the filename in my package. The way, we're handling it right now is, a shell script lists the files in the directory, and loads the filename into a table (using control file). This table is then queried to get the filename and the same is passed to the variable which stores the filename for processing.
I am looking for any other way, where I can avoiud having this temporary table to store the file name.
Can someone suggest me any alternative?
Related
I need to transfer around 20 CSV files inside a folder named ActivityPointer in an azure blob storage container to Azure SQL database in a single data factory pipeline, but ActivityPointer contains 20 CSV files and another folder named snapshots inside it. So when I try to create a pipeline and give * to select all the CSV files inside ActivityPointer it includes the snapshots folder too, which should not be included. Is there any possibilities to complete this task. Also I can't create another folder to transform the snapshots folder into it. What can I do now? Anyone can please help me out.
Assuming you want to copy all CSV files within ACtivityPointer folder,
You can use wildcard expression as below :
you can provide path till Active folder and than *.csv
Copy data is also considering the inner folder while using wildcards (even if we use .csv in wildcard file path). So, we have to validate whether it is a file or folder. Please look at the following demonstration.
First use Get Metadata on the required folder with field list as Child items. The debug output will be:
Now use this to iterate through child items using For each activity.
#activity('Get Metadata1').output.childItems
Inside for each, use if condition activity to check whether the current item is a file or not. Use the following condition.
#equals(item().type,'File')
When this is true, you can use copy data to complete copying the file to target table (Ignore the false case). I have create file_name parameter in my source dataset passing its value as #item().name().
This will help you to achieve your requirement. The following is the debug output. I have 4 files and 1 folder. The folder will be ignored, and the rest will be copied into the target table.
Create a stored procedure that will read the .csv file from oracle server path using read file operation, query the data in some X table and write the output in .csv file.
here after read .csv file, compare .csv file data with table data and need to update few columns in .csv file.
Oracle works best with data in the database. UPDATE is one of the most frequently used commands.
But, modifying a file which resides in some directory seems to be somewhat out of scope. There are other programming languages you should use, I believe. However, if a hammer is the only tool you have, every problem looks like a nail.
I can think of two options.
One is to load file into the database. Use SQL*Loader to do that if file resides on your PC, or - if you have access to the database server and DBA granted you read/write privileges on a directory (an Oracle object which points to a filesystem directory) - use it as an external table. Once you load data, modify it and export it back (i.e. create a new CSV file) using spool.
Another option is to use UTL_FILE package. It also requires access to the database server's directory. Using the A(ppend) option, you can add rows to the original file, but I don't think that you can edit it so this option - at the end - finishes like the previous one - with creating a new file (but this time using UTL_FILE).
Conclusion? Don't use a database management system to modify files. Use another tool.
I'd like to access a file which is stored in the binary table of a MSI installer from a custom action (VBScript, immediate execution).
My understanding is that the files from the binary table are extracted to some safe location and cleaned up after the installation. So the basic question would probably be: Can I determine from a custom action the path of this safe location, so that I can access the extracted files?
I found an alternative approach here. It reads the database from inside the CA and writes a temporary file itself, but does no cleanup. I have the impression that there must be a better solution.
I have a lot of script files (all files have SQL or Trans SQL) say around 2000 files that we used to add with the passage of the time. Now the problem is that in most of the scripts the owner of the database object is not defined and due to my current requirement I must have to set the owner of the database object. How can I update all the script files easily.
One approach is to open each file look for create word and replace/append the owner name before the create key word. I want to do it for all the script files in the directory.
Any ideas?
I want something like find replace.
Many thanks in advance.
I am working on a situation where I want to store my data in pig script into a file. This is pretty straight forward to do that, but I want file name to be derived from the data itself. So, I have a field in data as timestamp. I want to use say MAX(timestamp) as filename to store all the data for that day.
I know the usage of
STORE data INTO '$outputDir' USING org.apache.pig.piggybank.storage.MultiStorage('$outputDir', '2', 'none', ',');
But this variable "outputDir should be passed as the parameter. I want to set this value with a derived value of the field.
Any pointers will be really helpful.
Thanks & Regards,
Atul Aggarwal
In MultiStorage you specify a root directory because typically a HDFS installation is shared by many users, so you do not want data written anywhere. Hence you cannot change the root directory but you can specify which field is used to generate directory names within that directory (in your case 2). The Javadoc is helpful but I am guessing you have seen that already?