I am using IBM Mainframe TSO to view files from a dataset. I recently have been told to start FTPing the latest generation dataset everyday to a folder on my desktop. The problem is that the FTP script I have only lets me FTP a file with the exact name I put. Everyday the dataset changes.
How can I write a script that will FTP the latest generation? Please see example below of how the dataset changes:
Dataset
8/30/18 - KIBI.AL242422.REPORT.G6441V00
8/31/18- KIBI.AL242422.REPORT.G6442V00
9/1/18 - KIBI.AL242422.REPORT.G6443V00
9/4/18 - KIBI.AL242422.REPORT.G6444V00
9/5/18 - KIBI.AL242422.REPORT.G6445V00
command.bat
ftp -i -s:Command.txt
quit
command.txt
open sc01.sample.com
USER NAME
PASSWORD
get 'KIBI.AL242422.REPORT.G6441V00'
What your referring to are Generation Data Groups. You can refer to the files in relative form where (0) is the most current. (-1) is the previous generation, etc. In your case you want to access the dataset by relative reference. In your FTP client do the following:
cd KIBI
get AL242422.REPORT(0)
The system will determine which of the datasets is the one you want. Its a nice feature.
Related
When googling this question, it seems to have been asked, and partially (and poorly) answered a number of times, mostly for older versions.
Question: How can I download a CSV to a local file, with the below constraints? I'm designing in Spoon.
URL: Will always be the same. https://example.com/data/my.csv . The website prepares the csv and provides it back to the web client as a file download after about 4-5 seconds. In a browser this means it is downloaded as a .csv, and not displayed.
Authentication: The website does not require authentication for access. The data isn't sensitive.
Local file path: The downloaded CSV will overwrite the existing csv. eg: d:\data\my.csv . Ie, I can set this on a timer and have it download the newest csv every hour or so.
Proxy: It is quite likely I will need to traverse a network proxy. eg badproxy.mynetwork.internal:8080 and that proxy requires a username and password. It's far better if I can set this password in a single location so any future things created can reference it. Not really sure on how to approach this either.
The rest of my process focuses on addressing the content of the csv, and already works fine.
The processes I've found on google show using the Http Client component, though it's not particularly straightforward how this translates into a file being saved locally into a known location.
Thanks for any pointers.
PDI v9.0.0.0-423
The HTTP client step needs to be triggered. Use a Row generator step generating e.g. 1 empty row and link that with a hop to the HTTP client step.
for your solution , try this:
Data Grid -->HTTP Client-->CSV File Input->Text file output(extension with csv)
I have a requirement to generate the target file in Informatica with date/time appended to it. How will the Informatica FTP connection identify such dynamic file name with date appended to its name?
Also I would like to know if it is possible to FTP multiple files at a time via Informatica FTP connection. Please someone help me on this.
Its actually pretty simple, you just have to use the part of the file name that is constant and then place a *
for eg:
Myfile_20190607.txt
Myfile_20190507.txt
If i specify Myfile_2019* , this is good enough to pickup the files soecified above. You may have to play with the * and criteria to fit the files that you need.
Note: if you are sending files to third party, try to use SFTP instead of plain old ftp and most organization blocked ftp to outside ip's.
As far as I know until Informatica 9.x, it is neither possible to generate dynamic filename nor create multiple files using FTP connection. Only option was to create the files on Informatica server and then run a script to FTP them over to the destination server.
Here is how:
edit your workflow, choose variables tab, create a workflow
variable with datatype NSTRING; assume the variable name is
$wf_timestamp;
create an assignment task and assign TO_CHAR(SYSDATE,'YYYYMMDD')
to the variable in the assignment task;
edit session: Choose Mapping tab, choose your target; then
Connections; then edit FTP Value; then in the Remote Filename
attribute, enter your filename with the timestemp, eg,
myfile_$$$wf_timestamp.csv;
put your assignment before you session in your workflow.
that's it.
I have an Excel file at a shared location on windows environment. I have data stage server on Unix box. I want to read the excel file and load data to a Teradata table. I need help with reading the excel. One option for me is to transfer file to the server location and access it from there but can i read the excel from the shared folder in windows environment?
I tried to use ftp first in datastage. But getting the below error.
<FTP_Enterprise_18> Error occurred during initializeFromArgs().
<FTP_Enterprise_18> uri : ftp://server/path/file.xlsx is not valid remote file.
<main_program> Creation of a step finished with status = FAILED.
No - it is not possible to read it from a remote location so you will need to tranfer it first (if the shared location is not a SAMBA mount on the Unix maschine).
You can use the "Unstructured data" stage to read the Excel fle one it is on the Unix server.
I have a quick project I am working on for one of our VPs.
We have a few thousand CAD jobs stored on a network file share. The file structure is such that there is a parent folder for the CAD job. Part of the folder name contains the job number. Inside the folder, there are 1 to many .ini text files that contain the connection information I need.
What I need is a programatic way to search through all the folders and extract the job number from the folder name, and all the connection values from the ini files.
For example for a folder named CM8252390-3, the job number is 8252390-3. Inside this folder are 3 ini files. Inside the ini files are that look like this:
[Connection]
Name=IMP_Acme_3.5
[Origin]
X=-15.044784
Y=19.620095
Z=44.621395
So my program needs to give me the following result
Job Connection
8252390-3 IMP_Acme1_3.5
8252390-3 IMP_Acme2_3.5
8252390-3 IMP_Acme3_3.5
8254260-1 IMP_Acme3_2.4
8254260-1 IMP_Acme3_4.1
...continued for all folders in the network share
Any suggestion on the best way to do this. I am primarily an Oracle PL/SQL developer, but have some basic Windows batch and Unix shell experience. If I can get the data loaded into Oracle tables, I can search using PL/SQL tools, but is there a better way using shell, batch, or other tools?
Thank you.
I think this is a job for Powershell or vbScript. It would be easy to use these tools to write the information you need to one file.
This file should be written to an Oracle directory.
grant read permission to a database user on this directory
use utl_file to read the file or treat the file as an external table and expose it as a view
schedule a regular OS job to refresh or rebuild the list
I need to set up some sort of infrastructure to automatically FTP some files from one remote server to another. The FTP transaction will occur on a scheduled basis. Both these servers are Windows boxes and the location of the files that need to be FTP'ed will depend on the current date (the folder they sit in will be named the current day's date).
I would really hate to have to write something like that from scratch, so are there tools/utilities/the likes out there?
Windows command line ftp will read input from a text file. For example:
ftp -ni < ftpscript.txt
Where these options are used:
-n - Suppresses auto-login upon initial connection.
-i - Turns off interactive prompting during multiple file transfers.
The ftpscript.txt would look something like this:
open ftp.mycompany.com
user someuser password
binary
get /somedir/myfile.dat
It should be reasonably straightforward to write a script that outputs an FTP script containing the correct file name.
You would think that this would be an easy task. It isn't.
The best tool I've found is SyncBack and I think there is a free/lite version available.
If you're in a Windows environment consider using PowerShell and the System.Net.WebClient .NET Framework class like this post. This is a download example, but the WebClient object also provides an UploadFile method.