How can I extract values from a custom flat file header into variables? - etl

Question

I have been stuck for a while with this problem and I have no clue. I am trying to upload multiple CSV files which has dates but I wanted the dates stored as date variables so I use the date variables to form part of the column in a table using script componet and I have no idea how to create the dates as date variables in SSIS.
CSV files look as shown below when opened in Excel.
CSV data 1:
Relative Date: 02/01/2013
Run Date: 15/01/2013
Organisation,AreaCode,ACount
Chadwell,RM6,50
Primrose,RM6,60
CSV data 2:
Relative Date: 14/02/2013
Run Date: 17/02/2013
Organisation,AreaCode,ACount
Second Ave,E12,110
Fourth Avenue, E12,130
I want the Relative Date and Run Date stored as date variables. I hope I made sense.

Answer 1

Your best solution would be to use a Script Task in your control flow. With this you would pre-process your CSV files - you can easily parse the first two rows, retrieving your wanted dates and storing them into two variables created beforehand. (http://msdn.microsoft.com/en-us/library/ms135941.aspx)
Important to make sure when passing the variables into the script task you set them as ReadWriteVariables. Use these variables in any way you desire afterwards.
Updated Quick Walkthrough:
I presume that the CSV files you will want to import will be located in the same directory:
Add a Foreach Loop Container which will loop through the files in your specified directory and inside, a Script Task which will be responsible for parsing the two dates in each of your files and a Data Flow Task which you will use for your file import.
Create the variables you will be using - one for the FileName/Path, two for the two dates you want to retrieve. These you won't fill in as it will be done automatically in your process.
Set-up your Foreach Loop Container:
Select a Foreach File Enumerator
Select a directory folder that will contain your files. (Even better, add a variable that will take in a path you specify. This can then be read into the enumerator using its expression builder)
Wildcard for the files that will be searched in that directory.
You also need to map each filename the enumerator generates to the variable you created earlier.
Open up your Script Task, add the three variables to the ReadWriteVariables section. This is important, otherwise you won't be able to write to your variables.
This is the script I used for the purpose. Not necessarily the best, works for this example.
public void Main()
{
string filePath = this.Dts.Variables["User::FileName"].Value.ToString();
using (StreamReader reader = new System.IO.StreamReader(filePath))
{
string line = "";
bool getNext = true;
while (getNext && (line = reader.ReadLine()) != null)
{
if(line.Contains("Relative Date"))
{
string date = getDate(line);
this.Dts.Variables["User::RelativeDate"].Value = date;
// Test Event Information
bool fireAgain = false;
this.Dts.Events.FireInformation(1, "Rel Date", date,
"", 0, ref fireAgain);
}
else if (line.Contains("Run Date"))
{
string date = getDate(line);
this.Dts.Variables["User::RunDate"].Value = date;
// Test Event Information
bool fireAgain = false;
this.Dts.Events.FireInformation(1, "Run Date", date,
"", 0, ref fireAgain);
break;
}
}
}
Dts.TaskResult = (int)ScriptResults.Success;
}
private string getDate(string line)
{
Regex r = new Regex(#"\d{2}/\d{2}/\d{4}");
MatchCollection matches = r.Matches(line);
return matches[matches.Count - 1].Value;
}
The results from the execution of the Script Task for the two CSV files. The dates can now be used in any way you fancy in your Data Flow Task. Make sure you skip the first rows you don't need to import in your Source configuration.