I have to pick the files in order(first file first) from say a folder (C:\Users) and file name has the timestamp in it.
For example below are my files in C:\Users\ and the time stamp is after the first underscore i.e. 20170126102806 in the first file below. I have to loop through files and pick the first file and so on. so out of 5 files below,20170123-000011_20170126101823_AAA is the first file. How do I do this in SSIS?
1.20170123-000011_20170126102806_AAA
2.20170123-000011_20170126103251_AAA
3.20170123-000011_20170126101823_AAA
4.20170123-000011_20170126103305_AAA
5.20170123-000011_20170126102641_AAA
You can act in two ways:
use the foreach loop container to get the list of files, and then populate a database table.
Then, outside the foreach loop, use an Execute SQL to select from that table using an appropriate ORDER BY. Load an object variable with the result set. Then use a second foreach loop to step through the variable object and collect files.
use a Script Task to retrieve the contents of the folder (the list of files) and sort files then load an object variable with the dataset. Then use a foreach loop to step through the variable object to collect files.
I hope this help.
You could use a script task in a For Each Loop. Use the filename returned as the source to load each time.
using System.IO;
public void Main()
{
string filePath = "D:\\Temp";
DirectoryInfo dir = new DirectoryInfo(filePath);
var files = dir.GetFiles("*_AAA");//Or from a variable
DateTime fileCreateDate1 = File.GetCreationTime(filePath + "\\" + files[0]);
if (files.Length >= 2)
{
for (int i = 1; i < files.Length; i++)
{
DateTime fileCreateDate2 = File.GetCreationTime(filePath+ "\\" + files[i]);
if (fileCreateDate1 < fileCreateDate2)
{
fileCreateDate1 = fileCreateDate2;
}
}
}
Dts.Variables["User::FileToLoad"].Value = fileCreateDate1;
Dts.TaskResult = (int)ScriptResults.Success;
}
You will have to remove the file after it was loaded or else it will be loaded each time as it is the oldest or latest file.
There might be a bug or so, but have similar code that works. Just iron it out if needed.
Related
In my solution a had a folder with a few files. All this files have the Build Action "Embedded Resource".
With this code I can get a file:
assembly.GetManifestResourceStream(assembly.GetName().Name + ".Folder.File.txt");
But is there any way to get all *.txt files in this folder? A list of names or a method to iterate through them all?
You could check out
assembly.GetManifestResourceNames()
which returns an array of strings of all the resources contained. You could then filter that list to find all your *.txt files stored as embedded resources.
See MSDN docs for GetManifestResourceNames for details.
Try this, returns an array with all .txt files inside Folder directory.
private string[] GetAllTxt()
{
var executingAssembly = Assembly.GetExecutingAssembly();
string folderName = string.Format("{0}.Resources.Folder", executingAssembly.GetName().Name);
return executingAssembly
.GetManifestResourceNames()
.Where(r => r.StartsWith(folderName) && r.EndsWith(".txt"))
//.Select(r => r.Substring(folderName.Length + 1))
.ToArray();
}
NOTE: Uncomment the //.Select(... line in order to get the filename.
have a try with this. here you get all files
string[] embeddedResources = Assembly.GetAssembly(typeof(T)).GetManifestResourceNames();
T is of course your type. so you can use it generic
Just cracked this, use:
Assembly _assembly;
_assembly = Assembly.GetExecutingAssembly();
List<string> filenames = new List<string>();
filenames = _assembly.GetManifestResourceNames().ToList<string>();
List<string> txtFiles = new List<string>();
for (int i = 0; i < filenames.Count(); i++)
{
string[] items = filenames.ToArray();
if (items[i].ToString().EndsWith(".txt"))
{
txtFiles.Add(items[i].ToString());
}
}
How to search for each file in a single directory of xml files and get value of one element?
I tried to loop in Linq:
var files = Directory.GetFiles(C:\Users\valen\Downloads\2019-11-04 apmt, "*.xml", SearchOption.AllDirectories);
foreach (var file in files) {
var doc = XDocument.Load(file);
I need to find shipment element in <>.
It is hard to help when there is very little information about your XML given in OP. Assuming you are searching for Element ShipmentId in your xml files, you could search the Xml File using Linq. For example,
var listOfShipmentIds = new List<string>();
var files = Directory.GetFiles(folderToSearch, "*.xml", SearchOption.AllDirectories);
foreach(var file in files)
{
var dataRead = XDocument.Load(file)
.Root
.DescendantNodes()
.OfType<XElement>()
.Where(x=>x.Name.LocalName.Equals("ShipmentId"))
.Select(x=>x.Value);
listOfShipmentIds.AddRange(dataRead);
}
This would enable you to parse the Xml files and get the value of Element specified.
I have files in a directory like below:
first-file-name
2nd-file-name
3rd-file-name
.
.
n-file-name
I need to store each portion of file name in a separate variable because I want to insert these values in separate columns of table.
For this, I used the below script to get the each portion of a file name:
$var1=$item.BaseName.Split("-",3)[0]---------first
$var2=$item.BaseName.Split("-",3)[1]---------file
$var3=$item.BaseName.Split("-",3)[2]---------name
and can save these values in a variable. But the question is how can I do this for all files, if I use foreach loop then the variable values will be overwritten???
foreach(item in $items)
{
$var1=$item.BaseName.Split("-",3)[0]---------first
$var2=$item.BaseName.Split("-",3)[1]---------file
$var3=$item.BaseName.Split("-",3)[2]---------name
}
Here, in $items I got the file path using get-childitem.
I would create a PsCustomObject with the three parts:
$parts = $items | ForEach-Object {
[PsCustomObject]#{
FirstPart = $item.BaseName.Split("-",3)[0]
SecondPart = $item.BaseName.Split("-",3)[1]
ThirdPart = $item.BaseName.Split("-",3)[2]
}
}
now $parts is an array of these objects so you can access them using e. g.
$parts[0].FirstPart
in my current ASP.net MVC 3.0 project i am stuck with a situation.
I have four .txt files each has approximatly 100k rows of records
These files will be replaced with new files on weekly bases.
I need to Query data from these four text files, I am not able to choose the best and efficient way to do this.
3 ways I could think
Convert these text files to XML on a weekly basis and query it with Linq-XML
Run a batch import weekly from txt to SQL Server and query using Linq-Entities
avoid all conversions and query directly from text files.
Can any one suggest a best way to deal with this situation.
update:
Url of the Text File
I should connect to this file with credentials.
once i connect successfully, I will have the text file as below with Pipeline as Deliminator
This is the text file
Now i have to look up for the field highlighted in yellow and get the data in that row.
Note: First two lines of the text file are headers of the File.
Well As i Found a way my self. Hope this will be useful for any who are interested to get this done.
string url = "https://myurl.com/folder/file.txt";
WebClient request = new WebClient();
request.Credentials = new NetworkCredential(ConfigurationManager.AppSettings["UserName"], ConfigurationManager.AppSettings["Password"]);
Stream s = request.OpenRead(url);
using (StreamReader strReader = new StreamReader(s))
{
for (int i = 0; i <= 1; i++)
strReader.ReadLine();
while (!strReader.EndOfStream)
{
var CurrentLine = strReader.ReadLine();
var count = CurrentLine.Split('|').Count();
if (count > 3 && CurrentLine.Split('|')[3].Equals("SearchString"))
{
#region Bind Data to Model
//var Line = CurrentLine.Split('|');
//CID.RecordType = Line[0];
//CID.ChangeIdentifier = Line[1];
//CID.CoverageID = Convert.ToInt32(Line[2]);
//CID.NationalDrugCode = Line[3];
//CID.DrugQualifier = Convert.ToInt32(Line[4]);
#endregion
break;
}
}
s.Close();
}
request.Dispose();
I have a strange problem when deleteting records using linq, my suspicion is that it has something to do with the range variable (named source). After deleting a record all targets for a customer are retrieved using the following statement:
var q = from source in unitOfWork.GetRepository<db_Target>().Find()
where source.db_TargetBase.db_Person.fk_Customer == customerID
select source.FromLinq();
where FromLinq is in extention method on db_target:
public static Target FromLinq(this db_Target source)
{
return new Target
{
id = source.id,
LastModified = source.db_TargetBase.LastModified,
...
}
}
When a record is deleted both db_Target and db_TargetBase are deleted. When, for example, two users are deleting records, linq tries to retrieve a record for user2 which is deleted by user1, causing a crash on the LastModified = source.db_TargetBase.LastModified line because db_TargetBase is null.
When using the following code the problem does not occure and only the non-deleted records are retrieved:
var q = from source in unitOfWork.GetRepository<db_Target>().Find()
where source.db_TargetBase.db_Person.fk_Customer == customerID
select new Target
{
id = source.id,
LastModified = source.db_TargetBase.LastModified,
...
};
This spawns two questions:
What is happening here? Am I making a copy of the range variable source because I'm using it in a extention method?
How can I "wrap" the return new Target code? I am using this in multiple places and do not want to copy it every time. Making my code harder to maintain.
TIA,
JJ
In the first set of code - since the initializer lives an a non-translatable method (extension or otherwise), it cannot be translated - so it is run locally.
In the second set of code - the initializer is represented by an elementinit expression, which is translated (examine/compare the select clause of the generated sql for proof).
if you want to wrap this, you need to have an Expression<Func<db_Target, Target>> that anyone can grab and use in thier query. Fortunately, that's easy to do:
public Expression<Func<db_Target, Target>> GetFromLinqExpressionForTarget()
{
return
source => new Target
{
id = source.id,
LastModified = source.db_TargetBase.LastModified,
...
}
}
Which may be used like so:
var FromLinq = GetFromLinqExpressionForTarget();
var q =
(
from source in ...
...
...
select source
).Select(FromLinq);
Now ... I'm really running on a guess here and am only about 60% confident that my answer is correct. So if someone wants to confirm this, that'll make my day. :)