I am trying to use Parallel.For() to load n # of flat files into a database. Each iteration needs to create 2 config files in a folder and then run another process that uses the config files to know how to load the flat file into Oracle. Before I decided to use a parallel option, I tested the performance benefits by iterating 1000 times, each time creating the config files. I had better than a 2x increase in speed.
I then added the part where I call the other process, and I am getting periodic errors that some of the config files I am trying to create per iteration aren't there.
Here is my stripped-down code:
public static void FindFilesToLoad(int monitorId)
{
//left out code here that calls a stored procedure to get a list of ids
fileCount = idList.Count;
if (fileCount != 0)
{
Parallel.For(0, fileCount,
i =>
{
fileId = Convert.ToInt32(idList[i]);
//do the work here...
LoadFileIntoDatabase(monitorId, fileId);
});
}
}
public static void LoadFileIntoDatabase(int monitorId, int fileId)
{
stamp = Guid.NewGuid().ToString();
controlFile = CreateSqlLoaderControlFile(monitorId,stamp);
controlFileName = Path.GetFileName(controlFile);
parFile = CreateParFile(monitorId, controlFileName, stamp);
myCommand = #"CMD.EXE";
ProcessStartInfo startInfo = new ProcessStartInfo(myCommand)
{
WorkingDirectory = #"c:\temp\",
Arguments = #"/c SQLLDR CONTROL=" + controlFile + " PARFILE=" + parFile ,
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false
};
Process process = new Process();
process.StartInfo = startInfo;
process.Start();
process.WaitForExit();
exitCode = process.ExitCode;
}
Error example:
SQLLoader-500: Unable to open file (6f46ccfd-986e-427e-ab63-b7ce2396488e.ctl)
SQLLoader-553: file not found
SQL*Loader-509: System error: The system cannot find the file specified
Related
I am attempting to increase the speed at which files in my application download by downloading them in parallel. Previously I was downloading them sequentially and it worked fine but when I attempted to download them in parallel I ran into unexplained issues.
Here is my method in which I downloaded the files in sequence:
public IActionResult DownloadPartFiles([FromBody] FileRequestParameters parameters)
{
List<InMemoryFile> files = new List<InMemoryFile>();
for (int i = 0; i < parameters.FileNames.Length; i++)
{
InMemoryFile inMemoryFile = GetInMemoryFile(parameters.FileLocations[i], parameters.FileNames[i]).Result;
files.Add(inMemoryFile);
}
byte[] archiveFile = null;
using (MemoryStream archiveStream = new MemoryStream())
{
using (ZipArchive archive = new ZipArchive(archiveStream, ZipArchiveMode.Create, true))
{
foreach (InMemoryFile file in files)
{
ZipArchiveEntry zipArchiveEntry = archive.CreateEntry(file.FileName, CompressionLevel.Optimal);
using (MemoryStream originalFileStream = new MemoryStream(file.Content))
using (Stream zipStream = zipArchiveEntry.Open())
{
originalFileStream.CopyTo(zipStream);
}
}
}
archiveFile = archiveStream.ToArray();
}
return File(archiveFile, "application/octet-stream");
}
Here is the method changed to download the files in parallel:
public async Task<IActionResult> DownloadPartFiles([FromBody] FileRequestParameters parameters)
{
List<Task<InMemoryFile>> fileTasks = new List<Task<InMemoryFile>>();
for (int i = 0; i < parameters.FileNames.Length; i++)
{
if(i == parameters.FileNames.Length - 1)
{
int breakpoint = 0;
}
if(i == parameters.FileNames.Length)
{
int breakpoint = 0;
}
fileTasks.Add(Task.Run(() => GetInMemoryFile(parameters.FileLocations[i], parameters.FileNames[i])));
}
InMemoryFile[] fileResults = await Task.WhenAll(fileTasks);
byte[] archiveFile = null;
using (MemoryStream archiveStream = new MemoryStream())
{
using (ZipArchive archive = new ZipArchive(archiveStream, ZipArchiveMode.Create, true))
{
foreach (InMemoryFile file in fileResults)
{
ZipArchiveEntry zipArchiveEntry = archive.CreateEntry(file.FileName, CompressionLevel.Optimal);
using (MemoryStream originalFileStream = new MemoryStream(file.Content))
using (Stream zipStream = zipArchiveEntry.Open())
{
originalFileStream.CopyTo(zipStream);
}
}
}
archiveFile = archiveStream.ToArray();
}
return File(archiveFile, "application/octet-stream");
}
Here is the method that does the actual downloading:
private async Task<InMemoryFile> GetInMemoryFile(string fileLocation, string fileName)
{
InMemoryFile file;
using (HttpClient client = new HttpClient())
using (HttpResponseMessage response = await client.GetAsync(fileLocation))
{
byte[] fileContent = await response.Content.ReadAsByteArrayAsync();
file = new InMemoryFile(fileName, fileContent);
}
return file;
}
Now the issues I run into is after I changed DownloadPartFiles to get all the files in parallel my for loop is now seeming to go past its stop condition. For example, if parameters.FileNames.Length returns 12 the for loop should not run when i = 12 and it should exit the loop. However, in my testing it will continue to run when i = 12 and as one might expect I run into an out of bounds error. I tried to set breakpoints in my code to make sure that it was actually running past the stop condition and more weird behavior arose. In my for loop I included two if statements with breakpoint variables to break on. It will always break when i should be on its last loop but will never break when i is one after its expected last loop. It seems to skip that breakpoint when i is one past the expected stop condition. It will run fine if I step through the code while debugging but will out of bounds error when I let it run normally.
I'm not sure why this is happening but I am still new to asynchronous programming so maybe its just an oversight somewhere. Let me know if I need to explain anything further.
I make a critical mistake in that I tried to wrap an asynchronous method (my GetInMemoryFile method) in the Task.Run() method which is used to wrap synchronous methods to make them run asynchronously. This caused the weird behavior.
So in short I changed
fileTasks.Add(Task.Run(() => GetInMemoryFile(parameters.FileLocations[i], parameters.FileNames[i])));
To
fileTasks.Add(GetInMemoryFile(parameters.FileLocations[i], parameters.FileNames[i]));
I am saving my records using this code:
string path = Android.OS.Environment.ExternalStorageDirectory.AbsolutePath;
public string fileName { get; set; }
fileName = Path.Combine(path, "sample.wav");
if (!recorder.IsRecording)
{
recorder.StopRecordingOnSilence = TimeoutSwitch.IsToggled;
//Start recording
var audioRecordTask = await recorder.StartRecording();
BtnDoneRec.IsEnabled = false;
await audioRecordTask;
RecEditor.IsEnabled = true;
BtnDoneRec.IsEnabled = false;
PlayButton.IsEnabled = true;
var filePath = recorder.GetAudioFilePath();
if (filePath != null)
{
var stream = recorder.GetAudioFileStream();
using (var fileStream = new FileStream(fileName, FileMode.Create, FileAccess.Write))
{
stream.CopyTo(fileStream);
}
}
}
else
{
//stop recording ...
await recorder.StopRecording();
}
I want my record to have a specific name which is labeled with my RecEditor
using (var streamReader = new StreamReader(fileName))
{
File.Move("sample.wav", RecEditor.Text + ".wav");
}
So it will rename "sample.wav" to "RecEditor text.wav" every time I click my save button.
But when I click save, it gives me this record
System.IO.FileNotFoundException: 'Could not find file '/sample.wav'.'
The record is stored in /storage/emulated/0/sample.wav
The sample.wav is created in my device but I don't know why it give me 'Could not find file '/sample.wav'.' error. What am i doing wrong here?
I believe that what you're looking is something like this:
if(File.Exists(fileName))
{
var newFileName = Path.Combine(path, $"{RecEditor.Text}.wav");
File.Move(fileName, newFileName);
}
You don't need to open a new Stream as you are doing. Also, you need to put the full file path not only the file name.
You might want to validate that RecEditor.Text is not empty before using its value for the newfileName
Hope this helps.-
Is there any advantage to using a PowerShell script over a batch file?
By which I mean, if I have something which will run in either, is there anything to warrant detecting if PowerShell is installed and using it over a batch file if it is?
SCHTASKS works in exactly the same manner on both. Given that, is there any differentiator between them?
The following code executes the same code via both processes:
internal class Program
{
private static void Main()
{
const string COMMAND = #"SCHTASKS /QUERY /XML ONE";
Collection<PSObject> psObjects;
using (var ps = PowerShell.Create())
{
ps.AddScript(COMMAND);
psObjects = ps.Invoke();
}
var process = new Process
{
StartInfo = new ProcessStartInfo
{
UseShellExecute = false,
RedirectStandardOutput = true,
FileName = "cmd.exe",
Arguments = "/C " + COMMAND
}
};
process.Start();
string cmdTasks;
using (var reader = process.StandardOutput)
{
cmdTasks = reader.ReadToEnd();
}
}
}
The response object from the two approaches differs. The Process.Start() approach returns a string, which I would have to parse whereas invoking PowerShell gives me a collection of PSObject objects.
Howdy,
I have a project in VisualStudio which contains a folder 'xmlfiles' below the root node. This folder contains a file 'mensen.xml' which I try to open ...
However when I try to open that very file the debugger steps in and throws an exception.
I tried it with
if(File.Exists(#"/xmlfiles/mensen.xml") )
{
bool exists = true;
}
as well as:
FileStream fs = File.Open("/xmlfiles/mensen.xml", FileMode.Open);
TextReader textReader = new StreamReader(fs);
kantinen = (meineKantinen)deserializer.Deserialize(textReader);
textReader.Close();
Nothin is working :(.
How can I open a local file in the Phone7 Emulator?
If you are just opening it to read it then you can do the following (Assuming you have set the Build Action of the file to Resource):
System.IO.Stream myFileStream = Application.GetResourceStream(new Uri(#"/YOURASSEMBLY;component/xmlfiles/mensen.xml",UriKind.Relative)).Stream;
If you are attempting to read/write this file then you will need to copy it to Isolated Storage. (Be sure to add using System.IO.IsolatedStorage)
You can use these methods to do so:
private void CopyFromContentToStorage(String fileName)
{
IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication();
System.IO.Stream src = Application.GetResourceStream(new Uri(#"/YOURASSEMBLY;component/" + fileName,UriKind.Relative)).Stream;
IsolatedStorageFileStream dest = new IsolatedStorageFileStream(fileName, System.IO.FileMode.OpenOrCreate, System.IO.FileAccess.Write, store);
src.Position = 0;
CopyStream(src, dest);
dest.Flush();
dest.Close();
src.Close();
dest.Dispose();
}
private static void CopyStream(System.IO.Stream input, IsolatedStorageFileStream output)
{
byte[] buffer = new byte[32768];
long TempPos = input.Position;
int readCount;
do
{
readCount = input.Read(buffer, 0, buffer.Length);
if (readCount > 0) { output.Write(buffer, 0, readCount); }
} while (readCount > 0);
input.Position = TempPos;
}
In both cases, be sure the file is set to Resource and you replace the YOURASSEMBLY part with the name of your assembly.
Using the above methods, to access your file just do this:
IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication();
if (!store.FileExists(fileName))
{
CopyFromContentToStorage(fileName);
}
store.OpenFile(fileName, System.IO.FileMode.Append);
The emulator does not have access to the PC file system. You must deploy the file to the target (emulator). The easiest way to do this is mark the file as an embedded Resource. Set the file's Build Action to 'Resource' and then extract it at runtime with code something like this:
var res = Application.GetResourceStream(new Uri([nameOfYourFile], UriKind.Relative))
I'm trying to write a macro that will generate a plain-text list of files changed based on the list of files in the Pending Changes pane but I can't figure out how to do it. The server location of a file is the property that is formatted like this:
$/TfsName/SomeSolution/Web/SomeFolder/SomeFile1.aspx
$/TfsName/SomeSolution/Web/SomeFolder/SomeFile2.aspx
The closest I can get is opening the properties of the selected item in the pane, which isn't very useful:
DTE.ExecuteCommand ("TeamFoundationContextMenus.SourceControlPendingChangesSourceFiles.TfsContextPendingCheckinsPendingCheckinsProperties")
Edit: here's the entire code for the macro I have so far, the TODOs are where I need help:
Public Class Pending
Public Shared Sub Pending()
OutputClear()
OutputWriteLine("Files Changed:")
Dim outInfo As String = ""
DTE.Windows.Item("{2456BD12-ECF7-4988-A4A6-67D49173F564}").Activate() 'Pending Changes - Source Files
'TODO: loop through each changed file
'TODO: get TFS server location of each file
outInfo &= "some file name"
OutputWriteLine(outInfo)
End Sub
' snip: other supporting functions
End Class
Well I haven't been able to figure out how to do it with a macro yet, but thanks to Bob Hardister on twitter, I can use this command to get what I'm looking for:
"C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\tf.exe" status $/ProjectName/SubDir/ /recursive
...but what works even better is a command-line app that uses this code:
const string TFSSERVER = "http://TfsServer:8080";
static void Main(string[] args)
{
//http://blogs.msdn.com/b/buckh/archive/2006/03/15/552288.aspx
//http://blogs.msdn.com/b/jmanning/archive/2005/12/01/499033.aspx
string projectName = args[0];
TeamFoundationServer tfs = new TeamFoundationServer(TFSSERVER);
VersionControlServer versionControl = (VersionControlServer)tfs.GetService(typeof(VersionControlServer));
PendingSet[] sets = versionControl.GetPendingSets(new String[] { "$/Projects/" + projectName }, RecursionType.Full);
Console.WriteLine(versionControl.AuthenticatedUser + " pending changes for " + projectName + ":");
foreach (PendingSet set in sets)
{
if (set.Type == PendingSetType.Workspace && set.OwnerName == versionControl.AuthenticatedUser)
{
foreach (PendingChange pc in set.PendingChanges)
{
Console.WriteLine(pc.ServerItem);
}
}
}
}
Then I just added the compiled EXE call to the External Tools menu and use it within VS there.
Bonus Edit: Here's the VSS version (not as nice):
const string SSDIR = #"\\VssServer\VssShare";
static void Main(string[] args)
{
string projectName = args[0];
string userName = "user";
VSSDatabaseClass vss = new VSSDatabaseClass();
vss.Open(SSDIR + #"\srcsafe.ini", userName, userName);
VSSItem sourceItem = vss.get_VSSItem("$/Projects/" + projectName, false);
Console.WriteLine(userName + " pending checkins for " + projectName + ":");
int total = GetItems(sourceItem);
Console.WriteLine(total.ToString() + " total changes.");
}
const int VSSFILE_CHECKEDOUT_ME = 2;
const int VSSITEM_PROJECT = 0;
const int VSSITEM_FILE = 1;
public static int GetItems(IVSSItem originalItem)
{
int total = 0;
foreach (IVSSItem subItem in originalItem.get_Items(false))
{
if (subItem.Type == VSSITEM_FILE && subItem.IsCheckedOut == VSSFILE_CHECKEDOUT_ME)
{
Console.WriteLine(subItem.Spec);
total++;
}
else if (subItem.Type == VSSITEM_PROJECT)
{
total += GetItems(subItem);
}
}
return total;
}