Related
I am having a text file that has content in this manner.
One;Thomas;Newyork;2020-12-31 14:00:00;0
Two;David;London;2021-01-31 12:00:00;0
Three;James;Chicago;2021-01-20 15:00:00;0
Four;Edward;India;2020-12-25 15:00:00;0
In these entries according to date time, two are past entries and two are future entries. The last 0 in the string indicates the Flag. With the past entries that flag needs to be changed to 1.
Consider all the entries are separated with the array. I tried this block of code but its not working to solve the problem here.
for ($item=0 ; $item -lt $entries.count ; $item++)
{
if ($entries.DateTime[$item] -lt (Get-Date -Format "yyyy-MM-dd HH:mm:ss"))
{
$cont = Get-Content $entries -ErrorAction Stop
$string = $entries.number[$item] + ";" + $entries.name[$item] + ";" +
$entries.city[$item]+ ";" + $entries.DateTime[$item]
$lineNum = $cont | Select-String $string
$line = $lineNum.LineNumber + 1
$cont[$line] = $string + ";1"
Set-Content -path $entries
}
}
I am getting errors with this concept.
Output should come as:-
One;Thomas;Newyork;2020-12-31 14:00:00;1 ((Past Deployment with respect to current date)
Two;David;London;2021-01-31 12:00:00;0
Three;James;Chicago;2021-01-20 15:00:00;0
Four;Edward;India;2020-12-25 15:00:00;1 (Past Deployment with respect to current date)
This output needs to be overwritten on the file from where the content is extracted ie Entries.txt
param(
$exampleFileName = "d:\tmp\file.txt"
)
#"
One;Thomas;Newyork;2020-12-31 14:00:00;0
Two;David;London;2021-01-31 12:00:00;0
Three;James;Chicago;2021-01-20 15:00:00;0
Four;Edward;India;2020-12-25 15:00:00;0
"# | Out-File $exampleFileName
Remove-Variable out -ErrorAction SilentlyContinue
Get-Content $exampleFileName | ForEach-Object {
$out += ($_ -and [datetime]::Parse(($_ -split ";")[3]) -gt [datetime]::Now) ? $_.SubString(0,$_.Length-1) + "1`r`n" : $_ + "`r`n"
}
Out-File -InputObject $out -FilePath $exampleFileName
I try to make, the line from the first array is read from a file and is replaced with a line from the second array, so some times with different lines. I made a script, but I do not understand why it does not work.
$OldStrings = #(
"desktopwidth:i:1440",
"desktopheight:i:900",
"winposstr:s:0,1,140,60,1596,999"
)
$NewStrings = #(
"desktopwidth:i:1734",
"desktopheight:i:990",
"winposstr:s:0,1,50,7,1800,1036"
)
$LinesArray = Get-Content -Path 'C:\temp\My Copy\Default.rdp'
$LinesCount = $LinesArray.Count
for ($i=0; $i -lt $LinesCount; $i++) {
foreach ($OldString in $OldStrings) {
foreach ($NewString in $NewStrings) {
if ($LinesArray[$i] -like $OldString) {
$LinesArray[$i] = $LinesArray[$i] -replace $OldString, $NewString
Write-Host "`nline" $i "takes on value:" $LinesArray[$i] "`n" -ForegroundColor Gray
}
}
}
}
The file is probably why it is not read at all.
After executing the script, I see only
line 2 takes on value: desktopwidth:i:1734
line 3 takes on value: desktopwidth:i:1734
line 5 takes on value: desktopwidth:i:1734
You're looking through the string arrays twice. You want to do two loops, one for each line in the file AND another for each count in the lines you're replacing. I think this should work:
$OldStrings = #(
"desktopwidth:i:1440",
"desktopheight:i:900",
"winposstr:s:0,1,140,60,1596,999"
)
$NewStrings = #(
"desktopwidth:i:1734",
"desktopheight:i:990",
"winposstr:s:0,1,50,7,1800,1036"
)
$LinesArray = Get-Content -Path 'C:\temp\My Copy\Default.rdp'
# loop through each line
for ($i=0; $i -lt $LinesArray.Count; $i++)
{
for ($j=0;$j -lt $OldStrings.Count; $j++)
{
if ($LinesArray[$i] -match $OldStrings[$j])
{
$LinesArray[$i] = $LinesArray[$i] -replace $OldStrings[$j],$NewStrings[$j]
Write-Host "`nline" $i "takes on value:" $LinesArray[$i] "`n" -ForegroundColor Gray
}
}
}
$LinesArray | Set-Content -Path 'C:\temp\My Copy\Default.rdp'
You don't need to bother checking the lines to look for matches. Since you have the replacements ready just do the replacements outright anyway. Should be faster this way as well.
$stringReplacements = #{
"desktopwidth:i:1440" = "desktopwidth:i:1734"
"desktopheight:i:900" = "desktopheight:i:990"
"winposstr:s:0,1,140,60,1596,999" = "winposstr:s:0,1,50,7,1800,1036"
}
$path = 'C:\temp\My Copy\Default.rdp'
# Read the file in as a single string.
$fileContent = Get-Content $path | Out-String
# Iterate over each key value pair
$stringReplacements.Keys | ForEach-Object{
# Attempt the replacement for each key/pair search/replace pair
$fileContent =$fileContent.Replace($_,$stringReplacements[$_])
}
# Write changes back to file.
# $fileContent | Set-Content $path
$stringReplacements is a key value hash of search and replace strings. I don't see you writing the changes back to file so I left a line on the end for you to uncomment.
You could add in checks to do the replacements still if you value the write-host lines but I figured that was for debugging and you already know how to do that.
So I have a Powershell ISE ( tried running as administrator and w/o ) and it is created dummy .mdf file , 50 of them
Problem is that it holds onto the 1st and last , so copying or deleting them is not working ...
This is my script
param(
$amount = 50 # $(throw "Please give an amount of files to be created")
, $size = 5 # $(throw "Please give a the size of the files")
, $folder = "C:\dev\powershell\oldlocation" # $(throw "Please give an output folder wehere the files need to be created")
, $name = 'db' # $null
, $extension = '.mdf' # $null .mdf / .ldf
)
CLS
# Check for input
if(Test-Path $folder)
{
if($name -eq $null)
{
Write-Host "No filename given. Using default setting 'dummy'" -ForegroundColor Yellow
$name = 'dummy'
}
if($extension -eq $null)
{
Write-Host "No filename extension given. Using default setting '.txt'" -ForegroundColor Yellow
$extension = 'txt'
}
elseif($extension -contains '.')
{
$extension = $extension.Substring(($extension.LastIndexOf(".") + 1), ($extension.Length - 1))
}
for($i = 1; $i -le $amount; $i++)
{
$path = $folder + '\' + $name + '_' + $i + '.' + $extension
$file = [io.file]::Create($path)
$file.SetLength($size)
$file.Close
sleep 0.5
}
}
else{
Write-Host "The folder $folder doesn't exist" -ForegroundColor Red
Exit(0)
}
When () are omitted on a method, it returns the Overload Definitions. So the line where you are trying to close the file just needs ().
$file.Close()
If you see OverloadDefinitions ever returned, that's what to look for.
I'm trying to format large text files (~300MB) between 0 to 3 columns :
12345|123 Main St, New York|91110
23456|234 Main St, New York
34567|345 Main St, New York|91110
And the output should be:
000000000012345,"123 Main St, New York",91110,,,,,,,,,,,,
000000000023456,"234 Main St, New York",,,,,,,,,,,,,
000000000034567,"345 Main St, New York",91110,,,,,,,,,,,,
I'm new to powershell, but I've read that I should avoid Get-Content so I am using StreamReader. It is still much too slow:
function append-comma{} #helper function to append the correct amount of commas to each line
$separator = '|'
$infile = "\large_data.csv"
$outfile = "new_file.csv"
$target_file_in = New-Object System.IO.StreamReader -Arg $infile
If ($header -eq 'TRUE') {
$firstline = $target_file_in.ReadLine() #skip header if exists
}
while (!$target_file_in.EndOfStream ) {
$line = $target_file_in.ReadLine()
$a = $line.split($separator)[0].trim()
$b = ""
$c = ""
if ($dataType -eq 'ECN'){$a = $a.padleft(15,'0')}
if ($line.split($separator)[1].length -gt 0){$b = $line.split($separator)[1].trim()}
if ($line.split($separator)[2].length -gt 0){$c = $line.split($separator)[2].trim()}
$line = $a +',"'+$b+'","'+$c +'"'
$line -replace '(?m)"([^,]*?)"(?=,|$)', '$1' |append-comma >> $outfile
}
$target_file_in.close()
I am building this for other people on my team and wanted to add a gui using this guide:
http://blogs.technet.com/b/heyscriptingguy/archive/2014/08/01/i-39-ve-got-a-powershell-secret-adding-a-gui-to-scripts.aspx
Is there a faster way to do this in Powershell?
I wrote a script using Linux bash(Cygwin64 on Windows) and a separate one in Python. Both ran much faster, but I am trying to script something that would be "approved" on a Windows Platform.
All that splitting and replacing costs you way more time than you gain from the StreamReader. Below code cut execution time to ~20% for me:
$separator = '|'
$infile = "\large_data.csv"
$outfile = "new_file.csv"
if ($header -eq 'TRUE') {
$linesToSkip = 1
} else {
$linesToSkip = 0
}
Get-Content $infile | select -Skip $linesToSkip | % {
[int]$a, [string]$b, [string]$c = $_.split($separator)
'{0:d15},"{1}",{2},,,,,,,,,,,,,' -f $a, $b.Trim(), $c.Trim()
} | Set-Content $outfile
How does this work for you? I was able to read and process a 35MB file in about 40 seconds on a cheap ole workstation.
File Size: 36,548,820 bytes
Processed In: 39.7259722 seconds
Function CheckPath {
[CmdletBinding()]
param(
[Parameter(Mandatory=$True,
ValueFromPipeline=$True)]
[string[]]$Path
)
BEGIN {}
PROCESS {
IF ((Test-Path -LiteralPath $Path) -EQ $False) {Write-host "Invalid File Path $Path"}
}
END {}
}
$infile = "infile.txt"
$outfile = "restult5.txt"
#Check File Path
CheckPath $InFile
#Initiate StreamReader
$Reader = New-Object -TypeName System.IO.StreamReader($InFile);
#Create New File Stream Object For StreamWriter
$WriterStream = New-Object -TypeName System.IO.FileStream(
$outfile,
[System.IO.FileMode]::Create,
[System.IO.FileAccess]::Write);
#Initiate StreamWriter
$Writer = New-Object -TypeName System.IO.StreamWriter(
$WriterStream,
[System.Text.Encoding]::ASCII);
If ($header -eq $True) {
$Reader.ReadLine() |Out-Null #Skip First Line In File
}
while ($Reader.Peek() -ge 0) {
$line = $Reader.ReadLine() #Read Line
$Line = $Line.split('|') #Split Line
$OutPut = "$($($line[0]).PadLeft(15,'0')),`"$($Line[1])`",$($Line[2]),,,,,,,,,,,,"
$Writer.WriteLine($OutPut)
}
$Reader.Close();
$Reader.Dispose();
$Writer.Flush();
$Writer.Close();
$Writer.Dispose();
$endDTM = (Get-Date) #Get Script End Time For Measurement
Write-Host "Elapsed Time: $(($endDTM-$startDTM).totalseconds) seconds" #Echo Time elapsed
Regex is fast:
$infile = ".\large_data.csv"
gc $infile|%{
$x=if($_.indexof('|')-ne$_.lastindexof('|')){
$_-replace'(.+)\|(.+)\|(.+)',('$1,"$2",$3'+','*12)
}else{
$_-replace'(.+)\|(.+)',('$1,"$2"'+','*14)
}
('0'*(15-($x-replace'([^,]),.+','$1').length))+$x
}
I have another approach. Let powershell read the input file as a csv file, with a pipe character as delimiter. Then format the output the way you want it. I have not tested this for speed with large files.
$infile = "\large-data.csv"
$outfile = "new-file.csv"
import-csv $infile -header id,addr,zip -delimiter "|" |
% {'{0},"{1}",{2},,,,,,,,,,,,,' -f $_.id.padleft(15,'0'), $_.addr.trim(), $_.zip} |
set-content $outfile
Is it possible to redirect stdout from an external program to a variable and stderr from external programs to another variable in one run?
For example:
$global:ERRORS = #();
$global:PROGERR = #();
function test() {
# Can we redirect errors to $PROGERR here, leaving stdout for $OUTPUT?
$OUTPUT = (& myprogram.exe 'argv[0]', 'argv[1]');
if ( $OUTPUT | select-string -Pattern "foo" ) {
# do stuff
} else {
$global:ERRORS += "test(): oh noes! 'foo' missing!";
}
}
test;
if ( #($global:ERRORS).length -gt 0 ) {
Write-Host "Script specific error occurred";
foreach ( $err in $global:ERRORS ) {
$host.ui.WriteErrorLine("err: $err");
}
} else {
Write-Host "Script ran fine!";
}
if ( #($global:PROGERR).length -gt 0 ) {
# do stuff
} else {
Write-Host "External program ran fine!";
}
A dull example however I am wondering if that is possible?
One option is to combine the output of stdout and stderr into a single stream, then filter.
Data from stdout will be strings, while stderr produces System.Management.Automation.ErrorRecord objects.
$allOutput = & myprogram.exe 2>&1
$stderr = $allOutput | ?{ $_ -is [System.Management.Automation.ErrorRecord] }
$stdout = $allOutput | ?{ $_ -isnot [System.Management.Automation.ErrorRecord] }
The easiest way to do this is to use a file for the stderr output, e.g.:
$output = & myprogram.exe 'argv[0]', 'argv[1]' 2>stderr.txt
$err = get-content stderr.txt
if ($LastExitCode -ne 0) { ... handle error ... }
I would also use $LastExitCode to check for errors from native console EXE files.
You should be using Start-Process with -RedirectStandardError -RedirectStandardOutput options. This other post has a great example of how to do this (sampled from that post below):
$pinfo = New-Object System.Diagnostics.ProcessStartInfo
$pinfo.FileName = "ping.exe"
$pinfo.RedirectStandardError = $true
$pinfo.RedirectStandardOutput = $true
$pinfo.UseShellExecute = $false
$pinfo.Arguments = "localhost"
$p = New-Object System.Diagnostics.Process
$p.StartInfo = $pinfo
$p.Start() | Out-Null
$p.WaitForExit()
$stdout = $p.StandardOutput.ReadToEnd()
$stderr = $p.StandardError.ReadToEnd()
Write-Host "stdout: $stdout"
Write-Host "stderr: $stderr"
Write-Host "exit code: " + $p.ExitCode
This is also an alternative that I have used to redirect stdout and stderr of a command line while still showing the output during PowerShell execution:
$command = "myexecutable.exe my command line params"
Invoke-Expression $command -OutVariable output -ErrorVariable errors
Write-Host "STDOUT"
Write-Host $output
Write-Host "STDERR"
Write-Host $errors
It is just another possibility to supplement what was already given.
Keep in mind this may not always work depending upon how the script is invoked. I have had problems with -OutVariable and -ErrorVariable when invoked from a standard command line rather than a PowerShell command line like this:
PowerShell -File ".\FileName.ps1"
An alternative that seems to work under most circumstances is this:
$stdOutAndError = Invoke-Expression "$command 2>&1"
Unfortunately, you will lose output to the command line during execution of the script and would have to Write-Host $stdOutAndError after the command returns to make it "a part of the record" (like a part of a Jenkins batch file run). And unfortunately it doesn't separate stdout and stderr.
In case you want to get any from a PowerShell script and to pass a function name followed by any arguments you can use dot sourcing to call the function name and its parameters.
Then using part of James answer to get the $output or the $errors.
The .ps1 file is called W:\Path With Spaces\Get-Something.ps1 with a function inside named Get-It and a parameter FilePath.
Both the paths are wrapped in quotes to prevent spaces in the paths breaking the command.
$command = '. "C:\Path Spaces\Get-Something.ps1"; Get-It -FilePath "W:\Apps\settings.json"'
Invoke-Expression $command -OutVariable output -ErrorVariable errors | Out-Null
# This will get its output.
$output
# This will output the errors.
$errors
Copied from my answer on how to capture both output and verbose information in different variables.
Using Where-Object(The alias is symbol ?) is an obvious method, but it's a bit too cumbersome. It needs a lot of code.
In this way, it will not only take longer time, but also increase the probability of error.
In fact, there is a more concise method that separate different streams to different variable in PowerShell(it came to me by accident).
# First, declare a method that outputs both streams at the same time.
function thisFunc {
[cmdletbinding()]
param()
Write-Output 'Output'
Write-Verbose 'Verbose'
}
# The separation is done in a single statement.Our goal has been achieved.
$VerboseStream = (thisFunc -Verbose | Tee-Object -Variable 'String' | Out-Null) 4>&1
Then we verify the contents of these two variables
$VerboseStream.getType().FullName
$String.getType().FullName
The following information should appear on the console:
PS> System.Management.Automation.VerboseRecord
System.String
'4>&1' means to redirect the verboseStream to the success stream, which can then be saved to a variable, of course you can change this number to any number between 2 and 5.
Separately, preserving formatting
cls
function GetAnsVal {
param([Parameter(Mandatory=$true, ValueFromPipeline=$true)][System.Object[]][AllowEmptyString()]$Output,
[Parameter(Mandatory=$false, ValueFromPipeline=$true)][System.String]$firstEncNew="UTF-8",
[Parameter(Mandatory=$false, ValueFromPipeline=$true)][System.String]$secondEncNew="CP866"
)
function ConvertTo-Encoding ([string]$From, [string]$To){#"UTF-8" "CP866" "ASCII" "windows-1251"
Begin{
$encFrom = [System.Text.Encoding]::GetEncoding($from)
$encTo = [System.Text.Encoding]::GetEncoding($to)
}
Process{
$Text=($_).ToString()
$bytes = $encTo.GetBytes($Text)
$bytes = [System.Text.Encoding]::Convert($encFrom, $encTo, $bytes)
$encTo.GetString($bytes)
}
}
$all = New-Object System.Collections.Generic.List[System.Object];
$exception = New-Object System.Collections.Generic.List[System.Object];
$stderr = New-Object System.Collections.Generic.List[System.Object];
$stdout = New-Object System.Collections.Generic.List[System.Object]
$i = 0;$Output | % {
if ($_ -ne $null){
if ($_.GetType().FullName -ne 'System.Management.Automation.ErrorRecord'){
if ($_.Exception.message -ne $null){$Temp=$_.Exception.message | ConvertTo-Encoding $firstEncNew $secondEncNew;$all.Add($Temp);$exception.Add($Temp)}
elseif ($_ -ne $null){$Temp=$_ | ConvertTo-Encoding $firstEncNew $secondEncNew;$all.Add($Temp);$stdout.Add($Temp)}
} else {
#if (MyNonTerminatingError.Exception is AccessDeniedException)
$Temp=$_.Exception.message | ConvertTo-Encoding $firstEncNew $secondEncNew;
$all.Add($Temp);$stderr.Add($Temp)
}
}
$i++
}
[hashtable]$return = #{}
$return.Meta0=$all;$return.Meta1=$exception;$return.Meta2=$stderr;$return.Meta3=$stdout;
return $return
}
Add-Type -AssemblyName System.Windows.Forms;
& C:\Windows\System32\curl.exe 'api.ipify.org/?format=plain' 2>&1 | set-variable Output;
$r = & GetAnsVal $Output
$Meta2=""
foreach ($el in $r.Meta2){
$Meta2+=$el
}
$Meta2=($Meta2 -split "[`r`n]") -join "`n"
$Meta2=($Meta2 -split "[`n]{2,}") -join "`n"
[Console]::Write("stderr:`n");
[Console]::Write($Meta2);
[Console]::Write("`n");
$Meta3=""
foreach ($el in $r.Meta3){
$Meta3+=$el
}
$Meta3=($Meta3 -split "[`r`n]") -join "`n"
$Meta3=($Meta3 -split "[`n]{2,}") -join "`n"
[Console]::Write("stdout:`n");
[Console]::Write($Meta3);
[Console]::Write("`n");