Hi I have been trying to find a way to help me estimate how long it will take to move databases from one location to another. My online research has helped me through a few issues so far but I seem to be stuck because I have it using what seems to be the correct commands to see all files that would need to be counted but it comes back on a 5TB database as it will only take 22 milliseconds so either I have a faster network and server that I even knew or I screwed this up some how that I cannot see.
$item = get-childitem 'D:\SQL01' -Recurse
$d = "E:\SQL01"
$results = #()
$results = Foreach ($i in $item) {
Measure-Command -Expression {
Copy-Item -literalpath $i $d
}
}
($results | Measure-Object -Property TotalSeconds -Sum).Sum
$results -f "c"
Reading over this it seems fine and it even returns a the sum of time but there is no way that is accurate. Please leave a comment if anyone sees where I did something wrong or you think there is something I could try differently.
Here is an example of Write-Progress in action:
# Get all directories on D:\SQL01 recursively
$directories = Get-ChildItem 'D:\SQL01' -Directory -Recurse
# Set a destination folder
$destination = 'E:\SQL01'
$dirCount = $directories.count
$i=0;foreach($directory in $directories)
{
$progress = #{
Activity = "Copying - {0}" -f $directory.FullName
Status = "Folder $i of $dirCount"
PercentComplete = $i++ / $dirCount * 100
}
Write-Progress #progress
Copy-Item -Path $directory -Destination $destination -Recurse
}
Giving this little script in powershell:
$index = 1
$file = "C:\Users\myfile"
while ($index -le 100000)
{
$init_size = Write-Host((Get-Item $file).length/1KB)
<here is my command which populates $file>
$final_size = Write-Host((Get-Item $file).length/1KB)
$index ++
sleep 5
If ($final_size -eq $init_size) {break}
}
I don't understand why it breaks even if the init_size is different from the final_size.
Any suggestions?
Write-Host writes directly to the screen buffer and doesn't output anything, so the value of both $init_size and $final_size are effectively $null when you reach the if statement.
Do Write-Host $variable after assigning to $variable and it'll work:
$index = 1
$file = "C:\Users\myfile"
while ($index -le 100000) {
$init_size = (Get-Item $file).Length / 1KB
Write-Host $init_size
<here is my command which populates $file>
$final_size = (Get-Item $file).Length / 1KB
Write-Host $final_size
$index++
sleep 5
If ($final_size -eq $init_size) { break }
}
Calling Write-Host on the results of the assignment expression itself would work too:
Write-Host ($init_size = (Get-Item $file).Length / 1KB)
Running Powershell V4.0. Below is script to search through 1000's of files by date stamp in filename in given date range and find all matching strings. However Get-ChildItem cmdlet is extremely slow. I'm new to Powershell. Any way to make this more efficient? Perhaps read in batches?
######################################################
#Date ranges and filename
$startdate = [datetime]'01/26/2017'
$enddate = [datetime] '02/05/2017'
$Filename = "DACNBA0124IDT030"
######################################################
#Progress
######################################################
$array =
do {
$startdate.ToString('yyyy_MM_dd*')
$startdate = $startdate.AddDays(1)
}
until ($startdate -gt [datetime] $enddate)
$files = $array | foreach-object {"G:\Live Engineering\AsRuns\Test\$_*"}
write-host $files
$Matches = get-childitem $files -recurse -force -OutBuffer 20000 | select- string $Filename | Where -Verbose {$_.line -notlike '*.mxf'}
$Matches.Matches.Count > "C:\Users\user\Desktop\report app\Log.txt"
I have a helper function which I use it everyday at my work. I have the available in my GitHub repo.
I have two cmdlets in it which does the same and more...
hope will help you .
https://github.com/kvprasoon/PSReadScript
Feel free to post issues/Bugs and request there
regards,
Kvprasoon
I'm trying to format large text files (~300MB) between 0 to 3 columns :
12345|123 Main St, New York|91110
23456|234 Main St, New York
34567|345 Main St, New York|91110
And the output should be:
000000000012345,"123 Main St, New York",91110,,,,,,,,,,,,
000000000023456,"234 Main St, New York",,,,,,,,,,,,,
000000000034567,"345 Main St, New York",91110,,,,,,,,,,,,
I'm new to powershell, but I've read that I should avoid Get-Content so I am using StreamReader. It is still much too slow:
function append-comma{} #helper function to append the correct amount of commas to each line
$separator = '|'
$infile = "\large_data.csv"
$outfile = "new_file.csv"
$target_file_in = New-Object System.IO.StreamReader -Arg $infile
If ($header -eq 'TRUE') {
$firstline = $target_file_in.ReadLine() #skip header if exists
}
while (!$target_file_in.EndOfStream ) {
$line = $target_file_in.ReadLine()
$a = $line.split($separator)[0].trim()
$b = ""
$c = ""
if ($dataType -eq 'ECN'){$a = $a.padleft(15,'0')}
if ($line.split($separator)[1].length -gt 0){$b = $line.split($separator)[1].trim()}
if ($line.split($separator)[2].length -gt 0){$c = $line.split($separator)[2].trim()}
$line = $a +',"'+$b+'","'+$c +'"'
$line -replace '(?m)"([^,]*?)"(?=,|$)', '$1' |append-comma >> $outfile
}
$target_file_in.close()
I am building this for other people on my team and wanted to add a gui using this guide:
http://blogs.technet.com/b/heyscriptingguy/archive/2014/08/01/i-39-ve-got-a-powershell-secret-adding-a-gui-to-scripts.aspx
Is there a faster way to do this in Powershell?
I wrote a script using Linux bash(Cygwin64 on Windows) and a separate one in Python. Both ran much faster, but I am trying to script something that would be "approved" on a Windows Platform.
All that splitting and replacing costs you way more time than you gain from the StreamReader. Below code cut execution time to ~20% for me:
$separator = '|'
$infile = "\large_data.csv"
$outfile = "new_file.csv"
if ($header -eq 'TRUE') {
$linesToSkip = 1
} else {
$linesToSkip = 0
}
Get-Content $infile | select -Skip $linesToSkip | % {
[int]$a, [string]$b, [string]$c = $_.split($separator)
'{0:d15},"{1}",{2},,,,,,,,,,,,,' -f $a, $b.Trim(), $c.Trim()
} | Set-Content $outfile
How does this work for you? I was able to read and process a 35MB file in about 40 seconds on a cheap ole workstation.
File Size: 36,548,820 bytes
Processed In: 39.7259722 seconds
Function CheckPath {
[CmdletBinding()]
param(
[Parameter(Mandatory=$True,
ValueFromPipeline=$True)]
[string[]]$Path
)
BEGIN {}
PROCESS {
IF ((Test-Path -LiteralPath $Path) -EQ $False) {Write-host "Invalid File Path $Path"}
}
END {}
}
$infile = "infile.txt"
$outfile = "restult5.txt"
#Check File Path
CheckPath $InFile
#Initiate StreamReader
$Reader = New-Object -TypeName System.IO.StreamReader($InFile);
#Create New File Stream Object For StreamWriter
$WriterStream = New-Object -TypeName System.IO.FileStream(
$outfile,
[System.IO.FileMode]::Create,
[System.IO.FileAccess]::Write);
#Initiate StreamWriter
$Writer = New-Object -TypeName System.IO.StreamWriter(
$WriterStream,
[System.Text.Encoding]::ASCII);
If ($header -eq $True) {
$Reader.ReadLine() |Out-Null #Skip First Line In File
}
while ($Reader.Peek() -ge 0) {
$line = $Reader.ReadLine() #Read Line
$Line = $Line.split('|') #Split Line
$OutPut = "$($($line[0]).PadLeft(15,'0')),`"$($Line[1])`",$($Line[2]),,,,,,,,,,,,"
$Writer.WriteLine($OutPut)
}
$Reader.Close();
$Reader.Dispose();
$Writer.Flush();
$Writer.Close();
$Writer.Dispose();
$endDTM = (Get-Date) #Get Script End Time For Measurement
Write-Host "Elapsed Time: $(($endDTM-$startDTM).totalseconds) seconds" #Echo Time elapsed
Regex is fast:
$infile = ".\large_data.csv"
gc $infile|%{
$x=if($_.indexof('|')-ne$_.lastindexof('|')){
$_-replace'(.+)\|(.+)\|(.+)',('$1,"$2",$3'+','*12)
}else{
$_-replace'(.+)\|(.+)',('$1,"$2"'+','*14)
}
('0'*(15-($x-replace'([^,]),.+','$1').length))+$x
}
I have another approach. Let powershell read the input file as a csv file, with a pipe character as delimiter. Then format the output the way you want it. I have not tested this for speed with large files.
$infile = "\large-data.csv"
$outfile = "new-file.csv"
import-csv $infile -header id,addr,zip -delimiter "|" |
% {'{0},"{1}",{2},,,,,,,,,,,,,' -f $_.id.padleft(15,'0'), $_.addr.trim(), $_.zip} |
set-content $outfile
I have created the below script, which has been tested and working successfully, to amend the 'Last Modified Date' of all files contained in the selected folder.
$a = Get-Date "22/11/2012 10:00 AM"
$b = Get-ChildItem "C:\MyFiles"
foreach ($i in $b)
{
$i.LastWriteTime = $a
$a = $a.AddMinutes(1)
}
$b
I am just looking for some help amending this script to include all subfolders/file within the selected folder, as I am currently having to amend this manually to change the date within the sub-folders of "C:\MyFiles".. for e.g. "C:\MyFiles\A", "C:\MyFiles\B".. etc.
In addition.. I was also wondering how I can remove the line "$a = Get-Date "22/11/2012 10:00 AM" so that it automatically sets the date to todays, and I do not have to enter a date manually.
like this?
$a = get-date
$b = Get-ChildItem "C:\MyFiles" -recurse | ? { !$_.psiscontainer }
foreach ($i in $b)
{
$i.LastWriteTime = $a
$a = $a.AddMinutes(1)
}
$b
or if you don't need to add a minute after each file:
$dir = read-host "Insert path"
$b = Get-ChildItem $dir -recurse | ? { !$_.psiscontainer }
foreach ($i in $b)
{
$i.LastWriteTime = get-date
}
$b