Good afternoon Powershell wizards!
I am hoping someone can explain to me how I can fix this issue, and more importantly what the issue actually is!
I'm attempting to fix an old script I wrote years ago that searches for several dates on a files properties and picks one to use for renaming that file.
The issue I'm having is that when I use parseExact it fails for the date strings read from the files... but it works if I manually type the same string into powershell!
Please note that this script is only going to be ran on my PC and only needs to work with dates from my files formats so I'm not too worried about use of $null unless it's related.
See example below:
Write-Host "TEST 1"
$DateTime = [DateTime]::ParseExact("240720211515","ddMMyyyyHHmm",$null)
Write-Host $DateTime # WORKS!
Write-Host "TEST 2"
$DateTime2 = [DateTime]::ParseExact("240720211515","ddMMyyyyHHmm",$null)
Write-Host $DateTime2 # FAILS!
Looks the same right?
Here is a more real world example of what I'm up to that fails
$file = Get-Item "C:\SomeFolder\somefile.jpg"
$shellObject = New-Object -ComObject Shell.Application
$directoryObject = $shellObject.NameSpace( $file.Directory.FullName )
$fileObject = $directoryObject.ParseName( $file.Name )
$property = 'Date taken'
for(
$index = 5;
$directoryObject.GetDetailsOf( $directoryObject.Items, $index ) -ne $property;
++$index) { }
$photoDate = $directoryObject.GetDetailsOf($fileObject, $index)
Write-Host $photoDate # <-- This reads 03/08/2021 09:15
$output = [DateTime]::ParseExact($photoDate,"dd/MM/yyyy HH:mm",$null) # <-- This fails
Write-Host $output
# If i manually type in here it works.... If I copy and paste from the Write-Host it fails...
$someInput = "03/08/2021 09:15"
$workingOutput = [DateTime]::ParseExact($someInput,"dd/MM/yyyy HH:mm",$null)
Write-Host $workingOutput
For anyone else who comes across this, it seems like there are invisible characters being added. Thanks for the spot #SantiagoSquarzon
This fixes it for my particular purposes:
$photoDate = $directoryObject.GetDetailsOf($fileObject, $index)
$utfFree = $photoDate -replace "\u200e|\u200f", ""
I ran into this issue when I started exploring metadata with PowerShell. My solution was to create a regex "inverted whitelist" character class and delete (replace with '') all non-whitelist characters.
But then I learned of an alternate method avaiable to FolderItem obejcts: the ExtenedProperty() method.
Gets the value of a property from an item's property set. The property can be specified either by name or by the property set's format identifier (FMTID) and property identifier (PID).
It's primary strength being the feturn type corresponds to the property value type:
When this method returns, contains the value of the property, if it exists for the specified item. The value will have full typing—for example, dates are returned as dates, not strings.
Using this method to access date properties eliminates string parsing and the issues you encountered:
PS Pictures> $FileInfo = Get-Item "C:\Users\keith\Pictures\Leland\2009\Leland 191.JPG"
PS Pictures> $Shell = New-Object -ComObject shell.application
PS Pictures> $comFolder = $Shell.NameSpace($FileInfo.DirectoryName)
PS Pictures> $comFile = $comFolder.ParseName($FileInfo.Name)
PS Pictures>
PS Pictures> $comFolder.GetDetailsOf($null,12)
Date taken
PS Pictures> $comFolder.GetDetailsOf($comFile,12)
9/5/2009 2:06 PM
PS Pictures> $comFolder.GetDetailsOf($comFile,12).GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True String System.Object
PS Pictures> $comFile.ExtendedProperty("DateTaken")
PS Pictures> $comFile.ExtendedProperty("System.Photo.DateTaken")
Saturday, September 5, 2009 07:06:41 PM
PS Pictures> $comFile.ExtendedProperty("System.Photo.DateTaken").GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True DateTime System.ValueType
It also returns arrays for properties than can contain multiple values (Tags, Contributing Artists, etc/):
PS Pictures> $comFolder.GetDetailsOf($null,18)
Tags
PS Pictures> $comFolder.GetDetailsOf($comFile,18)
Leland; Tim; Jorge
PS Pictures> $comFolder.GetDetailsOf($comFile,18).GetTYpe()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True String System.Object
PS Pictures> $comFile.ExtendedProperty("System.KeyWords")
PS Pictures> $comFile.ExtendedProperty("System.Keywords")
Leland
Tim
Jorge
PS Pictures> $comFile.ExtendedProperty("System.Keywords").GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True String[] System.Array
But GetDetailsOf() is still useful, particularly for values that use an EnumList:
PS Pictures> $comFolder.GetDetailsOf($null,261)
Flash mode
PS Pictures> $comFolder.GetDetailsOf($comFile,261)
No flash, auto
PS Pictures> $comFile.ExtendedProperty("System.Photo.Flash")
24
Yeah something weird is going on with that string. (paste with control v in the console) It doesn't display right in stackoverflow, but ascii only goes up to code 0x7f (127). You can see the 200e and 200f hex codes. They look like little hooks. (emoji's would require extra handling with the surrogate characters)
$string = '# <-- This reads 03/08/2021 09:15'
function chardump {
param($string)
[char[]]$string |
% { [pscustomobject]#{Char = $_; Code = [int]$_ | % tostring x} }
}
chardump $string
Char Code
---- ----
# 23
20
< 3c
- 2d
- 2d
20
T 54
h 68
i 69
s 73
20
r 72
e 65
a 61
d 64
s 73
20
200e
0 30
3 33
/ 2f
200e
0 30
8 38
/ 2f
200e
2 32
0 30
2 32
1 31
20
200f
200e
0 30
9 39
: 3a
1 31
5 35
chardump $string | ? {[int]('0x' + $_.code) -gt 0x7f}
Char Code
---- ----
200e
200e
200e
200f
200e
Related
I'm looking for a command in Windows (either cmd or powershell) which will allow me to get which exact type of bus interface my storage device have, not only the type, but also the revision (SATA3, SATA4, NVME2.0 and so on).
For example:
> wmic diskdrive get InterfaceRevision
InterfaceRevision
SATA3
SATA4
NVME2.0
This will get you the bus but not the version.
Function Get-MaxLength {
<#
.SYNOPSIS
Finds the length of the longest item in collection.
.DESCRIPTION
Use this Function to get the length of the longest item in a
collection for use in format strings or other places where
needed.
.PARAMETER TestObj
The qualified object to be tested. See example!
.Parameter MinLen
The minimum length of the item (if using for formatting) which
should be the Label (title) length. Note if the object item
being tested does not have a Length property you MUST specify
the label length!
.OUTPUTS
Returns a numerical value
.EXAMPLE
$NameLen = Get-MaxLength -TestObj $DotNet.PSChildName
$VerLen = Get-MaxLength -TestObj $DotNet.Version
$RNLen = Get-MaxLength -TestObj $DotNet.Release -MinLen 11
#--- .Net Information ---
$fmtDotNet =
#{Expression={$_.PSChildName};Label=".Net Type";Width=$NameLen},
#{Expression={$_.Version};Label="Version No:";Width=$VerLen},
#{Expression={$_.Release};Label="Release No:";Width=$RNLen}
$Dotnet | Format-Table $fmtDotNet
#>
Param(
[Parameter(Mandatory=$True)]
[object] $TestObj,
[Parameter(Mandatory=$False)]
[int] $MinLen = 0,
[Parameter(Mandatory=$False)]
[int] $MaxLen = 0
)
$ErrorActionPreference = "SilentlyContinue"
foreach ($x in $TestObj) {
If ($x.Trim().length -gt $MinLen) {
$MinLen = $x.Trim().length
}
}
If ($MaxLen -ne 0) {
If ($MinLen -gt $MaxLen) {
$MinLen = $MaxLen
}
}
$ErrorActionPreference = "Continue"
Return ,$MinLen
} #End Function ----------- Get-MaxLength -------------------
Function PhysicalDiskTab {
#Physical Drive Info
$PhyDiskInfo = Get-Disk | Where-Object {$_.size -gt 0 } |
Sort-Object -Property DiskNumber
$SSD = $False
$AMArgs = #{Type = 'NoteProperty'
Name = 'SSD'
Value = 'No'}
$PhyDiskInfo | Add-Member #AMArgs
$AMArgs = #{Type = 'NoteProperty'
Name = 'Speed'
Value = '0'}
$PhyDiskInfo | Add-Member #AMArgs
$GCIArgs = #{
NameSpace = "root\Microsoft\Windows\Storage"
Class = "MSFT_PhysicalDisk"
ErrorAction = "SilentlyContinue"
}
$RotateSpeed = Get-CimInstance #GCIArgs |
Select-Object -Property DeviceID,#{Name="Speed/RPMs";
Expression={(&{If($_.MediaType -eq 0) {[Int]0}
Else {$_.SpindleSpeed/600000 -f "#,###"}})}} |
Sort-Object DeviceID
ForEach ($x in $phydiskinfo) {
ForEach ($Device in $RotateSpeed) {
If ($x.number -eq $Device.DeviceID) {
If ($Device.'Speed/RPMs' -eq 0) {
$SSD = $True
$x.SSD = "Yes"
} #End If
Else {
$x.Speed = $([Int]$Device.'SPeed/RPMs') -f "#,###"
}
} #End If ($x.number...
} #End ForEach ($Device
} #End ForEach $x
$DNLen =
Get-MaxLength -TestObj $PhyDiskInfo.Model -MinLen 4
$SNLen =
Get-MaxLength -TestObj $PhyDiskInfo.SerialNumber -MinLen 13
$fmtPhyDisk1 =
#{Expression={ '{0:N0}' -f $_.Number};
Label="Drive`n No.";Width=5;Align='Center'},
#{Expression={$_.Model};Label="`nName";Width=$DNLen},
#{Expression={$_.SSD};Label="`nSSD";Width=3;Align='left'},
#{Expression={ '{0:#,000.00}' -f ($_.Size/1gb)};
Label="Disk Size`n / GB";Width=9;align='right'},
#{Expression={$_.NumberOfPartitions};
Label="Parti`ntions";Width=5},
#{Expression={$_.PartitionStyle};Label="GPT`nMBR";Width=3},
#{Expression={(&{If ($_.IsBoot) {"Yes"} else {""}})};
Label="`nBoot";Width=5;Align="Left"},
#{Expression={(&{If ($_.BusType.GetType().Name -eq 'UInt16'){
(& {Switch ($_.BusType) {
0 {"Unknown"}
1 {"SCSI"}
2 {"ATAPI"}
3 {"ATA"}
4 {"1394"}
5 {"SSA"}
6 {"Fibre Channel"}
7 {"USB"}
8 {"RAID"}
9 {"iSCSI"}
10 {"SAS"}
11 {"SATA"}
12 {"SD"}
13 {"MMC"}
14 {"MAX"}
15 {"File Backed Virtual"}
16 {"Storage Spaces"}
17 {"NVMe"}
18 {"MS Reserved"}
Default {"Unknown"}}})}
Else{$_.BusType}})};Label="`nData Bus";Width=20}
$fmtPhyDisk2 =
#{Expression={ '{0:N0}' -f $_.Number};
Label="Drive`n No.";Width=5;Align='Center'},
#{Expression={$_.Model};Label="`nName";Width=$DNLen},
#{Expression={$_.SerialNumber.Trim()};
Label="`nSerial Number";Width=$SNLen;Align='left'},
#{Expression={
(&{If ($_.HealthStatus.GetType().Name -eq 'UInt16'){
(& {Switch ($_.HealthStatus) {
0 {"Healthy"}
1 {"Warning"}
2 {"Unhealthy"}
Default {"Unknown"}}})}
Else{$_.HealthStatus}})};Label="`nStatus";Width=7},
#{Expression={$_.Speed};
Label="Rotation`n RPMs ";Width=8;Align='Right'}
$PhyDiskInfo1 = $PhyDiskInfo |
Format-Table -Property $fmtPhyDisk1 -Wrap |
Out-String -Width $OStrWidth
$PhyDiskInfo2 = $PhyDiskInfo |
Format-Table -Property $fmtPhyDisk2 -Wrap |
Out-String -Width $OStrWidth
$PhyDiskTitle = "Physical Disk Information:" | Out-String
Return ,$($PhyDiskTitle + $PhyDiskInfo1 + $PhyDiskInfo2)
} #End Function ---------------- PhysicalDiskTab --------------
$OStrWidth = 79
PhysicalDiskTab
Output:
Physical Disk Information:
Drive Disk Size Parti GPT
No. Name SSD / GB tions MBR Boot Data Bus
----- ----- --- --------- ----- --- ----- ---------
0 Samsung SSD 960 Yes 232.89 4 GPT Yes NVMe
1 Samsung SSD 850 PRO 256GB Yes 238.47 2 GPT SATA
2 Samsung SSD 850 PRO 256GB Yes 238.47 1 GPT SATA
Drive Rotation
No. Name Serial Number Status RPMs
----- ----- -------------- ------- --------
0 Samsung SSD 960 0025_3853_81B0_B2EB. Healthy 0
1 Samsung SSD 850 PRO 256GB S39KNX0J687882W Healthy 0
2 Samsung SSD 850 PRO 256GB S39KNX0J688151N Healthy 0
This code was pulled from my CMsLocalPCInfoW10 program which can be downloaded from my shared OneDrive library.
I am trying to store a text file string which has a beginning and end that make it a substring of the original text file. I am new to Powershell so my methods are simple/crude. Basically my approach has been:
Roughly get what I want from the start of the string
Worry about trimming off what I don't want later
My minimum reproducible example is as follows:
# selectStringTest.ps
$inputFile = Get-Content -Path "C:\test\test3\Copy of 31832_226140__0001-00006.txt"
# selected text string needs to span from $refName up to $boundaryName
[string]$refName = "001 BARTLETT"
[string]$boundaryName = "001 BEECH"
# a rough estimate of the text file lines required
[int]$lines = 200
if (Select-String -InputObject $inputFile -pattern $refName) {
Write-Host "Selected shortened string found!"
# this selects the start of required string but with extra text
[string]$newFileStart = $inputFile | Select-String $refName -CaseSensitive -SimpleMatch -Context 0, $lines
}
else {
Write-Host "Selected string NOT FOUND."
}
# tidy up the start of the string by removing rubbish
$newFileStart = $newFileStart.TrimStart('> ')
# this is the kind of thing I want but it doesn't work
$newFileStart = $newFileStart - $newFileStart.StartsWith($boundaryName)
$newFileStart | Out-File tempOutputFile
As it is: the output begins correctly but I cannot remove text including and after $boundaryName
The original text file is OCR generated (Optical Character Recognition) So it is unevenly formatted. There are newlines in odd places. So I have limited options when it comes to delimiting.
I am not sure my if (Select-String -InputObject $inputFile -pattern $refName)is valid. It appears to work correctly. The general design seems crude. In that I am guessing how many lines I will need. And finally I have tried various methods of trimming the string from $boundaryName without success. For this:
string.split() not practical
replacing spaces with newlines in an array & looping through to elements of $boundaryName is possible but I don't know how to terminate the array at this point before returning it to string.
Any suggestions would be appreciated.
Abbreviated content of x2 200 listings single Copy of 31832_226140__0001-00006.txt file is:
Beginning of text file
________________
BARTLETT-BEDGGOOD
PENCARROW COMPOSITE ROLL
PAGE 6
PAGE 7
PENCARROW COMPOSITE ROLL
BEECH-BEST
www.
.......................
001 BARTLETT. Lois Elizabeth
Middle of text file
............. 15 St Ronans Av. Lower Hutt Marned 200 BEDGGOOD. Percy Lloyd
............15 St Ronans Av, Lower Mutt. Coachbuild
001 BEECH, Margaret ..........
End of text file
..............312 Munita Rood Eastbourne, Civil Eng 200 BEST, Dons Amy .........
..........50 Man Street, Wamuomata, Marned
SO NON
To use a regex across newlines, the file needs to be read as a single string. Get-Content -Raw will do that. This assumes that you do not want the lines containing refName and boundaryName included in the output
$c = Get-Content -Path '.\beech.txt' -Raw
$refName = "001 BARTLETT"
$boundaryName = "001 BEECH"
if ($c -match "(?smi).*$refName.*?`r`n(.*)$boundaryName.*?`r`n.*") {
$result = $Matches[1]
}
$result
More information at https://stackoverflow.com/a/12573413/447901
How close does this come to what you want?
function Process-File {
param (
[Parameter(Mandatory = $true, Position = 0)]
[string]$HeadText,
[Parameter(Mandatory = $true, Position = 1)]
[string]$TailText,
[Parameter(ValueFromPipeline)]
$File
)
Process {
$Inside = $false;
switch -Regex -File $File.FullName {
#'^\s*$' { continue }
"(?i)^\s*$TailText(?<Tail>.*)`$" { $Matches.Tail; $Inside = $false }
'^(?<Line>.+)$' { if($Inside) { $Matches.Line } }
"(?i)^\s*$HeadText(?<Head>.*)`$" { $Matches.Head; $Inside = $true }
default { continue }
}
}
}
$File = 'Copy of 31832_226140__0001-00006.txt'
#$Path = $PSScriptRoot
$Path = 'C:\test\test3'
$Result = Get-ChildItem -Path "$Path\$File" | Process-File '001 BARTLETT' '001 BEECH'
$Result | Out-File -FilePath "$Path\SpanText.txt"
This is the output:
. Lois Elizabeth
............. 15 St Ronans Av. Lower Hutt Marned 200 BEDGGOOD. Percy Lloyd
............15 St Ronans Av, Lower Mutt. Coachbuild
, Margaret ..........
I need to set some attributes for users and I need to be able to add a hexidecimal value for msExchArchiveGUID and my script errors at this point. I took a manual look and there is a drop down box that you choose from Hexidecimal, Binary, Decimal, or Octal and Hexidecimal is what I need for the value. My value for $Hex = 2a 1b ba 59 ff 5e 00 4c 89 b8 2a af fd c1 fc 23
Below is my script and all the values work except for the msExchArchiveGUID
# Import Information
$Information = Import-Csv "c:\scripts\GUIDIssue\UPNList.csv"
# Convert cloud archive GUID to AD Hex format
foreach ($Info in $Information) {
# Get AD Hex from GUID
[system.guid]$guid = $Info.CloudArchiveGUID
$Hex = ($Guid.ToByteArray() | foreach { $_.ToString('x2') }) -Join ' '
<#
Set Values for the following
Msexchrecipientdisplaytype 1073741824
msExchRecipientTypeDetails 1
msExchRemoteRecipientType 3
msexcharchiveguid “$Hex”
msexcharchivestatus 1
mailnickname “$Info.SAM”
#>
Set-ADUser `
-Identity $Info.SAM -Replace #{
Msexchrecipientdisplaytype="1073741824";
msExchRecipientTypeDetails="1";
msExchRemoteRecipientType="3";
msexcharchivestatus="1";
mailnickname=$Info.SAM;
msexcharchiveguid=$Hex
}
# Sync AD
Start-ADSyncSyncCycle -PolicyType Delta
}
I'm trying to write a PowerShell script to do some analysis on production log files.
I need to filter roughly 2.5 million lines of text by roughly 50-100 values - potentially 250,000,000 iterations if O(nm).
I've tried Get-Content | Select-String but this seems incredibly slow.
Is there any way to approach this without iterating over every line once for each value?
EDIT
So, the log files look a bit like this (datetime : process_id : log_level : message)
2016-01-30 14:01:22.349 [ 27] INFO XXX YYY XXXFX
2016-01-30 14:01:28.146 [ 16] INFO XXXD YY Z YYY XXXX
2016-01-30 14:01:28.162 [ 16] DEBUG YY XXXXX YY XX P YYY
2016-01-30 14:01:28.165 [ 16] DEBUG YY XXXXX YY XX YYY
2016-01-30 14:01:28.167 [ 16] DEBUG YY XXXXX YY XX YYY
2016-01-30 14:01:28.912 [ 27] INFO XXX YY XXGXXX YYYYYY YY XX
and I may be looking for the values D, F, G and Z.
The values could be strings of binary digits, hexadecimal digits, combinations of the two, strings of regular text and punctuation, or pipe-delimited values.
Rules of thumb:
StreamReader is faster than Get-Content is faster than Import-Csv.
String operation is faster than wildcard match is faster than regular expression match.
You probably want something like this if it's sufficient to check whether your log lines contain any of the given strings:
$reader = [IO.StreamReader]'C:\path\to\your.log'
$filters = 'foo', 'bar', ...
while ($reader.Peek() -ge 0) {
$line = $reader.ReadLine()
if ($filters | Where-Object {$line.Contains($_)}) {
$line
}
}
$reader.Close()
$reader.Dispose()
If you want to use a StreamWriter instead of just echoing the output simply adjust the code like this:
$reader = [IO.StreamReader]'C:\path\to\your.log'
$writer = [IO.StreamWriter]'C:\path\to\output.txt'
$filters = 'foo', 'bar', ...
while ($reader.Peek() -ge 0) {
$line = $reader.ReadLine()
if ($filters | Where-Object {$line.Contains($_)}) {
$writer.WriteLine($line)
}
}
$reader.Close(); $reader.Dispose()
$writer.Close(); $writer.Dispose()
Depending on the structure of your log lines as well as the filter values and how they need to be applied the filter logic may need adjustments. You need to actually show the log format and filter examples for that, though.
I would try a StreamReader + StreamWriter to speed up reading/writing as Get-Content is slow for big files. Also, I would try to make one regex (word OR word OR word etc.) to avoid hundreds of iterations. Ex:
$words = "foo","bar","donkey"
#Create regex-pattern (usually faster to match)
$regex = ($words | % { [regex]::Escape($_) }) -join '|'
$reader = New-Object System.IO.StreamReader -ArgumentList "c:\myinputfile.txt"
$writer = New-Object System.IO.StreamWriter -ArgumentList "c:\myOUTputfile.txt"
while (($line = $reader.ReadLine()) -ne $null) {
if($line -match $regex) { $writer.WriteLine($line) }
}
#Close writer
$writer.Close()
$writer.Dispose()
#Close reader
$reader.Close()
$reader.Dispose()
The command Get-Process gives output like below:
Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName
------- ------ ----- ----- ----- ------ -- -----------
65 6 1152 840 59 77.50 6048 Appx
78 8 2233 444 61 10.11 7878 Application
but I need a solution like below:
PM(K)=1152, ProcessName=Appx ; PM(K)=2233, ProcessName=Application
How do I parse output like above mentioned?
you can use -f to format your string :
PS>$resu=""
PS>gps | foreach {$resu+=("PM(K)= {0},appName={1};" -f ($_.pm/1KB),$_.name) }
PS>$resu
Try something like this:
$p = Get-Process | select #{n='PM(K)';e={$_.PM/1KB}}, ProcessName
($p | fl | Out-String) -replace "`n`n", ' ; ' -replace "`n", ', '