Changing the output results of Parser in Powershell - performance

So I have a parser that goes through two different logs, both .csv files, and checks for certain lines based off the regex code that I have chosen.
This one grabs the IDNumber from the beginning of the filename(1234-randomfile.csv), then adds the files location to a variable($Validate), then based on the regex, adds files to certain variables($Scriptdone, $Updatedone, $Failed) and starts the checks to see if they have them.
I am trying to make it so that the output is not line for line as the files I parse through have the same IDNumbers. So for example:
Output Currently:
1234 Script Completed
1234 Update Completed
How I want output:
1234 Script Completed Update Completed
Anyways, Thanks for all the assistance!
function Get-MR4RES {
[CmdletBinding()]
param (
[Parameter(Position = 0,
Mandatory = $True)]
[ValidateNotNullorEmpty()]
[ValidateScript( {Test-Path -Path $_ -PathType 'Any'})]
[String]
$Files,
[Parameter(Position = 1,
Mandatory = $false)]
[String]
$CSVPath) # End Param
begin {
# Setting Global Variables
$Scriptcompletedsuccess = '.+Script\scompleted\ssuccessfully.+' # 3:44:15 End function called, Script completed successfully at 3:44:15 on Tue 07/03/2018
$Updatecomplete = '\w+\s+\:\s\[\d+\:\d+\:\d+\]\s+\w+\scomplete' # STATUS : [03:43:07] Update complete
$FailedValidaton = '.+check\sfail.+'
$Fail1 = 'Validation Failed'
$Fail2 = 'Failed'
$Good1 = 'Script completed'
$Good2 = 'Update completed'
$array = #('IDNumber, Results')
$counter = 0
$FileList = (Get-ChildItem -Path $Files -File -Filter "*.log").FullName
$Done = ''
} # End begin
process {
# Do the following code in all the files in the filelist
foreach ($File in $fileList) {
# Test files variables to ensure is directory to ensure progress bar will be operational and needed
if ((Get-Item $Files) -is [System.IO.DirectoryInfo]) {
# Counts once per each file variable in filelist variable
$counter++
# Progress bar indicates the name of the current file and calculates percent based on current count verses total files in $filelist
Write-Progress -Activity 'Analyzing Files' -CurrentOperation $File -PercentComplete (($counter / $FileList.count) * 100)
}
# Calculates ID number based on filename, file name is -filtered in beginning to only contain properly named files
$IDNumber = [System.IO.Path]::GetFileName("$File").split('-')[0]
# Puts file into Variable to be IF Else
$Validate = Get-Content -Path $File
$Scriptdone = $Validate | Where-Object {$_ -match $Scriptcompletedsuccess}
$Updatedone = $Validate | where-object {$_ -match $Updatecomplete}
$Failed = $Validate | Where-Object {$_ -match $FailedValidaton}
# Check if the file HAS a FAILED validation
if($Failed){
# Creates an array of the data from each file that failed
$array += -join ("$IDNumber",', ',"$Fail1")
}
Elseif($Scriptdone){
$Done = $Good1
# Creates an array of the data from each file that script completed
$array += -join ("$IDNumber",', ',"$Done")
} # if the parser found "Update complete"
Elseif($Updatedone){
$Done = $Good2
# Creates an array of the data from each file that update is done
$array += -join ("$IDNumber",', ',"$Done")
} # End of Successful
Else{
# Creates an array of the data from each file that failed
$array += -join ("$IDNumber",', ',"$Fail2")
}
} # End of foreach
} # End process section
End {
# If CSVPath is used in get-command
if ($PSBoundParameters.ContainsKey('CSVPath')) {
# Pipe the array data to a CSV
Add-Content -Path $CSVPath -Value $array -Encoding ascii
}
# If no CSVPath is used in get-command
else {
# Out-put to console
Write-Output $array
} # End of else
} # End of the End
} # End of function

If you want to append new message to existing output you have to tell PowerShell to which entry it should add new info. As manipulating strings is not very intuitive in my opinion I'd suggest to use an object for that.
First you have to define data structure:
// Before ForEach
$array = #()
$properties = #{'ID'="";
'Results'=""}
// In ForEach
$object = New-Object –TypeName PSObject –Prop $properties
$object.ID = $IDNumber
Next, in your if you can set the value (this can also be done using Switch as suggested by #LotPings but let's leave it as it is for simplicity):
$object.Results = $Done // or $Fail or $Fail2
Then you should first check if the entry with such $ID already exists and if yes, add new result. If no, just add new element to the array. Something like this should work:
$line = $array | Where-Object ID -eq $object.id
if ($line) {
$line.Results += " $($object.Results)"
}
else {
$array += $object
}
Of course this will also require changing the way as you output you data (for example by using Export-Csv):
$array | Export-Csv $CSVPath -Append -NoTypeInformation

Related

robocopy adds hidden symbol when creating folders

What I do, is copying photo files from SD card to HDD using powershell ps1 file and Windows PowerShell ISE.
I get a taken date from image exif and add it to destination path.
The problem is that robocopy creates folders and adds strange prefix, which I do not want to have.
As a result I can see two subfolders with same name "2020", one folder created by hand and the other created by robocopy.
This prefix is only seen when I list folders with CMD.
The prefix not seen in output.log and in powershell.
$copy_from = "G:\DCIM\100MSDCF\"
$copy_to = "C:\Photos\"
function GetDateTaken {
param (
[Parameter(ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true)]
[Alias('FullName')]
[String]
$Path
)
begin {
$shell = New-Object -COMObject Shell.Application
}
process {
$returnvalue = 1 | Select-Object -Property Name, DateTaken, Folder
$returnvalue.Name = Split-Path $path -Leaf
$returnvalue.Folder = Split-Path $path
$shellfolder = $shell.Namespace($returnvalue.Folder)
$shellfile = $shellfolder.ParseName($returnvalue.Name)
$returnvalue.DateTaken = $shellfolder.GetDetailsOf($shellfile, 12)
$returnvalue.DateTaken
}
}
$file = Get-ChildItem -Path $copy_from -recurse -include ('*.jpg','*.arw')
$i = 0
$jpg = 0
$arw = 0
$logifile = 'output.log'
if ([System.IO.File]::Exists($logifile)) {
Clear-Content $logifile
Write-Host ("Logfile cleaned: $logifile")
} else {
try {
New-Item -Path . -Name $logifile | Out-Null
Write-Host ("New logfile created: $logifile")
}
catch {
"Failed to create $logifile"
}
}
foreach ($file in $file) {
if ($file.extension -eq '.JPG') { $jpg++ }
if ($file.extension -eq '.ARW') { $arw++ }
$i++
$datetaken = ($file.fullname | GetDateTaken).Split(' ')[0]
$datetaken_Day = $datetaken.Split('.')[0]
$datetaken_Month = $datetaken.Split('.')[1]
$datetaken_Year = $datetaken.Split('.')[2]
$TargetPath = "$copy_to$datetaken_Year\$datetaken_Month\$datetaken_Day\"
Write-Host ("$i. " + $file.Name + " `tDate taken: " + $datetaken)
robocopy $copy_from $TargetPath $file.Name /ts /fp /v /np /unilog+:$logifile | Out-Null
}
Write-Host ("`nTotal: " + $i + " files (" + $jpg + " JPG files, " + $arw + " ARW files)")
Not helps if write $TargetPath = $copy_to + $datetaken_Year + "\" + $datetaken_Month + "\" + $datetaken_Day + "\".
Not helps if I set /fat option to robocopy.
But, for example, when I set a year manualy, everything is ok $datetaken_Year = 2020
What should be fixed to create correct folder names?
Using the GetDetailsOf() method from the COM object returns localized results, which leads to your function on my Dutch machine returning the date in 'dd-MM-yyyy HH:mm' format (with invisible characters surrounding it).
A better approach IMO would be to get the date taken using System.Drawing.Imaging.Metafile to read the exif data as null-terminated byte array and parse the date from that as DateTime object using below function:
function Get-ExifDate {
# returns the 'DateTimeOriginal' property from the Exif metadata in an image file if possible
[CmdletBinding(DefaultParameterSetName = 'ByName')]
Param (
[Parameter(Mandatory = $true, ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true, Position = 0, ParameterSetName = 'ByName')]
[Alias('FullName', 'FileName')]
[ValidateScript({ Test-Path -Path $_ -PathType Leaf})]
[string]$Path,
[Parameter(Mandatory = $true, ValueFromPipeline = $true, Position = 0, ParameterSetName = 'ByObject')]
[System.IO.FileInfo]$FileObject
)
Begin {
Add-Type -AssemblyName 'System.Drawing'
}
Process {
# the function received a path, not a file object
if ($PSCmdlet.ParameterSetName -eq 'ByName') {
$FileObject = Get-Item -Path $Path -Force -ErrorAction SilentlyContinue
}
# Parameters for FileStream: Open/Read/SequentialScan
$streamArgs = #(
$FileObject.FullName
[System.IO.FileMode]::Open
[System.IO.FileAccess]::Read
[System.IO.FileShare]::Read
1024, # Buffer size
[System.IO.FileOptions]::SequentialScan
)
try {
$stream = New-Object System.IO.FileStream -ArgumentList $streamArgs
$metaData = [System.Drawing.Imaging.Metafile]::FromStream($stream)
# get the 'DateTimeOriginal' property (ID = 36867) from the metadata
# Tag Dec TagId Hex TagName Writable Group Notes
# ------- --------- ------- -------- ----- -----
# 36867 0x9003 DateTimeOriginal string ExifIFD (date/time when original image was taken)
# get the date taken as an array of bytes
$exifDateBytes = $metaData.GetPropertyItem(36867).Value
# transform to string, but beware that this string is Null terminated, so cut off the trailing 0 character
$exifDateString = [System.Text.Encoding]::ASCII.GetString($exifDateBytes).TrimEnd("`0")
# return the parsed date
return [datetime]::ParseExact($exifDateString, "yyyy:MM:dd HH:mm:ss", $null)
}
catch{
Write-Warning -Message "Could not read Exif data from '$($FileObject.FullName)'"
}
finally {
If ($metaData) {$metaData.Dispose()}
If ($stream) {$stream.Close()}
}
}
}
Another option would be to download and unzip ExifTool
(you can download the zip files from here)
Then use it like:
$exifTool = 'Path\To\Unzipped\ExifTool.exe' # don't forget to 'Unblock' after downloading
$file = 'Path\To\The\ImageFile' # fullname
# retrieve all date tags in the file
# -s2 (or -s -s) return short tag name add the colon directly after that
$allDates = & $exifTool -time:all -s2 $file
# try to find a line with tag 'DateTimeOriginal', 'CreateDate' or 'ModifyDate'
# which will show a date format of 'yyyy:MM:dd HH:mm:ss'
# and parse a DateTime object out of this string
$dateTaken = switch -Regex ($allDates) {
'^(?:DateTimeOriginal|CreateDate|ModifyDate):\s(\d{4}:\d{2}:\d{2} \d{2}:\d{2}:\d{2})' {
[datetime]::ParseExact($matches[1], 'yyyy:MM:dd HH:mm:ss', $null)
break
}
}
Short explanation of what the above returns
Both methods return the date the image was taken as a DateTime object, not a string.
This object has properties like .Year, .Month, .Day etc. It also has various methods like .AddDays(), .ToShortDateString(), .ToString() and a lot more.
If you do $datetaken = ($datetaken -split ' ')[0] as per your comment, you are asking PowerShell to implicitely convert it to a string using the default ToString() method.
You can use that ToString() method in your code if you give it the formatting string you need in between the brackets, anyway you like.
If you for instance do $dateTaken.ToString('yyyy\\MM\\dd'), you'll get a string 2020\10\08 if $dateTaken was today, which could serve as part of a file path.
In your code, you could do:
$TargetPath = Join-Path -Path $copy_to -ChildPath $dateTaken.ToString('yyyy\\MM\\dd')
# if that path does not exist yet, create it
if (!(Test-Path -Path $TargetPath -PathType Container)) {
$null = New-Item -Path $TargetPath -ItemType Directory
}
Then go ahead and copy the file to the now existing $TargetPath
Please have a look at all the standard format strings and custom format specifiers you can use on a DateTime object.

Powershell to Audit Folder Permissions Showing Group Membership

Background: I've been using Netwrix to audit permissions to network shares for a few years now and It's only ever worked smoothly 1 time..... So I've decided to move on to just an automated powershell script. I've run into a block. When I try to parse out the group members, it doesn't like the network name in front of the group name (TBANK). Then I also need to take the next step of just showing the name instead of the whole output of get-adgroupmember. Any help would be greatly appreciated as I'm very to to scripting with powershell. Current script below:
$OutFile = "C:\users\user1\Desktop\test.csv" # Insert folder path where you want to save your file and its name
$Header = "Folder Path,IdentityReference, Members,AccessControlType,IsInherited,InheritanceFlags,PropagationFlags"
$FileExist = Test-Path $OutFile
If ($FileExist -eq $True) {Del $OutFile}
Add-Content -Value $Header -Path $OutFile
$Folder = "\\server1.tbank.com\share1"
$ACLs = get-acl $Folder | ForEach-Object { $_.Access }
Foreach ($ACL in $ACLs){
$ID = $ACL.IdentityReference
$ID = $ID -replace 'TBANK\' , ''
$ACType = $ACL.AccessControlType
$ACInher = $ACL.IsInherited
$ACInherFlags = $ACL.InheritanceFlags
$ACProp = $ACL.PropagationFlags
$Members = get-adgroupmember $ID.
$OutInfo = $Folder + "," + $ID + "," + $Members + "," + $ACType + "," + $ACInher + "," + $ACInherFlags + "," + $ACProp
Add-Content -Value $OutInfo -Path $OutFile
}
First of all, there is a way better way to output a CSV file than by trying to write each row yourself (with the risk of missing out required quotes), called Export-Csv.
To use that cmdlet, you wil need to create an array of objects which is not hard to do.
$OutFile = "C:\users\user1\Desktop\test.csv" # Insert folder path where you want to save your file and its name
$Folder = "\\server1.tbank.com\share1"
# get the Acl.Access for the folder, loop through and collect PSObjects in variable $result
$result = (Get-Acl -Path $Folder).Access | ForEach-Object {
# -replace uses regex, so you need to anchor to the beginning of
# the string with '^' and escape the backslash by doubling it
$id = $_.IdentityReference -replace '^TBANK\\' # remove the starting string "TBANK\"
# Get-ADGroupMember can return users, groups, and computers. If you only want users, do this:
# $members = (Get-ADGroupMember -Identity $id | Where-Object { $_.objectClass -eq 'user'}).name -join ', '
$members = (Get-ADGroupMember -Identity $id).name -join ', '
# output an onbject with all properties you need
[PsCustomObject]#{
'Folder Path' = $Folder
'IdentityReference' = $id
'Members' = $members
'AccessControlType' = $_.AccessControlType
'IsInherited' = $_.IsInherited
'InheritanceFlags' = $_.InheritanceFlags -join ', '
'PropagationFlags' = $_.PropagationFlags -join ', '
}
}
# output on screen
$result | Format-List
# output to CSV file
$result | Export-Csv -Path $OutFile -Force -UseCulture -NoTypeInformation
I've added a lot of inline comments to hopefully make things clear for you.
The -UseCulture switch in the Export-Csv line makes sure the field delimiter that is used matches what is set in your system as list separator. This helps when opening the csv file in Excel.
P.S> the Get-ADGroupMember cmdlet also has a switch called -Recursive. With that, it will also get the members from groups inside groups

Replacing multiple lines in a file

I try to make, the line from the first array is read from a file and is replaced with a line from the second array, so some times with different lines. I made a script, but I do not understand why it does not work.
$OldStrings = #(
"desktopwidth:i:1440",
"desktopheight:i:900",
"winposstr:s:0,1,140,60,1596,999"
)
$NewStrings = #(
"desktopwidth:i:1734",
"desktopheight:i:990",
"winposstr:s:0,1,50,7,1800,1036"
)
$LinesArray = Get-Content -Path 'C:\temp\My Copy\Default.rdp'
$LinesCount = $LinesArray.Count
for ($i=0; $i -lt $LinesCount; $i++) {
foreach ($OldString in $OldStrings) {
foreach ($NewString in $NewStrings) {
if ($LinesArray[$i] -like $OldString) {
$LinesArray[$i] = $LinesArray[$i] -replace $OldString, $NewString
Write-Host "`nline" $i "takes on value:" $LinesArray[$i] "`n" -ForegroundColor Gray
}
}
}
}
The file is probably why it is not read at all.
After executing the script, I see only
line 2 takes on value: desktopwidth:i:1734
line 3 takes on value: desktopwidth:i:1734
line 5 takes on value: desktopwidth:i:1734
You're looking through the string arrays twice. You want to do two loops, one for each line in the file AND another for each count in the lines you're replacing. I think this should work:
$OldStrings = #(
"desktopwidth:i:1440",
"desktopheight:i:900",
"winposstr:s:0,1,140,60,1596,999"
)
$NewStrings = #(
"desktopwidth:i:1734",
"desktopheight:i:990",
"winposstr:s:0,1,50,7,1800,1036"
)
$LinesArray = Get-Content -Path 'C:\temp\My Copy\Default.rdp'
# loop through each line
for ($i=0; $i -lt $LinesArray.Count; $i++)
{
for ($j=0;$j -lt $OldStrings.Count; $j++)
{
if ($LinesArray[$i] -match $OldStrings[$j])
{
$LinesArray[$i] = $LinesArray[$i] -replace $OldStrings[$j],$NewStrings[$j]
Write-Host "`nline" $i "takes on value:" $LinesArray[$i] "`n" -ForegroundColor Gray
}
}
}
$LinesArray | Set-Content -Path 'C:\temp\My Copy\Default.rdp'
You don't need to bother checking the lines to look for matches. Since you have the replacements ready just do the replacements outright anyway. Should be faster this way as well.
$stringReplacements = #{
"desktopwidth:i:1440" = "desktopwidth:i:1734"
"desktopheight:i:900" = "desktopheight:i:990"
"winposstr:s:0,1,140,60,1596,999" = "winposstr:s:0,1,50,7,1800,1036"
}
$path = 'C:\temp\My Copy\Default.rdp'
# Read the file in as a single string.
$fileContent = Get-Content $path | Out-String
# Iterate over each key value pair
$stringReplacements.Keys | ForEach-Object{
# Attempt the replacement for each key/pair search/replace pair
$fileContent =$fileContent.Replace($_,$stringReplacements[$_])
}
# Write changes back to file.
# $fileContent | Set-Content $path
$stringReplacements is a key value hash of search and replace strings. I don't see you writing the changes back to file so I left a line on the end for you to uncomment.
You could add in checks to do the replacements still if you value the write-host lines but I figured that was for debugging and you already know how to do that.

Powershell - Speeding up writing to files

I wrote this script to find all of the folders in a directory and for each folder, check inside a common file if some strings exist and if not add them. I needed to insert strings in particular places. Not really knowing how to do this, I opted for simpler find and replace where the strings needed to be inserted. Anyway this script takes almost an hour to work through 800 files. I'm hoping some experienced members can point out ways to make my task quicker as I have only been working with Powershell for two days. Many Thanks!!!
# First find and replace items.
$FindOne =
$ReplaceOneA =
$ReplaceOneB =
$ReplaceOneC =
# Second find and replace items.
$FindTwo =
$ReplaceTwo =
# Strings to test if exist.
# To avoid duplicate entries.
$PatternOne =
$PatternTwo =
$PatternThree =
$PatternFour =
# Gets window folder names.
$FilePath = "$ProjectPath\$Station\WINDOW"
$Folders = Get-ChildItem $FilePath | Where-Object {$_.mode -match "d"}
# Adds folder names to an array.
$FolderName = #()
$Folders | ForEach-Object { $FolderName += $_.name }
# Adds code to each builder file.
ForEach ($Name in $FolderName) {
$File = "$FilePath\$Name\main.xaml"
$Test = Test-Path $File
# First tests if file exists. If not, no action.
If ($Test -eq $True) {
$StringOne = Select-String -pattern $PatternOne -path $File
$StringTwo = Select-String -pattern $PatternTwo -path $File
$StringThree = Select-String -pattern $PatternThree -path $File
$StringFour = Select-String -pattern $PatternFour -path $File
$Content = Get-Content $File
# If namespaces or object don't exist, add them.
If ($StringOne -eq $null) {
$Content = $Content -Replace $FindOne, $ReplaceOneA
}
If ($StringTwo -eq $null) {
$Content = $Content -Replace $FindOne, $ReplaceOneB
}
If ($StringThree -eq $null) {
$Content = $Content -Replace $FindOne, $ReplaceOneC
}
If ($StringFour -eq $null) {
$Content = $Content -Replace $FindTwo, $ReplaceTwo
}
$Content | Set-Content $File
}
}
# End of program.
You could try writing to the file with a stream, like this
$stream = [System.IO.StreamWriter] $File
$stream.WriteLine($content)
$stream.close()

Checking if registry value is equal to 1 is not working correctly

I have slopped together bits of PowerShell to remote query a list of machines, stored in a .csv file, for a registry value. If the registry key's value is equal to '1', the script should then create a text file using the machine's name as the name of the text file.
Everything works great. The script runs happily without any errors. The problem is that when I go back and remotely check a targeted registry value, I find that the value isn't 1. The script is simply creating a file for every line in the .csv.
What am I doing wrong?
EDIT*** I found a problem I had a typo in the $key variable for the registry path. 7/17/2013 2:21p
$File = Import-Csv 'c:\temp\machines.csv'
foreach ($line in $file)
{
$machinename = $line.machinename
trap [Exception] {continue}
$reg = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey("LocalMachine",$MachineName)
$key = "SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\WinLogon"
$regkey = ""
$regkey = $reg.opensubkey($key)
$keyValue = ""
$keyValue = $regKey.GetValue('AutoAdminLogon')
if ($keyValue = "1")
{
try
{
$textFile = New-Item -Path "c:\temp\autologin" -Name $MachineName -ItemType "File"
}
catch
{
$msg = $_
$msg
}
}
$Results = $MachineName , $keyValue
Write-host $Results
#Output Below Here:
}
In PowerShell = is an assignment operator, not a comparison operator. Change this line:
if ($keyValue = "1")
into this:
if ($keyValue -eq "1")
For more information see Get-Help about_Operators.
You're making this way too complicated, BTW. Something like this should suffice:
$keyname = 'SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\WinLogon'
Import-Csv 'C:\temp\machines.csv' | % {
$reg = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey("LocalMachine",
$_.machinename)
$key = $reg.OpenSubkey($keyname)
$value = $key.GetValue('AutoAdminLogon')
if ($value -eq "1") {
$filename = Join-Path "c:\temp\autologin" $_.machinename
try {
touch $filename
$textFile = Get-Item $filename
} catch {
$_
}
}
Write-Host ($_.machinename , $value)
}

Resources