Sort-Object doesn't remove duplicate strings - Powershell - windows

I have an array of paths and an array of exes I am matching against and getting the name of any exes that match under per path. I know for a fact there are not duplicate entries, and that the found exes only exist under just ONE of the paths in the array. But, when I do Sort-Object -Unique, there are duplicates and they are not removed.
Code:
$found_paths =#("C:\Program Files\Microsoft Office Servers\OFFICE15", "C:\Program Files\Microsoft Office\Office15");
$exes = #("MSOCF.DLL", "access.exe", "word.exe", "wordCnv.exe", "WordViewer.exe", "Excel.exe", "ExcelCnv.exe", "ExcelViewer.exe", "PowerPoint.exe",
"PowerPointViewer.exe", "PowerPointCnv.exe", "Publisher.exe", "Project.exe", "OneNote.exe", "InfoPath.exe Groove.exe", "FrontPage.exe",
"SharePointDesigner.exe", "Visio.exe", "VisioViewer.exe", "Lync.exeOutlook.exe", "WINPROJ.EXE");
foreach($path in $found_paths)
{
foreach($exe in $exes)
{
$found_files = Get-Item ([System.IO.Path]::Combine($path, $exe)) -EA Ignore;
$found_file = $found_files.Name | Sort-Object -Unique;
$found_file
}
}
Output:
MSOCF.DLL
WINPROJ.EXE
MSOCF.DLL
WINPROJ.EXE

This is because you are getting the same binary from a different location ($path) and your sort statement was inside a loop, where files are already unique.
$allFiles = foreach($path in $found_paths)
{
foreach($exe in $exes)
{
Get-Item ([System.IO.Path]::Combine($path, $exe)) -EA Ignore
}
}
$allFiles.Name | Sort-Object -Unique

Related

Active Directory - list full subgroup dependency of a given group and omit subgroups without any user

I have to list full subgroup dependency of specific group - filter -> only subgroups which contain at least 1 user.
I have tried this approach:
dsquery group -samid <specific_group> | dsget group -members -expand | dsquery * -filter "(&(objectclass=group))"
This is an adapted version of my answer here to omit groups without any user object.
Unfortunately, using Get-ADGroupMember together with switch -Recursive will not return members that are groups.
As the docs state:
If the Recursive parameter is specified, the cmdlet gets all members
in the hierarchy of the group that do not contain child objects.
To get an array of nested group objects within a certain parent group, you will need a recursive function like below:
function Get-NestedADGroup {
Param (
[Parameter(Mandatory = $true, ValueFromPipeline = $true, Position = 0)]
[ValidateNotNullOrEmpty()]
[Alias ('Identity')]
[string]$Group,
# the other parameters are optional
[string]$Server = $null,
[string]$SearchBase = $null,
[ValidateSet('Base', 'OneLevel', 'Subtree')]
[string]$SearchScope = 'Subtree'
)
$params = #{
Identity = $Group
SearchScope = $SearchScope
Properties = 'Members'
ErrorAction = 'SilentlyContinue'
}
if (![string]::IsNullOrWhiteSpace($Server)) { $params['Server'] = $Server }
if (![string]::IsNullOrWhiteSpace($SearchBase)) { $params['SearchBase'] = $SearchBase }
$adGroup = Get-ADGroup #params
if ($adGroup) {
if (-not $script:groupsHash.ContainsKey($Group)) {
# output this group object only if it has at least one user object
if (#($adGroup.Members | Where-Object {$_.objectClass -eq 'user'}).Count -gt 0) {
$adGroup
}
# avoid circular group references
$script:groupsHash[$Group] = $true
# and recurse to get the nested groups
foreach ($group in ($adGroup.Members | Where-Object {$_.objectClass -eq 'group'})) {
Get-NestedADGroup -Group $group.DistinguishedName -Server $Server -SearchBase $SearchBase
}
}
}
else {
Write-Warning "Group '$($Group)' could not be found.."
}
}
# create a Hashtable to avoid circular nested groups
$groupsHash = #{}
# call the function
$result = Get-NestedADGroup -Group 'SpecificGroup'
# output just the names if you like
$result.Name
# save to CSV
$result | Export-Csv -Path 'X:\Somewhere\SubgroupsWithAtLeastOneUser.csv' -NoTypeInformation
Here is an easy alternative leveraging Active Directory filtering capabilities. See inline comments to understand the logic.
Do note, this answer requires the ActiveDirectory Module available.
# Get the DN of the parent group (this is the initial group)
$parent = (Get-ADGroup parentGroup).distinguishedName
$param = #{
LDAPFilter = "(memberof:1.2.840.113556.1.4.1941:=$parent)"
Properties = "member"
}
# Find, recursively, all child groups Members Of this parent group
# and filter
Get-ADGroup #param | Where-Object {
# For each member of this child group,
# check if this member is an object of the Class `user`,
# if it is, we already can break the pipeline and return
# this child group, since we already know it has
# at least 1 user member
$_.member | Get-ADObject | Where-Object ObjectClass -EQ user |
Select-Object -First 1
}

How to get the highest value from list of Device Names

Generating a list of windows workstation computer names by reading the active directory and I need to find the highest number so that I can then assign a new device with the next available number - I am not having any success in doing this - how to do it? And as you can see from the list of names, I also have missing numbers in the sequence that ideally, I would like to fill in with new devices also...
The code I am using to get the list from AD is below.
((Get-ADComputer -Filter {operatingsystem -notlike "*server*" -and Name -like $NamingConvention -and enabled -eq "true"} -Credential $credential -server $ADServerIP).Name)
List of device names
PC01
PC28
PC29
PC30
PC31
PC32
PC33
PC34
PC35
PC36
PC37
PC38
PC40
PC41
PC42
PC43
PC44
PC45
PC46
PC47
PC27
PC48
PC26
PC24
PC179
PC18
PC180
PC181
PC182
PC183
PC184
PC185
PC186
PC187
PC188
PC189
PC19
PC190
PC191
PC192
PC21
PC22
PC23
PC25
PC178
PC49
PC51
PC77
PC78
PC79
PC80
PC81
PC83
PC84
PC85
PC87
PC88
PC89
PC90
PC91
PC92
PC93
PC94
PC95
PC96
PC97
PC76
PC50
PC75
PC72
PC52
PC53
PC54
PC55
PC56
PC57
PC59
PC60
PC61
PC62
PC63
PC64
PC65
PC66
PC67
PC68
PC69
PC70
PC71
PC73
PC98
PC177
PC175
PC115
PC116
PC117
PC118
PC119
PC12
PC120
PC121
PC122
PC123
PC124
PC125
PC126
PC127
PC128
PC129
PC13
PC130
PC131
PC114
PC132
PC113
PC111
PC02
PC03
PC04
PC06
PC08
PC09
PC10
PC100
PC101
PC102
PC103
PC104
PC105
PC106
PC107
PC108
PC109
PC11
PC110
PC112
PC176
PC133
PC135
PC158
PC159
PC16
PC160
PC161
PC162
PC163
PC164
PC165
PC166
PC167
PC168
PC169
PC17
PC170
PC171
PC172
PC173
PC174
PC157
PC134
PC156
PC154
PC136
PC137
PC138
PC139
PC14
PC140
PC141
PC142
PC143
PC144
PC145
PC146
PC147
PC148
PC149
PC150
PC151
PC152
PC153
PC155
PC99
Sort the pc names on their numeric values and select the last one:
$lastPC = (Get-ADComputer -Filter {operatingsystem -notlike "*server*" -and Name -like $NamingConvention -and enabled -eq "true"} -Credential $credential -server $ADServerIP).Name |
Sort-Object { [int]($_ -replace '\D+')} | Select-Object -Last 1
Here's a solution that will give you the highest number ($dataMax), the missing numbers ($dataMissing), and the next number to use ($dataNext). The next number to use will be either the 1st missing number, or if there are no missing numbers then it will be the highest number + 1
# load the computers list
$data = ((Get-ADComputer -Filter {operatingsystem -notlike "*server*" -and Name -like $NamingConvention -and enabled -eq "true"} -Credential $credential -server $ADServerIP).Name)
# create an array by splitting the data text using the "space" character as a delimiter
$data = $data.Split(" ")
# remove all the alpha characters ("PC"), leaving only the number values so it can be sorted easier
$dataCleaned = $data -replace "[^0-9]" , '' | sort { [int]$_ }
# after sorting the data, [-1] represents the last element in the array which will be the highest number
[int]$dataMax = $dataCleaned[-1]
# create a number range that represents all the numbers from 1 to the highest number
$range = 1..$dataMax | foreach-object { '{0:d2}' -f $_ }
# compare the created range against the numbers actually in the computer array to find the missing numbers
$dataMissing = #(compare $range $dataCleaned -PassThru)
# if there's a missing value, [0] represents the first element in the array of missing numbers
if ($dataMissing)
{
$dataNext = $dataMissing[0]
}
# if there's no missing values, the next value is the max value + 1
else
{
$dataMissing = "none"
$dataNext = $dataMax + 1
}
Write-Host "The highest number is:"('{0:d2}' -f $dataMax)
Write-Host "The missing numbers are: $dataMissing"
Write-Host "The next number to use is:" ('{0:d2}' -f $dataNext)
Assuming your list is exactly as it appears to be, then this appears to be one way to do it:
$List = 'PC01 PC28 PC29 PC30 PC31 PC32 PC33 PC34 PC35 PC36 PC37 PC38 PC40 PC41 PC42 PC43 PC44 PC45 PC46 PC47 PC27 PC48 PC26 PC24 PC179 PC18 PC180 PC181 PC182 PC183 PC184 PC185 PC186 PC187 PC188 PC189 PC19 PC190 PC191 PC192 PC21 PC22 PC23 PC25 PC178 PC49 PC51 PC77 PC78 PC79 PC80 PC81 PC83 PC84 PC85 PC87 PC88 PC89 PC90 PC91 PC92 PC93 PC94 PC95 PC96 PC97 PC76 PC50 PC75 PC72 PC52 PC53 PC54 PC55 PC56 PC57 PC59 PC60 PC61 PC62 PC63 PC64 PC65 PC66 PC67 PC68 PC69 PC70 PC71 PC73 PC98 PC177 PC175 PC115 PC116 PC117 PC118 PC119 PC12 PC120 PC121 PC122 PC123 PC124 PC125 PC126 PC127 PC128 PC129 PC13 PC130 PC131 PC114 PC132 PC113 PC111 PC02 PC03 PC04 PC06 PC08 PC09 PC10 PC100 PC101 PC102 PC103 PC104 PC105 PC106 PC107 PC108 PC109 PC11 PC110 PC112 PC176 PC133 PC135 PC158 PC159 PC16 PC160 PC161 PC162 PC163 PC164 PC165 PC166 PC167 PC168 PC169 PC17 PC170 PC171 PC172 PC173 PC174 PC157 PC134 PC156 PC154 PC136 PC137 PC138 PC139 PC14 PC140 PC141 PC142 PC143 PC144 PC145 PC146 PC147 PC148 PC149 PC150 PC151 PC152 PC153 PC155 PC99'
$NextNumber = ($List -split "\s" | ForEach-Object { if ($_ -match 'PC(?<Number>\d+)') { $Matches.Number } } | Measure-Object -Maximum).Maximum + 1
$NextNumber
"PC$NextNumber"

How save png as jpg without saving the file in dir

I'm using FromFile to get the image out of files, and it has the following error for the png's on the FromFile line:
Exception calling "FromFile" with "1" argument(s): "The given path's
format is not supported."
So, I'm trying to convert the bmp's to jpg, (see convert line above FromFile below) but all the examples I see (that seem usable) are saving the file. I don't want to save the file in the dir. All I need is the image format, so FromFile can use it like this example. I saw ConvertTo-Jpeg, but I don't think this is a standard powershell module, or don't see how to install it.
I saw this link, but I don't think that would leave the image in the format needed by FromFile.
This is my code:
$imageFile2 = Get-ChildItem -Recurse -Path $ImageFullBasePath -Include #("*.bmp","*.jpg","*.png") | Where-Object {$_.Name -match "$($pictureName)"} #$imageFile | Select-String -Pattern '$($pictureName)' -AllMatches
Write-Host $imageFile2
if($imageFile2.Exists)
{
if($imageFile2 -Match "png")
{
$imageFile2 | .\ConvertTo-Jpeg #I don't think this will work with FromFile below
}
$image = [System.Drawing.Image]::FromFile($imageFile2) step
}
else {
Write-Host "$($imageFile2) does not exist"
}
And then I put it in excel:
$xlsx = $result | Export-Excel -Path $outFilePath -WorksheetName $errCode -Autosize -AutoFilter -FreezeTopRow -BoldTopRow -PassThru # -ClearSheet can't ClearSheet every time or it clears previous data ###left off
$ws = $xlsx.Workbook.Worksheets[$errCode]
$ws.Dimension.Columns #number of columns
$tempRowCount = $ws.Dimension.Rows #number of rows
#only change width of 3rd column
$ws.Column(3).Width
$ws.Column(3).Width = 100
#Change all row heights
for ($row = 2 ;( $row -le $tempRowCount ); $row++)
{
#Write-Host $($ws.Dimension.Rows)
#Write-Host $($row)
$ws.Row($row).Height
$ws.Row($row).Height = 150
#place the image in spreadsheet
#https://github.com/dfinke/ImportExcel/issues/1041 https://github.com/dfinke/ImportExcel/issues/993
$drawingName = "$($row.PictureID)_Col3_$($row)" #Name_ColumnIndex_RowIndex
Write-Host $image
$picture = $ws.Drawings.AddPicture("$drawingName",$image)
$picture.SetPosition($row - 1, 0, 3 - 1, 0)
if($ws.Row($row).Height -lt $image.Height * (375/500)) {
$ws.Row($row).Height = $image.Height * (375/500)
}
if($ws.Column(3).Width -lt $image.Width * (17/120)){
$ws.Column(3).Width = $image.Width * (17/120)
}
}
Update:
I just wanted to reiterate that FromFile can't be used for a png image. So where Hey Scripting Guy saves the image like this doesn't work:
$image = [drawing.image]::FromFile($imageFile2)
I figured out that the $imageFile2 path has 2 filenames in it. It must be that two met the Get-ChildItem/Where-Object/match criteria. The images look identical, but have similar names, so will be easy to process. After I split the names, it does FromFile ok.

Windows Script to consolidate files

I have to work with a huge number of text files. I am able to consolidate the files into one single file. But I also have the use of the file name in my work and I would like to have it before the text of the file itself in excel format, preferably the first column should contain the names of files and the columns afterwards can contain the data.
Any help would be appreciated. Thanks.
Here's the Powershell script. You might need to modify it a bit to look for specific file extensions as now it's only looking for PS1 files
[System.Threading.Thread]::CurrentThread.CurrentCulture = New-Object System.Globalization.CultureInfo("en-US")
$excel = new-Object -comobject Excel.Application
$excel.visible = $false
$workBook = $excel.Workbooks.Add()
$sheet = $workBook.Sheets.Item(1)
$sheet.Name = "Files"
$sheet.Range("A1", "B1").Font.Bold = $true
$sheet.Range("A1","A2").ColumnWidth = 40
$sheet.Range("B1","B2").ColumnWidth = 100
$sheet.Cells.Item(1,1) = "Filename"
$sheet.cells.Item(1,2) = "Content"
$files = get-childitem C:\PST -recurse | where {$_.extension -eq ".ps1"}
$index = 2
foreach($file in $files)
{
$sheet.Cells.Item($index,1) = $file.FullName
$sheet.Cells.Item($index,2) = [System.IO.File]::ReadAllText($file.FullName)
$index++
}
$workBook.SaveAs("C:\PST\1.xlsx")
$excel.Quit()
Note: I'm not pretending that it's perfect, you still need to polish it and refactor it, but at least it will give you direction

How to split a huge folder?

We have a folder on Windows that's ... huge. I ran "dir > list.txt". The command lost response after 1.5 hours. The output file is about 200 MB. It shows there're at least 2.8 million files. I know the situation is stupid but let's focus the problem itself. If I have such a folder, how can I split it to some "manageable" sub-folders? Surprisingly all the solutions I have come up with all involve getting all the files in the folder at some point, which is a no-no in my case. Any suggestions?
Thank Keith Hill and Mehrdad. I accepted Keith's answer because that's exactly what I wanted to do but I couldn't quite get PS working quickly.
With Mehrdad's tip, I wrote this little program. It took 7+ hours to move 2.8 million files. So the initial dir command did finish. But somehow it didn't return to console.
namespace SplitHugeFolder
{
class Program
{
static void Main(string[] args)
{
var destination = args[1];
if (!Directory.Exists(destination))
Directory.CreateDirectory(destination);
var di = new DirectoryInfo(args[0]);
var batchCount = int.Parse(args[2]);
int currentBatch = 0;
string targetFolder = GetNewSubfolder(destination);
foreach (var fileInfo in di.EnumerateFiles())
{
if (currentBatch == batchCount)
{
Console.WriteLine("New Batch...");
currentBatch = 0;
targetFolder = GetNewSubfolder(destination);
}
var source = fileInfo.FullName;
var target = Path.Combine(targetFolder, fileInfo.Name);
File.Move(source, target);
currentBatch++;
}
}
private static string GetNewSubfolder(string parent)
{
string newFolder;
do
{
newFolder = Path.Combine(parent, Path.GetRandomFileName());
} while (Directory.Exists(newFolder));
Directory.CreateDirectory(newFolder);
return newFolder;
}
}
}
I use Get-ChildItem to index my whole C: drive every night into c:\filelist.txt. That's about 580,000 files and the resulting file size is ~60MB. Admittedly I'm on Win7 x64 with 8 GB of RAM. That said, you might try something like this:
md c:\newdir
Get-ChildItem C:\hugedir -r |
Foreach -Begin {$i = $j = 0} -Process {
if ($i++ % 100000 -eq 0) {
$dest = "C:\newdir\dir$j"
md $dest
$j++
}
Move-Item $_ $dest
}
The key is to do the move in a streaming manner. That is, don't collect up all the Get-ChildItem results into a single variable and then proceed. That would require all 2.8 million FileInfos to be in memory at once. Also, if you use the Name parameter on Get-ChildItem it will output a single string containing the file's path relative to the base dir. Even then, perhaps this size will just overwhelm the memory available to you. And no doubt, it will take quite a while to execute. IIRC correctly, my indexing script takes several hours.
If it does work, you should wind up with c:\newdir\dir0 thru dir28 but then again, I haven't tested this script at all so your mileage may vary. BTW this approach assumes that you're huge dir is a pretty flat dir.
Update: Using the Name parameter is almost twice as slow so don't use that parameter.
I found out the GetChildItem is the slowest option when working with many items in a directory.
Look at the results:
Measure-Command { Get-ChildItem C:\Windows -rec | Out-Null }
TotalSeconds : 77,3730275
Measure-Command { listdir C:\Windows | Out-Null }
TotalSeconds : 20,4077132
measure-command { cmd /c dir c:\windows /s /b | out-null }
TotalSeconds : 13,8357157
(with listdir function defined like this:
function listdir($dir) {
$dir
[system.io.directory]::GetFiles($dir)
foreach ($d in [system.io.directory]::GetDirectories($dir)) {
listdir $d
}
}
)
With this in mind, what I would do: I would stay in PowerShell but use more lowlevel approach with .NET methods:
function DoForFirst($directory, $max, $action) {
function go($dir, $options)
{
foreach ($f in [system.io.Directory]::EnumerateFiles($dir))
{
if ($options.Remaining -le 0) { return }
& $action $f
$options.Remaining--
}
foreach ($d in [system.io.directory]::EnumerateDirectories($dir))
{
if ($options.Remaining -le 0) { return }
go $d $options
}
}
go $directory (New-Object PsObject -Property #{Remaining=$max })
}
doForFirst c:\windows 100 {write-host File: $args }
# I use PsObject to avoid global variables and ref parameters.
To use the code you have to switch to .NET 4.0 runtime -- enumerating methods are new in .NET 4.0.
You can specify any scriptblock as -action parameter, so in your case it would be something like {Move-item -literalPath $args -dest c:\dir }.
Just try to list first 1000 items, I hope it will finish very quickly:
doForFirst c:\yourdirectory 1000 {write-host '.' -nonew }
And of course you can process all items at once, just use
doForFirst c:\yourdirectory ([long]::MaxValue) {move-item ... }
and each item should be processed immediately after it is returned. So the whole list is not read at once and then processed, but it is processed during reading.
How about starting with this:
cmd /c dir /b > list.txt
That should get you a list of all the file names.
If you're doing "dir > list.txt" from a powershell prompt, get-childitem is aliased as "dir". Get-childitem has known issues enumerating large directories, and the object collections it returns can get huge.

Resources