Sort lines within file based on folder in path - sorting

I have a file with below lines
c:\scripts\oltp\db1\scripts\scripts1.sql
c:\scripts\oltp\db1\tables\scripts1.sql
c:\scripts\oltp\db1\storedprocedures\scripts1.sql
c:\scripts\oltp\db1\functions\scripts1.sql
c:\scripts\oltp\db1\tables\scripts2.sql
c:\scripts\oltp\db1\storedprocedures\scripts2.sql
I am looking for a powershell script that can sort based on list as mentioned below...
Tables,storedprocedures,views,scripts,everything else.....
My expected output is
c:\scripts\oltp\db1\tables\scripts1.sql
c:\scripts\oltp\db1\tables\scripts2.sql
c:\scripts\oltp\db1\storedprocedures\scripts1.sql
c:\scripts\oltp\db1\storedprocedures\scripts2.sql
c:\scripts\oltp\db1\scripts\scripts1.sql
c:\scripts\oltp\db1\functions\scripts1.sql

Sort-Object -Property can take anonymous calculated properties as its argument. Put a switch inside the expression based on your sorting criteria:
function Test-CustomSort {
$ScriptNames = #(
'c:\scripts\oltp\db1\scripts\scripts1.sql'
'c:\scripts\oltp\db1\tables\scripts1.sql'
'c:\scripts\oltp\db1\storedprocedures\scripts1.sql'
'c:\scripts\oltp\db1\functions\scripts1.sql'
'c:\scripts\oltp\db1\tables\scripts2.sql'
'c:\scripts\oltp\db1\storedprocedures\scripts2.sql'
)
$ScriptNames | Sort-Object #{Expression={
switch(Split-Path -Parent $_ | Split-Path -Leaf){
"tables" { 1 }
"storedprocedures" { 2 }
"views" { 3 }
"scripts" { 4 }
default { 5 }
}
}}
}
Produces:
PS C:\> Test-CustomSort
c:\scripts\oltp\db1\tables\scripts1.sql
c:\scripts\oltp\db1\tables\scripts2.sql
c:\scripts\oltp\db1\storedprocedures\scripts1.sql
c:\scripts\oltp\db1\storedprocedures\scripts2.sql
c:\scripts\oltp\db1\scripts\scripts1.sql
c:\scripts\oltp\db1\functions\scripts1.sql

You could group the paths by directory, put them into a hashtable, then output the hashtable in the desired order.
$categories = 'tables', 'storedprocedures', 'views', 'scripts', 'functions'
$ht = #{}
Get-Content 'C:\path\to\your.txt' |
Group-Object { Split-Path -Parent $_ | Split-Path -Leaf } |
ForEach-Object { $ht[$_.Name] = $_.Group }
$categories | ForEach-Object {
$ht[$_] | Sort-Object
}

If the path is always the same : c:\scripts\oltp\db1\
(gc .\list.txt) | %{$_ -replace 'c:\\scripts\\oltp\\db1\\', ''} | Sort
Output
functions\scripts1.sql
scripts\scripts1.sql
storedprocedures\scripts1.sql
storedprocedures\scripts2.sql
tables\scripts1.sql
tables\scripts2.sql

Related

Is there a robocopy switch for gathering timestamps longer than 260 characters?

I am attempting to extract the date last modified from the files in a Windows directory. Here is my basic script:
Function Get-FolderItem {
[cmdletbinding(DefaultParameterSetName='Filter')]
Param (
[parameter(Position=0,ValueFromPipeline=$True,ValueFromPipelineByPropertyName=$True)]
[Alias('FullName')]
[string[]]$Path = $PWD,
[parameter(ParameterSetName='Filter')]
[string[]]$Filter = '*.*',
[parameter(ParameterSetName='Exclude')]
[string[]]$ExcludeFile,
[parameter()]
[int]$MaxAge,
[parameter()]
[int]$MinAge
)
Begin {
$params = New-Object System.Collections.Arraylist
$params.AddRange(#("/L","/E","/NJH","/NDL","/BYTES","/FP","/NC","/XJ","/R:0","/W:0","T:W","/TS","/UNILOG:c:\temp\test.txt"))
#params.AddRange(#("/L","/S","/NJH","/BYTES","/FP","/NC","/NDL","/TS","/XJ","/R:0","/W:0"))
If ($PSBoundParameters['MaxAge']) {
$params.Add("/MaxAge:$MaxAge") | Out-Null
}
If ($PSBoundParameters['MinAge']) {
$params.Add("/MinAge:$MinAge") | Out-Null
}
}
Process {
ForEach ($item in $Path) {
Try {
$item = (Resolve-Path -LiteralPath $item -ErrorAction Stop).ProviderPath
If (-Not (Test-Path -LiteralPath $item -Type Container -ErrorAction Stop)) {
Write-Warning ("{0} is not a directory and will be skipped" -f $item)
Return
}
If ($PSBoundParameters['ExcludeFile']) {
$Script = "robocopy `"$item`" NULL $Filter $params /XF $($ExcludeFile -join ',')"
} Else {
$Script = "robocopy `"$item`" NULL $Filter $params"
}
Write-Verbose ("Scanning {0}" -f $item)
Invoke-Expression $Script | Out-Null
get-content "c:\temp\test.txt" | ForEach {
Try {
If ($_.Trim() -match "^(?<Children>\d+)\s(?<FullName>.*)") {
$object = New-Object PSObject -Property #{
FullName = $matches.FullName
Extension = $matches.fullname -replace '.*\.(.*)','$1'
FullPathLength = [int] $matches.FullName.Length
FileHash = Get-FileHash -LiteralPath "\\?\$($matches.FullName)" |Select -Expand Hash
Created = ([System.IO.FileInfo] $matches.FullName).creationtime
LastWriteTime = ([System.IO.FileInfo] $matches.FullName).LastWriteTime
Characters = (Get-Content -LiteralPath "\\?\$($matches.FullName)" | Measure-Object -ignorewhitespace -Character).Characters
Owner = (Get-ACL $matches.Fullname).Owner
}
$object.pstypenames.insert(0,'System.IO.RobocopyDirectoryInfo')
Write-Output $object
} Else {
Write-Verbose ("Not matched: {0}" -f $_)
}
} Catch {
Write-Warning ("{0}" -f $_.Exception.Message)
Return
}
}
} Catch {
Write-Warning ("{0}" -f $_.Exception.Message)
Return
}
}
}
}
$a = Get-FolderItem "C:\TargetDirectory\Folder" | Export-Csv -Path C:\Temp\output.csv -Encoding Unicode
The script extracts the date last modified of filepaths less than 260 characters. It returns a nonsense date of 1600-12-31 4:00:00 PM for files longer than 260 characters. Here is the line that is not working:
LastWriteTime = ([System.IO.FileInfo] $matches.FullName).LastWriteTime
My first attempt to solve this problem was to find a command that began with Get- because such commands were useful in extracting filehashes, filepaths, character counts and owner names of files longer than 260 characters. For example:
Owner = (Get-ACL $matches.Fullname).Owner
Characters = (Get-Content -LiteralPath "\\?\$($matches.FullName)" | Measure-Object-ignorewhitespace -Character).Characters
FileHash = Get-FileHash -LiteralPath "\\?\$($matches.FullName)" |Select -Expand Hash
Get-Date however seemed to be about getting the current date.
In my second attempt, I went back to Boe Prox's original blogpost on this script and noticed that his script had two components that were missing from mine:
a robocopy switch /TS
Date = [datetime]$matches.Date
I added to my script however doing so return an error: WARNING: Cannot convert null to type "System.DateTime". I rechecked the file in the directory, and it clearly has a date.
I reexamined the documentation on Get-Date and tried
Date = Get-Date -Format o | ForEach-Object { $matches -replace ":", "." }
However, this returned WARNING: Cannot convert value "2018/03/05 18:06:54 C:TargetDirectory\Folder\Temp.csv to type "System.IO.FileInfo". Error: " Illegal characters in path."
(N.B. In other posts, people have suggested changing the server settings to permit the existence of files longer than 260 characters. This is not an option for me because I do not have access to the servers.)
Once you hit 260 characters in the path, you hit the old Windows MAX_PATH limitation. In order to get around that, you have to prepend your path with \\?\.
In your code above, you do that for Characters and FileHash but you don't do that when retrieving LastWriteTime. e.g. Changing the path to this will work:
Created = ([System.IO.FileInfo] "\\?\$($matches.FullName)").creationtime
LastWriteTime = ([System.IO.FileInfo] "\\?\$($matches.FullName)").LastWriteTime
The alternative way is to use the Get-ChildItem cmdlet along with \\?\ prepended to the path to retrieve most of the fields you want without having to query it multiple times:
get-content "c:\temp\test.txt" | ForEach {
Try {
If ($_.Trim() -match "^(?<Children>\d+)\s(?<FullName>.*)") {
$file = Get-ChildItem "\\?\$($matches.FullName)"
$object = New-Object PSObject -Property #{
FullName = $file.FullName
Extension = $file.Extension
FullPathLength = $file.FullName.Length
FileHash = Get-FileHash -LiteralPath "\\?\$($matches.FullName)" |Select -Expand Hash
Created = $file.CreationTime
LastWriteTime = $file.LastWriteTime
Characters = (Get-Content -LiteralPath "\\?\$($matches.FullName)" | Measure-Object -ignorewhitespace -Character).Characters
Owner = (Get-ACL $matches.Fullname).Owner
}
$object.pstypenames.insert(0,'System.IO.RobocopyDirectoryInfo')
Write-Output $object
} Else {
Write-Verbose ("Not matched: {0}" -f $_)
}
} Catch {
Write-Warning ("{0}" -f $_.Exception.Message)
Return
}
}

Powershell Script isn't returning anything (Running on MAC OS)

I've been messing with this powershell script (i installed powershell on my mac OS) I also modified the code a bit in the first line.
I am not getting any errors, just nothing happens.
$folder = “/Users/mbp/Desktop/nier_unpacked_2_extracted“
$files = gci -recurse $folder | where { ! $_.PSIsContainer }
$fileContents = $files | foreach { gc -encoding utf8 $_.fullname }
$lines = $fileContents | foreach { if ($_ -match "^JP: (.*)$") { $matches[1] } }
$chars = $lines | foreach { $_.ToCharArray() }
$groups = $chars | group-object
$totals = $groups | sort-object -desc -property count
Basically outputting japanese text characters and how often they show up.
This is the original code(before modification):
$folder = "F:\nier_unpacked_2_extracted"
$files = gci -recurse $folder | where { ! $_.PSIsContainer }
$fileContents = $files | foreach { gc -encoding utf8 $_.fullname }
$lines = $fileContents | foreach { if ($_ -match "^JP: (.*)$") { $matches[1] } }
$chars = $lines | foreach { $_.ToCharArray() }
$groups = $chars | group-object
$totals = $groups | sort-object -desc -property count
Here is the link to the resource i got the code from if that helps: https://dev.to/nyctef/extracting-game-text-from-nier-automata-1gm0
I'm not sure why nothing is returning unfortunately.
In PowerShell (as in most other programming languages), $totals = ... means that you assign the result of the expression at the right side is assigned to the variable ($totals) at the left side.
To display the contents of the variable ($totals), you might use the Write-Output $totals, Write-Host $totals, Out-Defualt $totals, along with a lot of other output cmdlets.
Anyways, in PowerShell, it is generally not necessary to use a cmdlet in instances where the output is displayed by default. For example:
$totals Enter

Sort a PowerShell hash table on a property of the value

I'm having an issue sorting a hash table. I've broken down my code to just bare necessities so as not to overwhelm anyone with my original script.
Write-Host "PowerShell Version = " ([string]$psversiontable.psversion)
$h = #{}
$Value = #{SortOrder=1;v1=1;}
$h.Add(1, $Value)
$Value = #{SortOrder=2;v1=1;}
$h.Add(2, $Value)
$Value = #{SortOrder=3;v1=1;}
$h.Add(3, $Value)
$Value = #{SortOrder=4;v1=1;}
$h.Add(4, $Value)
Write-Host "Ascending"
foreach($f in $h.GetEnumerator() | Sort-Object Value.SortOrder)
{
Write-Host $f.Value.SortOrder
}
Write-Host "Descending"
foreach($f in $h.GetEnumerator() | Sort-Object Value.SortOrder -descending)
{
Write-Host $f.Value.SortOrder
}
The output is
PowerShell Version = 3.0
Ascending
2
1
4
3
Descending
2
1
4
3
I'm sure this is just a simple case of not knowing the correct usage of Sort-Object. The sort works correctly on Sort-Object Name so maybe it has something to do with not knowing how to handle the Value.SortOrder?
Sort-Object accepts a property name or a script block used to sort. Since you're trying to sort on a property of a property, you'll need to use a script block:
Write-Host "Ascending"
$h.GetEnumerator() |
Sort-Object { $_.Value.SortOrder } |
ForEach-Object { Write-Host $_.Value.SortOrder }
Write-Host "Descending"
$h.GetEnumerator() |
Sort-Object { $_.Value.SortOrder } -Descending |
ForEach-Object { Write-Host $_.Value.SortOrder }
You can filter using the Where-Object cmdlet:
Write-Host "Ascending"
$h.GetEnumerator() |
Where-Object { $_.Name -ge 2 } |
Sort-Object { $_.Value.SortOrder } |
ForEach-Object { Write-Host $_.Value.SortOrder }
You usually want to put Where-Object before any Sort-Object cmdlets, since it makes sorting faster.
I was using a hash table as a frequency table, to count the occurrence of words in filenames.
$words = #{}
get-childitem *.pdf | foreach-object -process {
$name = $_.name.substring($_.name.indexof("-") + 1, $_.name.indexof(".") - $_.name.indexof("-") - 1)
$name = $name.replace("_", " ")
$word = $name.split(" ")[0]
if ( $words.contains($word) ){
$words[$word] = $words[$word] + 1
}else{
$words.add($word, 1)
}
}
$words.getenumerator() | sort-object -property value
It's that last line that does the magic, sorting the hash table on the value(frequency).

PowerShell scripts every developer should know [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
Windows PowerShell is out a quite long time now. In comparison to the the good old windows shell it's much more powerful.
Are there any scripts you use to speed up and simplify your every day work as an developer? If you can do magic with PowerShell -> please share it with us!
Update
Not really a script, but also very useful are PowerShell Community Extensions. The package contains a lot of new Cmdlets and PowerShell modifications.
I put together a bunch of scripts to work with Subversion at the command line. Most of them just use the --xml option to put various information in object form. Here are a couple of examples:
function Get-SvnStatus( [string[]] $Path = ".",
[string] $Filter = "^(?!unversioned|normal|external)",
[switch] $NoFormat )
{
# powershell chokes on "wc-status" and doesn't like two definitions of "item"
[xml]$status = ( ( Invoke-Expression "svn status $( $Path -join ',' ) --xml" ) -replace "wc-status", "svnstatus" ) `
-replace "item=", "itemstatus="
$statusObjects = $status.status.target | Foreach-Object { $_.entry } | Where-Object {
$_.svnstatus.itemstatus -match $Filter
} | Foreach-Object {
$_ | Select-Object #{ Name = "Status"; Expression = { $_.svnstatus.itemstatus } },
#{ Name = "Path"; Expression = { Join-Path ( Get-Location ) $_.path } }
} | Sort-Object Status, Path
if ( $NoFormat )
{
$statusObjects
}
else
{
$statusObjects | Format-Table -AutoSize
}
}
function Get-SvnLog( [string] $Path = ".",
[int] $Revision,
[int] $Limit = -1,
[switch] $Verbose,
[switch] $NoFormat )
{
$revisionString = ""
$limitString = ""
$verboseString = ""
if ( $Revision )
{
$revisionString = "--revision $Revision"
}
if ( $Limit -ne -1 )
{
$limitString = "--limit $Limit"
}
if ( $Verbose )
{
$verboseString = "--verbose"
}
[xml]$log = Invoke-Expression "svn log $( $path -join ',' ) --xml $revisionString $limitString $verboseString"
$logObjects = $log.log.logentry | Foreach-Object {
$logEntry = $_
$logEntry | Select-Object `
#{ Name = "Revision"; Expression = { [int]$logEntry.revision } },
#{ Name = "Author"; Expression = { $logEntry.author } },
#{ Name = "Date";
Expression = {
if ( $NoFormat )
{
[datetime]$logEntry.date
}
else
{
"{0:dd/MM/yyyy hh:mm:ss}" -f [datetime]$logEntry.date
}
} },
#{ Name = "Message"; Expression = { $logEntry.msg } } |
Foreach-Object {
# add the changed path information if the $Verbose parameter has been specified
if ( $Verbose )
{
$_ | Select-Object Revision, Author, Date, Message,
#{ Name = "ChangedPaths";
Expression = {
$paths = $logEntry.paths.path | Foreach-Object {
$_ | Select-Object `
#{ Name = "Change";
Expression = {
switch ( $_.action )
{
"A" { "added" }
"D" { "deleted" }
"M" { "modified" }
"R" { "replaced" }
default { $_.action }
}
} },
#{ Name = "Path"; Expression = { $_."#text" } }
}
if ( $NoFormat )
{
$paths
}
else
{
( $paths | Sort-Object Change | Format-Table -AutoSize | Out-String ).Trim()
}
}
}
}
else
{
$_
}
}
}
if ( $NoFormat )
{
$logObjects
}
else
{
$logObjects | Format-List
}
}
I have these aliased to svns and svnl, respectively. I talk about a few others here.
I use this one all the time because Windows Explorer's search for file contents never works for me:
Get-ChildItem -Recurse -Filter *.extension |
Select-String -List somestring |
Format-Table filename,linenumber -AutoSize
Just replace "extension" with the file extension of the file type you're interested in (or remove the -Filter parameter entirely) and replace "somestring" with the text you want to find in the file.
It's not a script, but in general it's helpful to learn when you can short-cut parameters, both by name and position.
By name, PowerShell just needs enough to narrow it down to one. For example, gci -r works but gci -f might be either -filter or -force.
Values specified without a parameter label are applied positionally. So if you want to specify -filter you could either do this:
gci -r -fil *.cs
Or provide . positionally as -path so you can also specify -filter positionally:
gci -r . *.cs
Any time you see something with proper capitalization, it's an indication I've used TAB completion. You should learn which things PS will complete for you -- it's quite good in V2.
Any time you see aliases in lowercase, it's something I typed from memory. You should memorize it too.
# grep example - find all using statements
dir -r -fil *cs | ss using
# advanced version
dir -fil *cs -r | ss '^using[^\(]+' | gpv line | sort -unique
# figure out how to query for drive free space (emphasis on "figure out" -- I can never remember things like this)
gcm *drive*
help Get-PSDrive -full
Get-PSDrive | gm
# now use it
Get-PSDrive | ? { $_.free -gt 1gb }
# pretend mscorlib.dll is an assembly you're developing and want to do some ad-hoc testing on
$system = [system.reflection.assembly]::LoadFile("c:\blah\...\mscorlib.dll")
$system | gm
$types = $a.GetTypes()
$types | gm
$types | ? { $_.ispublic -and $_.basetype -eq [system.object] } | sort name
$sbType = $types | ? { $_.name -eq "StringBuilder" }
# now that we've loaded the assembly, we could have also done:
# $sbType = [system.text.stringbuilder]
# but we may not have known it was in the Text namespace
$sb = new-object $sbType.FullName
$sb | gm
$sb.Append("asdf")
$sb.Append("jkl;")
$sb.ToString()

du in PowerShell?

How can I get a du-ish analysis using PowerShell? I'd like to periodically check the size of directories on my disk.
The following gives me the size of each file in the current directory:
foreach ($o in gci)
{
Write-output $o.Length
}
But what I really want is the aggregate size of all files in the directory, including subdirectories. Also I'd like to be able to sort it by size, optionally.
There is an implementation available at the "Exploring Beautiful Languages" blog:
"An implementation of 'du -s *' in Powershell"
function directory-summary($dir=".") {
get-childitem $dir |
% { $f = $_ ;
get-childitem -r $_.FullName |
measure-object -property length -sum |
select #{Name="Name";Expression={$f}},Sum}
}
(Code by the blog owner: Luis Diego Fallas)
Output:
PS C:\Python25> directory-summary
Name Sum
---- ---
DLLs 4794012
Doc 4160038
include 382592
Lib 13752327
libs 948600
tcl 3248808
Tools 547784
LICENSE.txt 13817
NEWS.txt 88573
python.exe 24064
pythonw.exe 24576
README.txt 56691
w9xpopen.exe 4608
I modified the command in the answer slightly to sort descending by size and include size in MB:
gci . |
%{$f=$_; gci -r $_.FullName |
measure-object -property length -sum |
select #{Name="Name"; Expression={$f}},
#{Name="Sum (MB)";
Expression={"{0:N3}" -f ($_.sum / 1MB) }}, Sum } |
sort Sum -desc |
format-table -Property Name,"Sum (MB)", Sum -autosize
Output:
PS C:\scripts> du
Name Sum (MB) Sum
---- -------- ---
results 101.297 106217913
SysinternalsSuite 56.081 58805079
ALUC 25.473 26710018
dir 11.812 12385690
dir2 3.168 3322298
Maybe it is not the most efficient method, but it works.
If you only need the total size of that path, one simplified version can be,
Get-ChildItem -Recurse ${HERE_YOUR_PATH} | Measure-Object -Sum Length
function Get-DiskUsage ([string]$path=".") {
$groupedList = Get-ChildItem -Recurse -File $path | Group-Object directoryName | select name,#{name='length'; expression={($_.group | Measure-Object -sum length).sum } }
foreach ($dn in $groupedList) {
New-Object psobject -Property #{ directoryName=$dn.name; length=($groupedList | where { $_.name -like "$($dn.name)*" } | Measure-Object -Sum length).sum }
}
}
Mine is a bit different; I group all of the files on directoryname, then walk through that list building totals for each directory (to include the subdirectories).
Building on previous answers, this will work for those that want to show sizes in KB, MB, GB, etc., and still be able to sort by size. To change units, just change "MB" to desired units in both "Name=" and "Expression=". You can also change the number of decimal places to show (rounding), by changing the "2".
function du($path=".") {
Get-ChildItem $path |
ForEach-Object {
$file = $_
Get-ChildItem -File -Recurse $_.FullName | Measure-Object -Property length -Sum |
Select-Object -Property #{Name="Name";Expression={$file}},
#{Name="Size(MB)";Expression={[math]::round(($_.Sum / 1MB),2)}} # round 2 decimal places
}
}
This gives the size as a number not a string (as seen in another answer), therefore one can sort by size. For example:
PS C:\Users\merce> du | Sort-Object -Property "Size(MB)" -Descending
Name Size(MB)
---- --------
OneDrive 30944.04
Downloads 401.7
Desktop 335.07
.vscode 301.02
Intel 6.62
Pictures 6.36
Music 0.06
Favorites 0.02
.ssh 0.01
Searches 0
Links 0
My own take using the previous answers:
function Format-FileSize([int64] $size) {
if ($size -lt 1024)
{
return $size
}
if ($size -lt 1Mb)
{
return "{0:0.0} Kb" -f ($size/1Kb)
}
if ($size -lt 1Gb)
{
return "{0:0.0} Mb" -f ($size/1Mb)
}
return "{0:0.0} Gb" -f ($size/1Gb)
}
function du {
param(
[System.String]
$Path=".",
[switch]
$SortBySize,
[switch]
$Summary
)
$path = (get-item ".").FullName
$groupedList = Get-ChildItem -Recurse -File $Path |
Group-Object directoryName |
select name,#{name='length'; expression={($_.group | Measure-Object -sum length).sum } }
$results = ($groupedList | % {
$dn = $_
if ($summary -and ($path -ne $dn.name)) {
return
}
$size = ($groupedList | where { $_.name -like "$($dn.name)*" } | Measure-Object -Sum length).sum
New-Object psobject -Property #{
Directory=$dn.name;
Size=Format-FileSize($size);
Bytes=$size`
}
})
if ($SortBySize)
{ $results = $results | sort-object -property Bytes }
$results | more
}

Resources