I want to move all pictures from several folders to one destination folder, if they listed in my txt-file.
The script works, but there are about 81k pictures and 450k names (eg samlpe-green-bigpic-detail-3.jpg) in the txt-file, it is damn slow.
Is there a way to script it, so it works faster?
$qpath = "c:\sample\picz\"
$Loggit = "c:\sample\pic_move.log"
$txtZeileU = "c:\sample\names.txt"
$d_pic = "C:\sample\moved_picz"
$arrZeileU = Get-Content -Path $txtZeileU
foreach ($Zeile in $arrZeileU) {
Get-ChildItem -Path $qpath -Recurse |
where {$_.Name –eq $Zeile} |
Move-Item -Destination $d_pic -Verbose -Force *>&1 |
Out-File -FilePath $Loggit -Append
}
Related
I want to search all drives using PowerShell on windows machine to get the list of all files along with their extensions -
Based on desired extension we pass in it like - *.mp3 or
Fetch all files with multiple extensions like - *.txt, *.mp3 etc.
I tried below script but its giving only information from where we are running it. But I want to scan whole machine.
Get-ChildItem -Path .\ -Filter ***.doc** -Recurse -File| Sort-Object Length -Descending | ForEach-Object { $_.BaseName }
Checkout the Get-PSDrive cmdlet. It returns a list of drives, and you can specify just disk drives with the -PSProvider FileSystem parameter:
foreach ( $drive in $(Get-PSDrive -PSProvider FileSystem) ) {
Get-ChildItem -Path $drive.Root -Filter ***.doc** -Recurse -File |
Sort-Object Length -Descending |
ForEach-Object { $_.BaseName }
}
Didn't test that but you get the idea.
Using -Include on Get-ChildItem will allow you to specify a list of extensions. The -ErrorAction will cause it to skip drives that are not available such as an unmounted CD drive.
Get-PSDrive -PSProvider FileSystem |
ForEach-Object {
Get-ChildItem -Path $_.Root -Recurse -Include '*.doc*', '*.txt' -ErrorAction SilentlyContinue |
ForEach-Object { $_.Name }
} |
ForEach-Object {[PSCustomObject]#{HashCode = $_.GetHashCode(); FullName = $_.FullName}}
} |
Export-Csv -Path $TempFile -NoTypeInformation -Encoding ASCII
Update:
Here is a better way. It will prevent unknown extensions from getting into the mix such as "Microsoft.NET.Sdk.Publish.Docker.targets."
$ExtensionList = #('.txt', '.doc', '.docx', '.mp3')
$TempFile = Join-Path -path $Env:TEMP -ChildPath "$($pid.ToString()).tmp"
Get-PSDrive -PSProvider FileSystem |
ForEach-Object {
Get-ChildItem -Path $_.Root -Recurse -ErrorAction SilentlyContinue |
Where-Object { $ExtensionList -contains $_.Extension } |
ForEach-Object {
[PSCustomObject]#{
HashCode = $_.GetHashCode();
DirectoryName = $_.DirectoryName
Name = $_.Name
}
}
} |
Export-Csv -Path $TempFile -Delimiter ';' -NoTypeInformation -Encoding ASCII
Write-Host "The temp file is $TempFile"
This is more than what the original question asked, but if you are going to go through the trouble of listing all your files, I suggest getting the filehash as well so you can determine if you have duplicates. A simple file name search will not detect if the same file has been saved with a different name. Adding to what #lit (https://stackoverflow.com/users/447901/lit) has posted:
$ExtensionList = #('.txt', '.doc', '.docx', '.mp3')
Get-PSDrive -PSProvider FileSystem |
ForEach-Object {
Get-ChildItem -Path $_.Root -Recurse -ErrorAction SilentlyContinue |
Where-Object { $ExtensionList -eq $_.Extension } |
## ForEach-Object { $_.Name, $_.FullName, $_.GetHashCode() }
Select-Object #{Name="Name";Expression={$_.Name}}, #{Name="Hash";Expression={$_.GetHashCode()}}, #{Name="FullName";Expression={$_.FullName}} |
Export-Csv -Path C:\Temp\testing.csv -NoTypeInformation -Append
}
The addition of the file hash will allow you to see if you have duplicates and the full name will allow you to see where they are located.
I am trying to write a script that copies the latest modified log file from one server to another server (servers are in different domains), while copying it should check for the credentials and then execute the script.
Please let me know if the script is correct or any corrections to be made.
$sourcePath = 'sourcepath'
$destPath = 'Destinationpath'
$compareDate = (Get-Date).AddDays(-1);
$LastFileCaptured = Get-ChildItem -Path $sourcePath |
where {$_.Extension.EndsWith('.log') -and $_.LastWriteTime -gt $compareDate } |
Sort LastAccessTime -Descending |
select -First 1 |
select -ExcludeProperty Name, LastAccessTime
Write-Host $LastFileCaptured.Name
$LastFileCaptured.LastAccessTime
$LastFileCaptured = Get-ChildItem -Recurse |
Where-Object{$_.LastWriteTime.AddDays(-1) -gt (Get-Date)}
Write-Host $LastFileCaptured
Get-ChildItem $sourcePath -Recurse -Include '.log' | Where-Object {
$_.LastWriteTime.AddDays(-1).ToString("yyyy/MM/dd") -gt (get-date).ToString("yyyy/mm/dd")
} | ForEach-Object {
$destDir = Split-Path ($_.FullName -replace [regex]::Escape($sourcePath), $destPath)
if (!(Test-Path $destDir)) {
New-Item -ItemType directory $destDir | Out-Null
}
Copy-Item $_ -Destination $destDir
}
The "correctness" of your script is determined easily by running it! But, while this isn't a direct answer, I would suggest robocopy for this task.
In particular note these options:
/mon: Monitors the source, and runs again when more than N changes are detected.
/maxage: Specifies the maximum file age (to exclude files older than N days or date).
I am looking for the simplest and least intrusive way to monitor the progress of md5 fingerprinting of large drives, many files (8 TB, 2 million).
What would be the best option, for example in case it gets stuck or begins an infinite loop, I can see the trouble file?
The code:
Get-childitem -recurse -file | select-object #{n="Hash";e={get-filehash -algorithm MD5 -path $_.FullName | Select-object -expandproperty Hash}},lastwritetime,length,fullname | export-csv "$((Get-Date).ToString("yyyyMMdd_HHmmss"))_filelistcsv_MD5_LWT_size_path_file.csv" -notypeinformation
aaaa
If you want to list progress, you need to know where your process will end, so you need to list all the files BEFORE you start operating on them.
Write-Host "Listing Files..." -Fore Yellow
$AllFiles = Get-ChildItem -Recurse -File
$CurrentFile = 0 ; $TotalFiles = $AllFiles.Count
Write-Host "Hashing Files..." -Fore Yellow
$AllHashes = foreach ($File in $AllFiles){
Write-Progress -Activity "Hashing Files" -Status "$($CurrentFile)/$($TotalFiles) $($File.FullName)" -PercentComplete (($CurrentFile++/$TotalFiles)*100)
[PSCustomObject]#{
File = $File.FullName
Hash = (Get-FileHash -LiteralPath $File.FullName -Algorithm MD5).Hash
LastWriteTime = $File.LastWriteTime
Size = $File.Length
}
}
$AllHashes | Export-Csv "File.csv" -NoTypeInformation
This will give you a nice header with a progress bar, which looks like this:
ISE:
Normal Shell:
I have a file transfer/sync job that is copying files from the main network into a totally secure network using a custom protocol (ie no SMB). The problem is that because I can't look back to see what files exist, the destination is filling up, as the copy doesn't remove any files it hasn't touched (like robocopy MIR does).
Initailly I wrote a script that:
1. Opens the log file and grabs the file paths out (this is quite quick and painless)
2. Does a Get-ChildItem on the destination folder (now using dir /s /b as it's way faster than gci)
3. Compared the two, and then removed the differences.
The problem is that there are more jobs that require this clean up but the log files are 100MB and the folders contain 600,000 files, so it's taking ages and using tons of memory. I actually have yet to see one finish. I'd really like some ideas on how to make this faster (memory/cpu use doesn't bother me too much but speed is essential.
$destinationMatch = "//server/fileshare/folder/"
the log file contains some headers and footers and then 600,000 lines like this one:
"//server/fileshare/folder/dummy/deep/tags/20140826/more_stuff/Deeper/2012-07-02_2_0.dat_v2" 33296B 0B completed
Here's the script:
[CmdletBinding(SupportsShouldProcess=$True)]
param(
[Parameter(Mandatory=$True)]
[String]$logName,
[Parameter(Mandatory=$True)]
[String]$destinationMatch
)
$logPath = [string]("C:\Logs\" + $logName)
$manifestFile = gci -Path $logPath | where {$_.name -match "manifest"} | sort creationtime -descending | select Name -first 1
$manifestFileName = [string]$manifestFile.name
$manifestFullPath = $logPath + "\" + $manifestFileName
$copiedList = #()
(gc $manifestFullPath -ReadCount 0) | where {$_.trim() -match $DestinationMatch} | % {
if ( $_ -cmatch '(?<=")[^"]*(?=")' ){
$copiedList += ($matches[0]).replace("/","\")
}
}
$dest = $destinationMatch.replace("/","\")
$actualPathString = (gci -Path $dest -Recurse | select fullname).fullnameCompare-Object -ReferenceObject $copiedList -DifferenceObject $actualPathString -PassThru | % {
$leaf = Split-Path $_ -leaf
if ($leaf.contains(".")){
$fsoData = gci -Path $_
if (!($fsoData.PSIsContainer)){
Remove-Item $_ -Force
}
}
}
$actualDirectory | where {$_.PSIsContainer -and #(gci -LiteralPath $_.FullName -Recurse -WarningAction SilentlyContinue -ErrorAction SilentlyContinue | where {!$_.PSIsContainer}).Length -eq 0} | remove-item -Recurse -Force
Ok, so let's assume that your file copy preserves the last modified date/time stamp. If you really need to pull a directory listing, and compare it against a log, I think you're doing a decent job of it. The biggest slow down is obviously going to be pulling your directory listing. I'll address that shortly. For right now I would propose the following modification of your code:
[CmdletBinding(SupportsShouldProcess=$True)]
param(
[Parameter(Mandatory=$True)]
[String]$logName,
[Parameter(Mandatory=$True)]
[String]$destinationMatch
)
$logPath = [string]("C:\Logs\" + $logName)
$manifestFile = gci -Path $logPath | where {$_.name -match "manifest"} | sort creationtime -descending | select -first 1
$RegExPattern = [regex]::escape($DestinationMatch)
$FilteredManifest = gc $manifestfile.FullPath | where {$_ -match "`"($RegexPattern[^`"]*)`""} |%{$matches[1] -replace '/','\'}
$dest = $destinationMatch.replace("/","\")
$DestFileList = gci -Path $dest -Recurse | select Fullname,Attributes
$DestFileList | Where{$FilteredManifest -notcontains $_.FullName -and $_.Attributes -notmatch "Directory"}|Remove-Item $_ -Force
$DestFileList | Where{$FilteredManifest -notcontains $_.FullName -and $_.Attributes -match "Directory" -and (gci -LiteralPath $_ -Recurse -WarningAction SilentlyContinue -ErrorAction SilentlyContinue).Length -eq 0}{Remove-Item $_ -Recurse -Force}
This stops you from duplicating efforts. There's no need to get your manifest file, and then assign different variables to different properties of the file object, just reference them directly. Then later when you pull your directory listing of the drive (the slow part here), keep the full name and attributes of the files/folders. That way you can easily filter against Attributes to see what's a directory and what not, so we can deal with files first, then clean up directories later after the files are cleaned up.
That script should be a bit more streamlined version of yours. Now, about pulling that directory listing... Here's the deal, using Get-ChildItem is going to be slower than some alternatives (such as dir /s /b) but it stops you from having to duplicate efforts by later checking what's a file, and what's a directory. I suppose if the actual files/folders that you are concerned with are a small percentage of the total, then the double work may actually be worth the time and effort to pull the list with something like dir /s /b, and then parse against the log, and only pull folder/file info for the specific items you need to address.
I am trying to move a list of files from one directory to another. The catch is, When the items are moved to the new directory, I want to automatically organize them.
Ex..
I have a folder of thousands of filenames.
All filenames are relative to a user's userID. Some users have multiple files in this folder, so it appends a number onto the end of the name. I.E. susy.txt, susy1.txt, susy2.txt, carl.txt, carl1.txt, etc...
What I am trying to do is create a specific folder (in the new directory) for each user that has multiple files, and move all associated files into that folder. So I notice there are multiple susy documents. So I want to create a folder named Susy and place susy.txt, susy1.txt, and susy2.txt into it... And so on for all files.
Is it even possible to do this as a batch file, if so can someone point me in the correct direction on doing this? I have a small amount of knowledge in writing batch scripts, and would like to take this as an opportunity to learn more.
This is very similar to a question I have asked earlier. File and Folder Manipulation in Powershell. I am very thankful for the responses I received, they helped me greatly. The answer from Adi Inbar was exactly what I needed, at the time. However, I was forced to make a modification, which I have tried myself.
Adi Inbar's Answer
Get-ChildItem | ?{$_.Name -match '(\D+)\d*\.txt'} | %{
md $matches[1] -ea SilentlyContinue
Move-Item $_ $matches[1]
}
Short sweet and too the point, exactly what I needed. However it only works for for files that are going to be organized but stay in the same parent folder.
This is what I have attempted:
Get-ChildItem –path "P:\My Documents\Org Test\Test1" | Where{$_.Name -match '(\D+)\d*\.txt'} | Foreach{
md P:\'My Documents'\'Org Test'\Test2\$matches[1] -ea SilentlyContinue
Move-Item $_ P:\'My Documents'\'Org Test'\Test2\$matches[1]
}
To my knowledge and basic understanding this should work... But I am getting an error saying Move-Item : Cannot create a file when that file already exists.
At P:\My Documents\Org Test\Test.ps1:3 char:3
+ Move-Item -Path P:\'My Documents'\'Org Test'\Test1\$_ -destination P:\'M ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : WriteError: (P:\My Documents...t1\Johnny123.txt:FileInfo) [Move-Item], I
+ FullyQualifiedErrorId : MoveFileInfoItemIOError,Microsoft.PowerShell.Commands.MoveItemCommand
I am sure that it is on the tip of my tongue, but I cannot get it. I have very basic powershell scripting experience and just need a quick fix.
EDIT:
I have been able to "resolve" my issue by using this script:
Get-ChildItem –path P:\'My Documents'\'PST Org Script'\Test1 | Foreach-Object{
move-item -Path $_.fullname -Destination "P:\My Documents\PST Org Script\Test2" -ea SilentlyContinue }
cd P:\'My Documents'\'PST Org Script'\Test2
Get-ChildItem | ?{$_.Name -match '(\D+)\d*\.txt'} | %{
md $matches[1] -ea SilentlyContinue
Move-Item $_ $matches[1]
}
I am curious. I feel like this can be done in the 3 lines of code I have above. This seems like a little redundant. But what do I know.
Thanks
Try this:
$srcPath = 'P:\My Documents\PST Org Script\Test1'
$dstPath = 'P:\My Documents\PST Org Script\Test2'
Get-ChildItem $srcPath | Where {$_.Name -match '(\D+)\d*\.txt'} |
Foreach {$targetDir = Join-Path $dstPath $matches[1]
md $targetDir -ea 0
Move-Item $_ $targetDir -WhatIf}
I have been able to resolve my issue using this:
Get-ChildItem –path P:\'My Documents'\'PST Org Script'\Test1 | Foreach-Object{
move-item -Path $_.fullname -Destination "P:\My Documents\PST Org Script\Test2" -ea SilentlyContinue }
cd P:\'My Documents'\'PST Org Script'\Test2
Get-ChildItem | ?{$_.Name -match '(\D+)\d*\.txt'} | %{
md $matches[1] -ea SilentlyContinue
Move-Item $_ $matches[1]
}
However I feel, there is a shorter, simpler way of doing this via some variation of the method I entered earlier in my original question.