Compare two folders recursively and save repeated files - windows

I'm trying to do something pretty weird. Any alternative solution could be analyzed but the process where I'm working is very difficult to change.
I have a folder to deploy in an IIS web server. This folder can contain any file such as dll, asmx, web.config, .exe, etc. We have a bat script that performs a full backup but a simple file deploy can last for an hour because of this.
I want some utility or to make a powershell script to compare the folder to deploy with the destination folder, and only backup the files (or folder) that has the same files on the first folder. This has to be done recursively and it also has to preserve folder structure in the destination server.
Edit: I'm currently working on a powershell script that will go like this (i'm definitely not a powershell expert):
Compare-Object $d1 $d2 | Where-Object {$_.SideIndicator -ne "=>" -and $_.InputObject -ne "*.ok*" -and $_.InputObject -ne "*.bat*" }
Any help or recommendation would be appreciated!
Regards

You won't be able to maintain file structure using compare-object... I got bored so here you go.
$sPath = "C:\test\"
$dPath = "C:\test2\"
$src = gci -recurse $sPath
$dest = gci -recurse $dPath
$src | % {
foreach($item in $dest) {
$t1 = ($item.FullName).trimStart($dpath)
$t2 = ($_.FullName).trimStart($sPath)
if($t1 -eq $t2) {copy-item $item.FullName $_.FullName -Force}
}
}
Just set $spath to src directory and $dPath to your deployment directory, and let her rip.

Related

Unzip Multiple Files into Different Directories

I have multiple zip files.
They are called folder(1).zip, folder(2).zip, folder(3).zip. Using PowerShell, when I attempt to unzip them all into unique folders using this...
Get-ChildItem 'c:\users\name\downloads' -Filter *.zip | Expand-Archive -DestinationPath 'c:\users\name\downloads' -Force
I get all of the files into one folder called "folder". How can I get the zip folders to unzip into separate folders?
Bonus question, is there a way, as part of this process, to rename each folder as it's coming out so folder(1).zip becomes Name-Here, folder(2).zip becomes Other-Name-Here, etc?
Thanks!
Because you specify only one destination path they will all be extracted into c:\users\name\downloads. I suppose the zip archives each contain a folder named "folder", so all contents from all archives end up together in c:\users\name\downloads\folder
You would have to specify a different destination path for each archive. Not sure what your naming convention should be, I have used a simple counter:
$counter = 0
Get-ChildItem 'c:\users\name\downloads' -Filter *.zip | foreach {
$destination = Join-Path $_.DirectoryName ("YourName" + $counter++)
Expand-Archive $_.FullName -DestinationPath $destination
}
Of course I suppose, now every of those folders will have the subfolder "folder", but if that's how the archives are built there's not really a way to change that. If you are absolutely sure that all archives have that subfolder, you could do something like this:
$counter = 0
Get-ChildItem 'c:\users\name\downloads' -Filter *.zip | foreach {
# expand to the root folder first
Expand-Archive $_.FullName -DestinationPath $_.DirectoryName
# now rename the extracted "folder" to whatever you like
Rename-Item (Join-Path $_.DirectoryName "folder") -NewName ("YourName" + $counter++)
}

Batch file to compress subdirectories individually with Windows native tools

I've seen variations of this question answered, but typically using something like 7zip. I'm trying to find a solution that will work with the capabilities that come with windows absent any additional tools.
I have a directory that contains several hundred subdirectories. I need to individually compress each subdirectory....so I'll wind up with several hundred zip files, one per subdirectory. This is on a machine at work where I don't have administrative privileges to install new software...hence the desire to stay away from 7zip, winRar, etc.
If this has already been answered elsewhere, my apologies...
Never tried that myself, but there is Compress-Archive:
The Compress-Archive cmdlet creates a zipped (or compressed) archive file from one or more specified files or folders. An archive file allows multiple files to be packaged, and optionally compressed, into a single zipped file for easier distribution and storage. An archive file can be compressed by using the compression algorithm specified by the CompressionLevel parameter.
Because Compress-Archive relies upon the Microsoft .NET Framework API System.IO.Compression.ZipArchive to compress files, the maximum file size that you can compress by using Compress-Archive is currently 2 GB. This is a limitation of the underlying API.
Here's a sample script I just hacked together:
# configure as needed
$source = "c:\temp"
$target = "d:\temp\test"
# grab source file names and list them
$files = gci $source -recurse
$files
# target exists?
if( -not (test-path $target)) {
new-item $target -type directory
}
# compress, I am using -force here to overwrite existing files
$files | foreach{
$dest = "$target\" + $_.name + ".zip"
compress-archive $_ $dest -CompressionLevel Optimal -force
}
# list target dir contents
gci $target -recurse
You may have to improve it a bit when it comes to subfolders. In the above version, subfolders are compressed as a whole into a single file. This might not exactly be what you want.
Get-ChildItem c:\path\of\your\folder | ForEach-Object {
$path = $_.FullName
Compress-Archive -Path $path -DestinationPath "$path.zip"
}
I put this, as a quick snippet. Don't hesitate to comment if this does not fit with your request.
In a folder X, there are subfolders Y1, Y2...
Y1.zip, Y2.zip... will be created.
use PowerShell go the the path that you would like to compress, do:
$folderlist = Get-ChildItem "."
foreach ($Folder in $folderlist) { Compress-Archive -path $Folder.Name -destinationPath "$($Folder.Name).zip"}

Extension/Tool to perform cleanup on project

I have been recently using Visual Studio 2017 and I'd like to have an external tool/extension/setting which deletes everything besides .sln, .vcxproj and sources. I have already tried CLean Project and Clean Solution extension but neither of those removes Debug folder. I have read something about PowerShell scripts but I have no idea how to use them and I don't want to run unknown code on my console.
PS: I know that VS has the cleanup function, but it only deletes executables. I also read something about modifying the project properties but that would be really unpleasant for many projects.
PSS: I am a student and I have many project directories. All I want is to have a neat way to store them.
PSSS: I have already configured my .gitignore file and I am using git. IS there a way to use it to perform cleanup?
You're using Git. You can simply reset your workspace to the last commit, removing all unversioned and ignored files. First make sure you have no pending changes, then perform a clean:
git clean -xdfn
-x: ignore the ignores (removes bin, obj, *.dll, ...)
-d: remove directories in addition to files
-f: force Git to actually do the job
-n: perform a dry run, which will list the files that will be removed.
Remove the n from the arguments to actually clean the workspace.
See also How do I clear my local working directory in git?.
Here a script I'm using that works well for me.
# PowerShell script that recursively deletes all 'bin' and 'obj' (or any other specified) folders inside current folder
$CurrentPath = (Get-Location -PSProvider FileSystem).ProviderPath
# recursively get all folders matching given includes, except ignored folders
$FoldersToRemove = Get-ChildItem .\ -include bin,obj -Recurse | where {$_ -notmatch '_tools' -and $_ -notmatch '_build'} | foreach {$_.fullname}
# recursively get all folders matching given includes
$AllFolders = Get-ChildItem .\ -include bin,obj -Recurse | foreach {$_.fullname}
# subtract arrays to calculate ignored ones
$IgnoredFolders = $AllFolders | where {$FoldersToRemove -notcontains $_}
# remove folders and print to output
if($FoldersToRemove -ne $null)
{
Write-Host
foreach ($item in $FoldersToRemove)
{
remove-item $item -Force -Recurse;
Write-Host "Removed: ." -nonewline;
Write-Host $item.replace($CurrentPath, "");
}
}
# print ignored folders to output
if($IgnoredFolders -ne $null)
{
Write-Host
foreach ($item in $IgnoredFolders)
{
Write-Host "Ignored: ." -nonewline;
Write-Host $item.replace($CurrentPath, "");
}
Write-Host
Write-Host $IgnoredFolders.count "folders ignored" -foregroundcolor yellow
}
# print summary of the operation
Write-Host
if($FoldersToRemove -ne $null)
{
Write-Host $FoldersToRemove.count "folders removed" -foregroundcolor green
}
else { Write-Host "No folders to remove" -foregroundcolor green }
Write-Host
# prevent closing the window immediately
$dummy = Read-Host "Completed, press enter to continue."
Copy and paste in a new file in the same directory of your .sln file.
I call it "CleanAll.ps1" but you can call it as you prefer.
The easiest way to run it?
Right-Click on the file > Run with powershell
It recursively delete bin and obj files of all sub-folders starting from the current path. You can of course personalize and add "debug" or any other folder once you get more familiar with the script.

How to use PowerShell Remove-Item to delete a directory with long name? [duplicate]

I'm writing a simple script to delete USMT migration folders after a certain amount of days:
## Server List ##
$servers = "Delorean","Adelaide","Brisbane","Melbourne","Newcastle","Perth"
## Number of days (-3 is over three days ago) ##
$days = -3
$timelimit = (Get-Date).AddDays($days)
foreach ($server in $servers)
{
$deletedusers = #()
$folders = Get-ChildItem \\$server\USMT$ | where {$_.psiscontainer}
write-host "Checking server : " $server
foreach ($folder in $folders)
{
If ($folder.LastWriteTime -lt $timelimit -And $folder -ne $null)
{
$deletedusers += $folder
Remove-Item -recurse -force $folder.fullname
}
}
write-host "Users deleted : " $deletedusers
write-host
}
However I keep hitting the dreaded Remove-Item : The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
I've been looking at workarounds and alternatives but they all revolve around me caring what is in the folder.
I was hoping for a more simple solution as I don't really care about the folder contents if it is marked for deletion.
Is there any native Powershell cmdlet other than Remove-Item -recurse that can accomplish what I'm after?
I often have this issue with node projects. They nest their dependencies and once git cloned, it's difficult to delete them. A nice node utility I came across is rimraf.
npm install rimraf -g
rimraf <dir>
Just as CADII said in another answer: Robocopy is able to create paths longer than the limit of 260 characters. Robocopy is also able to delete such paths. You can just mirror some empty folder over your path containing too long names in case you want to delete it.
For example:
robocopy C:\temp\some_empty_dir E:\temp\dir_containing_very_deep_structures /MIR
Here's the Robocopy reference to know the parameters and various options.
I've created a PowerShell function that is able to delete a long path (>260) using the mentioned robocopy technique:
function Remove-PathToLongDirectory
{
Param(
[string]$directory
)
# create a temporary (empty) directory
$parent = [System.IO.Path]::GetTempPath()
[string] $name = [System.Guid]::NewGuid()
$tempDirectory = New-Item -ItemType Directory -Path (Join-Path $parent $name)
robocopy /MIR $tempDirectory.FullName $directory | out-null
Remove-Item $directory -Force | out-null
Remove-Item $tempDirectory -Force | out-null
}
Usage example:
Remove-PathToLongDirectory c:\yourlongPath
This answer on SuperUser solved it for me: https://superuser.com/a/274224/85532
Cmd /C "rmdir /S /Q $myDir"
I learnt a trick a while ago that often works to get around long file path issues. Apparently when using some Windows API's certain functions will flow through legacy code that can't handle long file names. However if you format your paths in a particular way, the legacy code is avoided. The trick that solves this problem is to reference paths using the "\\?\" prefix. It should be noted that not all API's support this but in this particular case it worked for me, see my example below:
The following example fails:
PS D:\> get-childitem -path "D:\System Volume Information\dfsr" -hidden
Directory: D:\System Volume Information\dfsr
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a-hs 10/09/2014 11:10 PM 834424 FileIDTable_2
-a-hs 10/09/2014 8:43 PM 3211264 SimilarityTable_2
PS D:\> Remove-Item -Path "D:\System Volume Information\dfsr" -recurse -force
Remove-Item : The specified path, file name, or both are too long. The fully qualified file name must be less than 260
characters, and the directory name must be less than 248 characters.
At line:1 char:1
+ Remove-Item -Path "D:\System Volume Information\dfsr" -recurse -force
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : WriteError: (D:\System Volume Information\dfsr:String) [Remove-Item], PathTooLongExcepti
on
+ FullyQualifiedErrorId : RemoveItemIOError,Microsoft.PowerShell.Commands.RemoveItemCommand
PS D:\>
However, prefixing the path with "\\?\" makes the command work successfully:
PS D:\> Remove-Item -Path "\\?\D:\System Volume Information\dfsr" -recurse -force
PS D:\> get-childitem -path "D:\System Volume Information\dfsr" -hidden
PS D:\>
If you have ruby installed, you can use Fileman:
gem install fileman
Once installed, you can simply run the following in your command prompt:
fm rm your_folder_path
This problem is a real pain in the neck when you're developing in node.js on Windows, so fileman becomes really handy to delete all the garbage once in a while
This is a known limitation of PowerShell. The work around is to use dir cmd (sorry, but this is true).
http://asysadmin.tumblr.com/post/17654309496/powershell-path-length-limitation
or as mentioned by AaronH answer use \?\ syntax is in this example to delete build
dir -Include build -Depth 1 | Remove-Item -Recurse -Path "\\?\$($_.FullName)"
If all you're doing is deleting the files, I use a function to shorten the names, then I delete.
function ConvertTo-ShortNames{
param ([string]$folder)
$name = 1
$items = Get-ChildItem -path $folder
foreach ($item in $items){
Rename-Item -Path $item.FullName -NewName "$name"
if ($item.PSIsContainer){
$parts = $item.FullName.Split("\")
$folderPath = $parts[0]
for ($i = 1; $i -lt $parts.Count - 1; $i++){
$folderPath = $folderPath + "\" + $parts[$i]
}
$folderPath = $folderPath + "\$name"
ConvertTo-ShortNames $folderPath
}
$name++
}
}
I know this is an old question, but I thought I would put this here in case somebody needed it.
There is one workaround that uses Experimental.IO from Base Class Libraries project. You can find it over on poshcode, or download from author's blog. 260 limitation is derived from .NET, so it's either this, or using tools that do not depend on .NET (like cmd /c dir, as #Bill suggested).
Combination of tools can work best, try doing a dir /x to get the 8.3 file name instead. You could then parse out that output to a text file then build a powershell script to delete the paths that you out-file'd. Take you all of a minute. Alternatively you could just rename the 8.3 file name to something shorter then delete.
For my Robocopy worked in 1, 2 and 3
First create an empty directory lets say c:\emptydir
ROBOCOPY c:\emptydir c:\directorytodelete /purge
rmdir c:\directorytodelete
This is getting old but I recently had to work around it again. I ended up using 'subst' as it didn't require any other modules or functions be available on the PC this was running from. A little more portable.
Basically find a spare drive letter, 'subst' the long path to that letter, then use that as the base for GCI.
Only limitation is that the $_.fullname and other properties will report the drive letter as the root path.
Seems to work ok:
$location = \\path\to\long\
$driveLetter = ls function:[d-z]: -n | ?{ !(test-path $_) } | random
subst $driveLetter $location
sleep 1
Push-Location $driveLetter -ErrorAction SilentlyContinue
Get-ChildItem -Recurse
subst $driveLetter /D
That command is obviously not to delete files but can be substituted.
PowerShell can easily be used with AlphaFS.dll to do actual file I/O stuff
without the PATH TOO LONG hassle.
For example:
Import-Module <path-to-AlphaFS.dll>
[Alphaleonis.Win32.Filesystem.Directory]::Delete($path, $True)
Please see at Codeplex: https://alphafs.codeplex.com/ for this .NET project.
I had the same issue while trying to delete folders on a remote machine.
Nothing helped but... I found one trick :
# 1:let's create an empty folder
md ".\Empty" -erroraction silentlycontinue
# 2: let's MIR to the folder to delete : this will empty the folder completely.
robocopy ".\Empty" $foldertodelete /MIR /LOG+:$logname
# 3: let's delete the empty folder now:
remove-item $foldertodelete -force
# 4: we can delete now the empty folder
remove-item ".\Empty" -force
Works like a charm on local or remote folders (using UNC path)
Adding to Daniel Lee's solution,
When the $myDir has spaces in the middle it gives FILE NOT FOUND errors considering set of files splitted from space. To overcome this use quotations around the variable and put powershell escape character to skip the quatations.
PS>cmd.exe /C "rmdir /s /q <grave-accent>"$myDir<grave-accent>""
Please substitute the proper grave-accent character instead of <grave-accent>
SO plays with me and I can't add it :). Hope some one will update it for others to understand easily
Just for completeness, I have come across this a few more times and have used a combination of both 'subst' and 'New-PSDrive' to work around it in various situations.
Not exactly a solution, but if anyone is looking for alternatives this might help.
Subst seems very sensitive to which type of program you are using to access the files, sometimes it works and sometimes it doesn't, seems to be the same with New-PSDrive.
Any thing developed using .NET out of the box will fail with paths too long. You will have to move them to 8.3 names, PInVoke (Win32) calls, or use robocopy

How can I relocate MP4 files from one location to another?

I have a bunch of pictures and videos mixed in with each other on my external hard drive, and I want to move all video files from one location to another whilst keeping the same structure. Is this possible to to achieve without changing the current structure with the images? Example below:
Since the folders are in the root you can't use a top level folder and use where PSPath so instead try this:
gci 'C:\Folder A' | ? {$_.Extension -eq '.MP4'} | Move-Item -Destination 'D:\Folder A'
gci 'C:\Folder B' | ? {$_.Extension -eq '.MP4'} | Move-Item -Destination 'D:\Folder B'
That should move all the MP4 files to D:\ and leave behing JPEGs in C:\
The diagram shows that they are on the C drive (usualy not an External Drive) and only have a single directory. If this is th case, you would simple sort by File Type inside the Windows Explorer. A lot of information missing and the diagram doesn't seem to be what you are actually asking.
The diagram says the following:
"C:\Folder A\Image 1.jpg" move to "C:\Folder A\Image 1.jpg"
"C:\Folder B\Image 4.jpg" move to "C:\Folder B\Image 4.jpg"
As you can see this is redundant, because the file is already located in the path you want it to go to.
You just want to move the .MP4 files to the D drive.
"C:\Folder A\Video 1.MP4" move to "D:\Folder A\Video 1.MP4"
"C:\Folder A\Video 1.MP4" move to "D:\Folder B\Video 1.MP4"
If you wanted to keep the file structure naming the same. You would have to do something like this, but your question makes no sense. This is me trying to understand what you are trying to say without you telling us that you are downloading illegally, and that you want to cleanup.
# Variables
$sourcePath = 'Z:\Folder A'
$destPath = 'X:\Folder A'
# Movie folder Structure and File type
Get-ChildItem $sourcePath -Recurse -Include '*.MP4' | Foreach-Object `
{
$destDir = Split-Path ($_.FullName -Replace [regex]::Escape($sourcePath), $destPath)
if (!(Test-Path $destDir))
{
New-Item -ItemType directory $destDir | Out-Null
}
# Backup Files
#Copy-Item $_ -Destination $destDir
# Move Files
#Move-Item $_ -Destination $destDir
}

Resources