I have a powershell script that will look for and compress log files then make a new .7zip file, however currently it has trouble compressing folder with dates e.g. application_logs_2016-07-14
1: The script works fine if there are folders with char's however if the containing folder has a date e.g. (application_logs_2016-07-14) nothing is archived.
2: I need a zip log files that are older than 5 days old, dump.log.341.log, dump.log.342.log and dump.log.343.log should be converted as dump.log.341.zip, dump.log.342.zip, dump.log.343.zip.
Here is the current code, if any powershell guru's could advise i'd be very happy. Thanks in advance.
# See if 7 zip is installed
if (-not (test-path "$env:ProgramFiles\7-Zip\7z.exe")) {throw "$env:ProgramFiles\7-Zip\7z.exe needed"}
set-alias sz "$env:ProgramFiles\7-Zip\7z.exe"
# Define log location and target directory
$Logfile = "D:\stuff\Software\dump files\Archive.log"
$newdirectory = "D:\stuff\compressedlogs"
# Write to log file - start of archive
$now = Get-date -Format "yyyy-MM-dd HH:mm:ss"
Add-Content $Logfile -value "Archive started $now"
# Import what we want to back up from a list file
$List = Import-CSV "D:\stuff\Software\dump files\Compressionlist.txt"
ForEach ($Entry in $List){
$filepath = $($Entry.FilePath)
$Extension = $($Entry.Extension)
$Include = $($Entry.Include)
$Horizon = $($Entry.Horizon)
$Ext2Chg = $($Entry.Ext2Chg)
# Extract List of Files to process
$log = Get-ChildItem -Recurse -Path $filepath -Filter $Extension -Include $Include | Where-Object {$_.lastwriteTime -lt (((Get-Date).AddDays($Horizon)).date)}
# Archive each file found
ForEach ($file in $log) {
if ($file -ne $null) {
$name = $file.name
$newdate = $file.CreationTime | Get-Date -f "yyyy-MM-dd"
$newname = $file.BaseName + "___" + $newdate + $file.Extension
$directory = $file.DirectoryName
$zipfile = $newname.replace($Ext2Chg,".7z")
sz a -t7z "$newdirectory\$zipfile" "$directory\$name"
$now = Get-date -Format "yyyy-MM-dd HH:mm:ss"
Add-Content $Logfile -value "File $directory\$name archived to folder\new filename $newdirectory\$newname at $now"
Remove-Item $file
}
}
}
# Write to log file - end of archive
$now = Get-date -Format "yyyy-MM-dd HH:mm:ss"
Add-Content $Logfile -value "Archive completed $now"
# End
The script looks at a txt document to find what to archive
COMPRESSION LIST
Filepath,Extension,Include,Horizon,Ext2Chg
E:\APPLICATION\DUMP_logs_*,*.log,APP.dumps-currentlog.messages*,-5,.log
===============================================================
example folder structure
D:\application\server_log
(which will contain a log e.g. server_log_2016-07-14_00-00-00_0001.log) this will archive fine.
D:\application\application_log_2016-07-14
(which will contain a log e.g. APP.dumps-currentlog.messages.log) this will NOT archive.
Hope that make sense.
Related
I am creating a script that splits a target folder's files into subfolders of n length, where n is a number specified dynamically.
So basically, if Folder A has 9000 files, and I limit the number of files to 1000 per folder, the script would create nine sub-directories inside of Folder A with 1000 files each.
Here is working code:
param (
[Parameter(Mandatory,Position=0)]
[String]
$FileList,
[Parameter(Mandatory=$false,ValueFromPipelineByPropertyName)]
[Int32]
$NumFilesPerFolder = 1000,
[Parameter(Mandatory=$false,ValueFromPipelineByPropertyName)]
[Int32]
$FolderNumberPadding = 2
)
$Folders = Get-Content $FileList
Set-Location -LiteralPath ([IO.Path]::GetTempPath())
function Move-Files {
[CmdletBinding()]
param (
[Parameter(Mandatory,Position=0)]
[System.Collections.ArrayList]
$List,
[Parameter(Mandatory)]
[Int32]
$Index
)
$BaseFolder = [System.IO.Path]::GetDirectoryName($List[0])
$DestFolderName = $Index.ToString().PadLeft($FolderNumberPadding, '0')
$DestFolder = New-Item -Path (Join-Path $BaseFolder $DestFolderName) -Type Directory -Force
Move-Item $List -Destination $DestFolder -Force
}
foreach ($Folder in $Folders) {
$Files = Get-ChildItem -LiteralPath $Folder -File -Force
$filesidx = 1
$totalidx = $null
$groupidx = 0
$FilesToMove = [System.Collections.ArrayList]#()
foreach ($File in $Files) {
if($null -eq $totalidx){
$totalidx = $Files.Length
}
if($filesidx -eq 1){
$groupidx++
}
$FilesToMove.Add($File)
if($filesidx -eq $NumFilesPerFolder){
Move-Files -List $FilesToMove -Index $groupidx
$FilesToMove.Clear()
$filesidx = 1
}elseif($totalidx -eq 1){
Move-Files -List $FilesToMove -Index $groupidx
$FilesToMove.Clear()
break
}else{
$filesidx++
}
$totalidx--
}
}
Remove-Item $FileList -Force
$app = New-Object -ComObject Shell.Application
$appwin = $app.Windows()
foreach ($window in $appwin) {
if($window.Name -eq "File Explorer"){
$window.Refresh()
}
}
Invoke-VBMessageBox "Operation Complete" -Title "Operation Complete" -Icon Information -BoxType OKOnly
This code runs reasonably well, but it heavily bottlenecks when actually moving the files with Move-Item. I'd like to try and use RoboCopy here, but I am perplexed as to how I can implement it.
What I'm having trouble with is that the items I need to move are stored in a list (see the Move-Files function), and every item that needs to be moved are all in the same sub-directory. So I can't just do RoboCopy.exe C:\Source C:\Destination /mov.
How can I integrate RoboCopy here to accomplish my goal? I really need multi-threaded performance as this function will be responsible for moving thousands of files around in production on a frequent basis.
Any help would be greatly appreciated - please let me know if I can provide more information to further clarify my objective.
Thanks for any help at all!
Code:
$timestamp = (Get-Date).ToString('yyyy-MM-dd')
$originalSource = Get-ChildItem "D:\output\csv\*.csv", "D:\output\csv\Billing\*.csv" | Where-Object {($_.LastWriteTime -ge [datetime]::today)}
$source = $originalSource
$target = "D:\output\csv\bin\$timestamp.7z"
$housekeepZipFile = "D:\output\csv\bin\*"
####Using 7z to zip
if (-not (test-path "D:\bin\7-Zip\7z.exe")) {throw "D:\bin\7-Zip\7z.exe needed"}
set-alias sz "D:\bin\7-Zip\7z.exe"
sz a -mx=0 -mhe=on -m0=lzma2 $target $source
I have tried above powershell. When i pumped files into csv and billing folder, it will create archive file that I want.
When there is no today's date input file pumped into the csv and billing folder, 7zip random pull window files and create archive to me.
Question:
How can I set that only create archive if there's a file with current date or create archive (with empty folder) if there no file with current date. P/S: I tried to put where-object to filter lastwritetime but seems not useful enough.
you just have to add an if after calling Get-ChildItem
$timestamp = (Get-Date).ToString('yyyy-MM-dd')
$originalSource = Get-ChildItem "D:\output\csv\*.csv", "D:\output\csv\Billing\*.csv" | Where-Object {($_.LastWriteTime -ge [datetime]::today)}
if($originalSource) {
$source = $originalSource
$target = "D:\output\csv\bin\$timestamp.7z"
$housekeepZipFile = "D:\output\csv\bin\*"
####Using 7z to zip
if (-not (test-path "D:\bin\7-Zip\7z.exe")) {throw "D:\bin\7-Zip\7z.exe needed"}
set-alias sz "D:\bin\7-Zip\7z.exe"
sz a -mx=0 -mhe=on -m0=lzma2 $target $source
}
so only if files are found the archive will be created
Something very odd is going on with the "date modified" field of several folders on an exFAT external drive I have. A folder in which several files were recently added is still showing its date modified as its creation date. Even worse, another folder with recently added files is showing a date that precedes its creation date! Has anyone observed this and know what might be going on? I have checked online and found nothing useful/relevant regarding this. The same information shows up in both Explorer and in a command prompt so its not specific to Explorer
Run this PowerShell script. Close Explorer before running to avoid file-locking.
# -------------give each folder the highest modified date of it's files --------
function OneDir($dir)
{
# elaborate one folder, with given name
Set-Location -Path $dir.FullName
$maxd = Get-Date(0)
$files = Get-ChildItem -Recurse -Filter *.* | Where-Object { $_.PsIsContainer -eq $false }
for ($i=0; $i -lt $files.Count; $i++)
{
$file = $files[$i]
$cd = [datetime]($file.lastwritetime)
If ($cd -Gt $maxd)
{$maxd = $cd}
}
If ($files.Count -Gt 0)
{$dir.LastWriteTime = ($maxd)}
Write-Host ($dir.FullName) + " " + ($dir.LastWriteTime)
}
#------------------------- main ------------------------------------
$startDir = Read-Host 'Foldername to start with'
Set-Location -Path $startDir
$t = Get-ItemProperty $startDir
OneDir $t
$dirs = Get-ChildItem -Recurse -Filter *.* | Where-Object { $_.PSIsContainer }
for ($d=0; $d -lt $dirs.Count; $d++)
{
OneDir $dirs[$d]
}
Write-Host "Finished. press Enter"
cmd /c pause
I wrote the below PowerShell script to compress logs older than 30 days:
$LastWrite=(get-date).AddDays(-30).ToString("MM/dd/yyyy")
Get-ChildItem -Filter "server.log*" -Recurse -File | Where-Object
{$_.LastWriteTime -le $LastWrite}
Now, I am unable to get a compress command in PowerShell via which I can compress (zip/tar) the server.log* files older than 30 days.
Expecting a single command which I can use by adding a pipe sign in the above command.
You can use the Compress-Archive cmdlet to zip files if you have PowerShell version 5 or above:
$LastWrite = (get-date).AddDays(-30)
$Files = Get-ChildItem -Filter "server.log*" -Recurse -File | Where-Object {$_.LastWriteTime -le $LastWrite}
ForEach ($File in $Files) {
$File | Compress-Archive -DestinationPath "$($File.fullname).zip"
}
If you have an older version of Powershell you can use ZipFileExtensions' CreateEntryFromFile method, but there are a lot of considerations if you want a robust script that runs unattended.
In months of testing a script developed for this purpose, I encountered some issues that have made this small problem more complicated:
Will any of the files be locked? CreateEntryFromFile may fail if so.
Did you know that you can have multiple copies of the same file in a Zip archive? It's harder to extract them because you can't put them in the same folder. My script checks the file path and the archived file time stamp (+/- 2 seconds due to the lost date precision in Zip format) to determine if it's been already archived, and doesn't create a duplicate.
Are the files created in a time zone with Daylight Savings? Zip format doesn't preserve that attribute, and may lose or gain an hour when uncompressed.
Do you want to delete the original if it was successfully archived?
If unsuccessful due to a locked/missing file or very long path, should the process continue?
Will any error leave you with an unusable zip file? You need to Dispose() the archive to finalize it.
How many archives do you want to keep? I prefer one per run-month, adding new entries to an existing zip.
Do you want to preserve the relative path? Doing so will partially eliminate the problem of duplicates inside the zip file.
Mark Wragg's script should work if you don't care about these issues and you have Powershell 5, but it creates a zip for every log, which may not be what you want.
Here's the current version of the script - in case GitHub ever becomes unavailable:
#Sends $FileSpecs files to a zip archive if they match $Filter - deleting the original if $DeleteAfterArchiving is true.
#Files that have already been archived will be ignored.
param (
[string] $ParentFolder = "$PSScriptRoot", #Files will be stored in the zip with path relative to this folder
[string[]] $FileSpecs = #("*.log","*.txt","*.svclog","*.log.*"),
$Filter = { $_.LastWriteTime -lt (Get-Date).AddDays(-7)}, #a Where-Object function - default = older than 7 days
[string] $ZipPath = "$PSScriptRoot\archive-$(get-date -f yyyy-MM).zip", #create one archive per run-month - it may contain older files
[System.IO.Compression.CompressionLevel]$CompressionLevel = [System.IO.Compression.CompressionLevel]::Optimal,
[switch] $DeleteAfterArchiving = $true,
[switch] $Verbose = $true,
[switch] $Recurse = $true
)
#( 'System.IO.Compression','System.IO.Compression.FileSystem') | % { [void][System.Reflection.Assembly]::LoadWithPartialName($_) }
Push-Location $ParentFolder #change to the folder so we can get relative path
$FileList = (Get-ChildItem $FileSpecs -File -Recurse:$Recurse | Where-Object $Filter) #CreateEntryFromFile raises UnauthorizedAccessException if item is a directory
$totalcount = $FileList.Count
$countdown = $totalcount
$skipped = #()
Try{
$WriteArchive = [IO.Compression.ZipFile]::Open( $ZipPath, [System.IO.Compression.ZipArchiveMode]::Update)
ForEach ($File in $FileList){
Write-Progress -Activity "Archiving files" -Status "Archiving file $($totalcount - $countdown) of $totalcount : $($File.Name)" -PercentComplete (($totalcount - $countdown)/$totalcount * 100)
$ArchivedFile = $null
$RelativePath = (Resolve-Path -LiteralPath "$($File.FullName)" -Relative) -replace '^.\\'
$AlreadyArchivedFile = ($WriteArchive.Entries | Where-Object {#zip will store multiple copies of the exact same file - prevent this by checking if already archived.
(($_.FullName -eq $RelativePath) -and ($_.Length -eq $File.Length) ) -and
([math]::Abs(($_.LastWriteTime.UtcDateTime - $File.LastWriteTimeUtc).Seconds) -le 2) #ZipFileExtensions timestamps are only precise within 2 seconds.
})
If($AlreadyArchivedFile -eq $null){
If($Verbose){Write-Host "Archiving $RelativePath $($File.LastWriteTimeUtc -f "yyyyMMdd-HHmmss") $($File.Length)" }
Try{
$ArchivedFile = [System.IO.Compression.ZipFileExtensions]::CreateEntryFromFile($WriteArchive, $File.FullName, $RelativePath, $CompressionLevel)
}Catch{
Write-Warning "$($File.FullName) could not be archived. `n $($_.Exception.Message)"
$skipped += [psobject]#{Path=$file.FullName; Reason=$_.Exception.Message}
}
If($File.LastWriteTime.IsDaylightSavingTime() -and $ArchivedFile){#HACK: fix for buggy date - adds an hour inside archive when the zipped file was created during PDT (files created during PST are not affected). Not sure how to introduce DST attribute to file date in the archive.
$entry = $WriteArchive.GetEntry($RelativePath)
$entry.LastWriteTime = ($File.LastWriteTime.ToLocalTime() - (New-TimeSpan -Hours 1)) #TODO: This is better, but maybe not fully correct. Does it work in all time zones?
}
}Else{#Write-Warning "$($File.FullName) is already archived$(If($DeleteAfterArchiving){' and will be deleted.'}Else{'. No action taken.'})"
Write-Warning "$($File.FullName) is already archived - No action taken."
$skipped += [psobject]#{Path=$file.FullName; Reason="Already archived"}
}
If((($ArchivedFile -ne $null) -and ($ArchivedFile.FullName -eq $RelativePath)) -and $DeleteAfterArchiving) { #delete original if it's been successfully archived.
Try {
Remove-Item $File.FullName -Verbose:$Verbose
}Catch{
Write-Warning "$($File.FullName) could not be deleted. `n $($_.Exception.Message)"
}
}
$countdown = $countdown -1
}
}Catch [Exception]{
Write-Error $_.Exception
}Finally{
$WriteArchive.Dispose() #close the zip file so it can be read later
Write-Host "Sent $($totalcount - $countdown - $($skipped.Count)) of $totalcount files to archive: $ZipPath"
$skipped | Format-Table -Autosize -Wrap
}
Pop-Location
Here's a command line that will compress all server.log* files older than 30 days under the current folder:
.\ArchiveOldLogs.ps1 -FileSpecs #("server.log*") -Filter { $_.LastWriteTime -lt (Get-Date).AddDays(-30)}
I would like to copy files between folders. Just modified (CSV files with new entries) in current day and one day before.
Here is my code:
foreach ($file in (Get-ChildItem "D:\Shares\WinCAP Data\DAYPROT\OFS-222_2")) {
if ($file.LastWriteTime = (Get-Date).AddDays(-1)) {
Copy-Item -Path "D:\Shares\WinCAP Data\DAYPROT\OFS-222_2\*.csv" -Destination "\\Oracle\MP"
"copying $file"
} else {
"not copying $file"
}
}
What is wrong - any suggestions?
You need to compare the date with -gt otherwise your're looking for files that were copied at that EXACT time.
Note that doing (Get-Date).AddDays(-1) is perfectly valid but will give you anything modified in the last 24 hours.
$DestinationFolder = "\\Oracle\MP\"
$EarliestModifiedTime = (Get-date).AddDays(-1)
$Files = Get-ChildItem "D:\Shares\WinCAP Data\DAYPROT\OFS-222_2\*.csv"
foreach ($File in $Files) {
if ($File.LastWriteTime -gt $EarliestModifiedTime)
{
Copy-Item $File -Destination $DestinationFolder
Write-Host "Copying $File"
}
else
{
Write-Host "Not copying $File"
}
}
If you didn't want to write out the "Copying ..." and "Not Copying ..." then you could simplify this quite a bit.
$DestingationFolder = "\\Oracle\MP\"
$EarliestModifiedTime = (Get-date).AddDays(-1)
Get-ChildItem -File |? { $_.LastWriteTime -gt $EarliestModifiedTime } | Copy-Item -Destination $DestingationFolder
Finally, if you want to copy anything since the beginning of (eg midnight at the start of) yesterday then change the following line:
$EarliestModifiedTime = (Get-date).AddDays(-1).Date
#Mr Tree I have one more related question.
I got few times per day new file at the location D:\Shares\WinCAP Data\DAYPROT\OFS-HT (location 1) with fixed name abcDD.MM.YYYY.csv (abc03.09.2015.csv) and I have a service which every 10 minutes call my powershell script below. I made as you suggest before in upper posts. My goal is: 1. to check if there is new file with name abcDD.MM.YYYY.csv | 2. rename it into abcDD.MM.YYYYHT.csv and move it to "\Oracle\MP\PURO\" (location 2) folder where I need to rewrite it with existing for current day.
Problem is that if the file already exists on the location 2, script does not want to move it and rewrite it? Thanks for hints.
$DestingationFolder = "\\Oracle\MP\PURO\"
$EarliestModifiedTime = (Get-date).AddDays(-1)
Get-ChildItem "D:\Shares\WinCAP Data\DAYPROT\OFS-HT\*.csv" | ?{!($_.fullname -match "HT\.csv")} | Rename-Item -NewName { $_.Name -replace "\.csv", "HT.csv" }
$Files = Get-ChildItem "D:\Shares\WinCAP Data\DAYPROT\OFS-HT\*.csv" -File
foreach ($File in $Files) {
if ($File.LastWriteTime -gt $EarliestModifiedTime)
{
Move-Item $File -Destination $DestingationFolder
Write-Host "Moving $File"
}
else
{
Write-Host "Not moving $File"
}
}