how to fetch logs content between two dates in windows powershell? - windows

Question

In our project, we are getting AppDynamics logs(application logs) and machine logs and sometime the the size of the logs increase which eats out the disk size. What I am trying to do it is to get the content between two dates like 10 Nov and 13 Nov and delete the rest. Since we are working in windows environment, this needs to be done in powershell. It is easier to handle such things in linux but I am not good at powershell scripting. Below is the code snippet.
[AD Thread-Metric Reporter1] 10 Nov 2019 14:47:32,899 ERROR ManagedMonitorDelegate - Error sending metrics - will requeue for later transmission
com.singularity.ee.agent.commonservices.metricgeneration.metrics.MetricSendException: Connection back off limitation in effect: /controller/instance/702/metrics
at com.singularity.ee.agent.commonservices.metricgeneration.AMetricSubscriber.publish(AMetricSubscriber.java:350)
at com.singularity.ee.agent.commonservices.metricgeneration.MetricReporter.run(MetricReporter.java:113)
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run` ]
[AD Thread-Metric Reporter1] 11 Nov 2019 14:46:32,899 ERROR ManagedMonitorDelegate - Error sending metrics - will requeue for later transmission
com.singularity.ee.agent.commonservices.metricgeneration.metrics.MetricSendException: Connection back off limitation in effect: /controller/instance/702/metrics
at com.singularity.ee.agent.commonservices.metricgeneration.AMetricSubscriber.publish(AMetricSubscriber.java:350)
at com.singularity.ee.agent.commonservices.metricgeneration.MetricReporter.run(MetricReporter.java:113)
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run` ]
[extension-scheduler-pool-5] 13 Nov 2019 18:45:40,634 INFO ReportMetricsConfigSupplier - Basic metrics will be collected and reported through the SIM extension because SIM is enabled.
[extension-scheduler-pool-8] 14 Nov 2019 18:47:18,650 INFO ReportMetricsConfigSupplier - Basic metrics will be collected and reported through the SIM extension because SIM is enabled.` ]
Code Snippet with file paths
# Get Start Time
$startDTM = (Get-Date)
$zstart = Read-Host -prompt '
Enter your start date in "10 Nov" format. Start date must be earlier than stop date.'
$zstop = Read-Host -prompt 'Enter your stop date in "13 Nov" format. Stop date must be later than start date.'
# $zstart = '10 Nov'
# $zstop = '13 Nov'
$zstart= Select-String $zstart "$env:userprofile\Desktop\machine-log.txt" | Select-Object -ExpandProperty LineNumber
$zstop= Select-String $zstop "$env:userprofile\Desktop\machine-logtxt" | Select-Object -ExpandProperty LineNumber
$AppLog = gc $env:userprofile\Desktop\machine-log.txt
$i = 0
$array = #()
foreach ($line in $AppLog){
foreach-object { $i++ }
if (($i -ge $zstart) -and ($i -le $zstop))
{$array += $line}}
$array | Out-File -encoding ascii -filepath $env:userprofile\Desktop\logfile-output.txt
The ERROR i get while executing the script.
8923 8924 8925 8926 8927 8928 8929 8930 8931 8932 8933 8934 8935 8936 8937 8938 8939 8940 8941 8942 8943
8944 8945 8946 8947 8948". Error: "Cannot convert the "System.Object[]" value of type "System.Object[]" to
type "System.Int32"."
At C:\Users\xa_abbasmn\Documents\Logs\test.ps1:13 char:5
+ if (($i -ge $zstart) -and ($i -le $zstop))
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : ComparisonFailure
Could not compare "18" to "1 15 16 17 31 45 46 47 48 62 76 77 91 105 106 107 121 135 136 137 151 152 196 210
211 225 239 240 241 255 269 270 271 285 299 300 314 315 329 330 331 332 346 390 404 447 448 449 463 477 478
492 506 507 508 522 536 537 538 552 566 567 581 582 583 627 641 642 643 657 671 685 686 687
The powershell and windows version:
Name: Windows PowerShell ISE Host - Version : 5.1.14409.1018
Name: Microsoft Windows Server 2012 R2 Standard 64bit
your help will be highly obliged.
Best regards,

Answer 1

Windows 10 64-bit. Powershell 5. Does not require admin privileges.
How to quickly and efficiently extract text from a large logfile from input1 to input2 using powershell 5?
Sample logfile below.
For testing purposes copy your logfile to the desktop and name it logfile.txt
What is the default text editor in Windows Server 2012 R2 Standard 64-bit? See line 42.
What program do you have associated with .txt files? See line 42.
# Get Start Time
$startDTM = (Get-Date)
$zstart = Read-Host -prompt '
Enter your start date in "10 Nov" format (w/o quotes). Start date must be earlier than stop date.'
$zstop = Read-Host -prompt 'Enter your stop date in "13 Nov" format (w/o quotes). Stop date must be later than start date.'
# $zstart = '10 Nov'
# $zstop = '13 Nov'
$zstart= Select-String $zstart "$env:userprofile\Desktop\logfile.txt" | Select-Object -ExpandProperty LineNumber
$zstop= Select-String $zstop "$env:userprofile\Desktop\logfile.txt" | Select-Object -ExpandProperty LineNumber
$AppLog = gc $env:userprofile\Desktop\logfile.txt
$i = 0
$array = #()
foreach ($line in $AppLog){
foreach-object { $i++ }
if (($i -ge $zstart) -and ($i -le $zstop))
{$array += $line}}
$array | Out-File -encoding ascii -filepath $env:userprofile\Desktop\logfile-edited.txt
# begin get file size
$y = (Get-ChildItem "$env:userprofile\Desktop\logfile.txt" | Measure-Object -property length -sum)
$y = [System.Math]::Round(($y.Sum /1KB),2)
$z = (Get-ChildItem "$env:userprofile\Desktop\logfile-edited.txt" | Measure-Object -property length -sum)
$z = [System.Math]::Round(($z.Sum /1KB),2)
# end get file size
# get Stop Time
$endDTM = (Get-Date)
Write-Host "
Extracted $z KB from $y KB. The extraction took $(($endDTM-$startDTM).totalseconds) seconds"
start-process -wait notepad $env:userprofile\Desktop\logfile-edited.txt
remove-item $env:userprofile\Desktop\logfile-edited.txt
exit
logfile.txt:
[AD Thread-Metric Reporter1] 09 Nov 2019 14:48:32,899 ERROR ManagedMonitorDelegate - Error sending metrics
com.singularity.ee.agent.commonservices.metricgeneration.metrics.MetricSendException: Connection back off limitation
at com.singularity.ee.agent.commonservices.metricgeneration.AMetricSubscriber.publish(AMetricSubscriber.java:350)
at com.singularity.ee.agent.commonservices.metricgeneration.MetricReporter.run(MetricReporter.java:113)
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run ]
[AD Thread-Metric Reporter1] 10 Nov 2019 14:47:32,899 ERROR ManagedMonitorDelegate - Error sending metrics
com.singularity.ee.agent.commonservices.metricgeneration.metrics.MetricSendException: Connection back off limitation
at com.singularity.ee.agent.commonservices.metricgeneration.AMetricSubscriber.publish(AMetricSubscriber.java:350)
at com.singularity.ee.agent.commonservices.metricgeneration.MetricReporter.run(MetricReporter.java:113)
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run ]
[AD Thread-Metric Reporter1] 11 Nov 2019 14:46:32,899 ERROR ManagedMonitorDelegate - Error sending metrics
com.singularity.ee.agent.commonservices.metricgeneration.metrics.MetricSendException: Connection back off limitation
at com.singularity.ee.agent.commonservices.metricgeneration.AMetricSubscriber.publish(AMetricSubscriber.java:350)
at com.singularity.ee.agent.commonservices.metricgeneration.MetricReporter.run(MetricReporter.java:113)
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run ]
[extension-scheduler-pool-5] 13 Nov 2019 18:45:40,634 INFO ReportMetricsConfigSupplier - Basic metrics will be collected
[extension-scheduler-pool-8] 14 Nov 2019 18:47:18,650 INFO ReportMetricsConfigSupplier - Basic metrics will be collected
Results of running script on logfile.txt:
[AD Thread-Metric Reporter1] 10 Nov 2019 14:47:32,899 ERROR ManagedMonitorDelegate - Error sending metrics
com.singularity.ee.agent.commonservices.metricgeneration.metrics.MetricSendException: Connection back off limitation
at com.singularity.ee.agent.commonservices.metricgeneration.AMetricSubscriber.publish(AMetricSubscriber.java:350)
at com.singularity.ee.agent.commonservices.metricgeneration.MetricReporter.run(MetricReporter.java:113)
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run ]
[AD Thread-Metric Reporter1] 11 Nov 2019 14:46:32,899 ERROR ManagedMonitorDelegate - Error sending metrics
com.singularity.ee.agent.commonservices.metricgeneration.metrics.MetricSendException: Connection back off limitation
at com.singularity.ee.agent.commonservices.metricgeneration.AMetricSubscriber.publish(AMetricSubscriber.java:350)
at com.singularity.ee.agent.commonservices.metricgeneration.MetricReporter.run(MetricReporter.java:113)
at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run ]
[extension-scheduler-pool-5] 13 Nov 2019 18:45:40,634 INFO ReportMetricsConfigSupplier - Basic metrics will be collected
Powershell in four hours at Youtube
Parse and extract text with powershell at Bing
How to quickly and efficiently extract text from a large logfile from line x to line y using powershell 5?
How to quickly and efficiently extract text from a large logfile from date1 to date2 using powershell 5?