I need to use HTTP Apache piped logging to place all entries with response code "400" into a separate file. For that, I want to use PowerShell Select-String since it is the closest thing to the grep command in Linux:
I have the following batch to call a PowerShell script from within a cmd environment:
powershell -command "& { sls ,400, 'cmd -ca | select -exp line >> access_400.log }"
As far as I know, the "CustomLog" in httpd will be similar to this:
CustomLog "| 'call_powershell_script_that_will_extract_400_entries.ps1' " common
How can I make the piped output of CustomLog be input to the PowerShell script that will extract the entries with "400" response code?
Use the automatic PowerShell variable, $input, in your script.
I don't totally understand your script, but you should be able to extrapolate this similar example:
filter.bat
#powershell -command "$input | select-string -casesensitive -pattern '%1' | select-object -expandproperty line | out-file -append -encoding ASCII output.log"
$input will be filled with the console input for the BAT/CMD file.
To test, from the command line, use dir | filter.bat ".exe", then type output.log.
Notably, I'm using out-file instead of the powershell redirection operator so that the output file is not encoded in the powershell default, UTF-16, but as a more generally usable ASCII file.
Related
Currently, I'm using the following command to output filenames to a text file:
( Get-ChildItem -File ).BaseName | Out-File "Track List.txt" -Encoding utf8
How can I limit the filenames to a specific extension, e.g., .mp3?
The -Filter paramater accepts wildecards - ? for any single character, and * for zero or more matches
( Get-ChildItem -File -Filter *.mp3).BaseName | Out-File "Track List.txt" -Encoding utf8
See the Powershell documentaiton on -Filter and wildcards for more information
[wildcards]:
I tried to create a multiline Input to practice Select-String, expecting only a single matching line to be output, like I would normaly see it in an echo -e ... | grep combination. But the following command still gives me both lines. It seems to be the newline is only interpreted on final ouptut and Select-String still gets a single line of input
Write-Output "Hi`nthere" | Select-String -Pattern "i"
#
# Hi
# there
#
#
while I would expect it to return just
Hi
I used this version of PowerShell:
Get-Host | Select-Object Version
# 5.1.19041.906
Comparing with bash I would do the following for testing commands on multiline input in bash. I usually generate multiple lines with echo -e and then grep processes the individual lines.
echo -e "Hi\nthere" | grep "i"
# Hi
I hope someone can explain what I miss here in PowerShell? This problem seems like a basic misconception to me, where I also was not sure what to Google for.
Edits
[edit 1]: problem also for line ending with carriage return
Write-Output "Hi`r`nthere" | Select-String -Pattern "i"
I saw that separating with commas works as valid multiline input. So maybe the question is how to convert from newline to actual input line separation.
Write-Output "Hi","there" | Select-String -Pattern "i"
# Hi
[edit 2]: from edit 1 I found this stackoverflow-answer, where for me it now works with
Write-Output "Hi`nthere".Split([Environment]::NewLine) | Select-String -Pattern "i"
# or
Write-Output "Hi`nthere".Split("`n") | Select-String -Pattern "i"
Still may someone please explain why this is relevant here, but not in bash?
All the information is in the comments, but let me summarize and complement it:
PowerShell's pipeline is object-based, and Select-String operates on each input object - even if that happens to be a single multi-line string object, such as output by Write-Output "Hi`nthere"
It is only the output from external programs that is streamed line by line.
Therefore, you must split your multi-line string into individual lines in order to match them as such.
The best idiom for that is -split '\r?\n', because it recognizes both Windows-format CRLF and Unix-format LF-only newlines :
"Hi`nthere" -split '\r?\n' | Select-String -Pattern "i"
Note:
I've omitted Write-Output in favor of PowerShell's implicit output behavior (see the bottom section of this answer for more information).
For more information on how -split '\r?\n' works, see this answer.
Select-String doesn't directly output the matching lines (strings); instead it wraps them in match-information objects that provide metadata about each match. To get just the matching line (string):
In PowerShell (Core) 7+, add the -Raw switch.
In Windows PowerShell, pipe to ForEach-Object Line or wrap the entire call in (...).Line
I applied several filters to a text files using Powershell Get-Content and the -nomatch operator and then i spooled the result to a file.
gc file.txt | {?_ -notmatch 'excl1|excl2|excl3'} | out-file newfile.txt
What happens is that the output file (newfile.txt) has less lines, but it is reported by windows with a bigger size than file.txt.
Has someone ever encountered this behavior? How can I have the correct size reported by windows? I checked the number of rows, the file with less rows is reported as bigger in size.
I'm certain you have an encoding issue. By default Get-Content uses ascii whereas Out-File uses Unicode.
From TechNet
-Encoding
Specifies the type of character encoding used in the file. Valid values are "Unicode", "UTF7", "UTF8", "UTF32", "ASCII", "BigEndianUnicode", "Default", and "OEM". "Unicode" is the default.
Use -Enconding ascii with Out-File or just use Set-Content as it is the partner of Get-Content.
Get-Content file.txt | {?_ -notmatch 'excl1|excl2|excl3'} |
out-file -Encoding ascii newfile.txt
# or
Set-Content newfile.txt
Coming from the other direction if you have issues with your input file Get-Content in PowerShell v3 and above also supports -Enconding
Say I have several text files that contain the word 'not' and I want to find them and create a file containing the matches. In Linux, I might do
grep -r not *.txt > found_nots.txt
This works fine. In PowerShell, the following echos what I want to the screen
get-childitem *.txt -recurse | select-string not
However, if I pipe this to a file:
get-childitem *.txt -recurse | select-string not > found_nots.txt
It runs for ages. I eventually CTRL-C to exit and take a look at the found_nots.txt file which is truly huge. It looks as though PowerShell includes the output file as one of the files to search. Every time it adds more content, it finds more to add.
How can I stop this behavior and make it behave more like the Unix version?
Use the -Exclude option.
get-childitem *.txt -Exclude 'found_nots.txt' -recurse | select-string not > found_nots.txt
First easy solution is rename file output extension to another
I am trying a simple replace script to replace text in a app.cofig file. But it just processed and do nothing:
$old = 'Z:\gene'
$new = 'z:\gene\scripts'
Get-ChildItem z:\gene\scripts\Test\App.config -Recurse | Where {$_ -IS [IO.FileInfo]} |
% {
(Get-Content $_.FullName) -replace $old,$new | Set-Content $_.FullName
Write-Host "Processed: " + $_.FullName
}
Any Idea what I am doing wrong. As same script works fine for .txt file
Thanks
App.config is xml formatted but it's a text file as well, it should work the same. My guess is that you have a different values that your working on and they are not hitting. If you rename the file to app.txt does it work ? You might also consider using nant xmlpoke if you are running from nant script.