How to remove multiple lines after string in powershell? - shell

I have been at my internship for a couple days and recently was asked to do some Powershell scripting and was excited to learn some new things!
However, it has been a time consuming task because searching for things is so hard to find what you want.
Anyways, I was tasked with removing all sensitive data from a word file. It has not been too bad except until now. For example from a text file:
User pass created now moving on..
Password 7 ##########
All done
Which I had to remove all the numbers after searching the file for "Password 7" and similar tasks which did not take me too long.
Now, I have things such as that are a fixed length after:
Self-Service certificate ####### ######## #######
######## ######## ######## ########## #########
########## ##### ######## ########## ##########
With strings on multiple lines. I can remove the top line, but cannot figure out the next lines because they are just random numbers and I have nothing to search for. I have tried things like nr \n \r and many combinations. I am stumped.
$configFiles=get-childitem . *.txt -rec
foreach ($file in $configFiles)
{
$readIn = #(Get-Content $file.PSPath)
$readIn -replace "Password 7.*" , "Password 7 <REMOVED>" -replace "Secret 5.*" , "Secret 5 <REMOVED>" -replace "snmp-server community\s\S*" , "snmp-server community <REMOVED>" |
Set-Content $file.PSPath
}
That is my current code and it is working well so far. I have been messing around with the multi-line removal in a separate script. Thanks for the help.

Sounds like you are trying to delete a certificate from a Cisco configuration.
$config = #"
!
crypto ca certificate chain TP-self-signed-12345678
certificate self-signed 01
3082022B 30820194 A0030201 02020101 300D0609 2A864886 F70D0101 04050030
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D31 37363538
528BD5A8 E7E26C51 10BAB609 5B60228F C8DE0299 7BE85C2D 9769FF05 C295706F
3082022B 30820194 A0030201 02020101 300D0609 2A864886 F70D0101 04050030
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D31 37363538
528BD5A8 E7E26C51 10BAB609 5B60228F C8DE0299 7BE85C2D 9769FF05 C295706F
3082022B 30820194 A0030201 02020101 300D0609 2A864886 F70D0101 04050030
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D31 37363538
528BD5A8 E7E26C51 10BAB609 5B60228F C8DE0299 7BE85C2D 9769FF05 C295706F
quit
Username joe password bloggs
!
"#
$regex = [regex] '(?sm)certificate self-signed 01\s+([0-9A-F\s]+?)\s+quit'
$result = $config | Select-String -Pattern $regex
$cert = $result.Matches.Groups[1].Value
$censored = $config -replace $cert, '<REMOVED>'
Write-Output $censored
Output:
!
crypto ca certificate chain TP-self-signed-12345678
certificate self-signed 01
<REMOVED>
quit
Username joe password bloggs
!

The trick is to parse the entire text as a single string block and construct a regular expression that can detect the carriage returns embedded in the text.
By default Get-Content will return an array of strings separated at the carriage return. TO load in the test as a single text blob, use the -Raw parameter:
$readIn = Get-Content $file.PSPath -Raw
Then construct a regular expression that can detect the portion you want to remove, carriage returns and all. In this example here, I am assuming that the sensitive bit is 13 blocks of characters at least 5 character long separated by whitespace or carriage return:
$readIn -replace 'Self-Service certificate (\S{5,}[\s\n]+){13}', "Self-Service certificate <removed>`n" | Set-Content $file.PSPath

Related

Powershell - Can't unlock BitLocker as 256 characters long password contains special characters with single double quotes

Here's the background and I have no clue beyond this so tell me how to move ahead from this!
PS C:\> $SecureString = ConvertTo-SecureString "fjuksAS1337" -AsPlainText -Force
PS C:\> Unlock-BitLocker -MountPoint "E:" -Password $SecureString
My password here is:
cF;TA" X%jl"\G{d}rcVzNI=Inps#|P,o{~"k<+#?bm)PjQf^\c8EB! (cL.ZyA.v/yYQ#,!#gN'%"VwlNFs)(h\1Uf#cFdr7BU%zDA;&2R_3w3C3td-Nm,^VFE$cF>N{ol0Y~qR2i`Vm%Q#ckh0]#ZE!ijnirg5k?bj\L;88wBhg8QqO^/T64D#O6Q'H"")/I5(d4v7RC`jH=JH+,Zy*TY4MEf~.b7?;';zLEmB>F^S7aBrUfnN&(Vuhjw}Z3w5
As you see it has multiple single and double quotes which breaks the SecureString command output getting nowhere.
I need to use this password in order to unlock BitLocker drive as the UI throws wrong password error and recovery password of 48 digits is unfortunately lost!
Please help as I am having no idea here at all!
Use a single-quoted here-string.
Anything you put in there won't be escaped in any way.
Example
$PlainTextPassword = #'
cF;TA" X%jl"\G{d}rcVzNI=Inps#|P,o{~"k<+#?bm)PjQf^\c8EB! (cL.ZyA.v/yYQ#,!#gN'%"VwlNFs)(h\1Uf#cFdr7BU%zDA;&2R_3w3C3td-Nm,^VFE$cF>N{ol0Y~qR2i`Vm%Q#ckh0]#ZE!ijnirg5k?bj\L;88wBhg8QqO^/T64D#O6Q'H"")/I5(d4v7RC`jH=JH+,Zy*TY4MEf~.b7?;';zLEmB>F^S7aBrUfnN&(Vuhjw}Z3w5
'#
$SecureString = ConvertTo-SecureString $PlainTextPassword -AsPlainText -Force
For a quick reference:
# verbatim string. Nothing get processed. Single-quotes within the string need to
# be doubled down.
$sq = 'Hello '' world.'
# Expandable string. Stuff that can be expanded will be.
# Double-quotes within the string need to be doubled down or preceded by a backtick `
$dq = "Hello `" "" world"
# verbatim here string. single-quotes within the string do not need to be escaped
# This is the way to go for multiline strings
$sqh = #'
Hello ' World
'#
# Expandable here string. Double quotes within the string do not need to be escaped.
# This is the way to go for multiline strings
$dqh = #"
Hello " World
"#
Reference
About_Quoting_Rules

Powershell 7.x How to Select a Text Substring of Unknown Length Only Using Boundary Substrings

I am trying to store a text file string which has a beginning and end that make it a substring of the original text file. I am new to Powershell so my methods are simple/crude. Basically my approach has been:
Roughly get what I want from the start of the string
Worry about trimming off what I don't want later
My minimum reproducible example is as follows:
# selectStringTest.ps
$inputFile = Get-Content -Path "C:\test\test3\Copy of 31832_226140__0001-00006.txt"
# selected text string needs to span from $refName up to $boundaryName
[string]$refName = "001 BARTLETT"
[string]$boundaryName = "001 BEECH"
# a rough estimate of the text file lines required
[int]$lines = 200
if (Select-String -InputObject $inputFile -pattern $refName) {
Write-Host "Selected shortened string found!"
# this selects the start of required string but with extra text
[string]$newFileStart = $inputFile | Select-String $refName -CaseSensitive -SimpleMatch -Context 0, $lines
}
else {
Write-Host "Selected string NOT FOUND."
}
# tidy up the start of the string by removing rubbish
$newFileStart = $newFileStart.TrimStart('> ')
# this is the kind of thing I want but it doesn't work
$newFileStart = $newFileStart - $newFileStart.StartsWith($boundaryName)
$newFileStart | Out-File tempOutputFile
As it is: the output begins correctly but I cannot remove text including and after $boundaryName
The original text file is OCR generated (Optical Character Recognition) So it is unevenly formatted. There are newlines in odd places. So I have limited options when it comes to delimiting.
I am not sure my if (Select-String -InputObject $inputFile -pattern $refName)is valid. It appears to work correctly. The general design seems crude. In that I am guessing how many lines I will need. And finally I have tried various methods of trimming the string from $boundaryName without success. For this:
string.split() not practical
replacing spaces with newlines in an array & looping through to elements of $boundaryName is possible but I don't know how to terminate the array at this point before returning it to string.
Any suggestions would be appreciated.
Abbreviated content of x2 200 listings single Copy of 31832_226140__0001-00006.txt file is:
Beginning of text file
________________
BARTLETT-BEDGGOOD
PENCARROW COMPOSITE ROLL
PAGE 6
PAGE 7
PENCARROW COMPOSITE ROLL
BEECH-BEST
www.
.......................
001 BARTLETT. Lois Elizabeth
Middle of text file
............. 15 St Ronans Av. Lower Hutt Marned 200 BEDGGOOD. Percy Lloyd
............15 St Ronans Av, Lower Mutt. Coachbuild
001 BEECH, Margaret ..........
End of text file
..............312 Munita Rood Eastbourne, Civil Eng 200 BEST, Dons Amy .........
..........50 Man Street, Wamuomata, Marned
SO NON
To use a regex across newlines, the file needs to be read as a single string. Get-Content -Raw will do that. This assumes that you do not want the lines containing refName and boundaryName included in the output
$c = Get-Content -Path '.\beech.txt' -Raw
$refName = "001 BARTLETT"
$boundaryName = "001 BEECH"
if ($c -match "(?smi).*$refName.*?`r`n(.*)$boundaryName.*?`r`n.*") {
$result = $Matches[1]
}
$result
More information at https://stackoverflow.com/a/12573413/447901
How close does this come to what you want?
function Process-File {
param (
[Parameter(Mandatory = $true, Position = 0)]
[string]$HeadText,
[Parameter(Mandatory = $true, Position = 1)]
[string]$TailText,
[Parameter(ValueFromPipeline)]
$File
)
Process {
$Inside = $false;
switch -Regex -File $File.FullName {
#'^\s*$' { continue }
"(?i)^\s*$TailText(?<Tail>.*)`$" { $Matches.Tail; $Inside = $false }
'^(?<Line>.+)$' { if($Inside) { $Matches.Line } }
"(?i)^\s*$HeadText(?<Head>.*)`$" { $Matches.Head; $Inside = $true }
default { continue }
}
}
}
$File = 'Copy of 31832_226140__0001-00006.txt'
#$Path = $PSScriptRoot
$Path = 'C:\test\test3'
$Result = Get-ChildItem -Path "$Path\$File" | Process-File '001 BARTLETT' '001 BEECH'
$Result | Out-File -FilePath "$Path\SpanText.txt"
This is the output:
. Lois Elizabeth
............. 15 St Ronans Av. Lower Hutt Marned 200 BEDGGOOD. Percy Lloyd
............15 St Ronans Av, Lower Mutt. Coachbuild
, Margaret ..........

Windows Powershell script to find and replace a string after a particular string

I am currently working to convert AS3 class to JavaScript using Powershell script.
Below is the sample code needs to be converted.
package somePackageName
{
class someClassName
{
// other codes
}
}
I need the entire package block to be removed and "class someClassName{" should be converted to "function someClassName(){".
The "someClassName" can be any string.
And I need the output like this.
function someClassName()
{
}
This is what I tried.
$l1 = Get-Content $dest | Where-Object {$_ -like 'class'}
$arr = $l1 -split ' '
$n1 = "function "+ $arr[1] + "() " +$arr[2]
(Get-Content $dest) -creplace $l1, $n1 | Set-Content $dest
I can able to achieve what I intended if the opening brace is in same line as the package declaration line. As Powershell checks line by line, I am stuck if the opening brace present in next line.
Regex based solution
Depending on your willingness to post process this or accept leading spaces you could use this regex to remove the block outside of the class and replace with a function declaration. This is messier than it needs to be but safer since we cannot guess what // other codes is. You could just match the whole class block outright but if there are other curly braces in there it would muddy the regex.
PS M:\> (Get-Content -Raw $dest) -replace "(?sm).*?class (\w+)(.*)}",'function $1()$2'
function someClassName()
{
// other codes
}
See Regex101 for more detail on what the regex is doing.
Basically dump everything until the word class (first time). Then keep everything until the last closing brace
Note the leading space in the greater portion. This is honoring the existing space. To account for this we need to calculate the indentation. Simply removing all leading space would break existing indentation in the class/function.
So a solution like this might be preferred:
# Read in the file as a single string
$raw = (Get-Content -Raw $dest)
# Using the line that has the class declaration measure the number of spaces in front of it.
[void]($raw -match "(?m)^(\s+)class")
$leadingSpacesToRemove = $Matches[1].Length
# Remove the package block. Also remove a certain amount of leading space.
$raw -replace "(?sm).*?class (\w+)(.*)}",'function $1()$2' -replace "(?m)^\s{$leadingSpacesToRemove}"
Less regex
Seems filtering the lines with no leading spaces is an easy way to narrow down to what you want.
Get-Content $dest | Where-Object{$_.StartsWith(" ")}
From there we still need to replace the "class" and deal with the leading spaces. For those we are going to use similar solutions to what I showed above.
# Read in the file as a single string. Skipping the package wrapper since it has no leading spaces.
$classBlock = Get-Content $dest | Where-Object{$_.StartsWith(" ")}
# Get the class name and the number of leading spaces.
$classBlock[0] -match "^(\s+)class (\w+)" | Out-Null
$leadingSpacesToRemove = $matches[1].Length
$className = $matches[2]
# Output the new declaration and the trimmed block.
# Using an array to start so that piping output will be in one pipe
#("function $className()") + ($classBlock | Select -Skip 1) -replace "^\s{$leadingSpacesToRemove}"
Both solutions try to account for your exact specifications and account for the presence of weird stuff inside the class block.
I'd suggest using regex:
#class myclass -> function myclass()
#(Get-Content $dest) -creplace 'class\s(.+)', 'function $1()' |
Set-Content $dest
This will capture the class declaration and replace it with a backreference to the class name capture.

Bash Scripting - Search files for line above search criteria

I have 100s of config files, each 10,000 to 20,000 lines long. These are config files for hardware. I need to search through all the config files to find the "profile" associated with a given cert name. There are several different versions of hardware software so the configs files are somewhat different. However the profile name is always above the cert. The profile name does not necessarily contain the cert name.
Example of Profile Names:
clientssl_www.profile-cert
clientssl_www.example.com-cert
Example of Cert Name:
www.example.com.crt
Example sections of config:
profile clientssl clientssl_www.profile-cert {
defaults from clientssl
key "www.example.com.key"
cert "www.example.com.crt"
chain "Intermediate-bundle.crt"
options {
cipher server preference
dont insert empty fragments
no sslv2
}
}
ltm profile client-ssl /Common/clientssl_www.example.com-cert {
app-service none
cert /Common/www.example.com.crt
cert-key-chain {
www.example.com_www.example.com {
cert /Common/www.example.com.crt
chain /Common/Intermediate-bundle.crt
key /Common/www.example.com.key
}
}
chain /Common/Intermediate-bundle.crt
ciphers
key /Common/www.example.com.key
options { dont-insert-empty-fragments cipher-server-preference no-sslv2 }
}
I cannot read the config files line by line as there are millions of lines and it simply takes too long.
I can find the cert name with grep using something like this:
$ grep www.example.com *file.conf | egrep 'cert "|cert /Common'
Which gives me something like this:
cert "www.example.com.crt"
cert /Common/www.example.com.crt
cert /Common/www.example.com.crt
I need to find the 'profile name' that is above my search for a given cert name.
Any suggestions?
Thanks!
You can use -B option of grep which comes handy in such cases. From the man pages for grep:
-B NUM, --before-context=NUM
Print NUM lines of leading context before matching lines. Places a line containing a group separator (--) between contiguous groups
of matches. With the -o or --only-matching option, this has no effect and a warning is given.
So, the pattern match will now be:
$ grep www.example.com *file.conf | egrep -B3 'cert "|cert /Common'
Output:
profile clientssl clientssl_www.profile-cert {
defaults from clientssl
key "www.example.com.key"
cert "www.example.com.crt"
--
ltm profile client-ssl /Common/clientssl_www.example.com-cert {
app-service none
cert /Common/www.example.com.crt
cert-key-chain {
www.example.com_www.example.com {
cert /Common/www.example.com.crt
However, you will still need to figure out some common pattern in the line containing profile name to single them out. It becomes difficult in your example to filter it further because in the first case, the profile name is three lines before the cert " pattern whereas in the second example, it is two lines before cert / pattern.
Another approach which i find better is to find some pattern in the profile name itself. If all profile names contain the string profile or if they have a pattern such as clientssl.*-cert, then the following pattern match will do what you need:
$ grep www.example.com *file.conf | egrep 'profile|clientssl.*-cert'
Output:
profile clientssl clientssl_www.profile-cert {
ltm profile client-ssl /Common/clientssl_www.example.com-cert {
Even better, if you know that the profile name starts with clientssl_ and ends with -cert, then
$ grep www.example.com *file.conf | grep -o clientssl_.*-cert
Output:
clientssl_www.profile-cert
clientssl_www.example.com-cert
This may be madness, but whenever I see sample data that fits Tcl's syntax rules, I look to produce a Tcl solution:
#!/usr/bin/env tclsh
proc unknown {cmdname args} {
set data [lindex $args end]
if {[set idx [lsearch -exact $data "cert"]] != -1 && [string match $::cert_pattern [lindex $data [incr idx]]]} {
set idx [expr {$cmdname eq "profile" ? 1 : [lsearch -exact $args "profile"] + 2}]
puts [lindex [split [lindex $args $idx] /] end]
}
}
set cert_pattern "*[lindex $argv 0]*"
foreach file [lrange $argv 1 end] {
source $file
}
Then
$ ./cert.tcl www.example.com file.conf
file.conf
clientssl_www.profile-cert
clientssl_www.example.com-cert
I won't bother to explain how it works unless there's a hue and cry.

UNIX format files with Powershell

How do you create a unix file format in Powershell? I am using the following to create a file, but it always creates it in the windows format.
"hello world" | out-file -filepath test.txt -append
As I understand, the new line characters CRLF make it to be a Windows format file whereas the unix format needs only a LF at the end of the line. I tried replacing the CRLF with the following, but it didn't work
"hello world" | %{ $_.Replace("`r`n","`n") } | out-file -filepath test.txt -append
There is a Cmdlet in the PowerShell Community Extensions called ConvertTo-UnixLineEnding
One ugly-looking answer is (taking input from dos.txt outputting to unix.txt):
[string]::Join( "`n", (gc dos.txt)) | sc unix.txt
but I would really like to be able to make Set-Content do this by itself and this solution does not stream and therefore does not work well on large files...
And this solution will end the file with a DOS line ending as well... so it is not 100%
I've found that solution:
sc unix.txt ([byte[]][char[]] "$contenttext") -Encoding Byte
posted above, fails on encoding convertions in some cases.
So, here is yet another solution (a bit more verbose, but it works directly with bytes):
function ConvertTo-LinuxLineEndings($path) {
$oldBytes = [io.file]::ReadAllBytes($path)
if (!$oldBytes.Length) {
return;
}
[byte[]]$newBytes = #()
[byte[]]::Resize([ref]$newBytes, $oldBytes.Length)
$newLength = 0
for ($i = 0; $i -lt $oldBytes.Length - 1; $i++) {
if (($oldBytes[$i] -eq [byte][char]"`r") -and ($oldBytes[$i + 1] -eq [byte][char]"`n")) {
continue;
}
$newBytes[$newLength++] = $oldBytes[$i]
}
$newBytes[$newLength++] = $oldBytes[$oldBytes.Length - 1]
[byte[]]::Resize([ref]$newBytes, $newLength)
[io.file]::WriteAllBytes($path, $newBytes)
}
make your file in the Windows CRLF format. then convert all lines to Unix format in new file:
$streamWriter = New-Object System.IO.StreamWriter("\\wsl.localhost\Ubuntu\home\user1\.bashrc2")
$streamWriter.NewLine = "`n"
gc "\\wsl.localhost\Ubuntu\home\user1\.bashrc" | % {$streamWriter.WriteLine($_)}
$streamWriter.Flush()
$streamWriter.Close()
not a one-liner, but works for all lines, including EOF. new file now shows as Unix format in Notepad on Win11.
delete original file & rename new file to original, if you like:
ri "\\wsl.localhost\Ubuntu\home\user1\.bashrc" -Force
rni "\\wsl.localhost\Ubuntu\home\user1\.bashrc2" "\\wsl.localhost\Ubuntu\home\user1\.bashrc"
Two more examples on how you can replace CRLF by LF:
Example:
(Get-Content -Raw test.txt) -replace "`r`n","`n" | Set-Content test.txt -NoNewline
Example:
[IO.File]::WriteAllText('C:\test.txt', ([IO.File]::ReadAllText('C:\test.txt') -replace "`r`n","`n"))
Be aware, this does really just replace CRLF by LF. You might need to add a trailing LF if your Windows file does not contain a trailing CRLF.

Resources