I have a really bad habit of abusing tr.
I need to find another way, a different style.
all I want to so is print the list horizontally instead of vertically - so I can cut and past it into an email. Check out the use of the TR command. Just terrible.
$ cat /tmp/wig
update PTMM_ARCHIVE.FASTTRACK_USER set user_name = 'monohajoxx' where user_name = 'monohajo'
update PTMM_ARCHIVE.FASTTRACK_USER set user_name = 'wuemxx' where user_name = 'wuem'
update PTMM_ARCHIVE.FASTTRACK_USER set user_name = 'taraziemxx' where user_name = 'taraziem'
update PTMM_ARCHIVE.FASTTRACK_USER set user_name = 'mullankexx' where user_name = 'mullanke'
update PTMM_ARCHIVE.FASTTRACK_USER set user_name = 'fernanjaxx' where user_name = 'fernanja'
$ awk '{print $NF}' /tmp/wig | tr -d "'" | tr "\n" ", \s" ; echo "\n"
monohajo,wuem,taraziem,mullanke,fernanja,\n
Using awk
Here is one way to do it entirely with awk:
$ awk '{gsub(/'\''/,"",$NF); printf "%s%s",(NR>1?",":""),$NF} END{print "\\n"}' wig
monohajo,wuem,taraziem,mullanke,fernanja\n
The gsub command removes the single-quotes from the last field. The printf command prints the last field preceded by a comma if this isn't the first line. The final print statement finishes the line.
And, here is another:
$ awk '{printf "%s%s",(NR>1?",":""),substr($NF,2,length($NF)-2)} END{print "\\n"}' wig
monohajo,wuem,taraziem,mullanke,fernanja\n
This uses a similar printf statement but uses substr to remove the first and last characters of the last field.
Using sed
$ sed -nE "s/.*'([^']*)'/\1/"'; H; 1h; ${x; s/\n/,/g; s/$/\\n/; p}' wig
monohajo,wuem,taraziem,mullanke,fernanja\n
How it works:
-n tells sed not to print anything unless we explicitly ask it to.
-E tells sed to use extended regular expressions so that we don't have to type as many backslashes.
s/.*'([^']*)'/\1/
This removes everything from the line except for the last field single-quoted string (with the quotes are removed).
H; 1h;
H adds a newline to the hold space followed by a copy of the current pattern space (which now contains the last field, minus the quotes).
If this is the first line, however, the h command overwrites the hold space with just the current value of the pattern space (no newline).
${x; s/\n/,/g; s/$/\\n/; p}
On the last line, denoted by $, this does the following:
- `x` exchanges the hold and pattern spaces.
- `s/\n/,/g` converts all those newlines to commas.
- `s/$/\\n/` puts a `\n` at the end.
- `p` causes this pattern space to be printed.
Related
I have a string value stored in a variable:
PTYPE="Other Farm|Raised Ranch|Farm house|Other|A-Frame|Log Home"
I want to find & replace Other with some value like NOTHING. All values are stored in variables.
WhatToChange=Other
NewValue=NOTHING
echo $PTYPE|sed -e "s#${WhatToChange}#${NewValue}#g"
This is replacing all the occurances of Other and getting output like:
NOTHING Farm|Raised Ranch|Farm house|NOTHING|A-Frame|Log Home
Is there any way I can exactly change only the exact one? The place for ${WhatToChange} is variable.
As you have well defined fields and want an exact match, awk could be easier to use than sed; at the very least, you won't have to worry about escaping the strings for using it in the sed expression:
echo "Other Farm|Raised Ranch|Farm house|Other|A-Frame|Log Home" |
awk -v old="Other" -v new="NOTHING" \
'BEGIN {FS = OFS = "|"} {for(i=1;i<=NF;i++) if($i == old) $i = new} 1'
output:
Other Farm|Raised Ranch|Farm house|NOTHING|A-Frame|Log Home
To match either the exact character | or the beginning of the line, use ([|]|^).
To match either the exact character | or the end of the line, use ([|]|$).
To put a | back in place only when appropriate, store these in match groups, and refer to those groups with \1 or \2:
PTYPE="Other Farm|Raised Ranch|Farm house|Other|A-Frame|Log Home"
WhatToChange=Other
NewValue=NOTHING
sed -re "s#(^|[|])${WhatToChange}($|[|])#\1${NewValue}\2#g" <<<"$PTYPE"
...emits as output:
Other Farm|Raised Ranch|Farm house|NOTHING|A-Frame|Log Home
...and still works even if WhatToChange is matched at the beginning or end of the list.
For fun, some perl:
This is like #Charles's sed solution: Note the \Q...\E so that the "to change" value is treated as literal text.
echo "$PTYPE" | perl -spe '
s{ (?:^|\|)\K \Q$WhatToChange\E (?=\||$) }{$NewValue}gx
' -- -WhatToChange=Other -NewValue=NOTHING
This is like #Fravadona's awk solution:
echo "$PTYPE" | perl -F'[|]' -sane '
print join "|", map {$_ eq $WhatToChange ? $NewValue : $_} #F
' -- -WhatToChange=Other -NewValue=NOTHING
How about
echo ${PTYPE//$WhatToChange/$NewValue}
UPDATE:
I just realized that the replacement should happen only if WhatToChange is the whole content between two separators (|). In this case, we can do it in bash as well (without the need to revert to a child process):
if [[ $PTYPE =~ (.*[|]|^)$WhatToChange([|].*|$) ]]
echo "${BASH_REMATCH[1]}${NewValue}${BASH_REMATCH[2]}"
fi
UPDATE (based on the comment by Fravadona):
Used in this way, WhatToChange is interpreted as a regular expression. This can be useful, if you want to catch for instance variations of the string, for instance
WhatToChange='[Oo]ther' # to catch Other and other
If you always want to have a literal match, you have to quote the variable:
[[ $PTYPE =~ (.*[|]|^)"$WhatToChange"([|].*|$) ]]
This might work for you (GNU sed & bash):
<<<"$PTYPE" sed 'y/|/\n/;s/^'"$WhatToChange"'$/'"$NewValue"'/mg;y/\n/|/'
Input $PTYPE as a here-string into sed.
Translate | separators to newlines.
Replace $WhatToChange to $NewValue for each matched line.
Translate newlines back to |'s.
N.B. The use of the m flag in the substitution command allows sed to work in multiline mode and this presents each value between separators on its own line.
An alternative:
sed -z 'y/|/\x00/;s/^'"$WhatToChange"'$/'"$NewValue"'/mg;y/\x00/|/;' file
I have been tinkering with this for a while but can't quite figure it out. A sample line within the file looks like this:
"...~236 characters of data...Y YYY. Y...many more characters of data"
How would I use sed or awk to replace spaces with a B character only between positions 236 and 246? In that example string it starts at character 29 and ends at character 39 within the string. I would want to preserve all the text preceding and following the target chunk of data within the line.
For clarification based on the comments, it should be applied to all lines in the file and expected output would be:
"...~236 characters of data...YBBYYY.BBY...many more characters of data"
With GNU awk:
$ awk -v FIELDWIDTHS='29 10 *' -v OFS= '{gsub(/ /, "B", $2)} 1' ip.txt
...~236 characters of data...YBBYYY.BBY...many more characters of data
FIELDWIDTHS='29 10 *' means 29 characters for first field, next 10 characters for second field and the rest for third field. OFS is set to empty, otherwise you'll get space added between the fields.
With perl:
$ perl -pe 's/^.{29}\K.{10}/$&=~tr| |B|r/e' ip.txt
...~236 characters of data...YBBYYY.BBY...many more characters of data
^.{29}\K match and ignore first 29 characters
.{10} match 10 characters
e flag to allow Perl code instead of string in replacement section
$&=~tr| |B|r convert space to B for the matched portion
Use this Perl one-liner with substr and tr. Note that this uses the fact that you can assign to substr, which changes the original string:
perl -lpe 'BEGIN { $from = 29; $to = 39; } (substr $_, ( $from - 1 ), ( $to - $from + 1 ) ) =~ tr/ /B/;' in_file > out_file
To change the file in-place, use:
perl -i.bak -lpe 'BEGIN { $from = 29; $to = 39; } (substr $_, ( $from - 1 ), ( $to - $from + 1 ) ) =~ tr/ /B/;' in_file
The Perl one-liner uses these command line flags:
-e : Tells Perl to look for code in-line, instead of in a file.
-p : Loop over the input one line at a time, assigning it to $_ by default. Add print $_ after each loop iteration.
-l : Strip the input line separator ("\n" on *NIX by default) before executing the code in-line, and append it when printing.
-i.bak : Edit input files in-place (overwrite the input file). Before overwriting, save a backup copy of the original file by appending to its name the extension .bak.
I would use GNU AWK following way, for simplicity sake say we have file.txt content
S o m e s t r i n g
and want to change spaces from 5 (inclusive) to 10 (inclusive) position then
awk 'BEGIN{FPAT=".";OFS=""}{for(i=5;i<=10;i+=1)$i=($i==" "?"B":$i);print}' file.txt
output is
S o mBeBsBt r i n g
Explanation: I set field pattern (FPAT) to any single character and output field seperator (OFS) to empty string, thus every field is populated by single characters and I do not get superfluous space when print-ing. I use for loop to access desired fields and for every one I check if it is space, if it is I assign B here otherwise I assign original value, finally I print whole changed line.
Using GNU awk:
awk -v strt=29 -v end=39 '{ ram=substr($0,strt,(end-strt));gsub(" ","B",ram);print substr($0,1,(strt-1)) ram substr($0,(end)) }' file
Explanation:
awk -v strt=29 -v end=39 '{ # Pass the start and end character positions as strt and end respectively
ram=substr($0,strt,(end-strt)); # Extract the 29th to the 39th characters of the line and read into variable ram
gsub(" ","B",ram); # Replace spaces with B in ram
print substr($0,1,(strt-1)) ram substr($0,(end)) # Rebuild the line incorporating raw and printing the result
}'file
This is certainly a suitable task for perl, and saddens me that my perl has become so rusty that this is the best I can come up with at the moment:
perl -e 'local $/=\1;while(<>) { s/ /B/ if $. >= 236 && $. <= 246; print }' input;
Another awk but using FS="":
$ awk 'BEGIN{FS=OFS=""}{for(i=29;i<=39;i++)sub(/ /,"B",$i)}1' file
Output:
"...~236 characters of data...YBBYYY.BBY...many more characters of data"
Explained:
$ awk ' # yes awk yes
BEGIN {
FS=OFS="" # set empty field delimiters
}
{
for(i=29;i<=39;i++) # between desired indexes
sub(/ /,"B",$i) # replace space with B
# if($i==" ") # couldve taken this route, too
# $i="B"
}1' file # implicit output
With sed :
sed '
H
s/\(.\{236\}\)\(.\{11\}\).*/\2/
s/ /B/g
H
g
s/\n//g
s/\(.\{236\}\)\(.\{11\}\)\(.*\)\(.\{11\}\)/\1\4\3/
x
s/.*//
x' infile
When you have an input string without \r, you can use:
sed -r 's/(.{236})(.{10})(.*)/\1\r\2\r\3/;:a;s/(\r.*) (.*\r)/\1B\2/;ta;s/\r//g' input
Explanation:
First put \r around the area that you want to change.
Next introduce a label to jump back to.
Next replace a space between 2 markers.
Repeat until all spaces are replaced.
Remove the markers.
In your case, where the length doesn't change, you can do without the markers.
Replace a space after 236..245 characters and try again when it succeeds.
sed -r ':a; s/^(.{236})([^ ]{0,9}) /\1\2B/;ta' input
This might work for you (GNU sed):
sed -E 's/./&\n/245;s//\n&/236/;h;y/ /B/;H;g;s/\n.*\n(.*)\n.*\n(.*)\n.*/\2\1/' file
Divide the problem into 2 lines, one with spaces and one with B's where there were spaces.
Then using pattern matching make a composite line from the two lines.
N.B. The newline can be used as a delimiter as it is guaranteed not to be in seds pattern space.
I have a string like this:
/var/cpanel/users/joebloggs:DNS9=domain.example
I need to extract the username (joebloggs) from this string and store it in a variable.
The format of the string will always be the same with exception of joebloggs and domain.example so I am thinking the string can be split twice using cut?
The first split would split by : and we would store the first part in a variable to pass to the second split function.
The second split would split by / and store the last word (joebloggs) into a variable
I know how to do this in PHP using arrays and splits but I am a bit lost in bash.
To extract joebloggs from this string in bash using parameter expansion without any extra processes...
MYVAR="/var/cpanel/users/joebloggs:DNS9=domain.example"
NAME=${MYVAR%:*} # retain the part before the colon
NAME=${NAME##*/} # retain the part after the last slash
echo $NAME
Doesn't depend on joebloggs being at a particular depth in the path.
Summary
An overview of a few parameter expansion modes, for reference...
${MYVAR#pattern} # delete shortest match of pattern from the beginning
${MYVAR##pattern} # delete longest match of pattern from the beginning
${MYVAR%pattern} # delete shortest match of pattern from the end
${MYVAR%%pattern} # delete longest match of pattern from the end
So # means match from the beginning (think of a comment line) and % means from the end. One instance means shortest and two instances means longest.
You can get substrings based on position using numbers:
${MYVAR:3} # Remove the first three chars (leaving 4..end)
${MYVAR::3} # Return the first three characters
${MYVAR:3:5} # The next five characters after removing the first 3 (chars 4-9)
You can also replace particular strings or patterns using:
${MYVAR/search/replace}
The pattern is in the same format as file-name matching, so * (any characters) is common, often followed by a particular symbol like / or .
Examples:
Given a variable like
MYVAR="users/joebloggs/domain.example"
Remove the path leaving file name (all characters up to a slash):
echo ${MYVAR##*/}
domain.example
Remove the file name, leaving the path (delete shortest match after last /):
echo ${MYVAR%/*}
users/joebloggs
Get just the file extension (remove all before last period):
echo ${MYVAR##*.}
example
NOTE: To do two operations, you can't combine them, but have to assign to an intermediate variable. So to get the file name without path or extension:
NAME=${MYVAR##*/} # remove part before last slash
echo ${NAME%.*} # from the new var remove the part after the last period
domain
Define a function like this:
getUserName() {
echo $1 | cut -d : -f 1 | xargs basename
}
And pass the string as a parameter:
userName=$(getUserName "/var/cpanel/users/joebloggs:DNS9=domain.example")
echo $userName
What about sed? That will work in a single command:
sed 's#.*/\([^:]*\).*#\1#' <<<$string
The # are being used for regex dividers instead of / since the string has / in it.
.*/ grabs the string up to the last backslash.
\( .. \) marks a capture group. This is \([^:]*\).
The [^:] says any character _except a colon, and the * means zero or more.
.* means the rest of the line.
\1 means substitute what was found in the first (and only) capture group. This is the name.
Here's the breakdown matching the string with the regular expression:
/var/cpanel/users/ joebloggs :DNS9=domain.example joebloggs
sed 's#.*/ \([^:]*\) .* #\1 #'
Using a single Awk:
... | awk -F '[/:]' '{print $5}'
That is, using as field separator either / or :, the username is always in field 5.
To store it in a variable:
username=$(... | awk -F '[/:]' '{print $5}')
A more flexible implementation with sed that doesn't require username to be field 5:
... | sed -e s/:.*// -e s?.*/??
That is, delete everything from : and beyond, and then delete everything up until the last /. sed is probably faster too than awk, so this alternative is definitely better.
Using a single sed
echo "/var/cpanel/users/joebloggs:DNS9=domain.example" | sed 's/.*\/\(.*\):.*/\1/'
I like to chain together awk using different delimitators set with the -F argument. First, split the string on /users/ and then on :
txt="/var/cpanel/users/joebloggs:DNS9=domain.com"
echo $txt | awk -F"/users/" '{print$2}' | awk -F: '{print $1}'
$2 gives the text after the delim, $1 the text before it.
I know I'm a little late to the party and there's already good answers, but here's my method of doing something like this.
DIR="/var/cpanel/users/joebloggs:DNS9=domain.example"
echo ${DIR} | rev | cut -d'/' -f 1 | rev | cut -d':' -f1
I'm reading from a file and want to
Get the text for value where key = "key".
i.e.
key = value
Trim any white space around the value or the key
However, value can also contain an = because it's a base64 encoded field.
I've been using this previously:
key=`egrep 'key' myfile | cut -f2 -d'=' | sed 's/ //g'`
But, cut works globally. Is there a way of making it work on just the first item?
Or perhaps there more efficient way of doing this.
e.g. if myfile = ceph.client.keyring
[client.admin]
key = AQAa6HRVaDKxLxAANulnamD/5x2SBly7kPPatg==
auid = 0
caps mds = "allow"
caps mon = "allow *"
caps osd = "allow *"
I'm wanting to read into a variable the value for key (AQAa6HRVaDKxLxAANulnamD/5x2SBly7kPPatg==)
Using sed
Your your example myfile:
$ sed -rn '/^[[:space:]]*key[ =]/ s/[^=]*=[[:space:]]*//p' myfile
AQAa6HRVaDKxLxAANulnamD/5x2SBly7kPPatg==
How it works:
sed -rn
This tells sed to use extended regular expressions, -r, and not to print unless we explicitly ask it to, -n.
/^[[:space:]]*key[ =]/
This selects lines that optionally start with whitespace, followed by key, followed by either a blank or an equal sign, =.
s/[^=]*=[[:space:]]*//p
For those selected lines, this strips out everything before the first equal sign and any whitespace after the equal sign. The p option tells sed to print the resulting line.
Using awk
This assumes that, as in your example, the first equal sign is surrounded by spaces and that the value for key contains no whitespace:
$ awk '$1=="key"{print $3}' myfile
AQAa6HRVaDKxLxAANulnamD/5x2SBly7kPPatg==
How it works:
$1=="key"
This selects lines whose first field is key.
{print $3}
For those selected lines, this prints the third field.
Alternate sed solution
Doing the selecting and substituting in one step:
$ sed -rn 's/^[[:space:]]*key[[:space:]]=[[:space:]]*//p' myfile
AQAa6HRVaDKxLxAANulnamD/5x2SBly7kPPatg==
Using grep with Perl regex:
grep -oP "(?<=\skey\s=\s)[^\s]*" file
Output:
AQAa6HRVaDKxLxAANulnamD/5x2SBly7kPPatg==
Note: It assumes there's only one white space between key and =, & = and value
Or using sed:
sed -nr 's/.*\skey\s*=\s*([^\s]*).*/\1/p' file
AQAa6HRVaDKxLxAANulnamD/5x2SBly7kPPatg==
I have a string which i need to insert at a specific position in a file :
The file contains multiple semicolons(;) i need to insert the string just before the last ";"
Is this possible with SED ?
Please do post the explanation with the command as I am new to shell scripting
before :
adad;sfs;sdfsf;fsdfs
string = jjjjj
after
adad;sfs;sdfsf jjjjj;fsdfs
Thanks in advance
This might work for you:
echo 'adad;sfs;sdfsf;fsdfs'| sed 's/\(.*\);/\1 jjjjj;/'
adad;sfs;sdfsf jjjjj;fsdfs
The \(.*\) is greedy and swallows the whole line, the ; makes the regexp backtrack to the last ;. The \(.*\) make s a back reference \1. Put all together in the RHS of the s command means insert jjjjj before the last ;.
sed 's/\([^;]*\)\(;[^;]*;$\)/\1jjjjj\2/' filename
(substitute jjjjj with what you need to insert).
Example:
$ echo 'adad;sfs;sdfsf;fsdfs;' | sed 's/\([^;]*\)\(;[^;]*;$\)/\1jjjjj\2/'
adad;sfs;sdfsfjjjjj;fsdfs;
Explanation:
sed finds the following pattern: \([^;]*\)\(;[^;]*;$\). Escaped round brackets (\(, \)) form numbered groups so we can refer to them later as \1 and \2.
[^;]* is "everything but ;, repeated any number of times.
$ means end of the line.
Then it changes it to \1jjjjj\2.
\1 and \2 are groups matched in first and second round brackets.
For now, the shorter solution using sed : =)
sed -r 's#;([^;]+);$#; jjjjj;\1#' <<< 'adad;sfs;sdfsf;fsdfs;'
-r option stands for extented Regexp
# is the delimiter, the known / separator can be substituted to any other character
we match what's finishing by anything that's not a ; with the ; final one, $ mean end of the line
the last part from my explanation is captured with ()
finally, we substitute the matching part by adding "; jjjj" ans concatenate it with the captured part
Edit: POSIX version (more portable) :
echo 'adad;sfs;sdfsf;fsdfs;' | sed 's#;\([^;]\+\);$#; jjjjj;\1#'
echo 'adad;sfs;sdfsf;fsdfs;' | sed -r 's/(.*);(.*);/\1 jjjj;\2;/'
You don't need the negation of ; because sed is by default greedy, and will pick as much characters as it can.
sed -e 's/\(;[^;]*\)$/ jjjj\1/'
Inserts jjjj before the part where a semicolon is followed by any number of non-semicolons ([^;]*) at the end of the line $. \1 is called a backreference and contains the characters matched between \( and \).
UPDATE: Since the sample input has no longer a ";" at the end.
Something like this may work for you:
echo "adad;sfs;sdfsf;fsdfs"| awk 'BEGIN{FS=OFS=";"} {$(NF-1)=$(NF-1) " jjjjj"; print}'
OUTPUT:
adad;sfs;sdfsf jjjjj;fsdfs
Explanation: awk starts with setting FS (field separator) and OFS (output field separator) as semi colon ;. NF in awk stands for number of fields. $(NF-1) thus means last-1 field. In this awk command {$(NF-1)=$(NF-1) " jjjjj" I am just appending jjjjj to last-1 field.