I am trying to process the result of diff operation with sed. This is my diff output, which I pipe into sed
3d2
< 12-03-22_JET_D_CL_UR_l4053_0061 True_Warning All 9 149261
62a62
> 13-01-29_VUE_EPM3_v37_CSAV2_0370 True_Warning All 13 22125
68c68
< 13-05-14_Regular_Front_0062 True_Warning All 13 123383
---
> 13-05-14_Regular_Front_0062 True_Warning All 21 123383
119c119
< CADS4_PMP363_20130202_DPH_069 True_Warning All 13 233405
---
> CADS4_PMP363_20130202_DPH_069 True_Warning All 9 233409
149c149
< CADS4_PMP363_20130315_Fujifilm_UK_186 True_Warning All 21 18611
---
> CADS4_PMP363_20130315_Fujifilm_UK_186 True_Warning All 17 18615
I need to sort out the difference string and prepend the 3rd word in the strings with either "Old" or "New" - depending on the first character. My best effort so far is
diff new_jumps/true.jump old_jumps/true.jump | sed -n "/^[<>]/ s:\(.\) \(\S\+\) \(.\+\):\2 \1,\3: p" | replace ">" Old | replace "<" New
Which give me this result (exactly what I wanted).
12-03-22_JET_D_CL_UR_l4053_0061 New,True_Warning All 9 149261
13-01-29_VUE_EPM3_v37_CSAV2_0370 Old,True_Warning All 13 22125
13-05-14_Regular_Front_0062 New,True_Warning All 13 123383
13-05-14_Regular_Front_0062 Old,True_Warning All 21 123383
CADS4_PMP363_20130202_DPH_069 New,True_Warning All 13 233405
CADS4_PMP363_20130202_DPH_069 Old,True_Warning All 9 233409
CADS4_PMP363_20130315_Fujifilm_UK_186 New,True_Warning All 21 18611
CADS4_PMP363_20130315_Fujifilm_UK_186 Old,True_Warning All 17 18615
My question is - how can I change conditional expression within sed one-liner that will eliminate the need to use replace afterwards? (I assume that it is possible)
Thanks in advance
EDIT:
I know, I missed the option to chain sed expressions, but what I had in mind - is it possible to do it within one substitute operation?
By adding more commands to sed using semicolon (;), like this:
diff new_jumps/true.jump old_jumps/true.jump | sed -n "/^[<>]/ s:\(.\) \(\S\+\) \(.\+\):\2 \1,\3:; s/</New/gp; s/>/Old/gp"
With awk I get a faster response. Try this:
diff new_jumps/true.jump old_jumps/true.jump | awk '{ if($1=="<" || $1==">"){($1=="<")?temp="New,":temp="Old,";print $2,temp$3,$4,$5}}'
Here's another solution suggested by Jidder:
awk '/^</{i="old,"}/^>/{i="new,"}i{$2=$2" "i;print;i=0}'
#volcano: here is a one-liner solution in sed, but relies in the interaction with the shell. IMHO if you want to have only one sed substitution command, you cannot avoid that behavior: you have to output to the shell the information of which first character has been seen on the line, the shell somewhat does the mapping to "Old" or "New" strings, and gives the result back to sed.
So the one-liner is not exactly a one-liner because we have to define things in the shell... ;)
replace() { if [ "$1" == ">" ] ; then echo -n "Old"; else echo -n "New" ; fi }
export -f replace
sed -n '/^[<>]/ s:\(.\) \(\S\+\) \(.\+\):echo "\2 $(replace \\\1),\3";:ep' yourfile
Please note that the e flag to the substitution command is a GNU sed extension, we use it here to avoid calling the shell explicitly. If you don't use GNU sed, you can simply replace the last line above by the following:
sed -n '/^[<>]/ s:\(.\) \(\S\+\) \(.\+\):echo "\2 $(replace \\\1),\3";:p' yourfile | bash
The solution I am giving here has been inspired by that other one.
Please also note that all this gymnastics is avoidable if you accept to replace your three-letter tokens "Old" and "New" by their initials, because then we can neatly use the y command to first act in a tr fashion, likewise:
sed -n '/^[<>]/ y/<>/ON/; s:\(.\) \(\S\+\) \(.\+\):\2 \1,\3:p' yourfile
Related
In my bash script I have an external (received from user) string, which I should use in sed pattern.
REPLACE="<funny characters here>"
sed "s/KEYWORD/$REPLACE/g"
How can I escape the $REPLACE string so it would be safely accepted by sed as a literal replacement?
NOTE: The KEYWORD is a dumb substring with no matches etc. It is not supplied by user.
Warning: This does not consider newlines. For a more in-depth answer, see this SO-question instead. (Thanks, Ed Morton & Niklas Peter)
Note that escaping everything is a bad idea. Sed needs many characters to be escaped to get their special meaning. For example, if you escape a digit in the replacement string, it will turn in to a backreference.
As Ben Blank said, there are only three characters that need to be escaped in the replacement string (escapes themselves, forward slash for end of statement and & for replace all):
ESCAPED_REPLACE=$(printf '%s\n' "$REPLACE" | sed -e 's/[\/&]/\\&/g')
# Now you can use ESCAPED_REPLACE in the original sed statement
sed "s/KEYWORD/$ESCAPED_REPLACE/g"
If you ever need to escape the KEYWORD string, the following is the one you need:
sed -e 's/[]\/$*.^[]/\\&/g'
And can be used by:
KEYWORD="The Keyword You Need";
ESCAPED_KEYWORD=$(printf '%s\n' "$KEYWORD" | sed -e 's/[]\/$*.^[]/\\&/g');
# Now you can use it inside the original sed statement to replace text
sed "s/$ESCAPED_KEYWORD/$ESCAPED_REPLACE/g"
Remember, if you use a character other than / as delimiter, you need replace the slash in the expressions above wih the character you are using. See PeterJCLaw's comment for explanation.
Edited: Due to some corner cases previously not accounted for, the commands above have changed several times. Check the edit history for details.
The sed command allows you to use other characters instead of / as separator:
sed 's#"http://www\.fubar\.com"#URL_FUBAR#g'
The double quotes are not a problem.
The only three literal characters which are treated specially in the replace clause are / (to close the clause), \ (to escape characters, backreference, &c.), and & (to include the match in the replacement). Therefore, all you need to do is escape those three characters:
sed "s/KEYWORD/$(echo $REPLACE | sed -e 's/\\/\\\\/g; s/\//\\\//g; s/&/\\\&/g')/g"
Example:
$ export REPLACE="'\"|\\/><&!"
$ echo fooKEYWORDbar | sed "s/KEYWORD/$(echo $REPLACE | sed -e 's/\\/\\\\/g; s/\//\\\//g; s/&/\\\&/g')/g"
foo'"|\/><&!bar
Based on Pianosaurus's regular expressions, I made a bash function that escapes both keyword and replacement.
function sedeasy {
sed -i "s/$(echo $1 | sed -e 's/\([[\/.*]\|\]\)/\\&/g')/$(echo $2 | sed -e 's/[\/&]/\\&/g')/g" $3
}
Here's how you use it:
sedeasy "include /etc/nginx/conf.d/*" "include /apps/*/conf/nginx.conf" /etc/nginx/nginx.conf
It's a bit late to respond... but there IS a much simpler way to do this. Just change the delimiter (i.e., the character that separates fields). So, instead of s/foo/bar/ you write s|bar|foo.
And, here's the easy way to do this:
sed 's|/\*!50017 DEFINER=`snafu`#`localhost`\*/||g'
The resulting output is devoid of that nasty DEFINER clause.
It turns out you're asking the wrong question. I also asked the wrong question. The reason it's wrong is the beginning of the first sentence: "In my bash script...".
I had the same question & made the same mistake. If you're using bash, you don't need to use sed to do string replacements (and it's much cleaner to use the replace feature built into bash).
Instead of something like, for example:
function escape-all-funny-characters() { UNKNOWN_CODE_THAT_ANSWERS_THE_QUESTION_YOU_ASKED; }
INPUT='some long string with KEYWORD that need replacing KEYWORD.'
A="$(escape-all-funny-characters 'KEYWORD')"
B="$(escape-all-funny-characters '<funny characters here>')"
OUTPUT="$(sed "s/$A/$B/g" <<<"$INPUT")"
you can use bash features exclusively:
INPUT='some long string with KEYWORD that need replacing KEYWORD.'
A='KEYWORD'
B='<funny characters here>'
OUTPUT="${INPUT//"$A"/"$B"}"
Use awk - it is cleaner:
$ awk -v R='//addr:\\file' '{ sub("THIS", R, $0); print $0 }' <<< "http://file:\_THIS_/path/to/a/file\\is\\\a\\ nightmare"
http://file:\_//addr:\file_/path/to/a/file\\is\\\a\\ nightmare
Here is an example of an AWK I used a while ago. It is an AWK that prints new AWKS. AWK and SED being similar it may be a good template.
ls | awk '{ print "awk " "'"'"'" " {print $1,$2,$3} " "'"'"'" " " $1 ".old_ext > " $1 ".new_ext" }' > for_the_birds
It looks excessive, but somehow that combination of quotes works to keep the ' printed as literals. Then if I remember correctly the vaiables are just surrounded with quotes like this: "$1". Try it, let me know how it works with SED.
These are the escape codes that I've found:
* = \x2a
( = \x28
) = \x29
" = \x22
/ = \x2f
\ = \x5c
' = \x27
? = \x3f
% = \x25
^ = \x5e
sed is typically a mess, especially the difference between gnu-sed and bsd-sed
might just be easier to place some sort of sentinel at the sed side, then a quick pipe over to awk, which is far more flexible in accepting any ERE regex, escaped hex, or escaped octals.
e.g. OFS in awk is the true replacement ::
date | sed -E 's/[0-9]+/\xC1\xC0/g' |
mawk NF=NF FS='\xC1\xC0' OFS='\360\237\244\241'
1 Tue Aug 🤡 🤡:🤡:🤡 EDT 🤡
(tested and confirmed working on both BSD-sed and GNU-sed - the emoji isn't a typo that's what those 4 bytes map to in UTF-8 )
There are dozens of answers out there... If you don't mind using a bash function schema, below is a good answer. The objective below was to allow using sed with practically any parameter as a KEYWORD (F_PS_TARGET) or as a REPLACE (F_PS_REPLACE). We tested it in many scenarios and it seems to be pretty safe. The implementation below supports tabs, line breaks and sigle quotes for both KEYWORD and replace REPLACE.
NOTES: The idea here is to use sed to escape entries for another sed command.
CODE
F_REVERSE_STRING_R=""
f_reverse_string() {
: 'Do a string reverse.
To undo just use a reversed string as STRING_INPUT.
Args:
STRING_INPUT (str): String input.
Returns:
F_REVERSE_STRING_R (str): The modified string.
'
local STRING_INPUT=$1
F_REVERSE_STRING_R=$(echo "x${STRING_INPUT}x" | tac | rev)
F_REVERSE_STRING_R=${F_REVERSE_STRING_R%?}
F_REVERSE_STRING_R=${F_REVERSE_STRING_R#?}
}
# [Ref(s).: https://stackoverflow.com/a/2705678/3223785 ]
F_POWER_SED_ECP_R=""
f_power_sed_ecp() {
: 'Escape strings for the "sed" command.
Escaped characters will be processed as is (e.g. /n, /t ...).
Args:
F_PSE_VAL_TO_ECP (str): Value to be escaped.
F_PSE_ECP_TYPE (int): 0 - For the TARGET value; 1 - For the REPLACE value.
Returns:
F_POWER_SED_ECP_R (str): Escaped value.
'
local F_PSE_VAL_TO_ECP=$1
local F_PSE_ECP_TYPE=$2
# NOTE: Operational characters of "sed" will be escaped, as well as single quotes.
# By Questor
if [ ${F_PSE_ECP_TYPE} -eq 0 ] ; then
# NOTE: For the TARGET value. By Questor
F_POWER_SED_ECP_R=$(echo "x${F_PSE_VAL_TO_ECP}x" | sed 's/[]\/$*.^[]/\\&/g' | sed "s/'/\\\x27/g" | sed ':a;N;$!ba;s/\n/\\n/g')
else
# NOTE: For the REPLACE value. By Questor
F_POWER_SED_ECP_R=$(echo "x${F_PSE_VAL_TO_ECP}x" | sed 's/[\/&]/\\&/g' | sed "s/'/\\\x27/g" | sed ':a;N;$!ba;s/\n/\\n/g')
fi
F_POWER_SED_ECP_R=${F_POWER_SED_ECP_R%?}
F_POWER_SED_ECP_R=${F_POWER_SED_ECP_R#?}
}
# [Ref(s).: https://stackoverflow.com/a/24134488/3223785 ,
# https://stackoverflow.com/a/21740695/3223785 ,
# https://unix.stackexchange.com/a/655558/61742 ,
# https://stackoverflow.com/a/11461628/3223785 ,
# https://stackoverflow.com/a/45151986/3223785 ,
# https://linuxaria.com/pills/tac-and-rev-to-see-files-in-reverse-order ,
# https://unix.stackexchange.com/a/631355/61742 ]
F_POWER_SED_R=""
f_power_sed() {
: 'Facilitate the use of the "sed" command. Replaces in files and strings.
Args:
F_PS_TARGET (str): Value to be replaced by the value of F_PS_REPLACE.
F_PS_REPLACE (str): Value that will replace F_PS_TARGET.
F_PS_FILE (Optional[str]): File in which the replacement will be made.
F_PS_SOURCE (Optional[str]): String to be manipulated in case "F_PS_FILE" was
not informed.
F_PS_NTH_OCCUR (Optional[int]): [1~n] - Replace the nth match; [n~-1] - Replace
the last nth match; 0 - Replace every match; Default 1.
Returns:
F_POWER_SED_R (str): Return the result if "F_PS_FILE" is not informed.
'
local F_PS_TARGET=$1
local F_PS_REPLACE=$2
local F_PS_FILE=$3
local F_PS_SOURCE=$4
local F_PS_NTH_OCCUR=$5
if [ -z "$F_PS_NTH_OCCUR" ] ; then
F_PS_NTH_OCCUR=1
fi
local F_PS_REVERSE_MODE=0
if [ ${F_PS_NTH_OCCUR} -lt -1 ] ; then
F_PS_REVERSE_MODE=1
f_reverse_string "$F_PS_TARGET"
F_PS_TARGET="$F_REVERSE_STRING_R"
f_reverse_string "$F_PS_REPLACE"
F_PS_REPLACE="$F_REVERSE_STRING_R"
f_reverse_string "$F_PS_SOURCE"
F_PS_SOURCE="$F_REVERSE_STRING_R"
F_PS_NTH_OCCUR=$((-F_PS_NTH_OCCUR))
fi
f_power_sed_ecp "$F_PS_TARGET" 0
F_PS_TARGET=$F_POWER_SED_ECP_R
f_power_sed_ecp "$F_PS_REPLACE" 1
F_PS_REPLACE=$F_POWER_SED_ECP_R
local F_PS_SED_RPL=""
if [ ${F_PS_NTH_OCCUR} -eq -1 ] ; then
# NOTE: We kept this option because it performs better when we only need to replace
# the last occurrence. By Questor
# [Ref(s).: https://linuxhint.com/use-sed-replace-last-occurrence/ ,
# https://unix.stackexchange.com/a/713866/61742 ]
F_PS_SED_RPL="'s/\(.*\)$F_PS_TARGET/\1$F_PS_REPLACE/'"
elif [ ${F_PS_NTH_OCCUR} -gt 0 ] ; then
# [Ref(s).: https://unix.stackexchange.com/a/587924/61742 ]
F_PS_SED_RPL="'s/$F_PS_TARGET/$F_PS_REPLACE/$F_PS_NTH_OCCUR'"
elif [ ${F_PS_NTH_OCCUR} -eq 0 ] ; then
F_PS_SED_RPL="'s/$F_PS_TARGET/$F_PS_REPLACE/g'"
fi
# NOTE: As the "sed" commands below always process literal values for the "F_PS_TARGET"
# so we use the "-z" flag in case it has multiple lines. By Quaestor
# [Ref(s).: https://unix.stackexchange.com/a/525524/61742 ]
if [ -z "$F_PS_FILE" ] ; then
F_POWER_SED_R=$(echo "x${F_PS_SOURCE}x" | eval "sed -z $F_PS_SED_RPL")
F_POWER_SED_R=${F_POWER_SED_R%?}
F_POWER_SED_R=${F_POWER_SED_R#?}
if [ ${F_PS_REVERSE_MODE} -eq 1 ] ; then
f_reverse_string "$F_POWER_SED_R"
F_POWER_SED_R="$F_REVERSE_STRING_R"
fi
else
if [ ${F_PS_REVERSE_MODE} -eq 0 ] ; then
eval "sed -i -z $F_PS_SED_RPL \"$F_PS_FILE\""
else
tac "$F_PS_FILE" | rev | eval "sed -z $F_PS_SED_RPL" | tac | rev > "$F_PS_FILE"
fi
fi
}
MODEL
f_power_sed "F_PS_TARGET" "F_PS_REPLACE" "" "F_PS_SOURCE"
echo "$F_POWER_SED_R"
EXAMPLE
f_power_sed "{ gsub(/,[ ]+|$/,\"\0\"); print }' ./ and eliminate" "[ ]+|$/,\"\0\"" "" "Great answer (+1). If you change your awk to awk '{ gsub(/,[ ]+|$/,\"\0\"); print }' ./ and eliminate that concatenation of the final \", \" then you don't have to go through the gymnastics on eliminating the final record. So: readarray -td '' a < <(awk '{ gsub(/,[ ]+/,\"\0\"); print; }' <<<\"$string\") on Bash that supports readarray. Note your method is Bash 4.4+ I think because of the -d in readar"
echo "$F_POWER_SED_R"
IF YOU JUST WANT TO ESCAPE THE PARAMETERS TO THE SED COMMAND
MODEL
# "TARGET" value.
f_power_sed_ecp "F_PSE_VAL_TO_ECP" 0
echo "$F_POWER_SED_ECP_R"
# "REPLACE" value.
f_power_sed_ecp "F_PSE_VAL_TO_ECP" 1
echo "$F_POWER_SED_ECP_R"
IMPORTANT: If the strings for KEYWORD and/or replace REPLACE contain tabs or line breaks you will need to use the "-z" flag in your "sed" command. More details here.
EXAMPLE
f_power_sed_ecp "{ gsub(/,[ ]+|$/,\"\0\"); print }' ./ and eliminate" 0
echo "$F_POWER_SED_ECP_R"
f_power_sed_ecp "[ ]+|$/,\"\0\"" 1
echo "$F_POWER_SED_ECP_R"
NOTE: The f_power_sed_ecp and f_power_sed functions above was made available completely free as part of this project ez_i - Create shell script installers easily!.
Standard recommendation here: use perl :)
echo KEYWORD > /tmp/test
REPLACE="<funny characters here>"
perl -pi.bck -e "s/KEYWORD/${REPLACE}/g" /tmp/test
cat /tmp/test
don't forget all the pleasure that occur with the shell limitation around " and '
so (in ksh)
Var=">New version of \"content' here <"
printf "%s" "${Var}" | sed "s/[&\/\\\\*\\"']/\\&/g' | read -r EscVar
echo "Here is your \"text\" to change" | sed "s/text/${EscVar}/g"
If the case happens to be that you are generating a random password to pass to sed replace pattern, then you choose to be careful about which set of characters in the random string. If you choose a password made by encoding a value as base64, then there is is only character that is both possible in base64 and is also a special character in sed replace pattern. That character is "/", and is easily removed from the password you are generating:
# password 32 characters log, minus any copies of the "/" character.
pass=`openssl rand -base64 32 | sed -e 's/\///g'`;
If you are just looking to replace Variable value in sed command then just remove
Example:
sed -i 's/dev-/dev-$ENV/g' test to sed -i s/dev-/dev-$ENV/g test
I have an improvement over the sedeasy function, which WILL break with special characters like tab.
function sedeasy_improved {
sed -i "s/$(
echo "$1" | sed -e 's/\([[\/.*]\|\]\)/\\&/g'
| sed -e 's:\t:\\t:g'
)/$(
echo "$2" | sed -e 's/[\/&]/\\&/g'
| sed -e 's:\t:\\t:g'
)/g" "$3"
}
So, whats different? $1 and $2 wrapped in quotes to avoid shell expansions and preserve tabs or double spaces.
Additional piping | sed -e 's:\t:\\t:g' (I like : as token) which transforms a tab in \t.
An easier way to do this is simply building the string before hand and using it as a parameter for sed
rpstring="s/KEYWORD/$REPLACE/g"
sed -i $rpstring test.txt
I have a string that is sometimes
xxx.11_222_33_44_555.yyy
and sometimes
xxx.11_222_33_44.yyy
I would like to:
Check if has 4 occourances of _ (figured out how to do it).
If so - remove string's _33 (the 33 string changes, can be any number), so I am left with xxx.11_222_44.yyy.
Using sed :
sed 's/\(_[0-9]*\)_[0-9]*\(_[0-9]*_[0-9]*\)/\1\2/'
It matches the four underscores and replace the whole by the needed parts.
Test run :
$ echo "xxx.11_222_33_44_555.yyy" | sed 's/\(_[0-9]*\)_[0-9]*\(_[0-9]*_[0-9]*\)/\1\2/'
xxx.11_222_44_555.yyy
$ echo "xxx.11_222_33_44.yyy" | sed 's/\(_[0-9]*\)_[0-9]*\(_[0-9]*_[0-9]*\)/\1\2/'
xxx.11_222_33_44.yyy
perhaps something like this
echo "xxx.11_222_33_44.yyy" | sed -e's/\.\([0-9]\+\)_\([0-9]\+\)_\([0-9]\+\)_\([0-9]\+\)\./.\1_\2_\4./'
which checks if there are 4 groups of numbers separated by _ between the two dots and if yes, it leaves out the third group
try this;
echo "xxx.11_222_33_44_555.yyy" | awk -F'_' 'NF>4{print $1"_"$2"_"$4"_"$5};'
Solution using perl and Lookahead and Lookbehind
$ a="xxx.11_222_33_44_555.yyy"
$ perl -pe 's/\.\d+_\d+_\K\d+_(?=\d+_\d+\.)//' <<< "$a"
xxx.11_222_44_555.yyy
I want to write a small script to generate the location of a file in an NGINX cache directory.
The format of the path is:
/path/to/nginx/cache/d8/40/32/13febd65d65112badd0aa90a15d84032
Note the last 6 characters: d8 40 32, are represented in the path.
As an input I give the md5 hash (13febd65d65112badd0aa90a15d84032) and I want to generate the output: d8/40/32/13febd65d65112badd0aa90a15d84032
I'm sure sed or awk will be handy, but I don't know yet how...
This awk can make it:
awk 'BEGIN{FS=""; OFS="/"}{print $(NF-5)$(NF-4), $(NF-3)$(NF-2), $(NF-1)$NF, $0}'
Explanation
BEGIN{FS=""; OFS="/"}. FS="" sets the input field separator to be "", so that every char will be a different field. OFS="/" sets the output field separator as /, for print matters.
print ... $(NF-1)$NF, $0 prints the penultimate field and the last one all together; then, the whole string. The comma is "filled" with the OFS, which is /.
Test
$ awk 'BEGIN{FS=""; OFS="/"}{print $(NF-5)$(NF-4), $(NF-3)$(NF-2), $(NF-1)$NF, $0}' <<< "13febd65d65112badd0aa90a15d84032"
d8/40/32/13febd65d65112badd0aa90a15d84032
Or with a file:
$ cat a
13febd65d65112badd0aa90a15d84032
13febd65d65112badd0aa90a15f1f2f3
$ awk 'BEGIN{FS=""; OFS="/"}{print $(NF-5)$(NF-4), $(NF-3)$(NF-2), $(NF-1)$NF, $0}' a
d8/40/32/13febd65d65112badd0aa90a15d84032
f1/f2/f3/13febd65d65112badd0aa90a15f1f2f3
With sed:
echo '13febd65d65112badd0aa90a15d84032' | \
sed -n 's/\(.*\([0-9a-f]\{2\}\)\([0-9a-f]\{2\}\)\([0-9a-f]\{2\}\)\)$/\2\/\3\/\4\/\1/p;'
Having GNU sed you can even simplify the pattern using the -r option. Now you won't need to escape {} and () any more. Using ~ as the regex delimiter allows to use the path separator / without need to escape it:
sed -nr 's~(.*([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2}))$~\2/\3/\4/\1~p;'
Output:
d8/40/32/13febd65d65112badd0aa90a15d84032
Explained simple the pattern does the following: It matches:
(all (n-5 - n-4) (n-3 - n-2) (n-1 - n-0))
and replaces it by
/$1/$2/$3/$0
You can use a regular expression to separate each of the last 3 bytes from the rest of the hash.
hash=13febd65d65112badd0aa90a15d84032
[[ $hash =~ (..)(..)(..)$ ]]
new_path="/path/to/nginx/cache/${BASH_REMATCH[1]}/${BASH_REMATCH[2]}/${BASH_REMATCH[3]}/$hash"
Base="/path/to/nginx/cache/"
echo '13febd65d65112badd0aa90a15d84032' | \
sed "s|\(.*\(..\)\(..\)\(..\)\)|${Base}\2/\3/\4/\1|"
# or
# sed sed 's|.*\(..\)\(..\)\(..\)$|${Base}\1/\2/\3/&|'
Assuming info is a correct MD5 (and only) string
First of all - thanks to all of the responders - this was extremely quick!
I also did my own scripting meantime, and came up with this solution:
Run this script with a parameter of the URL you're looking for (www.example.com/article/76232?q=hello for example)
#!/bin/bash
path=$1
md5=$(echo -n "$path" | md5sum | cut -f1 -d' ')
p3=$(echo "${md5:0-2:2}")
p2=$(echo "${md5:0-4:2}")
p1=$(echo "${md5:0-6:2}")
echo "/path/to/nginx/cache/$p1/$p2/$p3/$md5"
This assumes the NGINX cache has a key structure of 2:2:2.
this question originated from string pattaren-matching using awk , basically we are splitting a line of text in multiple groups based on a regex pattern, and then printing two groups only. Now the question is can we right align a group while printing through sed?
below is an example
$cat input.txt
it is line one
it is longggggggg one
itttttttttt is another one
now
$sed -e 's/\(.*\) \(.*\) \(.*\) \(.*\)/\1 \3/g' input.txt
it splits and prints group 1 and 3, but the output is
it line
it longggggggg
itttttttttt another
my question is can we do it through sed so that the output comes as
it line
it longggggggg
itttttttttt another
I did it with awk but I feel it can be done through sed, but I am not able to get how I am going to get the length of the second group and then pad correct number of spaces in between the groups, I am open to any suggestions to try out.
This might work for you (GNU sed):
sed -r 's/^(.*) .* (.*) .*$/\1 \2/;:a;s/^.{1,40}$/ &/;ta;s/^( *)(\S*)/\2\1/' file
or:
sed -r 's/^(.*) .* (.*) .*$/printf "%-20s%20s" \1 \2/e' file
You can use looping in sed to achieve what you want:
#!/bin/bash
echo 'aa bb cc dd
11 22 33333333 44
ONE TWO THREEEEEEEEE FOUR' | \
sed -e 's/\(.*\) \(.*\) \(.*\) \(.*\)/\1 \3/g' \
-e '/\([^ ]*\) \([^ ]*\)/ { :x ; s/^\(.\{1,19\}\) \(.\{1,19\}\)$/\1 \2/g ; tx }'
The two 19's control the width of your columns. The :x is a label which is looped to by tx whenever the preceding substitution succeeded. (You could add a p; before tx to "debug" it.
It most easy to use awk in this case...
You could too use a bash loop to calculate the number of space and run this command on the line covered :
while read; do
# ... calculate $SPACE ...
echo $REPLY|sed "s/\([^\ ]*\)\ *[^\ ]*\ *\([^\ ]*\)/\1$SPACES\2/g"
done < file
But I prefer use awk for do all that (or other advanced shell languages ​​such as Perl, Python, PHP shell mode, ...)
TemplateSpace=" "
TemplateSize=${#TemplateSpace}
sed "
# split your group (based on word here but depend on your real need)
s/^ *\(\w\) \(\w\) \(\w\) \(\w\).*$/\1 \3/
# align
s/$/${TemplateSpace}/
s/^\(.\{${TemplateSize}\}\).*$/\1/
s/\(\w\) \(\w\)\( *\)/\1 \3\2/
"
or more simple for avoiding TemplateSize (and there are no dot in content)
TemplateSpace="............................................................."
and replace
s/^\(.\{${TemplateSize}\}.*$/\1/
by
s/^\(${TemplateSpace}\).*$/\1/
s/\./ /g
Del columns 2 and 4. Right justify resulting col 2 at line length of 23 chars.
sed -e '
s/[^ ]\+/ /4;
s/[^ ]\+//2;
s/^\(.\{23\}\).*$/\1/;
s/\(^[^ ]\+[ ]\+\)\([^ ]\+\)\([ ]\+\)/\1\3\2/;
'
or gnu sed with extended regex:
sed -r '
s/\W+\w+\W+(\w+)\W+\w+$/\1 /;
s/^(.{23}).*/\1/;
s/(+\W)(\w+)(\W+)$/\1\3\2/
'
This question is old, but I like to see it as a puzzle.
While I love the loop solution for its brevity, here is one without a loop or shell help.
sed -E "s/ \w+ (\w+) \w+$/ \1/;h;s/./ /g;s/$/# /;s/( *)#\1//;x;H;x;s/\n//;s/^( *)(\w+)/\2\1/"
or without extended regex
sed "s/ .* \(.*\) .*$/ \1/;h;s/./ /g;s/$/# /;s/\( *\)#\1//;x;H;x;s/\n//;s/^\( *\)\([^ ]*\)/\2\1/"
I have this file:
100: pattern1
++++++++++++++++++++
1:pattern2
9:pattern2
+++++++++++++++++++
79: pattern1
61: pattern1
+++++++++++++++++++
and I want to sort it like this:
++++++++++++++++++++
1:pattern2
9:pattern2
+++++++++++++++++++
61:pattern1
79:pattern1
100:pattern1
+++++++++++++++++++
Is it possible using Linux sort command only ?
If I had :
4:pat1
3:pat2
2:pat2
1:pat1
O/p should be:
1:pat1
++++++++++++
2:pat2
3:pat2
++++++++++++
4:pat1
So, want to sort on first group, but "group" on the pattern of second group.
Please note, the thing after : is a regex pattern not a literal.
Best you can do is to sort it according to the numerical values. But you cannot do anything with the "+"-string.
$ sort -n input
+++++++++++++++++++
+++++++++++++++++++
++++++++++++++++++++
1:wow
9:wow
61: this is it
79: this is it
100: this is it
I don't believe sort alone can do what you need.
Create a new shell script and put this in its contents (ie mysort.sh):
#!/bin/sh
IFS=$'\n' # This makes the for loop below split on newline instead of whitespace.
delim=+++++++++++++++++++
for l in `grep -v ^+| sort -g` # Ignore all + lines and sort by number
do
current=`echo $l | sed s/^[0-9]*://g` # Get what comes after the number
if [ ! -z "$prev" ] && [ "$prev" != "$current" ] # If it has changed...
then # then output a ++++ delimiter line.
echo $delim
fi
prev=$current
echo $l # Output this line.
done
To use it, pipe in the contents of your file like so:
cat input | sh mysort.sh
Probably not -- it's not in the sort of format sort(1) expects. And if you did it would be one of those amazing hacks, not easily used. If you have some sort of rule for what goes between the lines of plus signs, you can do it readily enough with an AWK or Perl or Python script.
If your input was space delimited, not ':' delimited:
sort -rk2 | uniq -D -f1
will do the grouping;
I guess you'd need to sort the 'subsections' later (unfortunately my sort(1) doesn't do composite key ordering. I do believe there are version that allow you to do sort -k2,1n and you'd be done at once).
use --all-repeated=separate instead of -D to get blank separators between groups. Look at man uniq for more ideas!
However, since your input is colon delimited, a hack is required:
sed 's/\([0123456789]\+\):/\1 /' t | sort -rk2 | uniq -D -f1
HTH