I need to store in a unix variable a string that can contain special characters.
In my case I am decrypting a text that returns to me Her$p7 that I need to store. Obviously that result can be any string (example i*Fi+K'7).
Would they know how I can save that result that I throw in a variable and use it like this
var=Her$p7
echo var
by example?
I use this for retrieve the string
echo "`${FDL_HOME}/bin/cypher2 DEC ${CF_KEYPFX} ${CF_KEYLEN} ${CF_PASS}== | grep "decrypt text" | gawk -F': ' '{print $2}'`"|perl -e print
The syntax is correct, if not robust, and the issue is in understanding string interpolation. In the standard string context, the dollar sign signifies to the shell that it's about to interpret a variable. Generally, this means to replace the variable with the value of the variable. Consider:
$ t1=Her$p7
$ t2="Her$p7"
$ t3='Her$p7'
$ t4="$(echo 'Her$p7')"
$ echo "t1: $t1; t2: $t2; t3: $t3, t4: $t4"
t1: Her; t2: Her; t3: Her$p7; t4: Her$p7
Note that while setting t1 (the first line) and t2, $p7 was interpreted as a variable (which you had not set), and thus was consequently replaced with it's value (empty/nothing). So, t1 and t2 were set to the value Her<nothing> -> Her.
In the third case, we used single quotes to tell the shell "no interpolation please; I mean strictly what I say". So, t3 is set to exactly the string you typed.
In the last case, we use the subshell operator ($( ... )) to set the variable t4 to the output of subshell command. In this case, we use double quotes to make sure we capture the entire output, but because we aren't typing the variable $p7, the shell won't interpolate the output of the command.
So, you should be good to go with something like:
$ yourVar=$(echo "`${FDL_HOME}/bin/cypher2 DEC ${CF_KEYPFX} ${CF_KEYLEN} ${CF_PASS}== | grep "decrypt text" | gawk -F': ' '{print $2}'`" | perl -e print)
Moving into the last decade, we could clean that up slightly by not using backticks for subshell operations:
$ yourVar=$(echo "$(${FDL_HOME}/bin/cypher2 DEC ${CF_KEYPFX} ${CF_KEYLEN} ${CF_PASS}== | grep "decrypt text" | gawk -F': ' '{print $2}')" | perl -e print)
Related
I have a string value stored in a variable:
PTYPE="Other Farm|Raised Ranch|Farm house|Other|A-Frame|Log Home"
I want to find & replace Other with some value like NOTHING. All values are stored in variables.
WhatToChange=Other
NewValue=NOTHING
echo $PTYPE|sed -e "s#${WhatToChange}#${NewValue}#g"
This is replacing all the occurances of Other and getting output like:
NOTHING Farm|Raised Ranch|Farm house|NOTHING|A-Frame|Log Home
Is there any way I can exactly change only the exact one? The place for ${WhatToChange} is variable.
As you have well defined fields and want an exact match, awk could be easier to use than sed; at the very least, you won't have to worry about escaping the strings for using it in the sed expression:
echo "Other Farm|Raised Ranch|Farm house|Other|A-Frame|Log Home" |
awk -v old="Other" -v new="NOTHING" \
'BEGIN {FS = OFS = "|"} {for(i=1;i<=NF;i++) if($i == old) $i = new} 1'
output:
Other Farm|Raised Ranch|Farm house|NOTHING|A-Frame|Log Home
To match either the exact character | or the beginning of the line, use ([|]|^).
To match either the exact character | or the end of the line, use ([|]|$).
To put a | back in place only when appropriate, store these in match groups, and refer to those groups with \1 or \2:
PTYPE="Other Farm|Raised Ranch|Farm house|Other|A-Frame|Log Home"
WhatToChange=Other
NewValue=NOTHING
sed -re "s#(^|[|])${WhatToChange}($|[|])#\1${NewValue}\2#g" <<<"$PTYPE"
...emits as output:
Other Farm|Raised Ranch|Farm house|NOTHING|A-Frame|Log Home
...and still works even if WhatToChange is matched at the beginning or end of the list.
For fun, some perl:
This is like #Charles's sed solution: Note the \Q...\E so that the "to change" value is treated as literal text.
echo "$PTYPE" | perl -spe '
s{ (?:^|\|)\K \Q$WhatToChange\E (?=\||$) }{$NewValue}gx
' -- -WhatToChange=Other -NewValue=NOTHING
This is like #Fravadona's awk solution:
echo "$PTYPE" | perl -F'[|]' -sane '
print join "|", map {$_ eq $WhatToChange ? $NewValue : $_} #F
' -- -WhatToChange=Other -NewValue=NOTHING
How about
echo ${PTYPE//$WhatToChange/$NewValue}
UPDATE:
I just realized that the replacement should happen only if WhatToChange is the whole content between two separators (|). In this case, we can do it in bash as well (without the need to revert to a child process):
if [[ $PTYPE =~ (.*[|]|^)$WhatToChange([|].*|$) ]]
echo "${BASH_REMATCH[1]}${NewValue}${BASH_REMATCH[2]}"
fi
UPDATE (based on the comment by Fravadona):
Used in this way, WhatToChange is interpreted as a regular expression. This can be useful, if you want to catch for instance variations of the string, for instance
WhatToChange='[Oo]ther' # to catch Other and other
If you always want to have a literal match, you have to quote the variable:
[[ $PTYPE =~ (.*[|]|^)"$WhatToChange"([|].*|$) ]]
This might work for you (GNU sed & bash):
<<<"$PTYPE" sed 'y/|/\n/;s/^'"$WhatToChange"'$/'"$NewValue"'/mg;y/\n/|/'
Input $PTYPE as a here-string into sed.
Translate | separators to newlines.
Replace $WhatToChange to $NewValue for each matched line.
Translate newlines back to |'s.
N.B. The use of the m flag in the substitution command allows sed to work in multiline mode and this presents each value between separators on its own line.
An alternative:
sed -z 'y/|/\x00/;s/^'"$WhatToChange"'$/'"$NewValue"'/mg;y/\x00/|/;' file
I am newbie in shell script , may be stupid query to experts, I am using following code to remove leading and trailing spaces from value, how do I assign output of echo variable to StringVar variable again or to other Variable. I am using ksh shell.
StringVar= ' abc '
echo StringVar | awk '{$1=$1};1'
x=$(command)
where x is the variable to which you want to assign the output of the command.
In your case, do not give space before or after the assignment operator =
StringVar=' abc '
x=$(echo "$StringVar" | awk '{$1=$1};1')
For every time the pattern shows up (In this example the case of a 2 digit number) I want to pass that pattern to a script and replace that pattern with the output of a script.
I'm using sed an example of what it should look like would be
echo 'siedi87sik65owk55dkd' | sed 's/[0-9][0-9]/.\/script.sh/g'
Right now this returns
siedi./script.shsik./script.showk./script.shdkd
But I would like it to return
siedi!!!87!!!sik!!!65!!!owk!!!55!!!dkd
This is what is in ./script.sh
#!/bin/bash
echo "!!!$1!!!"
It has to be replaced with the output. In this example I know I could just use a normal sed substitution but I don't want that as an answer.
sed is for simple substitutions on individual lines, that is all. Anything else, even if it can be done, requires arcane language constructs that became obsolete in the mid-1970s when awk was invented and are used today purely for the mental exercise. Your problem is not a simple substitution so you shouldn't try to use sed to solve it.
You're going to want something like:
awk '{
head = ""
tail = $0
while ( match(tail,/[0-9]{2}/) ) {
tgt = substr(tail,RSTART,RLENGTH)
cmd = "./script.sh " tgt
if ( (cmd | getline line) > 0) {
tgt = line
}
close(cmd)
head = head substr(tail,1,RSTART-1) tgt
tail = substr(tail,RSTART+RLENGTH)
}
print head tail
}'
e.g. using an echo in place of your script.sh command:
$ echo 'siedi87sik65owk55dkd' |
awk '{
head = ""
tail = $0
while ( match(tail,/[0-9]{2}/) ) {
tgt = substr(tail,RSTART,RLENGTH)
cmd = "echo !!!" tgt "!!!"
if ( (cmd | getline line) > 0) {
tgt = line
}
close(cmd)
head = head substr(tail,1,RSTART-1) tgt
tail = substr(tail,RSTART+RLENGTH)
}
print head tail
}'
siedi!!!87!!!sik!!!65!!!owk!!!55!!!dkd
Ed's awk solution is obviously the way to go here.
For fun, I tried to come up with a sed solution, and here is (a convoluted GNU sed) one that takes the pattern and the script to be run as parameters; the input is either read from standard input (i.e., you can pipe to it) or from a file supplied as the third argument.
For your example, we'd have infile with contents
siedi87sik65owk55dkd
siedi11sik22owk33dkd
(two lines to demonstrate how this works for multiple lines), then script with contents
#!/bin/bash
echo "!!!${1}!!!"
and finally the solution script itself, so. Usage is
./so pattern script [input]
where pattern is an extended regular expression as understood by GNU sed (with the -r option), script is the name of the command you want to run for each match, and the optional input is the name of the input file if input is not standard input.
For your example, this would be
./so '[[:digit:]]{2}' script infile
or, as a filter,
cat infile | ./so '[[:digit:]]{2}' script
with output
siedi!!!87!!!sik!!!65!!!owk!!!55!!!dkd
siedi!!!11!!!sik!!!22!!!owk!!!33!!!dkd
This is what so looks like:
#!/bin/bash
pat=$1 # The pattern to match
script=$2 # The command to run for each pattern
infile=${3:-/dev/stdin} # Read from standard input if not supplied
# Use sed and have $pattern and $script expand to the supplied parameters
sed -r "
:build_loop # Label to loop back to
h # Copy pattern space to hold space
s/.*($pat).*/.\/\"$script\" \1/ # (1) Extract last match and prepare command
# Replace pattern space with output of command
e
G # (2) Append hold space to pattern space
s/(.*)$pat(.*)/\1~~~\2/ # (3) Replace last match of pattern with ~~~
/\n[^\n]*$pat[^\n]*$/b build_loop # Loop if string contains match
:fill_loop # Label for second loop
s/(.*\n)(.*)\n([^\n]*)~~~([^\n]*)$/\1\3\2\4/ # (4) Replace last ~~~
t fill_loop # Loop if there was a replacement
s/(.*)\n(.*)~~~(.*)$/\2\1\3/ # (5) Final ~~~ replacement
" < "$infile"
The sed command works with two loops. The first one copies the pattern space to the hold space, then removes everything but the last match from the pattern space and prepares the command to be run. After the substitution with (1) in its comment, the pattern space looks like this:
./script 55
The e command (a GNU extension) then replaces the pattern space with the output of this command. After this, G appends the hold space to the pattern space (2). The pattern space now looks like this:
!!!55!!!
siedi87sik65owk55dkd
The substitution at (3) replaces the last match with a string hopefully not equal to the pattern and we get
!!!55!!!
siedi87sik65owk~~~dkd
The loop repeats if the last line of the pattern space still has a match for the pattern. After three loops, the pattern space looks like this:
!!!87!!!
!!!65!!!
!!!55!!!
siedi~~~sik~~~owk~~~dkd
The second loop now replaces the last ~~~ with the second to last line of the pattern space with substitution (4). The command uses lots of "not a newline" ([^\n]) to make sure we're not pulling the wrong replacement for ~~~.
Because of the way command (4) is written, the loop ends with one last substitution to go, so before command (5), we have this pattern space:
!!!87!!!
siedi~~~sik!!!65!!!owk!!!55!!!dkd
Command (5) is a simpler version of command (4), and after it, the output is as desired.
This seems to be fairly robust and can deal with spaces in the name of the script to be run as long as it's properly quoted when calling:
./so '[[:digit:]]{2}' 'my script' infile
This would fail if
The input file contains ~~~ (solvable by replacing all occurrences at the start, putting them back at the end)
The output of script contains ~~~
The pattern contains ~~~
i.e., the solution very much depends on ~~~ being unique.
Because nobody asked: so as a one-liner.
#!/bin/bash
sed -re ":b;h;s/.*($1).*/.\/\"$2\" \1/;e" -e "G;s/(.*)$1(.*)/\1~~~\2/;/\n[^\n]*$1[^\n]*$/bb;:f;s/(.*\n)(.*)\n([^\n]*)~~~([^\n]*)$/\1\3\2\4/;tf;s/(.*)\n(.*)~~~(.*)$/\2\1\3/" < "${3:-/dev/stdin}"
Still works!
A conceptually simpler multi-utility solution:
Using GNU utilities:
echo 'siedi87sik65owk55dkd' |
sed 's|[0-9]\{2\}|$(./script.sh &)|g' |
xargs -d'\n' -I% sh -c 'echo '\"%\"
Using BSD utilities (also works with GNU utilities):
echo 'siedi87sik65owk55dkd' |
sed 's|[0-9]\{2\}|$(./script.sh &)|g' | tr '\n' '\0' |
xargs -0 -I% sh -c 'echo '\"%\"
The idea is to use sed to translate the tokens of interest lexically into a string containing shell command substitutions that invoke the target script with the token, and then pass the result to the shell for evaluation.
Note:
Any embedded " and $ characters in the input must be \-escaped.
xargs -d'\n' (GNU) and tr '\n' '\0' / xargs -0 (BSD) are only needed to correctly preserve whitespace in the input - if that is not needed, the following POSIX-compliant solution will do:
echo 'siedi87sik65owk55dkd' |
sed 's|[0-9]\{2\}|$(./script.sh &)|g' | tr '\n' '\0' |
xargs -I% sh -c 'printf "%s\n" '\"%\"
Can you please help me solve this puzzle? I am trying to print the location of a string (i.e., line #) in a file, first to the std output, and then capture that value in a variable to be used later. The string is “my string”, the file name is “myFile” which is defined as follows:
this is first line
this is second line
this is my string on the third line
this is fourth line
the end
Now, when I use this command directly at the command prompt:
% awk ‘s=index($0, “my string”) { print “line=” NR, “position= ” s}’ myFile
I get exactly the result I want:
% line= 3, position= 9
My question is: if I define a variable VAR=”my string”, why can’t I get the same result when I do this:
% awk ‘s=index($0, $VAR) { print “line=” NR, “position= ” s}’ myFile
It just won’t work!! I even tried putting the $VAR in quotation marks, to no avail? I tried using VAR (without the $ sign), no luck. I tried everything I could possibly think of ... Am I missing something?
awk variables are not the same as shell variables. You need to define them with the -v flag
For example:
$ awk -v var="..." '$0~var{print NR}' file
will print the line number(s) of pattern matches. Or for your case with the index
$ awk -v var="$Var" 'p=index($0,var){print NR,p}' file
using all uppercase may not be good convention since you may accidentally overwrite other variables.
to capture the output into a shell variable
$ info=$(awk ...)
for multi line output assignment to shell array, you can do
$ values=( $(awk ...) ); echo ${values[0]}
however, if the output contains more than one field, it will be assigned it's own array index. You can change it with setting the IFS variable, such as
$ IFS=$(echo -en "\n\b"); values=( $(awk ...) )
which will capture the complete lines as the array values.
I want to get the string between <sometag param=' and '>
I tried to use the method from Get any string between 2 string and assign a variable in bash to get the "x":
echo "<sometag param='x'><irrelevant stuff='nonsense'>" | tr "'" _ | sed -n 's/.*<sometag param=_\(.*\)_>.*/\1/p'
The problem (apart from low efficiency because I just cannot manage to escape the apostrophe correctly for sed) is that sed matches the maximum, i.e. the output is:
x_><irrelevant stuff=_nonsense
but the correct output would be the minimum-match, in this example just "x"
Thanks for your help
You are probably looking for something like this:
sed -n "s/.*<sometag param='\([^']*\)'>.*/\1/p"
Test:
echo "<sometag param='x'><irrelevant stuff='nonsense'>" | sed -n "s/.*<sometag param='\([^']*\)'>.*/\1/p"
Results:
x
Explanation:
Instead of a greedy capture, use a non-greedy capture like: [^']* which means match anything except ' any number of times. To make the pattern stick, this is followed by: '>.
You can also use double quotes so that you don't need to escape the single quotes. If you wanted to escape the single quotes, you'd do this:
-
... | sed -n 's/.*<sometag param='\''\([^'\'']*\)'\''>.*/\1/p'
Notice how that the single quotes aren't really escaped. The sed expression is stopped, an escaped single quote is inserted and the sed expression is re-opened. Think of it like a four character escape sequence.
Personally, I'd use GNU grep. It would make for a slightly shorter solution. Run like:
... | grep -oP "(?<=<sometag param=').*?(?='>)"
Test:
echo "<sometag param='x'><irrelevant stuff='nonsense'>" | grep -oP "(?<=<sometag param=').*?(?='>)"
Results:
x
You don't have to assemble regexes in those cases, you can just use ' as the field separator
in="<sometag param='x'><irrelevant stuff='nonsense'>"
IFS="'" read x whatiwant y <<< "$in" # bash
echo "$whatiwant"
awk -F\' '{print $2}' <<< "$in" # awk