Assigning a value having semicolon (';') to a variable in bash - shell

I'm trying to escape ('\') a semicolon (';') in a string on unix shell (bash) with sed. It works when I do it directly without assigning the value to a variable. That is,
$ echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'
hello\;
$
However, it doesn't appear to work when the above command is assigned to a variable:
$ result=`echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'`
$
$ echo $result
hello;
$
Any idea why?
I tried by using the value enclosed with and without quotes but that didn't help. Any clue greatly appreciated.
btw, I first thought the semicolon at the end of the string was somehow acting as a terminator and hence the shell didn't continue executing the sed (if that made any sense). However, that doesn't appear to be an issue. I tried by using the semicolon not at the end of the string (somewhere in between). I still see the same result as before. That is,
$ echo "hel;lo" | sed 's/\([^\\]\);/\1\\;/g'
hel\;lo
$
$ result=`echo "hel;lo" | sed 's/\([^\\]\);/\1\\;/g'`
$
$ echo $result
hel;lo
$

You don't need sed (or any other regex engine) for this at all:
s='hello;'
echo "${s//;/\;}"
This is a parameter expansion which replaces ; with \;.
That said -- why are you trying to do this? In most cases, you don't want escape characters (which are syntax) to be inside of scalar variables (which are data); they only matter if you're parsing your data as syntax (such as using eval), which is a bad idea for other reasons, and best avoided (or done programatically, as via printf %q).

I find it interesting that the use of back-ticks gives one result (your result) and the use of $(...) gives another result (the wanted result):
$ echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'
hello\;
$ z1=$(echo "hello;" | sed 's/\([^\\]\);/\1\\;/g')
$ z2=`echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'`
$ printf "%s\n" "$z1" "$z2"
hello\;
hello;
$
If ever you needed an argument for using the modern x=$(...) notation in preference to the older x=`...` notation, this is probably it. The shell does an extra round of backslash interpretation with the back-ticks. I can demonstrate this with a little program I use when debugging shell scripts called al (for 'argument list'); you can simulate it with printf "%s\n":
$ z2=`echo "hello;" | al sed 's/\([^\\]\);/\1\\;/g'`
$ echo "$z2"
sed
s/\([^\]\);/\1\;/g
$ z1=$(echo "hello;" | al sed 's/\([^\\]\);/\1\\;/g')
$ echo "$z1"
sed
s/\([^\\]\);/\1\\;/g
$ z1=$(echo "hello;" | printf "%s\n" sed 's/\([^\\]\);/\1\\;/g')
$ echo "$z1"
sed
s/\([^\\]\);/\1\\;/g
$
As you can see, the script executed by sed differs depending on whether you use x=$(...) notation or x=`...` notation.
s/\([^\]\);/\1\;/g # ``
s/\([^\\]\);/\1\\;/g # $()
Summary
Use $(...); it is easier to understand.

You need to use four (three also work). I guess its because it's interpreted twice, first one by the sed command and the second one by the shell when reading the content of the variable:
result=`echo "hello;" | sed 's/\([^\\]\);/\1\\\\;/g'`
And
echo "$result"
yields:
hello\;

Related

Using value inside a variable without expanding

I am trying to find and replace a specific text content using the sed command and to run it via a shell script.
Below is the sample script that I am using:
fp=/asd/filename.txt
fd="sed -i -E 's ($2).* $2:$3 g' ${fp}"
eval $fd
and executing the same by passing the arguments:
./test.sh update asd asdfgh
But if the argument string contains $ , it breaks the commands and it is replacing with wrong values, like
./test.sh update asd $apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1.
How can I make sure that the values inside the variables are not expanded because of the $?
Updated
sh file test.sh
set -xv
fp="/asd/filename.txt"
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
text file filename.txt
hello:world
Outputs
1)
./test.sh update hello WORLD
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
++ sed -iE 's/(hello).*/hello:WORLD/g' /asd/filename.txt
2)
./test.sh update hello '$apr1$hosgaxyv$D0KXp5dCyZ2BUYCS9BmHu1'
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
++ sed -iE 's/(hello).*/hello:'\''$'\''apr1'\''$'\''hosgaxyv'\''$'\''D0KXp5dCyZ2BUYCS9BmHu1/g' /asd/filename.txt
In both the case , its not replacing the content
You don't need eval here at all:
fp=/asd/filename.txt
sed -i -E "s/(${2//'$'/'\$'}).*/\1:${3//'$'/'\$'}/g" "$fp"
The whole sed command is in double quotes so variables can expand.
I've replaced the blank as the s separator with / (doesn't really matter in the example).
I've used \1 to reference the first capture group instead of repeating the variable in the substitution.
Most importantly, I've used ${2//'$'/'\$'} instead of $2 (and similar for $3). This escapes every $ sign as \$; this is required because of the double quoting, or the $ get eaten by the shell before sed gets to see them.
When you call your script, you must escape any $ in the input, or the shell tries to expand them as variable names:
./test.sh update asd '$apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1.'
Put the command-line arguments that are filenames in single quotes:
./test.sh update 'asd' '$apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1'
must protect all the script arguments with quotes if having space and special shell char, and escape it if it's a dollar $, and -Ei instead of -iE even better drop it first for test, may add it later if being really sure
I admit i won't understant your regex so let's just get in the gist of solution, no need eval;
fp=/asd/filename.txt
sed -Ei "s/($2).*/$2:$3/g" $fp
./test.sh update asd '\$apr1\$HnIF6bOt\$9m3NzAwr.aG1Yp.t.bpIS1.'

Why isn't sed's output stored with its multiple backslashes in bash?

I am trying to store the output of sed into a variable, however the output differs from what I expect.
My test is the following one
$ foo="this is (foo)"
$ x="$(sed 's/(/\\\\(/g' <<< $foo)"
The result expected is :
$ echo $x
this is \\(foo)
The result I get is :
$ echo $x
this is \(foo)
However, when I do not assign the output to a variable, then result is the expected one :
$ sed 's/(/\\\\(/g' <<< $foo
this is \\(foo)
Why is the storage of my output failing?
NOTE :
I also tried the following command lines which all ended to the same result :
$ x=`sed 's/(/\\\\(/g' <<< $foo`
$ x=$(sed 's/(/\\\\(/g' <<< $foo)
$ x=`echo $foo | sed 's/(/\\\\(/g'`
$ x=$(echo $foo | sed 's/(/\\\\(/g')
Don't trust echo: The POSIX specification leaves enough leeway in its implementation that you can't trust its output to correctly represent the value at hand. Instead, use printf:
foo="this is (foo)"
x="$(sed 's/(/\\\\(/g' <<<"$foo")"
printf '%s\n' "$x"
...properly emits...
this is \\(foo)
From the POSIX specification for echo, emphasis added:
The following operands shall be supported:
string
A string to be written to standard output. If the first operand is -n, or if any of the operands contain a backslash ( '\' ) character, the results are implementation-defined.
Similarly, from the APPLICATION USAGE section of that same document:
It is not possible to use echo portably across all POSIX systems unless both -n (as the first argument) and escape sequences are omitted.
The printf utility can be used portably to emulate any of the traditional behaviors of the echo utility [...]

How to print current bash prompt?

The question is simple. I want to evaluate current value of PS1 in my bash script.
All materials on google point to tutorials on pimping it up, but I want to evaluate to see how would it be rendered by my current terminal, or at least by some terminal.
Is there any software/function that would help me achieve that? Of course I'd like to have all escaped characters evaluated, so echo $PS1 is not that useful in my case.
Bash 4.4+ solution using parameter transformation for a prompt string: echo "${PS1#P}"
[adamhotep#tabasco ~]$ echo "the prompt is '${PS1#P}'"
the prompt is '[adamhotep#tabasco ~]$'
[adamhotep#tabasco ~]$ TEST_STRING='\u is dining at \t using \s \V'
[adamhotep#tabasco ~]$ echo "${TEST_STRING}"
\u is dining at \t using \s \V
[adamhotep#tabasco ~]$ echo "${TEST_STRING#P}"
adamhotep is dining at 21:45:10 using bash 5.0.3
[adamhotep#tabasco ~]$
From the Bash Reference Manual page on Shell Parameter Expansion:
${parameter#operator}
Parameter transformation. The expansion is either a transformation of the value of parameter or information about parameter itself, depending on the value of operator.
Each operator is a single letter:
Q The expansion is a string that is the value of parameter quoted in a
format that can be reused as input.
E The expansion is a string that is the value of parameter with backslash
escape sequences expanded as with the $'…' quoting mechanism.
P The expansion is a string that is the result of expanding the value of
parameter as if it were a prompt string (see PROMPTING below).
A The expansion is a string in the form of an assignment statement or
declare command that, if evaluated, will recreate parameter with its
attributes and value.
a The expansion is a string consisting of flag values representing
parameter's attributes.
If parameter is # or *, the operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with # or *, the operation is applied to each member of the array in turn, and the expansion is the resultant list.
(See also this answer from duplicate question Echo expanded PS1.)
 
Z Shell (zsh) can do this with ${(%%)PS1} or with its print builtin's -P flag:
[adamhotep#tabasco ~]% echo "the prompt is '${(%%)PS1}'"
the prompt is '[adamhotep#tabasco ~]%'
[adamhotep#tabasco ~]% print -P "the prompt is '$PS1'"
the prompt is '[adamhotep#tabasco ~]%'
[adamhotep#tabasco ~]% TEST_STRING="%n is dining at %* using %N $ZSH_VERSION"
[adamhotep#tabasco ~]% echo "$TEST_STRING"
%n is dining at %* using %N 5.7.1
[adamhotep#tabasco ~]% echo "${(%%)TEST_STRING}"
adamhotep is dining at 11:49:01 using zsh 5.7.1
[adamhotep#tabasco ~]% print -P "$TEST_STRING"
adamhotep is dining at 11:49:07 using zsh 5.7.1
[adamhotep#tabasco ~]%
The Zsh Expansion and Subsitution manual tells us:
Parameter Expansion Flags. If the opening brace is directly followed by an opening parenthesis, the string up to the matching closing parenthesis will be taken as a list of flags. In cases where repeating a flag is meaningful, the repetitions need not be consecutive; for example, (q%q%q) means the same thing as the more readable (%%qqq). The following flags are supported:…
%    Expand all % escapes in the resulting words in the same way as in
prompts (see Prompt Expansion). If this flag is given twice, full
prompt expansion is done on the resulting words, depending on the
setting of the PROMPT_PERCENT, PROMPT_SUBST and PROMPT_BANG options.
From the Zsh Builtins documentation for print:
-P    Perform prompt expansion (see Prompt Expansion). In combination with -f, prompt escape sequences are parsed only within interpolated arguments, not within the format string.
I would get it like this:
echo $PS1
And then edit it with an editor. After that for the test (this is set while the session is active):
PS1='\[\033[1m\]\[\033[34m\]\u\[\033[90m\]#\[\033[01;35m\]\h:\[\033[01;32m\]\W\[\033[0m\]$ '
(\u is for user, \h is for host, \w is for full path and \W is for short path)
And if I like it I will make it permanent by changing the value of PS1 in ~/.bashrc
P.S.:
If you want to see all global variables:
printenv
OR:
printenv <name_of_var_to_see>
One more possibility, using script utility (part of bsdutils package on ubuntu):
$ TEST_PS1="\e[31;1m\u#\h:\n\e[0;1m\$ \e[0m"
$ RANDOM_STRING=some_random_string_here_that_is_not_part_of_PS1
$ script /dev/null <<-EOF | awk 'NR==2' RS=$RANDOM_STRING
PS1="$TEST_PS1"; HISTFILE=/dev/null
echo -n $RANDOM_STRING
echo -n $RANDOM_STRING
exit
EOF
<prints the formatted prompt properly here>
script command generates a file specified & the output is also shown on stdout. If filename is omitted, it generates a file called typescript.
Since we are not interested in the log file in this case, filename is specified as /dev/null. Instead the stdout of the script command is passed to awk for further processing.
The entire code can also be encapsulated into a function.
Also, the output prompt can also be assigned to a variable.
This approach also supports parsing of PROMPT_COMMAND...
EDIT:
It appears that the new version of script echoes the piped stdin in the typescript. To handle that, the above mechanism can be changed to:
$ TEST_PS1="\e[31;1m\u#\h:\n\e[0;1m\$ \e[0m"
$ RANDOM_STRING=some_random_string_here_that_is_not_part_of_PS1
$ script /dev/null <<-EOF | awk '{old=current; current=$0;} END{print old}' RS=$RANDOM_STRING
PS1="$TEST_PS1"; HISTFILE=/dev/null
alias $RANDOM_STRING=true
$RANDOM_STRING
$RANDOM_STRING
EOF
<prints the formatted prompt properly here>
Explanation:
Try entering these commands manually on the terminal. Copy these commands under the heredoc as they are and paste with mouse middle click. The script command's stdout would contain something very similar.
e.g. With above case, the output of the script command gives this:
PS1="\e[31;1m\u#\h:\n\e[0;1m$ \e[0m"; HISTFILE=/dev/null
alias some_random_string_here_that_is_not_part_of_PS1=true
some_random_string_here_that_is_not_part_of_PS1
some_random_string_here_that_is_not_part_of_PS1
\e[0m"; HISTFILE=/dev/nullhsane-dev : ~/Desktop $ PS1="\e[31;1m\u#\h:\n\e[0;1m$
anishsane#anishsane-dev:
$ alias some_random_string_here_that_is_not_part_of_PS1=true
anishsane#anishsane-dev:
$ some_random_string_here_that_is_not_part_of_PS1
anishsane#anishsane-dev:
$ some_random_string_here_that_is_not_part_of_PS1
anishsane#anishsane-dev:
$ exit
Split that stdout with "some_random_string_here_that_is_not_part_of_PS1" as delimiter (record separator of awk) and print the last but one record.
EDIT2:
Another mechanism (using bash source code and gdb):
$ gdb -batch -p $$ -ex 'call bind_variable("expanded_PS1", decode_prompt_string (get_string_value ("PS1")), 0)'
$ echo "$expanded_PS1"
<prints the formatted prompt properly here>
There is a tiny issue here though. The \[ or \] strings in PS1 will get printed as \1/\2 respectively. You can remove those with tr -d '\1\2' <<< "$expanded_PS1"
If you get error like gdb failed to attach to the process (seems to happen in ubuntu :-\ ), run gdb with sudo.
Another way to do this would be to eval echoing your prompt to handle any expansion (was not sure why the brackets remain). This is most likely less robust than #anishsane's method, but may be a little quicker:
show-prompt() {
eval 'echo -en "'$PS1'"' | sed -e 's#\\\[##g' -e 's#\\\]##g'
}
# To show it in a function registered with `complete -F` on
# a single tab, and keep the user's input:
show-prompt
echo -n "${COMP_WORDS[#]}"
Had tinkered with that regarding this GitHub issue.
Try the below command
echo $PS1 |
sed -e s/'\\d'/"$(date +'%a %b %_d')"/g |
sed -e s/'\\t'/"$(date +'%T')"/g |
sed -e s/'\\#'/"$(date +'%r')"/g |
sed -e s/'\\T'/"$(date +'%r'| awk {'print $1'})"/g |
sed -e s/'\\e'//g | sed -e s/'\\h'/"$HOSTNAME"/g |
sed -e s/'\\h'/"$HOSTNAME"/g |
sed -e s/'\\H'/"$HOSTNAME"/g |
sed -e s/'\\u'/"$USER"/g |
sed -e s#'\\W'#"$(pwd)"#g |
sed -e s/'\\w'/"$(pwd | sed -e s#$HOME#'~'#g )"/g |
sed -e s/"\\\\"//g |
sed -e s/"\\["//g |
sed -e s/"\\]"/*/g |
cut -d'*' -f2 |
cut -d';' -f2 |
sed s/\ //g |
sed -e s/[a-z]$/"$([ "$USER" != "root" ] && echo \$ || echo \#)"/g
Edit the /etc/bashrc file
you can use this as example and check the output
# If id command returns zero, you’ve root access.
if [ $(id -u) -eq 0 ];
then # you are root, set red colour prompt
PS1="\\[$(tput setaf 1)\\]\\u#\\h:\\w #\\[$(tput sgr0)\\]"
else # normal
PS1="[\\u#\\h:\\w] $"
fi

Escaping backslash in AWK in command substituion

I am trying to escape backslash in AWK. This is a sample of what I am trying to do.
Say, I have a variable
$echo $a
hi
The following works
$echo $a | awk '{printf("\\\"%s\"",$1)'}
\"hi"
But, when I am trying to save the output of the same command to a variable using command substitution, I get the following error:
$ q=`echo $a | awk '{printf("\\\"%s\"",$1)'}`
awk: {printf("\\"%s\"",$1)}
awk: ^ backslash not last character on line
I am unable to understand why command substitution is breaking the AWK. Thanks a lot for your help.
Try this:
q=$(echo $a | awk '{printf("\\\"%s\"",$1)}')
Test:
$ a=hi
$ echo $a
hi
$ q=$(echo $a | awk '{printf("\\\"%s\"",$1)}')
$ echo $q
\"hi"
Update:
It will, it just gets a littler messier.
q=`echo $a | awk '{printf("\\\\\"%s\"",$1)}'`
Test:
$ b=hello
$ echo $b
hello
$ t=`echo $b | awk '{printf("\\\\\"%s\"",$1)}'`
$ echo $t
\"hello"
Reference
Quoting inside backquoted commands is somewhat complicated, mainy
because the same token is used to start and to end a backquoted
command. As a consequence, to nest backquoted commands, the backquotes
of the inner one have to be escaped using backslashes. Furthermore,
backslashes can be used to quote other backslashes and dollar signs
(the latter are in fact redundant). If the backquoted command is
contained within double quotes, a backslash can also be used to quote a
double quote. All these backslashes are removed when the shell reads
the backquoted command. All other backslashes are left intact.
The new $(...) avoids these troubles.
Don't get into bad habits with backticks, quoting and parsing shell variables to awk The correct way to do this is:
$ shell_var="hi"
$ awk -v awk_var="$shell_var" -v c='\' 'BEGIN{printf "%s%s\n",c,awk_var}'
\hi
$ res=$(awk -v awk_var="$shell_var" -v c='\' 'BEGIN{printf "%s%s\n",c,awk_var}')
$ echo "$res"
\hi

How to replace .. from string in bash script?

I have to remove .. character from a file in Bash script. Example:
I have some string like:
some/../path/to/file
some/ab/path/to/file
And after replace, it should look like
some/path/to/file
some/ab/path/to/file
I have used below code
DUMMY_STRING=/../
TEMP_FILE=./temp.txt
sed s%${DUMMY_STRING}%/%g ${SRC_FILE} > ${TEMP_FILE}
cp ${TEMP_FILE} ${SRC_FILE}
It is replacing the /../ in line 1; but it is also removing the line /ab/ from second line. This is not desired. I understand it is considering /../ as some regex and /ab/ matches this regex. But I want only those /../ to be replaced.
Please provide some help.
Thanks,
NN
The . is a metacharacter in sed meaning 'any character'. To suppress its special meaning, escape it with a backslash:
sed -e 's%/\.\./%/%g' $src_file > $temp_file
Note that you are referring to different files after you eliminate the /../ like that. To refer to the same name as before (in the absence of symlinks, which complicate things), you would need to remove the directory component before the /../. Thus:
some/../path/to/file
path/to/file
refer to the same file, assuming some is a directory and not a symlink somewhere else, but in general, some/path/to/file is a different file (though symlinks could be used to confound that assertion).
$ x="some/../path/to/file
> some/ab/path/to/file
> /some/path/../to/another/../file"
$ echo "$x"
some/../path/to/file
some/ab/path/to/file
/some/path/../to/another/../file
$ echo "$x" | sed -e 's%/\.\./%/%g'
some/path/to/file
some/ab/path/to/file
/some/path/to/another/file
$ echo "$x" | sed -e "s%/\.\./%/%g"
some/path/to/file
some/ab/path/to/file
/some/path/to/another/file
$ echo "$x" | sed -e s%/\.\./%/%g
some/path/file
some/path/file
/some/path/to/another/file
$ echo "$x" | sed -e s%/\\.\\./%/%g
some/path/to/file
some/ab/path/to/file
/some/path/to/another/file
$
Note the careful use of double quotes around the variable "$x" in the echo commands. I could have used either single or double quotes in the assignment and would have gotten the same result.
Test on Mac OS X 10.7.4 with the standard sed (and shell is /bin/sh, aka bash 3.2.x), but the results would be the same on any system.

Resources