I am trying to run a complex grep command from shell (currently zsh on MacOS, but bash would be ok)
I want to pass variables, i.e. $1 and $2, to the command : grep -e 'something $1' -e 'somethingelse $2' file
For instance my script:
#/bin/zsh
echo ------
echo grep -e "'"something $1"'" -e "'"somethingelse $2"'" file
echo ------
grep -e "'"something $1"'" -e "'"somethingelse $2"'" file
This doesn't work with:
% ~/scripts/test cat mouse
------
grep -e 'something cat' -e 'somethingelse mouse' file
------
grep: cat': No such file or directory
grep: mouse': No such file or directory
Any idea?
Don't try to add single-quotes when you run the command; just put double-quotes around the pattern (including the parameter):
#/bin/zsh
echo ------
echo grep -e "'something $1'" -e "'somethingelse $2'" file
echo ------
grep -e "something $1" -e "somethingelse $2" file
Note that when echoing it, I used single-quotes inside the double-quotes. They'll be printed, so it'll look ok, but the shell won't treat them as syntactically significant. When actually running grep, you don't want single-quotes at all.
Well, unless the something contains escapes or dollar signs; in that case, you can either escape them:
grep -e "\$ometh\\ng $1" -e "\$ometh\\nge\\se $2" file
Or mix single- and double-quoting, with single-quotes around the fixed pattern part, and double-quotes just around the parameter part:
grep -e '$ometh\ng '"$1" -e '$ometh\nge\se '"$2" file
I don't know why you want grep to see your quotes. Assuming your literal string something does not contain spaces or other characters which are significant to the shell (most notable filename expansion wildcards) and you are using zsh,
grep something$1 FILE
would work. Of course if you have spaces in or around your something, you need to quote it:
grep 'something '$1 FILE # Significant space between something and $1
or
grep "something $1" FILE
Since you also mentioned bash: In bash, only the last form (using double quotes) makes sense, because if $1 contained spaces, bash would do word splitting.
Related
I am trying to find and replace a specific text content using the sed command and to run it via a shell script.
Below is the sample script that I am using:
fp=/asd/filename.txt
fd="sed -i -E 's ($2).* $2:$3 g' ${fp}"
eval $fd
and executing the same by passing the arguments:
./test.sh update asd asdfgh
But if the argument string contains $ , it breaks the commands and it is replacing with wrong values, like
./test.sh update asd $apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1.
How can I make sure that the values inside the variables are not expanded because of the $?
Updated
sh file test.sh
set -xv
fp="/asd/filename.txt"
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
text file filename.txt
hello:world
Outputs
1)
./test.sh update hello WORLD
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
++ sed -iE 's/(hello).*/hello:WORLD/g' /asd/filename.txt
2)
./test.sh update hello '$apr1$hosgaxyv$D0KXp5dCyZ2BUYCS9BmHu1'
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
++ sed -iE 's/(hello).*/hello:'\''$'\''apr1'\''$'\''hosgaxyv'\''$'\''D0KXp5dCyZ2BUYCS9BmHu1/g' /asd/filename.txt
In both the case , its not replacing the content
You don't need eval here at all:
fp=/asd/filename.txt
sed -i -E "s/(${2//'$'/'\$'}).*/\1:${3//'$'/'\$'}/g" "$fp"
The whole sed command is in double quotes so variables can expand.
I've replaced the blank as the s separator with / (doesn't really matter in the example).
I've used \1 to reference the first capture group instead of repeating the variable in the substitution.
Most importantly, I've used ${2//'$'/'\$'} instead of $2 (and similar for $3). This escapes every $ sign as \$; this is required because of the double quoting, or the $ get eaten by the shell before sed gets to see them.
When you call your script, you must escape any $ in the input, or the shell tries to expand them as variable names:
./test.sh update asd '$apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1.'
Put the command-line arguments that are filenames in single quotes:
./test.sh update 'asd' '$apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1'
must protect all the script arguments with quotes if having space and special shell char, and escape it if it's a dollar $, and -Ei instead of -iE even better drop it first for test, may add it later if being really sure
I admit i won't understant your regex so let's just get in the gist of solution, no need eval;
fp=/asd/filename.txt
sed -Ei "s/($2).*/$2:$3/g" $fp
./test.sh update asd '\$apr1\$HnIF6bOt\$9m3NzAwr.aG1Yp.t.bpIS1.'
I have a shell script that processes some files. The problem is that there might be white spaces in file names, I did:
#!/bin/sh
FILE=`echo $FILE | sed -e 's/[[:space:]]/\\ /g'`
cat $FILE
So the variable FILE is a file name which is passed in from some other program. It may contain white spaces. I used sed to escape white space with \ in order to make the command line utilities be able to process it.
The problem is that it doesn't work. echo $FILE | sed -e 's/[[:space:]]/\\ /g' itself works as expected, but when assigned to FILE, the escape char \ disappeared again. As a result, cat will interpret it as more than 1 arguments. I wonder why it behaves like this? Is there anyway to avoid it? And what if there're multiple white spaces, say some terrible file.txt, which should be replaced by some\ \ \ terrible\ \ file.txt. Thanks.
Don't try to put escape characters inside your data -- they're only honored as syntax (that is, backslashes have meaning when found in your source code, not your data).
That is to say, the following works perfectly, exactly as given:
file='some terrible file.txt'
cat "$file"
...likewise if the name comes from a glob result or similar:
# this operates in a temporary directory to not change the filesystem you're running it in
tempdir=$(mktemp -d "${TMPDIR:-/tmp}/testdir.XXXXXX") && (
cd "$tempdir" || exit
echo 'example' >'some terrible file.txt'
for file in *.txt; do
printf 'Found file %q with the following contents:\n' "$file"
cat "$file"
done
rm -rf "$tempdir"
)
Don’t make it more complicated than it is.
cat "$FILE"
That’s all you need. Note the quotes around the variable. They prevent the variable from being expanded and split at whitespace. You should always write your shell programs like that. Always put quotes around all your variables, unless you really want the shell to expand them.
for i in $pattern; do
That would be ok.
The question is simple. I want to evaluate current value of PS1 in my bash script.
All materials on google point to tutorials on pimping it up, but I want to evaluate to see how would it be rendered by my current terminal, or at least by some terminal.
Is there any software/function that would help me achieve that? Of course I'd like to have all escaped characters evaluated, so echo $PS1 is not that useful in my case.
Bash 4.4+ solution using parameter transformation for a prompt string: echo "${PS1#P}"
[adamhotep#tabasco ~]$ echo "the prompt is '${PS1#P}'"
the prompt is '[adamhotep#tabasco ~]$'
[adamhotep#tabasco ~]$ TEST_STRING='\u is dining at \t using \s \V'
[adamhotep#tabasco ~]$ echo "${TEST_STRING}"
\u is dining at \t using \s \V
[adamhotep#tabasco ~]$ echo "${TEST_STRING#P}"
adamhotep is dining at 21:45:10 using bash 5.0.3
[adamhotep#tabasco ~]$
From the Bash Reference Manual page on Shell Parameter Expansion:
${parameter#operator}
Parameter transformation. The expansion is either a transformation of the value of parameter or information about parameter itself, depending on the value of operator.
Each operator is a single letter:
Q The expansion is a string that is the value of parameter quoted in a
format that can be reused as input.
E The expansion is a string that is the value of parameter with backslash
escape sequences expanded as with the $'…' quoting mechanism.
P The expansion is a string that is the result of expanding the value of
parameter as if it were a prompt string (see PROMPTING below).
A The expansion is a string in the form of an assignment statement or
declare command that, if evaluated, will recreate parameter with its
attributes and value.
a The expansion is a string consisting of flag values representing
parameter's attributes.
If parameter is # or *, the operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with # or *, the operation is applied to each member of the array in turn, and the expansion is the resultant list.
(See also this answer from duplicate question Echo expanded PS1.)
Z Shell (zsh) can do this with ${(%%)PS1} or with its print builtin's -P flag:
[adamhotep#tabasco ~]% echo "the prompt is '${(%%)PS1}'"
the prompt is '[adamhotep#tabasco ~]%'
[adamhotep#tabasco ~]% print -P "the prompt is '$PS1'"
the prompt is '[adamhotep#tabasco ~]%'
[adamhotep#tabasco ~]% TEST_STRING="%n is dining at %* using %N $ZSH_VERSION"
[adamhotep#tabasco ~]% echo "$TEST_STRING"
%n is dining at %* using %N 5.7.1
[adamhotep#tabasco ~]% echo "${(%%)TEST_STRING}"
adamhotep is dining at 11:49:01 using zsh 5.7.1
[adamhotep#tabasco ~]% print -P "$TEST_STRING"
adamhotep is dining at 11:49:07 using zsh 5.7.1
[adamhotep#tabasco ~]%
The Zsh Expansion and Subsitution manual tells us:
Parameter Expansion Flags. If the opening brace is directly followed by an opening parenthesis, the string up to the matching closing parenthesis will be taken as a list of flags. In cases where repeating a flag is meaningful, the repetitions need not be consecutive; for example, (q%q%q) means the same thing as the more readable (%%qqq). The following flags are supported:…
% Expand all % escapes in the resulting words in the same way as in
prompts (see Prompt Expansion). If this flag is given twice, full
prompt expansion is done on the resulting words, depending on the
setting of the PROMPT_PERCENT, PROMPT_SUBST and PROMPT_BANG options.
From the Zsh Builtins documentation for print:
-P Perform prompt expansion (see Prompt Expansion). In combination with -f, prompt escape sequences are parsed only within interpolated arguments, not within the format string.
I would get it like this:
echo $PS1
And then edit it with an editor. After that for the test (this is set while the session is active):
PS1='\[\033[1m\]\[\033[34m\]\u\[\033[90m\]#\[\033[01;35m\]\h:\[\033[01;32m\]\W\[\033[0m\]$ '
(\u is for user, \h is for host, \w is for full path and \W is for short path)
And if I like it I will make it permanent by changing the value of PS1 in ~/.bashrc
P.S.:
If you want to see all global variables:
printenv
OR:
printenv <name_of_var_to_see>
One more possibility, using script utility (part of bsdutils package on ubuntu):
$ TEST_PS1="\e[31;1m\u#\h:\n\e[0;1m\$ \e[0m"
$ RANDOM_STRING=some_random_string_here_that_is_not_part_of_PS1
$ script /dev/null <<-EOF | awk 'NR==2' RS=$RANDOM_STRING
PS1="$TEST_PS1"; HISTFILE=/dev/null
echo -n $RANDOM_STRING
echo -n $RANDOM_STRING
exit
EOF
<prints the formatted prompt properly here>
script command generates a file specified & the output is also shown on stdout. If filename is omitted, it generates a file called typescript.
Since we are not interested in the log file in this case, filename is specified as /dev/null. Instead the stdout of the script command is passed to awk for further processing.
The entire code can also be encapsulated into a function.
Also, the output prompt can also be assigned to a variable.
This approach also supports parsing of PROMPT_COMMAND...
EDIT:
It appears that the new version of script echoes the piped stdin in the typescript. To handle that, the above mechanism can be changed to:
$ TEST_PS1="\e[31;1m\u#\h:\n\e[0;1m\$ \e[0m"
$ RANDOM_STRING=some_random_string_here_that_is_not_part_of_PS1
$ script /dev/null <<-EOF | awk '{old=current; current=$0;} END{print old}' RS=$RANDOM_STRING
PS1="$TEST_PS1"; HISTFILE=/dev/null
alias $RANDOM_STRING=true
$RANDOM_STRING
$RANDOM_STRING
EOF
<prints the formatted prompt properly here>
Explanation:
Try entering these commands manually on the terminal. Copy these commands under the heredoc as they are and paste with mouse middle click. The script command's stdout would contain something very similar.
e.g. With above case, the output of the script command gives this:
PS1="\e[31;1m\u#\h:\n\e[0;1m$ \e[0m"; HISTFILE=/dev/null
alias some_random_string_here_that_is_not_part_of_PS1=true
some_random_string_here_that_is_not_part_of_PS1
some_random_string_here_that_is_not_part_of_PS1
\e[0m"; HISTFILE=/dev/nullhsane-dev : ~/Desktop $ PS1="\e[31;1m\u#\h:\n\e[0;1m$
anishsane#anishsane-dev:
$ alias some_random_string_here_that_is_not_part_of_PS1=true
anishsane#anishsane-dev:
$ some_random_string_here_that_is_not_part_of_PS1
anishsane#anishsane-dev:
$ some_random_string_here_that_is_not_part_of_PS1
anishsane#anishsane-dev:
$ exit
Split that stdout with "some_random_string_here_that_is_not_part_of_PS1" as delimiter (record separator of awk) and print the last but one record.
EDIT2:
Another mechanism (using bash source code and gdb):
$ gdb -batch -p $$ -ex 'call bind_variable("expanded_PS1", decode_prompt_string (get_string_value ("PS1")), 0)'
$ echo "$expanded_PS1"
<prints the formatted prompt properly here>
There is a tiny issue here though. The \[ or \] strings in PS1 will get printed as \1/\2 respectively. You can remove those with tr -d '\1\2' <<< "$expanded_PS1"
If you get error like gdb failed to attach to the process (seems to happen in ubuntu :-\ ), run gdb with sudo.
Another way to do this would be to eval echoing your prompt to handle any expansion (was not sure why the brackets remain). This is most likely less robust than #anishsane's method, but may be a little quicker:
show-prompt() {
eval 'echo -en "'$PS1'"' | sed -e 's#\\\[##g' -e 's#\\\]##g'
}
# To show it in a function registered with `complete -F` on
# a single tab, and keep the user's input:
show-prompt
echo -n "${COMP_WORDS[#]}"
Had tinkered with that regarding this GitHub issue.
Try the below command
echo $PS1 |
sed -e s/'\\d'/"$(date +'%a %b %_d')"/g |
sed -e s/'\\t'/"$(date +'%T')"/g |
sed -e s/'\\#'/"$(date +'%r')"/g |
sed -e s/'\\T'/"$(date +'%r'| awk {'print $1'})"/g |
sed -e s/'\\e'//g | sed -e s/'\\h'/"$HOSTNAME"/g |
sed -e s/'\\h'/"$HOSTNAME"/g |
sed -e s/'\\H'/"$HOSTNAME"/g |
sed -e s/'\\u'/"$USER"/g |
sed -e s#'\\W'#"$(pwd)"#g |
sed -e s/'\\w'/"$(pwd | sed -e s#$HOME#'~'#g )"/g |
sed -e s/"\\\\"//g |
sed -e s/"\\["//g |
sed -e s/"\\]"/*/g |
cut -d'*' -f2 |
cut -d';' -f2 |
sed s/\ //g |
sed -e s/[a-z]$/"$([ "$USER" != "root" ] && echo \$ || echo \#)"/g
Edit the /etc/bashrc file
you can use this as example and check the output
# If id command returns zero, you’ve root access.
if [ $(id -u) -eq 0 ];
then # you are root, set red colour prompt
PS1="\\[$(tput setaf 1)\\]\\u#\\h:\\w #\\[$(tput sgr0)\\]"
else # normal
PS1="[\\u#\\h:\\w] $"
fi
I'm trying to escape ('\') a semicolon (';') in a string on unix shell (bash) with sed. It works when I do it directly without assigning the value to a variable. That is,
$ echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'
hello\;
$
However, it doesn't appear to work when the above command is assigned to a variable:
$ result=`echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'`
$
$ echo $result
hello;
$
Any idea why?
I tried by using the value enclosed with and without quotes but that didn't help. Any clue greatly appreciated.
btw, I first thought the semicolon at the end of the string was somehow acting as a terminator and hence the shell didn't continue executing the sed (if that made any sense). However, that doesn't appear to be an issue. I tried by using the semicolon not at the end of the string (somewhere in between). I still see the same result as before. That is,
$ echo "hel;lo" | sed 's/\([^\\]\);/\1\\;/g'
hel\;lo
$
$ result=`echo "hel;lo" | sed 's/\([^\\]\);/\1\\;/g'`
$
$ echo $result
hel;lo
$
You don't need sed (or any other regex engine) for this at all:
s='hello;'
echo "${s//;/\;}"
This is a parameter expansion which replaces ; with \;.
That said -- why are you trying to do this? In most cases, you don't want escape characters (which are syntax) to be inside of scalar variables (which are data); they only matter if you're parsing your data as syntax (such as using eval), which is a bad idea for other reasons, and best avoided (or done programatically, as via printf %q).
I find it interesting that the use of back-ticks gives one result (your result) and the use of $(...) gives another result (the wanted result):
$ echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'
hello\;
$ z1=$(echo "hello;" | sed 's/\([^\\]\);/\1\\;/g')
$ z2=`echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'`
$ printf "%s\n" "$z1" "$z2"
hello\;
hello;
$
If ever you needed an argument for using the modern x=$(...) notation in preference to the older x=`...` notation, this is probably it. The shell does an extra round of backslash interpretation with the back-ticks. I can demonstrate this with a little program I use when debugging shell scripts called al (for 'argument list'); you can simulate it with printf "%s\n":
$ z2=`echo "hello;" | al sed 's/\([^\\]\);/\1\\;/g'`
$ echo "$z2"
sed
s/\([^\]\);/\1\;/g
$ z1=$(echo "hello;" | al sed 's/\([^\\]\);/\1\\;/g')
$ echo "$z1"
sed
s/\([^\\]\);/\1\\;/g
$ z1=$(echo "hello;" | printf "%s\n" sed 's/\([^\\]\);/\1\\;/g')
$ echo "$z1"
sed
s/\([^\\]\);/\1\\;/g
$
As you can see, the script executed by sed differs depending on whether you use x=$(...) notation or x=`...` notation.
s/\([^\]\);/\1\;/g # ``
s/\([^\\]\);/\1\\;/g # $()
Summary
Use $(...); it is easier to understand.
You need to use four (three also work). I guess its because it's interpreted twice, first one by the sed command and the second one by the shell when reading the content of the variable:
result=`echo "hello;" | sed 's/\([^\\]\);/\1\\\\;/g'`
And
echo "$result"
yields:
hello\;
I have to remove .. character from a file in Bash script. Example:
I have some string like:
some/../path/to/file
some/ab/path/to/file
And after replace, it should look like
some/path/to/file
some/ab/path/to/file
I have used below code
DUMMY_STRING=/../
TEMP_FILE=./temp.txt
sed s%${DUMMY_STRING}%/%g ${SRC_FILE} > ${TEMP_FILE}
cp ${TEMP_FILE} ${SRC_FILE}
It is replacing the /../ in line 1; but it is also removing the line /ab/ from second line. This is not desired. I understand it is considering /../ as some regex and /ab/ matches this regex. But I want only those /../ to be replaced.
Please provide some help.
Thanks,
NN
The . is a metacharacter in sed meaning 'any character'. To suppress its special meaning, escape it with a backslash:
sed -e 's%/\.\./%/%g' $src_file > $temp_file
Note that you are referring to different files after you eliminate the /../ like that. To refer to the same name as before (in the absence of symlinks, which complicate things), you would need to remove the directory component before the /../. Thus:
some/../path/to/file
path/to/file
refer to the same file, assuming some is a directory and not a symlink somewhere else, but in general, some/path/to/file is a different file (though symlinks could be used to confound that assertion).
$ x="some/../path/to/file
> some/ab/path/to/file
> /some/path/../to/another/../file"
$ echo "$x"
some/../path/to/file
some/ab/path/to/file
/some/path/../to/another/../file
$ echo "$x" | sed -e 's%/\.\./%/%g'
some/path/to/file
some/ab/path/to/file
/some/path/to/another/file
$ echo "$x" | sed -e "s%/\.\./%/%g"
some/path/to/file
some/ab/path/to/file
/some/path/to/another/file
$ echo "$x" | sed -e s%/\.\./%/%g
some/path/file
some/path/file
/some/path/to/another/file
$ echo "$x" | sed -e s%/\\.\\./%/%g
some/path/to/file
some/ab/path/to/file
/some/path/to/another/file
$
Note the careful use of double quotes around the variable "$x" in the echo commands. I could have used either single or double quotes in the assignment and would have gotten the same result.
Test on Mac OS X 10.7.4 with the standard sed (and shell is /bin/sh, aka bash 3.2.x), but the results would be the same on any system.