I am new to shell scripts. I want to read a file line by line, which contains arguments and if the arguments contains any spaces in it, I want to replace it by enclosing with quotes.
For example if the file (test.dat) contains:
-DtestArgument1=/path/to a/text file
-DtestArgument2=/path/to a/text file
After parsing the above file, shell script should prepare the string with following:
-DtestArgument1="/path/to a/text file" -DtestArgument2="/path/to a/text file"
Here is my shell script:
while read ARGUMENT; do
ARGUMENT=`echo ${ARGUMENT} | tr "\n" " "`
if [[ "${ARGUMENT}" =~ " " ]]; then
ARGUMENT=`echo $ARGUMENT | sed 's/\^(-D.*\)=(.*)/\1=\"\2\"/g'`
NEW_ARGUMENT="${NEW_ARGUMENT} ${ARGUMENT}"
else
echo "doesn't contains spaces"
NEW_ARGUMENT="${NEW_ARGUMENT} ${ARGUMENT}"
fi
done < test.dat
But it's throwing the following error:
sed: -e expression #1, char 28: Unmatched ) or \)
The code should be compatible with all shells.
I think you should simplify the problem. Rather than worrying about spaces, just quote the argument after the =. Something like:
sed -e 's/=/="/' -e 's/$/"/' test.dat | paste -s -d\ -
Should be sufficient. If you really care about spaces, you could try something like:
sed -e '/=.* /{ s/=/="/; s/$/"/; }' test.dat | paste -s -d\ -
That will only notice spaces after the =. Just use / / if you really want to change any line that has a space anywhere.
There's no need to use a while/read loop: just let sed read the file directly.
The sed parentheses should be escaped:
ARGUMENT=`echo $ARGUMENT | sed "s/\^\(-D.*\)=\(.*\)/\1=\"\2\"/g"`
One place you did, in 3 places you forgot... BTW, I generally use " quotation.
If you prefer '-style, do like this:
ARGUMENT=`echo $ARGUMENT | sed 's/\^(-D.*)=(.*)/\1="\2"/g'`
Related
I have a Bash script working fine locally, now I am trying to put it in Jenkinsfile to run as its pipeline:
stage('Update Cloudfront'){
steps {
sh '''
#!/bin/bash
YAML_FILE="path/to/values.yaml"
DATE="$(date '+%d-%m-%Y')"
wget https://www.cloudflare.com/ips-v4 && wget https://www.cloudflare.com/ips-v6
CLOUDFLARE_NEW=$(awk '{printf fmt,$1}' fmt="%s\n" ips-v4 ips-v6 | paste -sd, -)
CLOUDFLARE_OLD=$(yq -r .controller.config.proxy-real-ip-cidr $YAML_FILE | sed -E 's/\,37\.16\.11\.30\/32//')
if [[ "$CLOUDFLARE_NEW" == "$CLOUDFLARE_OLD" ]]; then
echo "No need to do anything"
else
echo "Cloudflare IP ranges change detected, updating Nginx value file"
CLOUDFLARE_NEW=$(awk '{printf fmt,$1}' fmt="%s\n" ips-v4 ips-v6 | paste -sd, -) yq e '.controller.config.proxy-real-ip-cidr = env(CLOUDFLARE_NEW)' -i $YAML_FILE
echo "Add third party IP range"
yq e '.controller.config.proxy-real-ip-cidr +=",1.2.3.4/32"' -i $YAML_FILE
fi
'''
}
}//end stage('Update Cloudfront')
Unfortunately it won't work:
WorkflowScript: 73: unexpected char: '\' # line 73, column 113.
cidr $YAML_FILE | sed -E \\"s/\,37\.16\.
^
I've tried to escape it with \\"s/\,37\.16\.11\.30\/32//\\" etc. but it doesn't work either. I've tried with double and single quotes with no luck.
You can avoid all the escaping by using a character class and different regex delimiters, like so:
sed -e 's#,37[.]16[.]11[.]30/32##'
In the event you do need to escape something though, simply doubling the backslash should do it:
sed -e 's/,37\\.16\\.11\\.30\\/32//'
Though, given the number of levels involved here, it might need double escaping:
sed -e 's/,37\\\\.16\\\\.11\\\\.30\\\\/32//'
Below is the piece of code of my bash script, I want to get duplicate output of that script.
This is how my script runs
#bash check_script -a used_memory
Output is: used_memory: 812632
Desired Output: used_memory: 812632 | used_memory: 812632
get_vals() {
metrics=`command -h $hostname -p $port -a $pass info | grep -w $opt_var | cut -d ':' -f2 > ${filename}`
}
output() {
get_vals
if [ -s ${filename} ];
then
val1=`cat ${filename}`
echo "$opt_var: $val1"
# rm $filename;
exit $ST_OK;
else
echo "Parameter not found"
exit $ST_UK
fi
}
But when i used echo "$opt_var: $val1 | $opt_var: $val1" the output become: | used_memory: 812632
$opt_var is an argument.
I had a similar problem when capturing results from cat with Windows-formatted text files. One way to circumvent this issue is to pipe your result to dos2unix, e.g.:
val1=`cat ${filename} | dos2unix`
Also, if you want to duplicate lines, you can use sed:
sed 's/^\(.*\)$/\1 | \1/'
Then pipe it to your echo command:
echo "$opt_var: $val1" | sed 's/^\(.*\)$/\1 | \1/'
The sed expression works like that:
's/<before>/<after>/' means that you want to substitute <before> with <after>
on the <before> side: ^.*$ is a regular expression meaning you get the entire line, ^\(.*\)$ is basically the same regex but you get the entire line and you capture everything (capturing is performed inside the \(\) expression)
on the <after> side: \1 | \1 means you write the 1st captured expression (\1), then the space character, then the pipe character, then the space character and then the 1st captured expression again
So it captures your entire line and duplicates it with a "|" separator in the middle.
Below is the snippet of a shell script from a larger script. It removes the quotes from the string that is held by a variable. I am doing it using sed, but is it efficient? If not, then what is the efficient way?
#!/bin/sh
opt="\"html\\test\\\""
temp=`echo $opt | sed 's/.\(.*\)/\1/' | sed 's/\(.*\)./\1/'`
echo $temp
Use tr to delete ":
echo "$opt" | tr -d '"'
NOTE: This does not fully answer the question, removes all double quotes, not just leading and trailing. See other answers below.
There's a simpler and more efficient way, using the native shell prefix/suffix removal feature:
temp="${opt%\"}"
temp="${temp#\"}"
echo "$temp"
${opt%\"} will remove the suffix " (escaped with a backslash to prevent shell interpretation).
${temp#\"} will remove the prefix " (escaped with a backslash to prevent shell interpretation).
Another advantage is that it will remove surrounding quotes only if there are surrounding quotes.
BTW, your solution always removes the first and last character, whatever they may be (of course, I'm sure you know your data, but it's always better to be sure of what you're removing).
Using sed:
echo "$opt" | sed -e 's/^"//' -e 's/"$//'
(Improved version, as indicated by jfgagne, getting rid of echo)
sed -e 's/^"//' -e 's/"$//' <<<"$opt"
So it replaces a leading " with nothing, and a trailing " with nothing too. In the same invocation (there isn't any need to pipe and start another sed. Using -e you can have multiple text processing).
If you're using jq and trying to remove the quotes from the result, the other answers will work, but there's a better way. By using the -r option, you can output the result with no quotes.
$ echo '{"foo": "bar"}' | jq '.foo'
"bar"
$ echo '{"foo": "bar"}' | jq -r '.foo'
bar
There is a straightforward way using xargs:
> echo '"quoted"' | xargs
quoted
xargs uses echo as the default command if no command is provided and strips quotes from the input, see e.g. here. Note, however, that this will work only if the string does not contain additional quotes. In that case it will either fail (uneven number of quotes) or remove all of them.
If you came here for aws cli --query, try this. --output text
You can do it with only one call to sed:
$ echo "\"html\\test\\\"" | sed 's/^"\(.*\)"$/\1/'
html\test\
The shortest way around - try:
echo $opt | sed "s/\"//g"
It actually removes all "s (double quotes) from opt (are there really going to be any more double quotes other than in the beginning and the end though? So it's actually the same thing, and much more brief ;-))
The easiest solution in Bash:
$ s='"abc"'
$ echo $s
"abc"
$ echo "${s:1:-1}"
abc
This is called substring expansion (see Gnu Bash Manual and search for ${parameter:offset:length}). In this example it takes the substring from s starting at position 1 and ending at the second last position. This is due to the fact that if length is a negative value it is interpreted as a backwards running offset from the end of parameter.
Update
A simple and elegant answer from Stripping single and double quotes in a string using bash / standard Linux commands only:
BAR=$(eval echo $BAR) strips quotes from BAR.
=============================================================
Based on hueybois's answer, I came up with this function after much trial and error:
function stripStartAndEndQuotes {
cmd="temp=\${$1%\\\"}"
eval echo $cmd
temp="${temp#\"}"
eval echo "$1=$temp"
}
If you don't want anything printed out, you can pipe the evals to /dev/null 2>&1.
Usage:
$ BAR="FOO BAR"
$ echo BAR
"FOO BAR"
$ stripStartAndEndQuotes "BAR"
$ echo BAR
FOO BAR
This is the most discrete way without using sed:
x='"fish"'
printf " quotes: %s\nno quotes: %s\n" "$x" "${x//\"/}"
Or
echo $x
echo ${x//\"/}
Output:
quotes: "fish"
no quotes: fish
I got this from a source.
Linux=`cat /etc/os-release | grep "ID" | head -1 | awk -F= '{ print $2 }'`
echo $Linux
Output:
"amzn"
Simplest ways to remove double quotes from variables are
Linux=`echo "$Linux" | tr -d '"'`
Linux=$(eval echo $Linux)
Linux=`echo ${Linux//\"/}`
Linux=`echo $Linux | xargs`
All provides the Output without double quotes:
echo $Linux
amzn
I know this is a very old question, but here is another sed variation, which may be useful to someone. Unlike some of the others, it only replaces double quotes at the start or end...
echo "$opt" | sed -r 's/^"|"$//g'
If you need to match single or double quotes, and only strings that are properly quoted. You can use this slightly more complex regex...
echo $opt | sed -E "s|^(['\"])(.*)\1$|\2|g"
This uses backrefences to ensure the quote at the end is the same as at the start.
In Bash, you could use the following one-liner:
[[ "${var}" == \"*\" || "${var}" == \'*\' ]] && var="${var:1:-1}"
This will remove surrounding quotes (both single and double) from the string stored in var while keeping quote characters inside the string intact. Also, this won't do anything if there's only a single leading quote or only a single trailing quote or if there are mixed quote characters at start/end.
Wrapped in a function:
#!/usr/bin/env bash
# Strip surrounding quotes from string [$1: variable name]
function strip_quotes() {
local -n var="$1"
[[ "${var}" == \"*\" || "${var}" == \'*\' ]] && var="${var:1:-1}"
}
str="'hello world'"
echo "Before: ${str}"
strip_quotes str
echo "After: ${str}"
My version
strip_quotes() {
while [[ $# -gt 0 ]]; do
local value=${!1}
local len=${#value}
[[ ${value:0:1} == \" && ${value:$len-1:1} == \" ]] && declare -g $1="${value:1:$len-2}"
shift
done
}
The function accepts variable name(s) and strips quotes in place. It only strips a matching pair of leading and trailing quotes. It doesn't check if the trailing quote is escaped (preceded by \ which is not itself escaped).
In my experience, general-purpose string utility functions like this (I have a library of them) are most efficient when manipulating the strings directly, not using any pattern matching and especially not creating any sub-shells, or calling any external tools such as sed, awk or grep.
var1="\"test \\ \" end \""
var2=test
var3=\"test
var4=test\"
echo before:
for i in var{1,2,3,4}; do
echo $i="${!i}"
done
strip_quotes var{1,2,3,4}
echo
echo after:
for i in var{1,2,3,4}; do
echo $i="${!i}"
done
I use this regular expression, which avoids removing quotes from strings that are not properly quoted, here the different outputs are shown depending on the inputs, only one with begin-end quote was affected:
echo '"only first' | sed 's/^"\(.*\)"$/\1/'
Output: >"only first<
echo 'only last"' | sed 's/^"\(.*\)"$/\1/'
Output: >"only last"<
echo '"both"' | sed 's/^"\(.*\)"$/\1/'
Output: >both<
echo '"space after" ' | sed 's/^"\(.*\)"$/\1/'
Output: >"space after" <
echo ' "space before"' | sed 's/^"\(.*\)"$/\1/'
Output: > "space before"<
STR='"0.0.0"' ## OR STR="\"0.0.0\""
echo "${STR//\"/}"
## Output: 0.0.0
There is another way to do it. Like:
echo ${opt:1:-1}
If you try to remove quotes because the Makefile keeps them, try this:
$(subst $\",,$(YOUR_VARIABLE))
Based on another answer: https://stackoverflow.com/a/10430975/10452175
I used many times [``] to capture output of command to a variable. but with following code i am not getting right output.
#!/bin/bash
export XLINE='($ZWP_SCRIP_NAME),$ZWP_LT_RSI_TRIGGER)R),$ZWP_RTIMER'
echo 'Original XLINE'
echo $XLINE
echo '------------------'
echo 'Extract all word with $ZWP'
#works fine
echo $XLINE | sed -e 's/\$/\n/g' | sed -e 's/.*\(ZWP[_A-Z]*\).*/\1/g' | grep ZWP
echo '------------------'
echo 'Assign all word with $ZWP to XVAR'
#XVAR doesn't get all the values
export XVAR=`echo $XLINE | sed -e 's/\$/\n/g' | sed -e 's/.*\(ZWP[_A-Z]*\).*/\1/g' | grep ZWP` #fails
echo "$XVAR"
and i get:
Original XLINE
($ZWP_SCRIP_NAME),$ZWP_LT_RSI_TRIGGER)R),$ZWP_RTIMER
------------------
Extract all word with $ZWP
ZWP_SCRIP_NAME
ZWP_LT_RSI_TRIGGER
ZWP_RTIMER
------------------
Assign all word with $ZWP to XVAR
ZWP_RTIMER
why XVAR doesn't get all the values?
however if i use $() to capture the out instead of ``, it works fine. but why `` is not working?
Having GNU grep you can use this command:
XVAR=$(grep -oP '\$\KZWP[A-Z_]+' <<< "$XLINE")
If you pass -P grep is using Perl compatible regular expressions. The key here is the \K escape sequence. Basically the regex matches $ZWP followed by one or more uppercase characters or underscores. The \K after the $ removes the $ itself from the match, while its presence is still required to match the whole pattern. Call it poor man's lookbehind if you want, I like it! :)
Btw, grep -o outputs every match on a single line instead of just printing the lines which match the pattern.
If you don't have GNU grep or you care about portability you can use awk, like this:
XVAR=$(awk -F'$' '{sub(/[^A-Z_].*/, "", $2); print $2}' RS=',' <<< "$XLINE")
First, the smallest change that makes your code "work":
echo "$XLINE" | tr '$' '\n' | sed -e 's/.*\(ZWP[_A-Z]*\).*/\1/g' | grep ZWP_
The use of tr replaces a sed expression that didn't actually do what you thought it did -- try looking at its output to see.
One sane alternative would be to rely on GNU grep's -o option. If you can't do that...
zwpvars=( ) # create a shell array
zwp_assignment_re='[$](ZWP_[[:alnum:]_]+)(.*)' # ...and a regex
content="$XLINE"
while [[ $content =~ $zwp_assignment_re ]]; do
zwpvars+=( "${BASH_REMATCH[1]}" ) # found a reference
content=${BASH_REMATCH[2]} # stuff the remaining content aside
done
printf 'Found variable: %s\n' "${zwpvars[#]}"
How to remove extra spaces in variable HEAD?
HEAD=" how to remove extra spaces "
Result:
how to remove extra spaces
Try this:
echo "$HEAD" | tr -s " "
or maybe you want to save it in a variable:
NEWHEAD=$(echo "$HEAD" | tr -s " ")
Update
To remove leading and trailing whitespaces, do this:
NEWHEAD=$(echo "$HEAD" | tr -s " ")
NEWHEAD=${NEWHEAD%% }
NEWHEAD=${NEWHEAD## }
Using awk:
$ echo "$HEAD" | awk '$1=$1'
how to remove extra spaces
Take advantage of the word-splitting effects of not quoting your variable
$ HEAD=" how to remove extra spaces "
$ set -- $HEAD
$ HEAD=$*
$ echo ">>>$HEAD<<<"
>>>how to remove extra spaces<<<
If you don't want to use the positional paramaters, use an array
ary=($HEAD)
HEAD=${ary[#]}
echo "$HEAD"
One dangerous side-effect of not quoting is that filename expansion will be in play. So turn it off first, and re-enable it after:
$ set -f
$ set -- $HEAD
$ set +f
This horse isn't quite dead yet: Let's keep beating it!*
Read into array
Other people have mentioned read, but since using unquoted expansion may cause undesirable expansions all answers using it can be regarded as more or less the same. You could do
set -f
read HEAD <<< $HEAD
set +f
or you could do
read -rd '' -a HEAD <<< "$HEAD" # Assuming the default IFS
HEAD="${HEAD[*]}"
Extended Globbing with Parameter Expansion
$ shopt -s extglob
$ HEAD="${HEAD//+( )/ }" HEAD="${HEAD# }" HEAD="${HEAD% }"
$ printf '"%s"\n' "$HEAD"
"how to remove extra spaces"
*No horses were actually harmed – this was merely a metaphor for getting six+ diverse answers to a simple question.
Here's how I would do it with sed:
string=' how to remove extra spaces '
echo "$string" | sed -e 's/ */ /g' -e 's/^ *\(.*\) *$/\1/'
=> how to remove extra spaces # (no spaces at beginning or end)
The first sed expression replaces any groups of more than 1 space with a single space, and the second expression removes any trailing or leading spaces.
echo -e " abc \t def "|column -t|tr -s " "
column -t will:
remove the spaces at the beginning and at the end of the line
convert tabs to spaces
tr -s " " will squeeze multiple spaces to single space
BTW, to see the whole output you can use cat - -A: shows you all spacial characters including tabs and EOL:
echo -e " abc \t def "|cat - -A
output: abc ^I def $
echo -e " abc \t def "|column -t|tr -s " "|cat - -A
output:
abc def$
Whitespace can take the form of both spaces and tabs. Although they are non-printing characters and unseen to us, sed and other tools see them as different forms of whitespace and only operate on what you ask for. ie, if you tell sed to delete x number of spaces, it will do this, but the expression will not match tabs. The inverse is true- supply a tab to sed and it will not match spaces, even if the number of them is equal to those in a tab.
A more extensible solution that will work for removing either/both additional space in the form of spaces and tabs (I've tested mixing both in your specimen variable) is:
echo $HEAD | sed 's/^[[:blank:]]*//g'
or we can tighten-up #Frontear 's excellent suggestion of using xargs without the tr:
echo $HEAD | xargs
However, note that xargs would also remove newlines. So if you were to cat a file and pipe it to xargs, all the extra space- including newlines- are removed and everything put on the same line ;-).
Both of the foregoing achieved your desired result in my testing.
Try this one:
echo ' how to remove extra spaces ' | sed 's/^ *//g' | sed 's/$ *//g' | sed 's/ */ /g'
or
HEAD=" how to remove extra spaces "
HEAD=$(echo "$HEAD" | sed 's/^ *//g' | sed 's/$ *//g' | sed 's/ */ /g')
I would make use of tr to remove the extra spaces, and xargs to trim the back and front.
TEXT=" This is some text "
echo $(echo $TEXT | tr -s " " | xargs)
# [...]$ This is some text
echo variable without quotes does what you want:
HEAD=" how to remove extra spaces "
echo $HEAD
# or assign to new variable
NEW_HEAD=$(echo $HEAD)
echo $NEW_HEAD
output: how to remove extra spaces