I can print a range of lines from a file using this cmd:
sed -n 267975,1000000p < dump-gq1-sample > dump267975
but how to print to the end? I tried
sed -n 267975,$p < dump-gq1-sample > dump267975
and it gives me:
sed: 1: "267975,": expected context address
You're the victim of Shell Parameter Expansion
sed -n 267975,$p < dump-gq1-sample > dump267975
is received by sed as
sed -n 267975, < dump-gq1-sample > dump267975
because the p variable is undefined.
You should quote your parameter with single quotes '
sed -n '267975,$p' < dump-gq1-sample > dump267975
See https://www.gnu.org/software/bash/manual/html_node/Shell-Expansions.html For the full list of existing shell expansions.
the single-quote response does not work in cases where the start/end of range are passed to sed as a variable.
including a space between the end-symbol $ and p in single quotes works on my systems to prevent unintended expansion of $p
sed -n '267975,$ p' ${INPUT_FILE} > ${OUTPUT_FILE} #double quotes work here too
If you need to pass the initial line as a variable, double quotes are required for the expansion of $L1 so the space is generally a good idea:
L1=267975
sed -n "${L1},$ p" ${INPUT_FILE} > dump${L1}
Related
I have a Bash script working fine locally, now I am trying to put it in Jenkinsfile to run as its pipeline:
stage('Update Cloudfront'){
steps {
sh '''
#!/bin/bash
YAML_FILE="path/to/values.yaml"
DATE="$(date '+%d-%m-%Y')"
wget https://www.cloudflare.com/ips-v4 && wget https://www.cloudflare.com/ips-v6
CLOUDFLARE_NEW=$(awk '{printf fmt,$1}' fmt="%s\n" ips-v4 ips-v6 | paste -sd, -)
CLOUDFLARE_OLD=$(yq -r .controller.config.proxy-real-ip-cidr $YAML_FILE | sed -E 's/\,37\.16\.11\.30\/32//')
if [[ "$CLOUDFLARE_NEW" == "$CLOUDFLARE_OLD" ]]; then
echo "No need to do anything"
else
echo "Cloudflare IP ranges change detected, updating Nginx value file"
CLOUDFLARE_NEW=$(awk '{printf fmt,$1}' fmt="%s\n" ips-v4 ips-v6 | paste -sd, -) yq e '.controller.config.proxy-real-ip-cidr = env(CLOUDFLARE_NEW)' -i $YAML_FILE
echo "Add third party IP range"
yq e '.controller.config.proxy-real-ip-cidr +=",1.2.3.4/32"' -i $YAML_FILE
fi
'''
}
}//end stage('Update Cloudfront')
Unfortunately it won't work:
WorkflowScript: 73: unexpected char: '\' # line 73, column 113.
cidr $YAML_FILE | sed -E \\"s/\,37\.16\.
^
I've tried to escape it with \\"s/\,37\.16\.11\.30\/32//\\" etc. but it doesn't work either. I've tried with double and single quotes with no luck.
You can avoid all the escaping by using a character class and different regex delimiters, like so:
sed -e 's#,37[.]16[.]11[.]30/32##'
In the event you do need to escape something though, simply doubling the backslash should do it:
sed -e 's/,37\\.16\\.11\\.30\\/32//'
Though, given the number of levels involved here, it might need double escaping:
sed -e 's/,37\\\\.16\\\\.11\\\\.30\\\\/32//'
I am using the sed command on Ubuntu to replace content.
This initial command comes from here.
sed -i '$ s/$/ /replacement/' "$DIR./result/doc.md"
However, as you can see, I have a slash in the replacement. The slash causes the command to throw:
sed: -e expression #1, char 9: unknown option to `s'
Moreover, my replacement is stored in a variable.
So the following will not work because of the slash:
sed -i "$ s/$/ $1/" "$DIR./result/doc.md"
As stated here and in duplicate, I should use another delimiter. If I try with #:
sed -i "$ s#$# $1#" "$DIR./result/doc.md"
It gives the error:
sed: -e expression #1, char 42: unterminated `s' command
My question is:
How can I use a variable in this command as well as other delimiter than / ?
Don't use sed here; perl and awk allow more robust approaches.
sed doesn't allow variables to be passed out-of-band from code, so they always need to be escaped. Use a language without that limitation, and you have code that always works, no matter what characters your data contains.
The Short Answer: Using perl
The below is taken from BashFAQ #21:
inplace_replace() {
local search=$1; shift; local replace=$1; shift
in="$search" out="$replace" perl -pi -e 's/\Q$ENV{"in"}/$ENV{"out"}/g' "$#"
}
inplace_replace '#' "replacement" "$DIR/result/doc.md"
The Longer Answer: Using awk
...or, using awk to do a streaming replacement, and a shell function to make that file replacement instead:
# usage as in: echo "in should instead be out" | gsub_literal "in" "out"
gsub_literal() {
local search=$1 replace=$2
awk -v s="${search//\\/\\\\}" -v r="${rep//\\/\\\\}" 'BEGIN {l=length(s)} {o="";while (i=index($0, s)) {o=o substr($0,1,i-1) r; $0=substr($0,i+l)} print o $0}'
}
# usage as in: inplace_replace "in" "out" /path/to/file1 /path/to/file2 ...
inplace_replace() {
local search=$1 replace=$2 retval=0; shift; shift
for file; do
tempfile=$(mktemp "$file.XXXXXX") || { retval |= $?; continue; }
if gsub_literal "$search" "$replace" <"$file" >"$tempfile"; then
mv -- "$tempfile" "$file" || (( retval |= $? ))
else
rm -f -- "$tempfile" || (( retval |= $? ))
fi
done
}
TL;DR:
Try:
sed -i '$ s#$# '"$1"'#' "$DIR./result/doc.md"
Long version:
Let's start with your original code:
sed -i '$ s/$/ /replacement/' "$DIR./result/doc.md"
And let's compare it to the code you referenced:
sed -i '$ s/$/abc/' file.txt
We can see that they don't exactly match up. I see that you've correctly made this substitution:
file.txt --> "$DIR./result/doc.md"
That looks fine (although I do have my doubts about the . after $DIR ). However, the other substitution doesn't look great:
abc --> /replacement
You actually introduced another delimeter /. However, if we replace the delimiters with '#' we get this:
sed -i '$ s#$# /replacement#' "$DIR./result/doc.md"
I think that the above is perfectly valid in sed/bash. The $# will not be replaced by the shell because it is single quoted. The $DIR variable will be interpolated by the shell because it is double quoted.
Looking at one of your attempts:
sed -i "$ s#$# $1#" "$DIR./result/doc.md"
You will have problems due to the shell interpolation of $# in the double quotes. Let's correct that by replacing with single quotes (but leaving $1 unquoted):
sed -i '$ s#$# '"$1"'#' "$DIR./result/doc.md"
Notice the '"$1"'. I had to surround $1 with '' to basically unescape the surrounding single quotes. But then I surrounded the $1 with double quotes so we could protect the string from white spaces.
Use shell parameter expansion to add escapes to the slashes in the variable:
$ cat file
foo
bar
baz
$ set -- ' /repl'
$ sed "s/$/$1/" file
sed: 1: "s/$/ /repl/": bad flag in substitute command: 'r'
$ sed "s/$/${1//\//\\\/}/" file
foo /repl
bar /repl
baz /repl
That is a monstrosity of leaning toothpicks, but it serves to transform this:
sed "s/$/ /repl/"
into
sed "s/$/ \/repl/"
The same technique can be used for whatever you choose as the sed s/// delimiter.
I am using this command to copy certain line from one file to another.Its working fine.No issue with it.
sed -f <(sed -e '1,10d; 12,$d; x; s/.*/10a\\/;p; x' ../log/file2.txt ) ../log/file4.txt > ../log/file5.txt
The problem is instead of 10, I want to use variable VAR1 (where var1=10).
The $VAR1 is not working.
I tried this command.
sed -f <(sed -e '1,$VAR1d; 12,$d; x; s/.*/10a\\/;p; x' ../log/file2.txt ) ../log/file4.txt > ../log/file5.txt
Please help me.
At first, use double quotes. It will enable BASH to process string.
Next, escape backslashes (even those that supposed to be escape symbols for sed) - because bash will escape them too.
I suppose it should be
sed -f <(sed -e "1,${VAR1}d; 12,\$d; x; s/.*/10a\\\\/;p; x" ../log/file2.txt ) ../log/file4.txt > ../log/file5.txt
Try double qoutes and curly braces:
sed -f <(sed -e "1,${VAR1}d; 12,\$d; x; s/.*/10a\\\\/;p; x" ../log/file2.txt ) ../log/file4.txt > ../log/file5.txt
I prefer mix of single & double quotes:
Single quotes around the text that must not be expanded. (Removes need to escape special characters.)
Double quotes around the text that needs to be expanded.
e.g. Your case would look like:
sed -f <(sed -e '1,'"$VAR1"'d; 12,$d; x; s/.*/10a\\/;p; x' ../log/file2.txt ) ../log/file4.txt > ../log/file5.txt
I have problem with this sed oneliner
sed -i -n "1h; 1!H; ${g; :a s/\(Name=\"$key\".*<\!\[CDATA\[\"\)$val\(\"\]\]>\)/\1$deval\2/;ta p}"
Obviously I need to expand variable key, val and deval in sed. So I need the " around sed command.
With this command I get
bash: !H: event not found
escaping the ! corrects it
sed -i -n "1h; 1\!H; ${g; :a s/\(Name=\"$key\".*<\!\[CDATA\[\"\)$val(\"\]\]>\)/\1$deval\2/;ta p}"
With this I get
bash: sed -i -n "1h; 1\!H; ${g; :a s/\(Name=\"$key\".*<\!\[CDATA\[\"\)$val\(\"\]\]>\)/\1$deval\2/;ta p}" :bad substitution
So I guess the { is a problem. Trying to fix it like this
sed -i -n "1h; 1\!H; $\{g; :a s/\(Name=\"$key\".*<\!\[CDATA\[\"\)$val(\"\]\]>\)/\1$deval\2/;ta p}"
yields
sed: -e expression 1, char 6: unknown command: "\"
What is going on here? How can I make this work?
event not found is only a problem in interactive shells because histexpand is enabled by default. If you either run set +H first or put it in a script and run it from there, Bash will leave your !s alone.
${..} is variable substitution syntax (so a mangled value gives bad substitution). Let sed treat it as a block of commands to do on the final line by escaping the $, as in \${ .. }.
In full:
set +H
key="foo"
val="bar"
deval="puppies"
echo 'Name="foo" <![CDATA["bar"]]>' > file
sed -i -n "1h; 1!H; \${g; :a s/\(Name=\"$key\".*<!\[CDATA\[\"\)$val\(\"\]\]>\)/\1$deval\2/;ta p}" file
cat file
Will print Name="foo" <![CDATA["puppies"]]>
You can use separate single-quoted strings:
sed -i -n '1h; 1!H; ${g; :a s/\(Name='"$key"'.*<\!\[CDATA\[\"\)'"$val"'\(\"\]\]>\)/\1'"$deval"'\2/;ta p}'
When I execute commands in Bash (or to be specific, wc -l < log.txt), the output contains a linebreak after it. How do I get rid of it?
If your expected output is a single line, you can simply remove all newline characters from the output. It would not be uncommon to pipe to the tr utility, or to Perl if preferred:
wc -l < log.txt | tr -d '\n'
wc -l < log.txt | perl -pe 'chomp'
You can also use command substitution to remove the trailing newline:
echo -n "$(wc -l < log.txt)"
printf "%s" "$(wc -l < log.txt)"
If your expected output may contain multiple lines, you have another decision to make:
If you want to remove MULTIPLE newline characters from the end of the file, again use cmd substitution:
printf "%s" "$(< log.txt)"
If you want to strictly remove THE LAST newline character from a file, use Perl:
perl -pe 'chomp if eof' log.txt
Note that if you are certain you have a trailing newline character you want to remove, you can use head from GNU coreutils to select everything except the last byte. This should be quite quick:
head -c -1 log.txt
Also, for completeness, you can quickly check where your newline (or other special) characters are in your file using cat and the 'show-all' flag -A. The dollar sign character will indicate the end of each line:
cat -A log.txt
One way:
wc -l < log.txt | xargs echo -n
If you want to remove only the last newline, pipe through:
sed -z '$ s/\n$//'
sed won't add a \0 to then end of the stream if the delimiter is set to NUL via -z, whereas to create a POSIX text file (defined to end in a \n), it will always output a final \n without -z.
Eg:
$ { echo foo; echo bar; } | sed -z '$ s/\n$//'; echo tender
foo
bartender
And to prove no NUL added:
$ { echo foo; echo bar; } | sed -z '$ s/\n$//' | xxd
00000000: 666f 6f0a 6261 72 foo.bar
To remove multiple trailing newlines, pipe through:
sed -Ez '$ s/\n+$//'
There is also direct support for white space removal in Bash variable substitution:
testvar=$(wc -l < log.txt)
trailing_space_removed=${testvar%%[[:space:]]}
leading_space_removed=${testvar##[[:space:]]}
If you want to print output of anything in Bash without end of line, you echo it with the -n switch.
If you have it in a variable already, then echo it with the trailing newline cropped:
$ testvar=$(wc -l < log.txt)
$ echo -n $testvar
Or you can do it in one line, instead:
$ echo -n $(wc -l < log.txt)
If you assign its output to a variable, bash automatically strips whitespace:
linecount=`wc -l < log.txt`
printf already crops the trailing newline for you:
$ printf '%s' $(wc -l < log.txt)
Detail:
printf will print your content in place of the %s string place holder.
If you do not tell it to print a newline (%s\n), it won't.
Adding this for my reference more than anything else ^_^
You can also strip a new line from the output using the bash expansion magic
VAR=$'helloworld\n'
CLEANED="${VAR%$'\n'}"
echo "${CLEANED}"
Using Awk:
awk -v ORS="" '1' log.txt
Explanation:
-v assignment for ORS
ORS - output record separator set to blank. This will replace new line (Input record separator) with ""