The bash manual says regarding command substitution:
Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command, with any trailing newlines deleted.
Demonstration - 3 characters, newlines first:
$ output="$(printf "\n\nx")"; echo -n "$output" | wc -c
3
Here the newlines are not at the end, and do not get removed, so the count is 3.
Demonstration - 3 characters, newlines last:
$ output="$(printf "x\n\n")"; echo -n "$output" | wc -c
1
Here the newlines are removed from the end, so the count is 1.
TL;DR
What is a robust work-around to get the binary-clean output of a command into a variable?
Bonus points for Bourne shell compatibility.
The only way to do it in a "Bourne compatible" way is to use external utilities.
Beside writting one in c, you can use xxd and expr (for example):
$ output="$(printf "x\n\n"; printf "X")" # get the output ending in "X".
$ printf '%s' "${output}" | xxd -p # transform the string to hex.
780a0a58
$ hexstr="$(printf '%s' "${output}" | xxd -p)" # capture the hex
$ expr "$hexstr" : '\(.*\)..' # remove the last two hex ("X").
780a0a
$ hexstr="$(expr "$hexstr" : '\(.*\)..') # capture the shorter str.
$ printf "$hexstr" | xxd -p -r | wc -c # convert back to binary.
3
Shortened:
$ output="$(printf "x\n\n"; printf "X")"
$ hexstr="$(printf '%s' "${output}" | xxd -p )"
$ expr "$hexstr" : '\(.*\)..' | xxd -p -r | wc -c
3
The command xxd is being used for its ability to convert back to binary.
Note that wc will fail with many UNICODE characters (multibyte chars):
$ printf "Voilà" | wc -c
6
$ printf "★" | wc -c
3
It will print the count of bytes, not characters.
The length of a variable ${#var} will also fail in older shells.
Of course, to get this to run in a Bourne shell you must use `…` instead of $(…).
In bash, the ${parameter%word} form of Shell Parameter Expansion can be used:
$ output="$(printf "x\n\n"; echo X)"; echo -n "${output%X}" | wc -c
3
This is substitution is also specified by POSIX.1-2008.
Related
I want to use grep to determine if a string contains a substring. (The plan is to use that result as the test command in a bash if statement.)
Here, I check the length of the output of grep . :
$ echo "abc" | grep "j" | wc -c
0
Since wc -c shows zero, I know that grep returned an empty string.
But if I save output from grep to a variable before calling wc -c, I get a different value:
$ match=$(echo "abc" | grep "j")
$ echo "$match" | wc -c
1
The output from grep now is a string with 1 character in it.
I suspect that it's a newline in there:
$ echo $match
$
Why is there now an extra character in $match, and how can I keep that from happening?
match does not contain a newline, but echo "$match" writes a newline (but see note below). In the first case, you are directly passing the output of grep to wc, but in the second case you are passing the output of grep plus a newline to wc.
But don't do this at all. There is no need to introduce wc into the problem. Just test the value returned by grep. eg:
if ... | grep -q "$pattern"; then echo "$pattern was found in the input"; fi
Note, echo "$match" is bad practice. For example, consider if $match expands to the string -e or -n. It is much more robust to use printf '%s' "$match"
Consider the following:
$ echo "$match" | od -c
0000000 \n
0000001
$ echo "$match" | wc -c
1
$ printf "$match" | od -c
0000000
$ printf "$match" | wc -c
0
The only difference in the 2 commands is that echo appends a \n on the end of the ouput.
Of course, we can make the printf generate the same result by explicitly adding a \n on the end:
$ printf "$match\n" | od -c
0000000 \n
0000001
$ printf "$match\n" | wc -c
1
I'm trying to generate a 2048 bit long prime number, this is my code so far:
#!/bin/bash
generate_random() {
hex=$(head -c 256 /dev/urandom | xxd -p)
bc <<< "ibase=16; $hex"
}
p=$(generate_random)
echo "$p"
While running the script I get (standard_in) 1: syntax error followed by random zeroes.
Anyone knows what is causing this error and how can I fix it? I've tried with bash -x, but it doesn't add any useful information.
First, bc understands only upper-case letters as hex digits (at least by default). Second, you have separators in your xxd output, so you generate multiple numbers with bc later.
This should work:
#!/bin/bash
generate_random() {
hex=$(head -c 256 /dev/urandom | xxd -p -u | tr -d '\n')
bc <<< "ibase=16; $hex"
}
p=$(generate_random)
echo "$p"
-u flag to xxd instructs it to output upper-case letters as digits, and tr removes separators.
Example output:
84404284040092528807148386035025161100484110236893077703095592941720\
00537078513504880246726730474236368181068985417211434943913923235822\
01284401417146606673073772989889733010524123703686975444423088406509\
44767677616371794606797386146855833950295071249000795855185540560405\
62673903614333076371092344026999031152809898928396395497832309795471\
93897215963003601022703133486344387720277877558264139632520964120681\
97764906669023878701319760947789227343517474218584987497204300184084\
62846775760153647010072072799120566180042021620262646969602253704108\
06274157727080642084167983313757899766696995668747042179553171962777\
5716
To remove newline separators and backslashes, you can do
p_joined=$(echo "$p" | sed -z 's=\\\n==g')
echo "$p_joined"
instead.
An alternative way might be
printf -v hex '%s' $(od -v -An -N256 -x /dev/urandom)
read dec < <(bc <<< "ibase=16; ${hex^^}")
echo $dec
I have UUID, 3abbea88-c77d-11eb-b8bc-0242ac130003 and I want to take first 16 character of this string and want Hexadecimal string of first 16 characters using shell script.
I tried,
code=$(echo -n ${${ID##*:}:0:16} | od -A n -t x1)
HEX_ID=$(echo ${code//[[:blank:]]/})
Any better way ?
Expected Output : 33616262656138382d633737642d3131
Using od you can simply limit the number of read characters using the -N option:
HEX_ID=$(od -A n -t x1 -N 16 <<< ${ID##*:} | tr -dc '[:xdigit:]')
Edit: tr is used to suppress non-hexadecimal characters, namely whitespaces and potential newlines.
Perl to the rescue!
perl -le 'print unpack "H32", shift' 3abbea88-c77d-11eb-b8bc-0242ac130003
-l adds newlines to print
unpack takes a string and expands it to a list of values based on a template. H32 means "take characters and interpret them as 32 hex values".
shift reads the first command line argument.
Or, using xxd and head:
echo 3abbea88-c77d-11eb-b8bc-0242ac130003 | xxd -p | head -c32
That's certainly a useless echo.
Probably avoid uppercase for your private variables.
uuid='3abbea88-c77d-11eb-b8bc-0242ac130003'
tmp=${uuid//-/}
hex_id=$(od -A n -t x1 <<<${tmp:0:13})
hex_id=${hex_id//[[:blank:]]/}
hex_id=${hex_id%0a}
The here string unattractively supplies trailing newline to od which we have to trim off.
Bash-only:
while read -r -N 1 c # read input string 1 char at a time
do [[ "$c" == " " ]] || # skip embedded spaces
printf "%02X" "$( # output the hexidecimal value of
printf "%d" \'$c # the ASCII decimal ordinal of $c
)"
done <<< "${text##*:}" # ignoring the leading trash to the :
echo # newline-teminate the output
All in one line:
while read -rn1 c;do [[ "$c" == " " ]]||printf %02X $(printf "%d" \'$c);done<<<"${text##*:}";echo
This is not the fastest approach...
hexdump does it all:
hexdump -n 16 -ve '1/1 "%.2x"'
-n 16 means only process the first 16 bytes
-e '1/1 "%.2x"' means display each byte using given printf format
-v means display normally (without this, it replaces dupe sections with * 🤷)
echo '3abbea88-c77d-11eb-b8bc-0242ac130003' | hexdump -n 16 -ve '1/1 "%.2x"'
output:
33616262656138382d633737642d3131
Problem
(1) Given a string, I replace spaces with $'\n' using sed:
echo "one two" | sed 's/ /$'"'"'\\n'"'"'/g'
This outputs:
# one$'\n'two
(2) Note that echoing this output of (1):
echo one$'\n'two
results in:
# one
# two
(3) I echo the output of (1) in another way, by piping the output of (1) into xargs echo:
echo "one two" | sed 's/ /$'"'"'\\n'"'"'/g' | xargs echo
But I don't get the same output as (2):
# one$\ntwo
Question
What does xargs do when formatting the input of a string containing $'\n'?
Why is echoing a string with $'\n' not the same as using xargs echo on the same string?
When you write
echo one$'\n'two
at the command line, bash replaces the "$'\n'" with a newline. But when you pass it to xargs no such replacement can happen.
But piping it to xargs will still not do what you want, since by default xargs uses the newline as an argument separator:
$ echo "one two" | tr ' ' '\n' | xargs echo
one two
You must tell xargs to use a different separator, even if it is a bogus one:
$ echo "one two" | tr ' ' '\n' | xargs -0 echo
one
two
Unsure if answering your question, but a trick I've used in the past for similar cases is to use printf instead, which loops over passed arguments in a loop (if not enough % to consume them), e.g.:
$ printf "%s\n" one two
one
two
Use shell own white-space separator if above are in a single string
$ args="one two"
$ printf "%s\n" $args
one
two
Just for completeness, feed to xargs -n1 with some foo scriptlet
$ printf "%s\n" one two |xargs -n1 sh -c 'echo [$(date -R)] foo=$1' --
[Sun, 03 Jun 2018 21:34:17 -0300] foo=one
[Sun, 03 Jun 2018 21:34:17 -0300] foo=two
When I execute commands in Bash (or to be specific, wc -l < log.txt), the output contains a linebreak after it. How do I get rid of it?
If your expected output is a single line, you can simply remove all newline characters from the output. It would not be uncommon to pipe to the tr utility, or to Perl if preferred:
wc -l < log.txt | tr -d '\n'
wc -l < log.txt | perl -pe 'chomp'
You can also use command substitution to remove the trailing newline:
echo -n "$(wc -l < log.txt)"
printf "%s" "$(wc -l < log.txt)"
If your expected output may contain multiple lines, you have another decision to make:
If you want to remove MULTIPLE newline characters from the end of the file, again use cmd substitution:
printf "%s" "$(< log.txt)"
If you want to strictly remove THE LAST newline character from a file, use Perl:
perl -pe 'chomp if eof' log.txt
Note that if you are certain you have a trailing newline character you want to remove, you can use head from GNU coreutils to select everything except the last byte. This should be quite quick:
head -c -1 log.txt
Also, for completeness, you can quickly check where your newline (or other special) characters are in your file using cat and the 'show-all' flag -A. The dollar sign character will indicate the end of each line:
cat -A log.txt
One way:
wc -l < log.txt | xargs echo -n
If you want to remove only the last newline, pipe through:
sed -z '$ s/\n$//'
sed won't add a \0 to then end of the stream if the delimiter is set to NUL via -z, whereas to create a POSIX text file (defined to end in a \n), it will always output a final \n without -z.
Eg:
$ { echo foo; echo bar; } | sed -z '$ s/\n$//'; echo tender
foo
bartender
And to prove no NUL added:
$ { echo foo; echo bar; } | sed -z '$ s/\n$//' | xxd
00000000: 666f 6f0a 6261 72 foo.bar
To remove multiple trailing newlines, pipe through:
sed -Ez '$ s/\n+$//'
There is also direct support for white space removal in Bash variable substitution:
testvar=$(wc -l < log.txt)
trailing_space_removed=${testvar%%[[:space:]]}
leading_space_removed=${testvar##[[:space:]]}
If you want to print output of anything in Bash without end of line, you echo it with the -n switch.
If you have it in a variable already, then echo it with the trailing newline cropped:
$ testvar=$(wc -l < log.txt)
$ echo -n $testvar
Or you can do it in one line, instead:
$ echo -n $(wc -l < log.txt)
If you assign its output to a variable, bash automatically strips whitespace:
linecount=`wc -l < log.txt`
printf already crops the trailing newline for you:
$ printf '%s' $(wc -l < log.txt)
Detail:
printf will print your content in place of the %s string place holder.
If you do not tell it to print a newline (%s\n), it won't.
Adding this for my reference more than anything else ^_^
You can also strip a new line from the output using the bash expansion magic
VAR=$'helloworld\n'
CLEANED="${VAR%$'\n'}"
echo "${CLEANED}"
Using Awk:
awk -v ORS="" '1' log.txt
Explanation:
-v assignment for ORS
ORS - output record separator set to blank. This will replace new line (Input record separator) with ""