Simple conditional unexpectedly fails in bash if statement - bash

I've spent an embarrassingly long time trying to understand why the second conditional in the "foo" script below fails but the first one succeeds.
Please note:
The current directory contains two files: bar and foo.
All three strings $s1, $s2 and $s3 are equal according to hexdump.
Thanks in advance for any help.
Session: (Running on a Centos7 host):
>ls
bar foo
>cat foo
#!/bin/bash
s1="bar foo"
s2="bar foo"
s3=`ls`
echo -n $s1 | hexdump -C
echo -n $s2 | hexdump -C
echo -n $s3 | hexdump -C
if [ "$s1" = "$s2" ]; then # True
echo s1 = s2
fi
if [ "$s1" = "$s3" ]; then # NOT true! Why?
echo s1 = s3
fi
>foo
00000000 62 61 72 20 66 6f 6f |bar foo|
00000007
00000000 62 61 72 20 66 6f 6f |bar foo|
00000007
00000000 62 61 72 20 66 6f 6f |bar foo|
00000007
s1 = s2
>

Quote the variables when echoing.
echo -n "$s3" | hexdump -C
You'll see a newline between the file names, as ls uses -1 when the output is redirected.

Your demo would be more convincing with echo -n "$s1" etc. That would show that there's a newline in the middle of s3 where there's a space in s1 and s2. The echo without the double quotes mangles the newline into a space (and generally each sequence of one or more white space characters in the string into a single space).
Given:
#!/bin/bash
s1="bar foo"
s2="bar foo"
s3=`ls`
echo -n "$s1" | hexdump -C
echo -n "$s2" | hexdump -C
echo -n "$s3" | hexdump -C
if [ "$s1" = "$s2" ]; then # True
echo s1 = s2
fi
if [ "$s1" = "$s3" ]; then # NOT true because s3 contains a newline!
echo s1 = s3
fi
I get:
$ sh foo
00000000 2d 6e 20 62 61 72 20 66 6f 6f 0a |-n bar foo.|
0000000b
00000000 2d 6e 20 62 61 72 20 66 6f 6f 0a |-n bar foo.|
0000000b
00000000 2d 6e 20 62 61 72 0a 66 6f 6f 0a |-n bar.foo.|
0000000b
s1 = s2
$ bash foo
00000000 62 61 72 20 66 6f 6f |bar foo|
00000007
00000000 62 61 72 20 66 6f 6f |bar foo|
00000007
00000000 62 61 72 0a 66 6f 6f |bar.foo|
00000007
s1 = s2
$

Related

bash substitution after glob not working?

I encounter a strange behaviour with bash string substitution.
I expected the same substitution on $r1 and $var to yield the exact same results.
both strings seem to have the same value.
But It is not the case and I can't understand what I am missing....
maybe is because of the glob? I just don't know... I am not pure IT guys and maybe it's something that will be evident for you.
(bottom a Repl.it link)
mkdir -p T21805
touch T21805/T21805_SI-GA-D8-BH25N7DSXY_S1_L001_R1_001.fastq.gz
r1=T21805/*R1*
echo $r1;
echo ${r1%%_S1*z}
var=T21805/T21805_SI-GA-D8-BH25N7DSXY_S1_L001_R1_001.fastq.gz
echo ${var%%_S1*z}
echo $r1| hexdump -C
echo $var | hexdump -C
output :
echo $r1
T21805/T21805_SI-GA-D8-BH25N7DSXY_S1_L001_R1_001.fastq.gz
echo ${r1%%_S1*z}
T21805/T21805_SI-GA-D8-BH25N7DSXY_S1_L001_R1_001.fastq.gz
echo ${var%%_S1*z}
T21805/T21805_SI-GA-D8-BH25N7DSXY
echo $r1| hexdump -C
00000000 54 32 31 38 30 35 2f 54 32 31 38 30 35 5f 53 49
|T21805/T21805_SI|
00000010 2d 47 41 2d 44 38 2d 42 48 32 35 4e 37 44 53 58
|-GA-D8-BH25N7DSX|
00000020 59 5f 53 31 5f 4c 30 30 31 5f 52 31 5f 30 30 31
|Y_S1_L001_R1_001|
00000030 2e 66 61 73 74 71 2e 67 7a 0a
|.fastq.gz.| 0000003a
echo $var | hexdump -C
00000000 54 32 31 38 30 35 2f 54 32 31 38 30 35 5f 53 49
|T21805/T21805_SI|
00000010 2d 47 41 2d 44 38 2d 42 48 32 35 4e 37 44 53 58
|-GA-D8-BH25N7DSX|
00000020 59 5f 53 31 5f 4c 30 30 31 5f 52 31 5f 30 30 31
|Y_S1_L001_R1_001|
00000030 2e 66 61 73 74 71 2e 67 7a 0a
|.fastq.gz.| 0000003a
Repl.it
I am interested on understanding why this is not working, I can achieve my desire output using sed for example.
Glob expansion doesn't happen at assignment time.
$ mkdir -p T21805
$ touch T21805/T21805_SI-GA-D8-BH25N7DSXY_S1_L001_R1_001.fastq.gz
$ touch T21805/T21805_SI-GA-D8-BH25N7DSXY_S1_L001_R1_002.fastq.gz
$ r1=T21805/*R1*
$ printf '%s\n' "$r1"
T21805/*R1*
$ printf '%s\n' $r1
T21805/T21805_SI-GA-D8-BH25N7DSXY_S1_L001_R1_001.fastq.gz
T21805/T21805_SI-GA-D8-BH25N7DSXY_S1_L001_R1_002.fastq.gz
It happens after the unquoted r1 has been expanded. When you write ${r1%%_S1*z}, the value of r1 doesn't contain the string S1; only after ${r1} expands is there an S1 you could match against.
If you set an array, the assignment rules are different. The glob expands before the assignment, and so you can do your filtering on each element of the array.
$ r1=( T21805/*R1* )
$ printf '%2\n' "${r1[#]}"
T21805/T21805_SI-GA-D8-BH25N7DSXY_S1_L001_R1_001.fastq.gz
T21805/T21805_SI-GA-D8-BH25N7DSXY_S1_L001_R1_002.fastq.gz
$ printf '%s\n' "${r1[#]%%_S1*z}"
T21805/T21805_SI-GA-D8-BH25N7DSXY
T21805/T21805_SI-GA-D8-BH25N7DSXY
I ran it after set -xv to see the contents of r1.
$ r1=T21805/*R1*
+ r1='T21805/*R1*'
$ var=T21805/T21805_SI-GA-D8-BH25N7DSXY_S1_L001_R1_001.fastq.gz
+ var=T21805/T21805_SI-GA-D8-BH25N7DSXY_S1_L001_R1_001.fastq.gz
The r1 of$ {r1 %% _ S1 * z}isT21805 / * R1 *.
r1 does not include_S1 * z.

Append to a string in bash

I'm trying to get a download URL using curl and awk and want to append something to that URL afterwards.
Here some snipped of my code:
IMAGE=$(curl -I -s https://downloads.raspberrypi.org/raspbian_lite_latest | awk '/Location/ {print $2}')
CHECKSUM="$IMAGE.sha256"
echo $IMAGE
echo $CHECKSUM
What I'm getting is that it is somehow replacing parts at the beginning.
https://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2018-11-15/2018-11-13-raspbian-stretch-lite.zip
.sha256/downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2018-11-15/2018-11-13-raspbian-stretch-lite.zip
I'm a bit helpless, because the following works as expected:
A="https""://abc.org/a_b/a.zip" # looks weird, but full URLs are not allowed here
B="$A.sha256"
echo $B
What am I doing wrong?
When you hexdump your string, you see that is uses windows line endings (with carriage return):
echo $IMAGE | hexdump -C
00000000 68 74 74 70 73 3a 2f 2f 64 6f 77 6e 6c 6f 61 64 |https://download|
00000010 73 2e 72 61 73 70 62 65 72 72 79 70 69 2e 6f 72 |s.raspberrypi.or|
00000020 67 2f 72 61 73 70 62 69 61 6e 5f 6c 69 74 65 2f |g/raspbian_lite/|
00000030 69 6d 61 67 65 73 2f 72 61 73 70 62 69 61 6e 5f |images/raspbian_|
00000040 6c 69 74 65 2d 32 30 31 38 2d 31 31 2d 31 35 2f |lite-2018-11-15/|
00000050 32 30 31 38 2d 31 31 2d 31 33 2d 72 61 73 70 62 |2018-11-13-raspb|
00000060 69 61 6e 2d 73 74 72 65 74 63 68 2d 6c 69 74 65 |ian-stretch-lite|
00000070 2e 7a 69 70 0d 0a |.zip..|
00000076
To fix that, use
IMAGE=$(curl -I -s https://downloads.raspberrypi.org/raspbian_lite_latest | awk '/Location/ {print $2}' | tr -d "\r")
The problem apparently is, that your $IMAGE contains / ends in a trailing '\r(carriage return). So you've actually appended ".sha256" as you expected to"something\r.sha256" which when being echoed means.... something, cursor back to the beginning of the line, .sha256. Long story short, strip that '\r`. E.g:
IMAGE=$(curl -I -s https://downloads.raspberrypi.org/raspbian_lite_latest | awk '/Location/ {sub(/\r$/, "", $2); print $2}')
Since you are using bash you can use substring replacement, ie. replace the \r in IMAGEvar:
$ CHECKSUM="${IMAGE/$'\r'/}.sha256"
$ echo $CHECKSUM
https://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2018-11-15/2018-11-13-raspbian-stretch-lite.zip.sha256
or prepare for it in the awk part by setting the record separator RS:
... | awk -v RS="\r?\n" '/Location/ {print $2}'
Tested with gawk, mawk and original-awk. Surprisingly busybox awk removed it by itself:
$ echo -e \\r | busybox awk '{print $1}' | hexdump -C
00000000 0a |.|
but for example:
$ echo -e \\r | gawk '{print $1}' | hexdump -C
00000000 0d 0a |..|

Two variables not equaling each-other for if statement

Hello I am having a problem with checking two variables to see whether or not they are both equal. I have the following script:
Output=$(sudo defaults read /System/Library/User\ Template/English.lproj/Library/Preferences/com.apple.SetupAssistant | grep -o "DidSeeCloudSetup = 1")
Output2=$(sudo defaults read /System/Library/User\ Template/English.lproj/Library/Preferences/com.apple.SetupAssistant | grep -o "LastSeenCloudProductVersion")
Check="DidSeeCloudSetup = 1"
Check2="LastSeenCloudProductVersion"
echo "$Output"
echo "$Check"
if [ "$Output" = "$Check" ]
then
echo "OK"
else
echo "FALSE"
Even though they both contain the same thing it always comes out false... any ideas why?
There is a special character (hex: 10) between $ and Check in your if clause:
00000000 69 66 20 5b 20 22 24 4f 75 74 70 75 74 22 20 3d |if [ "$Output" =|
00000010 20 22 24 10 43 68 65 63 6b 22 20 5d 0a | "$.Check" ].|

New to awk and sed, How could I improve this? Multiple sed and awk commands

This is the script I've constructed
It takes a list of files according to the extension supplied as an argument.
It then removes everything before the pattern 00000000: in those files.
The pattern 00000000: is preceded by the string <pre>, it then removes those five first characters.
The script then removes the last three lines of the file
The script the outputs only the hexdump data of the file.
The script runs xxd to convert the hexdump to a file.jpg
if [[ $# -eq 0 ]] ; then
echo 'Run script as ./hexconv ext'
exit 0
fi
for file in *.$1
do
filename=$(basename $file)
extension="${filename##*.}"
filename="${filename%.*}"
sed -n '/00000000:/,$p' $file | sed '1s/^.....//' | head -n -3 | awk '{print $2" "$3" "$4" "$5" "$6" "$7" "$8" "$9" "$10" "$11" "$12" "$13" "$14" "$15" "$16" "$17}' | xxd -p -r > $filename.jpg
done
It works as I want it too, but I suspect there are things to improve it by, but alas, I am a novice in the use of awk and sed.
Excerpt from file
<th>response-head:</th>
<td>HTTP/1.1 200 OK
Date: Sun, 15 Dec 2013 04:27:04 GMT
Server: PWS/8.0.18
X-Px: ms h0-s34.p6-lhr ( h0-s35.p6-lhr), ht-d h0-s35.p6-lhr.cdngp.net
Etag: "4556354-9fbf8-4e40387aadfc0"
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, max-age=0
Accept-Ranges: bytes
Content-Length: 654328
Content-Type: image/jpeg
Last-Modified: Thu, 15 Aug 2013 21:55:19 GMT
Pragma: no-cache
</td>
</tr>
</table>
<hr/>
<pre>00000000: ff d8 ff e0 00 10 4a 46 49 46 00 01 01 01 00 48 ......JFIF.....H
00000010: 00 48 00 00 ff e1 00 18 45 78 69 66 00 00 49 49 .H......Exif..II
00000020: 2a 00 08 00 00 00 00 00 00 00 00 00 00 00 ff ed *...............
00000030: 00 48 50 68 74 73 68 70 20 33 2e 30 00 .HPhotoshop 3.0.
00000040: 38 42 49 4d 04 04 00 00 00 00 00 1c 01 5a 00 8BIM..........Z.
00000050: 03 1b 25 47 1c 02 00 00 02 00 02 00 38 42 49 4d ..%G........8BIM
00000060: 04 25 00 00 00 00 00 10 fc e1 89 c8 b7 c9 78 .%.............x
00000070: 34 62 34 07 58 77 eb ff e1 03 a5 68 74 74 70 /4b4.Xw.....http
00000080: 3a 6e 73 2e 61 64 62 65 2e 63 6d ://ns.adobe.com/
00000090: 78 61 70 31 2e 30 00 3c 78 70 61 63 6b xap/1.0/.<?xpack
000000a0: 65 74 20 62 65 67 69 6e 3d 22 ef bb bf 22 20 69 et begin="..." i
000000b0: 64 3d 22 57 35 4d 30 4d 70 43 65 68 69 48 7a 72 d="W5M0MpCehiHzr
000000c0: 65 53 7a 4e 54 63 7a 6b 63 39 64 22 3e 20 3c eSzNTczkc9d"?> <
000000d0: 78 3a 78 6d 70 6d 65 74 61 20 78 6d 6c 6e 73 3a x:xmpmeta xmlns:
000000e0: 78 3d 22 61 64 62 65 3a 6e 73 3a 6d 65 74 61 x="adobe:ns:meta
000000f0: 22 20 78 3a 78 6d 70 74 6b 3d 22 41 64 62 /" x:xmptk="Adob
00000100: 65 20 58 4d 50 20 43 72 65 20 35 2e 30 2d 63 e XMP Core 5.0-c
00000110: 30 36 31 20 36 34 2e 31 34 30 39 34 39 2c 20 32 061 64.140949, 2
00000120: 30 31 30 31 32 30 37 2d 31 30 3a 35 37 3a 010/12/07-10:57:
Although #CodeGnome is right and this might belong to Code Review SE, here you go anyway:
Slightly more efficient to combine the multiple sed commands into one, for example:
sed -n -e 's/^<pre>//' -e '/00000000:/,$p'
I decided to retract this part, as I'm not all that sure it's any better or clearer. Your version is fine, except that s/^<pre>// is better than s/^.....//.
Use exit 1 when checking the number of arguments to signal an error
What is for file in *. there? Iterate for all files ending with a dot? Typo?
Unless you're 100% sure the filenames will never contain spaces, you should quote them, but don't quote where you don't need, for example:
filename=$(basename "$file") # need to quote
extension=${filename##*.} # no need,
filename=${filename%.*} # no need
sed ... "$file" # need to quote
... | xxd > "$filename".jpg # need to quote
The last awk could be shorter and less error prone as a loop:
... | awk '{printf $2; for (i=3; i<=17; ++i) printf " " $i; print ""}'
It seems you want to learn. You might be interested in this other answer too: What are the rules to write robust shell scripts?
The error message should be sent to stderr, should not hard-code the name of the script in case you rename it later, and should exit with a nonzero value.
if (( ! $# )); then
echo >&2 "Run script as '$0' \$extension"
exit 1
fi
If you're going to put the then on the same line as the if, then you should put the do on the same line as the for, too, for consistency:
for file in *.$1; do
Using file for the full name and filename for the basename is confusing variable name choice. I would use basename for the variable, to match the operation. And you need to quote the parameter expansion:
basename=$(basename "$file")
But you don't need to quote the right hand side of an assignment:
extension=${basename##*.}
The part of a filename without the extension is sometimes called the root (in vi and csh :-modifiers, you get it with :r)... using that name would be less confusing than changing an existing variable and reusing it:
root=${basename%.*}
As far as the actual pipeline, I would reorder it to put the head before the awk, since the sed and the head are all about what lines to print out and should be grouped together before the awk which modifies those selected lines. I would also use a loop and printf to make the awk a little more wieldy:
sed -n '/0\{8\}:/,$p' "$file" |
head -n -3 |
awk '{ printf "%s", $2; for (f=3;f<=17;++f) { printf " %s", $f }; print "" }' |
xxd -p -r > "$root.jpg"
done

Append to end of each line with sed on Windows?

I am using sed on Windows (the GNU port).
I execute:
$> sed "s/$/./" < /data.txt
And get:
.ne
.wo
.hree
But expect.
one.
two.
three.
The following works though I don't think it should. The way I read it is "replace the last character of the line with a period." I'm afraid it won't work consistently when used elsewhere. The intent isn't to replace the last character with a period but to append a period.
$> sed "s/.$/./" < /data.txt
I am not sure if the file encoding or something specific to windows is causing the issues I'm having or if it's just lack of experience with sed. Ideas?
hexdump -C sheds some light:
$ sed 's/$/./' < t.dos | hexdump -C
00000000 6f 6e 65 0d 2e 0a 74 77 6f 0d 2e 0a 74 68 72 65 |one...two...thre|
00000010 65 0d 2e 0a |e...|
00000014
There, 2e is the dot, the 0d before it is carriage return aka \r, and after that is the newline aka \n. In other words, sed treats \r as the end of the line instead of \r\n together, and thus \r is still part of the line, so it puts the dot after it, then adds back the newline as usual.
I think this does what you want, but it's not exactly pretty:
$ sed 's/.$/.\r/' < t.dos | hexdump -C
00000000 6f 6e 65 2e 0d 0a 74 77 6f 2e 0d 0a 74 68 72 65 |one...two...thre|
00000010 65 2e 0d 0a |e...|
00000014
The above is not so good, because it will only work if the input is in dos format, otherwise it will break the file. A better solution might be to first strip any \r and add them back manually later, like this:
$ tr -d '\r' < t.dos | sed -e 's/$/.\r/' | hexdump -C
00000000 6f 6e 65 2e 0d 0a 74 77 6f 2e 0d 0a 74 68 72 65 |one...two...thre|
00000010 65 2e 0d 0a |e...|
00000014

Resources