How to combine two lines from the bash prompt? - bash

Question

I am trying to craft a script to perform curl requests on webservers and parse out the "server" and "Location." This way I can easily import it into my excel tables without having to reformat.
My current script:
curl -sD - -o /dev/null -A "Mozilla/4.0" http://site/ | sed -e '/Server/p' -e '/Location/!d' | paste - -
Expected/Desired output:
Server: Apache Location: http://www.site
Current output:
From curl:
HTTP/1.1 301 Moved permanently
Date: Sun, 16 Nov 2014 20:14:01 GMT
Server: Apache
Set-Cookie: USERNAME=;path=/
Set-Cookie: CFID=16581239;path=/
Set-Cookie: CFTOKEN=32126621;path=/
Location: http://www.site
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
Piped into 'sed':
Server: Apache
Location: http://www.site
Piped into 'paste':
Server: Location: http://www.site
Why does paste immediately 'paste' after the first space? How do I get it to format correctly? I'm open to other methods, but keep in mind, the responses from the 'curl' request will be different lengths.
Thanks

Answer 1

Output of "curl" contains "return" i.e. \r character(s) which will cause that behaviour.
curl -sD - -o /dev/null -A "Mozilla/4.0" http://site/ | tr -d '\r'| sed -e '/Server/p' -e '/Location/!d' | paste - -
tr -d '\r' filters out all carriage return characters.
About line ends
While Linux/Unix uses "LF" (Line Feed, \n) line ends many other systems use "CR LF" (Carriage Return Line Feed \r\n) line ends. That can cause weard looking results unless you are prepared for it. Let's see some examples without \r and the same with \r.
Concatenation of strings:
a=$(echo -e "Please notice don't delete your files in /<config_dir> ")
b=$(echo -e "without hesitation ")
echo "$a""$b"
Result:
Please notice don't delete your files in /<config_dir> without hesitation
We get somewhat different result if lines end with CR LF:
a=$(echo -e "Please notice don't delete your files in /<config_dir> \r")
b=$(echo -e "without hesitation \r")
echo "$a""$b"
Result:
without hesitation delete your files in /<config_dir>
What might happen with programs which modify text only if matching string is at line end ?
Let's remove "ny" if it appears at line end:
echo "Stackoverflow is funny" | sed 's/ny$//g'
Result:
Stackoverflow is fun
The same wirh CR LF ending line:
echo -e "Stackoverflow is funny\r" | sed 's/ny$//g'
Result:
Stackoverflow is funny
sed works as designed because the line does not end to "ny" but "ny CR".
The teaching of all this is to be prepared for unexpected input data. In most cases it may be a good idea to filter out \r from data copletely since it's seldom needed for anything useful in BASH script. Filtering out unwanted character(s) is simple with "tr":
tr -d '\r'