Shell script, escape newlines but emit others? - bash

Question

Given a filename, I want to write a shell-script which emits the following, and pipes it into a process:
Content-Length:<LEN><CR><LF>
<CR><LF>
{ "jsonrpc":"2.0", "params":{ "text":"<ESCAPED-TEXT>" } }
where <ESCAPED-TEXT> is the content of the file but its CRs, LFs and quotation marks have been escaped as \r and \n and \" (and I guess all other JSON escapes will eventually be needed as well), and where <LEN> is the length of final JSON line that includes the escaped text.
Here's my current bash-script solution. It works but is ugly as heck.
(
TXT=`cat ~/a.py | sed -E -e :a -e '$!N; s/\n/\\\n/g; ta' | sed 's/"/\\\"/g'`
CMD='{"jsonrpc":"2.0", "params":{ "text":{"'${TXT}'"}} }'
printf "Content-Length: ${#CMD}\r\n\r\n"
echo -n "${CMD}"
) | pyls
Can anyone suggest how to do this cleaner, please?
This sed script only replaces LFs, not CRs. It accumulates each line into the buffer and then does a s//g to replace all LFs in it. I couldn't figure out anything cleaner that still worked on both Linux and OSX/BSD.
I used both printf and echo. First printf because I do want to emit the CRLFCRLF after the Content-Length header, and you apparently need printf for that because the behavior of echo with escapes isn't uniform across platforms. Next echo because I don't want the \r and \n literals inside TXT to be unescaped, which printf would do.
Context: there's a standard called "Language Server Protocol". Basically you run something like the pyls I'm running here, and you pipe in JsonRPC to it over stdin, and it pipes back stuff. Different people have written language servers for Python (the pyls I'm using here), and C#, and C++, and Typescript, and PHP, and OCaml, and Go, and Java, and each person tends to write their language server in their own language.
I want to write a test-harness which can send some example JsonRPC packets into any such server.
I figured it'd be better to write my test-harness in just the common basic shell-scripting stuff that's available on all platforms out of the box. That way everyone can use my test-harness against their language server. (If I wrote it on Python instead, say, it'd be easier for me to write, but it would force the C# folks to learn+install python just to run it, and likewise the Typescript, PHP, OCaml, Go and other folks.)

Answer 1

a.py:
print("alfa")
print("bravo")
Awk script:
{
gsub("\r", "\\r")
gsub("\42", "\\\42")
z = z $0 "\\n"
}
END {
printf "Content-Length: %d\r\n", length(z) + 42
printf "\r\n"
printf "{\42jsonrpc\42: \0422.0\42, \42params\42: {\42text\42: \42%s\42}}", z
}
Result:
Content-Length: 81
{"jsonrpc": "2.0", "params": {"text": "print(\"alfa\")\r\nprint(\"bravo\")\r\n"}}

Answer 2

Can anyone suggest how to do this cleaner, please?
I guess all other JSON escapes will eventually be needed as well
If I already had Python at my disposal, I'd try really, really hard to use the standard Python JSON encoder, at least for the string escaping part. Why hack together something that kind of works when you can use something known to work that you already are halfway familiar with?
If I didn't have Python, I like Steve Penny's solution. Rules of thumb:
to process sets of files, use the shell
to process data in a file, use awk
if sed can't do it trivially, see rule #2
If you know a little awk, his solution is easy to understand almost at a glance. I would call that "cleaner". If you don't know awk, this would seem to be an excellent opportunity to become acquainted.

Answer 3

I think the main problem with your script is not using format strings with printf. The usual way that printf is used is with various special characters in the format string (like %s, %b, etc) and a list of additional arguments that are substituted into the format string.
That is, when you say "[I used] echo because I don't want the \r and \n literals to be unescaped, which printf would do", the problem is just not using printf "%s" "$string".
Anyway, here's an idea of how to use this stuff to get everything done in bash with no external tools:
escapes=('\n' '\r' '\"') # the escapes we want to put into the output
txt=$(< ~/a.py); # read the file into a variable
for esc in "${escapes[#]}"; do
# escapes are evaluated in a %b string w/ printf
# using -v puts the result into a variable
printf -v lit '%b' "$esc"
# use built-in ${string//pattern/replacement} expansion
txt=${txt//$lit/$esc}
done
txt='{"jsonrpc":"2.0", "params":{ "text":{"'$txt'"}} }'
# escapes in the format string are expanded
# but escapes in the argument substituted for %s are not
printf 'Content-Length: %s\r\n\r\n%s' "${#txt}"
"$txt"