Why do these small D programs behave differently? - algorithm

Question

I wrote a D implementation of the nul2pfb utility from http://www.dwheeler.com/essays/filenames-in-shell.html, as the link to the source code was broken and I wanted to try to learn D. I noticed that it was rather slow (could barely keep up with the find -print0 that was passing it data, when it should be far faster as it need not do anywhere near as many system calls).
The first implementation works correctly (tested with zsh and bash printf built-ins, as well as /usr/bin/printf). The second, though much faster (probably due to far fewer calls to write()), repeates the first part of its output many times, and fails to output the remainder of its output. What is causing this difference? I am a newbie to D and do not understand.
Working code:
import std.stdio;
import std.conv;
void main()
{
foreach (ubyte[] mybuff; chunks(stdin, 4096)) {
encodebuff (mybuff);
}
}
#safe void encodebuff (ubyte[] mybuff) {
foreach (ubyte i; mybuff) {
char b = to!char(i);
switch (i) {
case 'a': .. case 'z':
case 'A': .. case 'Z':
case '0': .. case '9':
case '/':
case '.':
case '_':
case ':': writeChar(b); break;
default: writeOctal(b); break;
case 0: writeChar ('\n'); break;
case '\\': writeString(`\\`); break;
case '\t': writeString(`\t`); break;
case '\n': writeString(`\n`); break;
case '\r': writeString(`\r`); break;
case '\f': writeString(`\f`); break;
case '\v': writeString(`\v`); break;
case '\a': writeString(`\a`); break;
case '\b': writeString(`\b`); break;
}
}
}
#trusted void writeString (string a)
{
write (a);
}
#trusted void writeOctal (int a)
{
writef ("\\%.4o", a); // leading 0 needed for for zsh printf '%b'
}
#trusted void writeChar (char a)
{
write (a);
}
The broken version:
import std.stdio;
import std.conv;
import std.string;
void main()
{
foreach (ubyte[] mybuff; chunks(stdin, 4096)) {
encodebuff (mybuff);
}
}
#safe void encodebuff (ubyte[] mybuff) {
char[] outstring;
foreach (ubyte i; mybuff) {
switch (i) {
case 'a': .. case 'z':
case 'A': .. case 'Z':
case '0': .. case '9':
case '/':
case '.':
case '_':
case ':': outstring ~= to!char(i); break;
case 0: outstring ~= '\n'; break;
default: char[5] mystring;
formatOctal(mystring, i);
outstring ~= mystring;
break;
case '\\': outstring ~= `\\`; break;
case '\t': outstring ~= `\t`; break;
case '\n': outstring ~= `\n`; break;
case '\r': outstring ~= `\r`; break;
case '\f': outstring ~= `\f`; break;
case '\v': outstring ~= `\v`; break;
case '\a': outstring ~= `\a`; break;
case '\b': outstring ~= `\b`; break;
}
writeString (outstring);
}
}
#trusted void writeString (char[] a)
{
write (a);
}
#trusted void formatOctal (char[] b, ubyte a)
{
sformat (b, "\\%.4o", a); // leading 0 needed for zsh printf '%b'
}
Tests: (note that filelist is a NUL-delimited list of files generated by find -print0 on my home directory, and filelist2.txt is generated from filelist by filelist.txt sed -e 's/\x0/\n/g' > filelist2.txt and is thus the corresponding list of newline-delimited filenames).
# the sed script escapes the backslashes so xargs does not clobber them
diff filelist2.txt <(<filelist.txt char2code2 | sed -e 's/\\/\\\\/g' | xargs /usr/bin/printf "%b\n")
# from within zsh
bash -c 'diff filelist2.txt <(for i in "$(<filelist.txt char2code)"; do printf "%b\n" "$i"; done)'
# from within zsh and bash
diff filelist.txt <(for i in $(char2code <filelist.txt); do printf '%b\0' "$i"; done)
# from within zsh, bash, and dash
for i in $(char2code <filelist.txt); do printf '%b\0' "$i"; done | diff - filelist.txt
A script I made as an acid test:
#!/bin/bash
# this creates a completely random list of NUL-delimited strings
a=''
trap 'rm -f "$a"' EXIT
a="$(mktemp)";
</dev/urandom sed -e 's/\x0\x0/\x0/g' | dd count=2048 of="$a"
test -s "$a" || exit 1
printf '\0' >> "$a"
for i in $("$#" < "$a")
do
printf '%b\0' "$i"
done | diff - "$a"
What is the reason for the difference?
EDIT: I have implemented the changes suggested by #yaz and #MichalMinich and am still seeing wrong results. Specifically, find -print0 | char2code2 (the name of the program, which is in my $PATH) from my home directory results in an exit status of 1 and no output. However, it works from a subsidiary directory with far fewer items. My revised source is below:
import std.stdio;
import std.conv;
import std.format;
import std.array;
void main()
{
foreach (ubyte[] mybuff; chunks(stdin, 4096)) {
encodebuff (mybuff);
}
writeln();
}
void encodebuff (ubyte[] mybuff) {
auto buffer = appender!string();
foreach (ubyte i; mybuff) {
switch (i) {
case 'a': .. case 'z':
case 'A': .. case 'Z':
case '0': .. case '9':
case '/':
case '.':
case '_':
case ':': buffer.put(to!char(i)); break;
case 0: buffer.put('\n'); break;
default: formatOctal(buffer, i); break;
case '\\': buffer.put(`\\`); break;
case '\t': buffer.put(`\t`); break;
case '\n': buffer.put(`\n`); break;
case '\r': buffer.put(`\r`); break;
case '\f': buffer.put(`\f`); break;
case '\v': buffer.put(`\v`); break;
case '\a': buffer.put(`\a`); break;
case '\b': buffer.put(`\b`); break;
}
}
writeString (buffer.data);
// writef(stderr, "Wrote a line\n");
}
#trusted void writeString (string a)
{
write (a);
}
#trusted void formatOctal(Writer)(Writer w, ubyte a)
{
formattedWrite(w, "\\%.4o", a); // leading 0 needed for zsh printf '%b'
}

Answer 1

You need to take writeString outside the foreach in encodebuff. Currently you're writing outstring on each loop without clearing it. The issue #Michal Minich pointed is valid too.

Answer 2

One reason could be that you are appending always 5 chars of char[5] mystring. Function sformat in formatOctal returns the string formatted which might have less than 5 chars (probably slice of the buffer), you should use that string to append to outstring.
Performance advice: use Appender instead of ~= for better performance when building string.

Answer 3

The real problem turned out to be with my LDC installation. It used shared libraries, which aren't supported by its version of druntime.
Recompiling LDC to used static libraries fixed the problem.