Why do I need a temp file with pg_dump? - bash

Question

I have been receiving errors from pgdumplib and have boiled down the problem to how the output of pg_dump is redirected.
This is what I would like to do, but it consistently fails with RuntimeError: Unsupported data format:
F=/tmp/test_Fc_format.dump
ssh codimd "sudo -u codimd bash -c 'cd /; pg_dump -d codimd -Fc'" >$F
python -c "import pgdumplib; dump = pgdumplib.load('$F')"
I can work around the problem by redirecting the file on the remote machine. This sequence works consistently:
F=/tmp/test_Fc_format.dump
ssh codimd "sudo -u codimd bash -c 'cd /; pg_dump -d codimd -Fc >/tmp/934354 && cat /tmp/934354'" >$F
python -c "import pgdumplib; dump = pgdumplib.load('$F')"
Note that the only difference is that the second sequence adds >/tmp/934354 && cat /tmp/934354, i.e. redirecting the output of pg_dump to a file on the remote machine first, and then sending that to stdout. In both cases, the resulting file is the same size (though not identical because the database is online).
Both local and remote machines run Ubuntu 20.04.
Why is this extra step necessary, and is there a better way to solve this issue?
Update 1:
This, too, gives the error:
F=/tmp/test_Fc_format.dump
ssh codimd "sudo -u codimd bash -c 'cd /; pg_dump -d codimd -Fc |tee /tmp/934354 >/dev/null && cat /tmp/934354'" >$F
python -c "import pgdumplib; dump = pgdumplib.load('$F')"
In other words, to work, pg_dump seems to require a redirect to a local file or the -f option.
Update 2:
Here is the full list of differences between the bad and good version of the database, after using hd on each (note pg_dump never produces the same output twice):
2c2
< 00000010 00 11 00 00 00 00 19 00 00 00 00 16 00 00 00 00 |................|
---
> 00000010 00 2e 00 00 00 00 19 00 00 00 00 16 00 00 00 00 |................|
349c349
< 000015c0 31 38 01 01 00 00 00 01 00 00 00 00 00 00 00 00 |18..............|
---
> 000015c0 31 38 01 01 00 00 00 02 b0 30 00 00 00 00 00 00 |18.......0......|
370c370
< 00001710 31 34 01 01 00 00 00 01 00 00 00 00 00 00 00 00 |14..............|
---
> 00001710 31 34 01 01 00 00 00 02 1e 5f 00 00 00 00 00 00 |14......._......|
387c387
< 00001820 00 00 01 00 00 00 00 00 00 00 00 00 b7 0b 00 00 |................|
---
> 00001820 00 00 02 21 d9 60 00 00 00 00 00 00 b7 0b 00 00 |...!.`..........|
399c399
< 000018e0 01 00 00 00 00 00 00 00 00 00 bf 0b 00 00 00 01 |................|
---
> 000018e0 02 91 0a 4c 01 00 00 00 00 00 bf 0b 00 00 00 01 |...L............|
412,413c412,413
< 000019b0 03 00 00 00 32 32 30 01 01 00 00 00 01 00 00 00 |....220.........|
< 000019c0 00 00 00 00 00 00 ba 0b 00 00 00 01 00 00 00 00 |................|
---
> 000019b0 03 00 00 00 32 32 30 01 01 00 00 00 02 ce 0b 4c |....220........L|
> 000019c0 01 00 00 00 00 00 ba 0b 00 00 00 01 00 00 00 00 |................|
425c425
< 00001a80 35 01 01 00 00 00 01 00 00 00 00 00 00 00 00 00 |5...............|
---
> 00001a80 35 01 01 00 00 00 02 0b ee 4c 01 00 00 00 00 00 |5........L......|
438c438
< 00001b50 00 00 01 00 00 00 00 00 00 00 00 00 b8 0b 00 00 |................|
---
> 00001b50 00 00 02 28 ee 4c 01 00 00 00 00 00 b8 0b 00 00 |...(.L..........|
456c456
< 00001c70 01 00 00 00 01 00 00 00 00 00 00 00 00 00 c7 0b |................|
---
> 00001c70 01 00 00 00 02 45 ee 4c 01 00 00 00 00 00 c7 0b |.....E.L........|
Update 3:
It turns out that this has nothing to do with ssh. I think pg_dump requires a seekable file as output. Here I demonstrate that inserting |cat before redirecting the output file causes the file to be corrupted. If true, is this a bug in pg_dump?
$ pg_dump -d codimd -Fc >/tmp/good
$ python3 -c "import pgdumplib; dump = pgdumplib.load('/tmp/good')"
$ # no error
$ pg_dump -d codimd -Fc |cat >/tmp/bad
$ python3 -c "import pgdumplib; dump = pgdumplib.load('/tmp/bad')"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/lib/python3.8/dist-packages/pgdumplib/__init__.py", line 24, in load
return dump.Dump(converter=converter).load(filepath)
File "/usr/local/lib/python3.8/dist-packages/pgdumplib/dump.py", line 254, in load
raise RuntimeError('Unsupported data format')
RuntimeError: Unsupported data format
$

Answer 1

Some testing on my end:
F=test_dmp.out
pg_dump -d test -U postgres -Fc > $F | ls -al test_dmp.out
-rw-r--r-- 1 aklaver users 0 Sep 30 14:06 test_dmp.out
pg_dump -d test -U postgres -Fc > temp_file.out && cat temp_file.out > $F | ls -al test_dmp.out
-rw-r--r-- 1 aklaver users 97488 Sep 30 14:08 test_dmp.out
pg_dump -d test -U postgres -Fc -f $F | ls -al test_dmp.out
-rw-r--r-- 1 aklaver users 97488 Sep 30 14:08 test_dmp.out
I know that not an answer as to why > does not work, but it gives an alternative. For some reason > creates an empty file. The other options do not.
UPDATE.
It seems to have something to do with the SSH transfer:
F=test_dump.out
#Using local only.
pg_dump -d test -U postgres -Fc > $F
python -c "import pgdumplib; dump = pgdumplib.load('$F'); print('Database: {}'.format(dump.dbname))"
Database: test
#Using SSH, different database
ssh arkansas "sudo -u aklaver bash -c 'cd /; pg_dump -d redmine -U postgres -Fc'" >$F
python -c "import pgdumplib; dump = pgdumplib.load('$F'); print('Database: {}'.format(dump.dbname))"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/aklaver/py_virt/py37/lib/python3.7/site-packages/pgdumplib/__init__.py", line 24, in load
return dump.Dump(converter=converter).load(filepath)
File "/home/aklaver/py_virt/py37/lib/python3.7/site-packages/pgdumplib/dump.py", line 254, in load
raise RuntimeError('Unsupported data format')
RuntimeError: Unsupported data format
#Though the file itself ends up being ok. The below does not error out.
#There seems to be something asynchronous going on. In other words
#pgdumplib is reading the file before it is complete.
pg_restore -f test.sql test_dump.out