SignTool sporadically reports EXEC This file format cannot be signed because it is not recognized - signtool

Question

I'm using the signtool utility to sign the MSI and EXE files on Teamcity as the part of Continuous Integration process. Most of the time it works as expected but sometimes builds fail with the error message below.
[09:04:00][Exec] C:\TeamCity\buildAgent\work\f1f2a8269bd3cd0e\MyProject.Setup.AllUsers\..\SigningTool\signtool.exe sign /f C:\TeamCity\buildAgent\work\f1f2a8269bd3cd0e\MyProject.Setup.AllUsers\..\Certificate\MyCompany.p12 /d "MyCompany" /p MyPassword! /v /tr http://sha256timestamp.ws.symantec.com/sha256/timestamp /td SHA256 C:\TeamCity\buildAgent\work\f1f2a8269bd3cd0e\MyProject.Setup.AllUsers\bin\Debug\MyProject.msi
[09:04:00][Exec] The following certificate was selected:
[09:04:00][Exec] Issued to: MyCompany
[09:04:00][Exec]
[09:04:00][Exec] Issued by: Sectigo RSA Code Signing CA
[09:04:00][Exec]
[09:04:00][Exec] Expires: Wed Apr 27 23:59:59 2022
[09:04:00][Exec]
[09:04:00][Exec] SHA1 hash: ***
[09:04:00][Exec]
[09:04:00][Exec]
[09:04:00][Exec] Done Adding Additional Store
[09:04:00][Exec]
[09:04:00][Exec] Number of files successfully Signed: 0
[09:04:00][Exec]
[09:04:00][Exec] Number of warnings: 0
[09:04:00][Exec]
[09:04:00][Exec] Number of errors: 1
[09:04:00][Exec]
[09:04:00][Exec] EXEC This file format cannot be signed because it is not recognized.
[09:04:00][Exec] EXEC An error occurred while attempting to sign: C:\TeamCity\buildAgent\work\f1f2a8269bd3cd0e\MyProject.Setup.AllUsers\bin\Debug\MyProject.msi
[09:04:00][Exec]
[09:04:00][Exec] C:\TeamCity\buildAgent\work\f1f2a8269bd3cd0e\packages\WiX.3.11.2\tools\wix2010.targets(2832, 5): error MSB3073: The command "C:\TeamCity\buildAgent\work\f1f2a8269bd3cd0e\MyProject.Setup.AllUsers\..\SigningTool\signtool.exe sign /f C:\TeamCity\buildAgent\work\f1f2a8269bd3cd0e\MyProject.Setup.AllUsers\..\Certificate\MyCompany.p12 /d "MyCompany" /p MyPassword! /v /tr http://sha256timestamp.ws.symantec.com/sha256/timestamp /td SHA256 C:\TeamCity\buildAgent\work\f1f2a8269bd3cd0e\MyProject.Setup.AllUsers\bin\Debug\MyProject.msi" exited with code 1.
Once failed all subsequent builds start failing with the same error message until I restart the build machine. The version of the signtool used is 10.0.17763.132. What is more confusing is when I log in to the build machine and sign the same executables with the same command line it gets signed successfully with no issues.

Answer 1

Something very similar happened to me in a different context. There, too, an executable file became abruptly unserviceable and a system reboot was required.
The cause was a process stopping and waiting for input confirmation with an output file (in your case, this would be the signed executable) still open. The confirmation never came (it was a headless machine) and the process waited forever until a reboot.
We found out the cause by running Process Explorer and checking out all open file handles containing the unwriteable file name. At that point we discovered that it was kept open by another process, and terminating this second process allowed to restart the cycle without rebooting the whole machine.
In the end, in my case, it was the input stream to the hung process that was malfunctioning; we changed the workflow and the problem disappeared.
I suggest as soon as the machine malfunctions again, you run Process Explorer and examine which processes are active and what are they doing.