I have a stupid question about homebrew: Why are executables that I install via homebrew trusted by MacOS (gatekeeper)? i.e. after installation I can run an executable and don't get a security popup and don't have to allow an exception - why is that?
I initially thought that homebrew might sign/notarize the binaries in their CI, but looking at some random executables it doesn't look like they have a signature: spctl -a -v $(which <some-executable-installed-with-homebrew>).
edit: meaning executables installed from bottles (pre-compiled binaries, not source packages compiled on my local machine)
There is no quarantining flag for a CLI app downloaded with curl.
Home-brew, uses UNIX core tools to download the bottles, and thus they don't have this flag set.
Next home-brew also ad-hoc signs binaries.
Don't confuse code sign with notarisation.
Notarisation is where Apple vouches for software signed with a dev cert private key.
They cannot notarise ad-hoc signed software (like home-brew bottles) by definition.
Now when my executable is NOT notarized it terminates with "Killed: 9", regardless if there's a quarantine attribute or not.
This is happening, I would speculate because the binary here isnt ad-hoc signed. Nothing to do with notarisation.
I bet you are on Apple Silicon right?
Related
I'm working to correctly distribute a software bundle for Mac ARMS (M1, M2...) which consists in a zip containing a bunch of command line utilities and dynamic libraries built using the command line tools (cc, c++ etc).
So far I've been able to sign all my executable files and my dynamic libraries with codesign using my company Developer ID certificate and to successfully notarize the zip.
Unfortunately when I download the zip using the browser my command line tools show a strange behavior: if I double click on of them from finder I get an error message saying that the identity of the developer can't be confirmed but if I run from the terminal most of the utilities work. I've noticed that if I download the last jdk as .tar.gz the behavior with the java executable is the same so I'm wondering if it's the expected behavior for command line utilities.
Still one of the utilities is not working because it is supposed to load the dynamic libraries I get an error saying that relative paths are not allowed in hardened programs.
My questions are:
Is the behavior described above for command line executable files expected?
Is there a way to allow my program that loads dynamic library to work by loading dynamic libraries as it used to do with our unsigned bundles for Intel Macs?
Does anyone knows if it's possible to notarize and distribute such a bundle in .tar.gz format rather than zip?
Thanks!
For macOS notarized software that means the executable is going to have the hard-coded path to the library and won’t work with a relative path. Usually that means setting RPATH to the hardpath
/Applications/myapp.app/Contents/Frameworks/
and aliasing your binaries from
/Applications/myapp.app/Contents/MacOS
But that is just the most common arrangement.
The hard-coded path is part of the “hardened runtime” which is a requirement to pass apple notary, an anti-malware scanner. Changing the assigned RPATH with otool is possible but invalidates the code signature, blocking execution on machines with default security without code signing at least ad hoc.
See
https://wiki.lazarus.freepascal.org/Code_Signing_for_macOS
Launching the executable directly from terminal does not run thru launchservices the way it does when you launch in finder. Launchservices begins by checking the signature against the ticket and checking the ticket, so a bare executable will be missing that info.
Thanks to the suggestions of Richard Barber I've finally understood how to correctly code-sing, notarize and deploy our bundles on Mac with hardened runtimes.
I'm leaving the steps I've taken below for future reference.
Here are the steps I needed to run:
Made sure all the dynamic executable files and libraries have referenced their dependencies without using relative paths. The relative paths may be substituted with macros like #rpath using install_name_tool
Code-singed all executable files and all the dynamic libraries suing the codesign tool with a valid Apple Distribution certificate and developer/organization key
Once signed put everything in a ZIP and sent for notarization using xcrun notarytool submit command
If the notarization succeeds the contents of the ZIP can be extracted and put in a .tar.gz without losing the validity
Once the tar.gz is downloaded on a testing machine our command line tools can be correctly used by the terminal. As suggested by Richard Barber they can't be run from Finder, but even the JDK in tar.gz format shows the same behavior, so happy with that.
We have a multi-platform suite of command-line executables and libraries that we ported to Mac. The file layout was
/Applications/
(company folder)/
(our UI).app
(product name)/
bin/
...executalbes...
lib/
...dylibs...
(other stuff)/...
This was shipped in a DMG that was codesigned, as was the app. This worked OK until Catalina.
Now on Catalina, we have codesigned all the executables, dylibs, apps (including ones nested in the top-level app's framework), frameworks, and the DM itself. When we notarize it the resulting JSON log lists no issues. However, when I run any of our executables that depends on one of our dylibs I get a pop-up telling me the "developer cannot be identified". Even though it has been signed and notarized OK. Running codesign with -dvvv option includes the following:
SHA-256 hash choice
list of Authority entries terminating in Apple Root CA
TeamIdentifier entry
Timestamp
Runtime: 10.13.0
Question How can I fix this, or at least get Gatekeeper to tell me why it's not accepting this file? Maybe a log, or an equivalent of spctl --assess for files rather than apps?
Observations
This only happens when
the OS is Catalina
it's under the /Applications folder
outside of /Applications (e.g. in a folder on desktop) it only happens sometimes (and sometimes it claims a dylib can't be loaded on first attempt, then succeeds if tried a moment later)
the executable depends on one or more of our dylibs (standalone ones run OK)
the executable has the com.apple.quarantine xattr set
I've tried to mix'n'match between clean and downloaded (i.e. quarantine-xattr'd files) and the problem only arises if the executable is quarantined; it doesn't care if a non-quarantined executable loads a quaratined dylib
The signing operation was done via codesign with args "--deep --strict --timestamp --options runtime", and then verified
EDIT: I've since updated this to include some the Hardened Runtime entitlements to fix another build issue, but it hasn't helped with this one
The executables depend on the dylibs via #rpath (as reported by otool -L)
EDIT: I'ver tried replacing the #rpath with #executable_path/../lib in each case, and that hasn't helped
One more tidbit that Apple haven't added to their documentation; if you are distributing a dylib rather than a full app bundle, you can't staple the notarisation results to it. So far, so documented, and Catalina should verify the notarisation online when the software is installed.
However, the notarised dylib will be rejected on the build machine unless you add it to a container (i.e. zip/dmg), remove the original copy, then extract it from the zip again after the notarisation has been fully accepted (i.e. you have received the success email). It appears that the notarisation is only checked when the file is created.
You can always run it from within XCode, but not on it's own, without removing and replacing the binary.
I maintain OS software which is distributed as an Mach-O executable in a zip. It's not packaged as an app. I can codesign this without problems but all references to notarization that I can find are talking about .apps. Does anyone know about notarizing simple executables?
Yes. You can notarize plain executables. Some caveats...
The upload process does not accept executables, they must be encoded. Zipping them is the easiest way. The notarization is for the underlying executable, you can unzip, re-zip, etc. But the transfer process needs the enclosing zip.
You can't staple to the executable. Gatekeeper can verify against apple's servers, instead of the stapled ticket. (And presumably, the quarantine bit is then removed)
The spctl -a -vvv -t install command will display a bunch of info.
$ file go-hello-notarized-APPLEID
go-hello-notarized-APPLEID: Mach-O 64-bit executable x86_64
$ spctl -a -vvv -t install go-hello-notarized-APPLEID
go-hello-notarized-APPLEID: accepted
source=Notarized Developer ID
origin=Developer ID Application: Example, Inc (APPLEID)
There's info in https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution/customizing_the_notarization_workflow but it's hard to tell until you try it all.
I develop a Mac app with Qt5, so outside Xcode. I want GateKeeper to allow my app to run on clients' computers rather than issuing the "Can't be opened because the identity of the developer cannot be confirmed" warning.
I have successfully digitally signed the app but GateKeeper still comes with this complaint. I have an Apple developer certificate (I am the Team Agent) and my keychain says it is valid. I also have installed two Apple root certificates.
I use the command line utility codesign to digitally sign all the binaries inside the app folder and in addition I digitally sign the app folder itself. In all cases the response of codesign is informative and displays no error. With codesign I can check that indeed all the binaries are signed, running
$ codesign --verify --deep --verbose=2 MyApp.app
shows that all binaries are validated. And in addition it reports:
MyApp.app: valid on disk
MyApp.app: satisfies its Designated Requirement
Running:
$ codesign -v --verbose=4 --display MyApp.app
gives
Executable=/Users/xxx/trunk/yyy/deploy/release/MyApp.app/Contents/MacOS/MyApp
Identifier=aaaa.MyApp
Format=bundle with Mach-O thin (x86_64)
CodeDirectory v=20200 size=12461 flags=0x0(none) hashes=616+3 location=embedded
Hash type=sha1 size=20
CDHash=d1c12c783dac0e8d9a2b749fb896b11558cec8b6
Signature size=8532
Authority=Developer ID Application: XXXXX
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=29 jul. 2015 12;04:40
Info.plist entries=8
TeamIdentifier=YYYYY
Sealed Resources version=2 rules=12 files=10
Internal requirements count=1 size=180
which seems OK.
Running
$ spctl -a -t exec -vv MyApp.app
on all binaries gives as result
MyApp.app: accepted
source=Developer ID
origin=Developer ID Application: XXXX
which also seems OK
Running the XCode command line tool check-signature
on the app or on the binaries inside the app folder:
$ ./check-signature /Users/xxx/trunk/yyy/release/MyApp.app
gives as result
(c) 2014 Apple Inc. All rights reserved.
YES
which in all cases is the desired result.
But GateKeeper still does not accept the app and complains about the fact that the developer cannot be confirmed.
[added by author on Friday July 17, 2015]
I think I have found the problem. I do not know whether it is a feature or an OSX bug. I was helped a lot by stackoverflow question 19551298.
Whenever a file is downloaded from the internet it gets an extended file attribute com.apple.quarantine associated with it. When double-clicking on this downloaded file in Finder, GateKeeper has two possibilities:
When the file is not signed it issues the "Unidentified developer etc" message
When the file is digitally signed it issues the "Developer cannot be confirmed etc" message
In both cases the MessageBox has only one button, an OK button. When this button is clicked nothing happens, apart from the MessageBox closing.
If the extended attribute is deleted (xattr -d) the applications runs, signed or not.
The behavior is different when the applications is started by right-mouse-button-clicking in Finder on the app and then click on the "open" menu action. Again one of the two messageboxes is shown, but now with an extra button to allow the user to open the app anyway. Again the only difference between signed and not signed is the "Unidentified" or the "Not confirmed" message. I do not expect my customers to be able to tell the differende. As a result signing the app is an exercise in futility.
On the basis of the Apple Support Documentation I expected another, much nicer behavior of GateKeeper when double clicking a downloaded app (perhaps the documentation is outdated, or I misread it):
if the app is signed GateKeeper should show a MessageBox with "Downloaded from the internet etc" and a button with "Proceed anyway?"
if the app is not signed a MessageBox with a single OK button and a text "Unidentified developer etc.."
Sorry for answering my own question, but I see no other way as editing the original question would lead to spaghetti text.
I finally solved my problem. First the credit: (i) The answer to my other stackoverflow question was very useful and (ii) I got very good (paid) advice from an official Apple developer, by filing a so-called Technical Support Incident (TSI).
On the basis of all this I am now able to give here a very concise recipe of how getting your Mac app successfully treated by GateKeeper. After detailing the recipe I will show what my original mistake was.
Goal: After having developed a Mac app outside Xcode to have GateKeeper issuing the warning "Downloaded from the Internet ..." with three buttons, one of which is "open".
Failure: When GateKeeper issues a warning with either the text ".. unidentified developer.." or the text ".. unconfirmed developer .. " with - in both cases - a messagebox with a single OK button.
Getting your app GateKeeper-ready involves three steps:
Make your app standalone with no unacceptable external dependencies. The only acceptable external dependences are system libraries. All other dependencies should have been copied to your MyApp.app folder. GateKeeper rejects any app that has non-system external dependencies
Binaries should not be located at illegal positions inside the MyApp.app folder. Libraries go into MyApp/Contents/Frameworks and the executable goes into MyApp/Contents/MacOS
All binaries inside MyApp should be digitally signed. Then the MyApp.app folder should be signed. For this signing an Apple "Developer ID Application ..." certificate is necessary
Our recipe is automatic. All the work is done by one script. In case of Qt Creator we use a qmake script where we access the system shell through the $$system command. When using either of the (Xcode) system commands codesign, spctl or check-signature we assume you have redirected stderr to stdout as outlined in answer to question . Otherwise you will not be able to catch the system response when running these utilities. In the following we will not explicitly show this redirection.
HERE IS OUR RECIPE
A. Making the app stand-alone:
copy (with a script) all the needed binaries to the MyApp.folder
run (with a script) install_name_tool -change and install_name_tool -id such that all dependences inside the app are of the relative type #executable_path/../MacOS.. or #executable_path/../Frameworks
run (with a script) otool -L on all binaries inside the MyApp.app folder and flag any illegal dependence, like "#rpath..." or absolute file paths not being system paths. Note that otool -L is not guaranteed to find all dependencies. Plugins are often beyond the horizon of otool. That is why you need the next check.
start a terminal at the location "MyApp.app/Contents/MacOS". Run export DYLD_PRINT_LIBRARIES=1. Then run inside the same terminal window ./MyApp. Your terminal will fill up with over hundred loaded libraries. Check this list again for forbidden libraries (libraries present on your computer, but not on the computer of your customers).
proof of the pudding is in the eating. We use the MacInCloud virtual machines and check whether or not our app runs there. Alternative solution could be the Mac of a relative who is not a developer. Or you could also create a new user ("test") on your own Mac and copy the app to its Download (or Desktop folder, or ...). In the latter case you must temporarily rename the root folder of your IDE as otherwise the user "test" will find the missing binaries there.
B Signing the app
Signing: With our script we run codesign --force --verify --verbose --sign \"Developer ID Application: ....\" \"/path/to/binary\" on all the binaries in the app and then on the app folder itself. In each case the system response is caught. It should contain in each case the string "signed Mach-O thin".
Verification: Run (with a script) command codesign --verify --verbose \"/path/to/binary\" on each binary in your app and on the app itself and catch the system response. It should in each case contain the strings "valid on disk" and "satisfies its Designated Requirement".
GateKeeper check: Run (with a script) spctl -a -t exec -vv /path/to/binary\" on each binary and on the app folder itself. The system response is caught. It should contain in all cases the string "accepted source".
check-signature: Run (with a script) check-signature \"/path/to/banary\" on each binary and on the app folder itself. The system response is caught. It should contain the string "YES" in each case.
C External check
zip your app into a single zip file. Upload to one of your cloud servers
GateKeepers keeps a long list (typically hundreds of items) of exceptions on its general gate-keeper role. Your app must not be in that list if you want to test GateKeeper. Rather than editing this list a much simpler trick is creating a new user on your Mac. Log in to that user and download the zip file from the Internet cloud server. Finder will automatically uncompress it. Click on it. If GateKeeper tells you that it can open the application but it warns you at the same time that it is downloaded from the Internet, it is time to grab a (white) beer.
Here the desired GateKeeper warning:
My mistake
I did much of the installing and signing without explicitly checking the result for each binary. After that I would use otool -L on a number of binaries but not on all. I missed the fact that upgrading to Qt 5.5 from an earlier Qt version the binary libqminimal.dylib has acquired an extra dependency, viz.: QtDBus. I had not noticed it, but GateKeeper did.
Qt developers might wonder why we not just use macdeployqt for deploying Qt application on a Mac. In the first place we do not like not to use ill-documented black-box utilities. On Internet fora there are quite a number of people reporting issues with macdeployqt. In addition the Qt libraries can have different install locations (as reported by otool-L) when comparing different Qt versions. When we have a new Qt version our script will immediately start to yell about forbidden dependencies. In this way we get information about what has changed in this new version.
adlag's question and self-answer was invaluable in helping me overcome the same issue. However, as good as his recipe is, some statements are not quite right, so I'd like to offer a few additional points.
It's not necessary to replace #rpath entries in your binaries and dynamic libraries with #executable_path statements. #rpath statements are fine so long as the actual rpath entries embedded in the binaries are not absolute. You can find plenty of valid Qt app bundles that use rpath. You can make it work doing what adlag said, but you may be making work for yourself.
See jil's comment above for how to use otool -l $file | grep -A2 LC_RPATH and install_name_tool -delete_rpath $path $file to inspect and remove the embedded paths in your binaries and libraries
See https://developer.apple.com/library/content/technotes/tn2206/_index.html#//apple_ref/doc/uid/DTS40007919-CH1-TNTAG207 for a clear explanation of why GateKeeper complains about paths in your binaries, and how you can see the specific complaint in syslog.
If you have a problem with absolute paths, you should first try to fix your build, rather than use install_name_tool after the fact.
If you're using cmake, this is likely helpful: https://cmake.org/Wiki/CMake_RPATH_handling#Mac_OS_X_and_the_RPATH
Don't run spctl -a -t exec -vv /path/to/binary on dylib files. You will get errors about the resource envelope. This is expected, and not a problem.
In my experience, macdeployqt works fine. I solved the problem by changing my build, such that the absolute paths did not get into the offending dylib file (libquazip). I still used install_name_tool to remove the absolute paths to the Qt installation. I then used macdeployqt to create the bundle, sign the bundle and create the DMG file.
My two bits:
To really verify codesigning, I had to either upload my DMG to a
server and download it using a browser or set the quarantine attribute manually:
APP_PATH="Any.app"
xattr -w com.apple.quarantine '0081;5a37dc6a;Google Chrome;F15F7E1C-F894-4B7D-91B4-E110D11C4858' "$APP_PATH"
xattr -l "$APP_PATH" # You should see the quarantine attribute here
open "$APP_PATH"
If your app is correctly signed, you should see a system dialog with
an "Open" button.
I found the value of the quarantine attribute by looking at another
.app downloaded from the internet. I don't know what the value
means.
I don't really understand why the spctl command says "accepted" even
if the Gatekeeper service denies opening the app.
I had the "unidentified developer" message box because my Qt frameworks were referenced as "#rpath/QtCore.framework". Changing it to "#application_path/../Frameworks/QtCore.framework" using the install_name_tool fixed the issue in my app.
Figured out the problem after lot of tries.
In my case:
The Pop Message - damaged application came due to libraries were missing.
I Had created .app file using QT.
To generate dmg i was using deploymacqt command tool.
deploymacqt tool creates dynamic libraries inside .app, so basically if we codesign before creating dmg, this alter will manipulate code sign.
So the proper fix is.
# Create dmg using
deploymacqt <yourapp.app> -dmg
# Open resulted dmg file, copy <yourapp.app> to different folder(let's say /Documents/<yourapp.app>)
# Codesign the /Documents/<yourapp.app> using
codesign --deep --force --verify --verbose --sign "Developer ID Application: <developerid>" <yourapp.app>
# Verify using
codesign --verify --verbose=4 <yourapp.app>
* you should see something like this
<yourapp.app>: valid on disk
<yourapp.app>: satisfies its Designated Requirement
# Now create again the dmg file using dropdmg(https://c-command.com/dropdmg/) application, download, install dropdmg. set the cofiguration preferences with your developer id certificate in signing option.
# drag and drop <yourapp.app> to dropdmg app, wait for creation of dmg to complete. voila you have now successfully created dmg with proper developer id certification.
# verify resulted dmg again using
codesign --verify --verbose=4 <yourapp.dmg>
# you can also verify with gatekeeper
spctl -a -t exec -vv <yourapp.dmg>
once you are done with these, you will not see pop message saying app is damaged or broken or unidentified developer.
You need to verify all your rpath with command line otool -l of your executable and yours Frameworks. If you have a local rpath (ex: /user/name/Qt/) in your executable delete it (with this command install_name_tool -delete_rpath).
I'm developing a product that includes kernel extension and have found a weird problem in one of our testing machines that I can't find a solution for.
In my development machine, (OSX 10.8.3 and latest Xcode) I codesign our kext like this:
$ codesign -s "Developer ID Application: Mycompany" my.kext
my.kext: signed bundle with Mach-O thin (x86_64) [com.mycompany.kext]
it all goes fine, my.kext/Contents/MacOS/mykext binary is modified (signature added) and a folder my.kext/Contents/_CodeSignature is created with a file CodeResources in it.
When loading this kext on one of our testing machines (OSX 10.7.5 with Xcode 3.2.6, Darwin Kernel 11.4.2 x86_64), it refuses to do so:
kxld[com.mycompany.kext]: The Mach-O file is malformed: Invalid segment type in MH_KEXT_BUNDLE kext: 29.
Can't load kext com.mycompany.kext - link failed.
Failed to load executable for kext com.mycompany.kext.
Kext com.mycompany.kext failed to load (0xdc008016).
If I load the module unsigned, there is no problem. Also tried signing the kext from Xcode and not from command-line, with the same results.
I moved the signing certificate to that troublesome computer, and signed the kext there. The signing process goes different:
$ codesign -v -s "Developer ID Application: Mycompany" my.kext
my.kext: signed bundle with generic [com.mycompany.kext]
Once signed, the kext executable at my.kext/Contents/MacOS/mykext is unmodified, and the folder Contents/_CodeSignature includes more files: CodeDirectory, CodeRequirements, CodeResources and CodeSignature. This signed kext seems to work on all devices so far.
So the question is:
What's going on in here? What am I doing wrong in my signing process? How can I create a signature in a updated device that will work on this "outdated" machine? I understand that the target machine is refusing load of the kext because it does not undertand the signed binary. Signing from this device creates some kind of detached signature where the binary is untouched. I can't get my codesign to do that, the -D option seems useless and won't create a _CodeSignature folder inside the bundle.
Update
As of XCode 4.6, the problem still persists. Only i386 kexts are signed in a backwards-compatible way. x64 and mixed arch kexts can't be loaded by some 10.6 and 10.7 kernels due to them not understanding the signature embedded into the binaries.
The codesign command-line tool has an undocumented --no-macho flag for this purpose, but seems to be unimplemented.
Update 2
The problem still persists as of Xcode 4.6.2 4.6.3
Preamble: Explanation of what's going on
The older kernel linker/loaders can't handle certain types of load commands in the kext's Mach-O object code, including the LC_CODE_SIGNATURE section. This has also caused problems with e.g. mixed 32-bit/64-bit kexts built using Xcode 4.5.x, where the toolchain added various other sections that the Lion and Snow Leopard kernel linkers weren't expecting. (this bug is fixed in 4.6.x)
Apple hasn't released any specific info on codesigning kexts that I can find. If you look at their own kexts, some are signed, and some are not. (the open source ones seem to be unsigned as far as I can tell) If you look at the Mach-O sections in the binaries for their signed kexts (using otool -l), you will notice that LC_CODE_SIGNATURE is absent, unlike .app bundle binaries, where this inline signature is now the default. This is the case even for kexts that ship with Mountain Lion.
So the solution for supporting older versions is to place the signature in a separate file rather than letting codesign insert a signature section into the binary.
Solution
I found the undocumented --no-macho flag in the codesign source code and that seems to do the trick. No LC_CODE_SIGNATURE section, and the signature ends up in _CodeSignature/CodeSignature.
I believe the solution can be found under the BSD and Kernel Features section at the bottom of the OS X v10.9 Mavericks page in the What's New in OS X document. Unfortunately, I'm not sure if I can disclose the information here as it falls under the pre-release category. However, for those of you who have a paid Mac Dev account, here's the URL:
https://developer.apple.com/library/prerelease/mac/releasenotes/MacOSX/WhatsNewInOSX/Articles/MacOSX10_9.html#//apple_ref/doc/uid/TP40013207-CH100