MDJ 2020.02.24: Finally, Notarization
The Weekly Attitudinal accepts the necessity of inconveniencing everyone
At last, your Attitudinal circles back to the item of note that started this journal down this exquisitely twisted and lengthy path: notarized software. The need for Apple to scan and, effectively, sign every app you launch in macOS Catalina or later. The ability for Apple to stop such programs from opening if the company deems it critical. The confusion for users, the time taken from developers, the black-box nature of a system few understand and even fewer praise.
You will understand from the previous installment (MDJ 2020.02.20) that the Attitudinal begrudgingly regards notarization as an absolute must in the computing world of 2020. For that matter, previous comments on notarization in this series have pretty much given that away:
• “In the carrot-and-stick metaphor, notarization is supposed to be the carrot—the good thing you get for free by submitting your code to Apple for a return ticket that Gatekeeper checks. The stick is supposed to be the extra hoops to open non-notarized programs on first launch.” (MDJ 2020.02.03)
• MDJ noted that Apple is likely scanning for a lot more malware than third-party antivirus utilities and that the company could be performing statistical analysis on binaries to look for where to optimate macOS itself (MDJ 2020.02.12)
• This journal also pointed you to Apple’s descriptions of what kinds of misbehavior could cause the company to revoke a notarization or Developer ID certificate and emphasized how no one has evidence of Apple abusing this authority (MDJ 2020.02.14)
• The Attitudinal has even given you a one-line takeaway already: “Notarization lets Apple kill a bad app immediately, and then step up to revoke the Developer ID certificate if the infection was no accident.” (MDJ 2020.02.20) This may lead to making notarization even less optional than it is now, and now its level of optionality is “not very.”
What’s left to cover? The things that went wrong, the things that aren’t wrong (but that people still bemoan), and the lack of options for a more freewheeling future.
Where Apple screwed up
Where should the Attitudinal even begin? Most of the errors here have one root cause, but let’s get through them in the usual order.
It should not be controversial that developer tools and an operating system need to be symbiotic. First, the OS developers often need updates to tools to make new features possible. About a decade ago, one of the significant security developments was address space layout randomization, or ASLR. Since every process gets its own address space in macOS, an attacker who had found a vulnerability in a program like Pages could easily exploit it, because the code would always load at the same address in Pages’s own address space. Why wouldn’t it? Each address space started with the OS’s own libraries loaded at their typical locations, and then the system loaded the application at the usual address. Starting in Mac OS X 10.7, these addresses were adjusted randomly for each new process and its libraries. (Lion also introduced the sandbox. It was all so innocent back then.)
ASLR may seem like an obvious step now, but implementing fully position-independent code required changes in both the OS (in the dynamic linker) and Xcode (in the compilers and the static linker). The advent of the sandbox and, later, the Hardened Runtime also require changes in both places—in Xcode to set the right flags and create the correct binaries, and in the OS to interpret those flags and binaries.
For these reasons, you’d think it would be Best Practices for low-level developer tools and their paired system counterparts to be completed, documented, and stable a full version ahead of anything requiring them. That, of course, is not the case. That would require Apple to invest significantly more in both areas of development than the company currently budgets.
The Road to Notary
The Attitudinal asserts this lack of resources caused unnecessary pain. Apple first announced the concept and process of notarization at WWDC 2018, noting it was optional in the then-upcoming macOS Mojave (10.14). When that OS was released, it provided a “streamlined Gatekeeper dialog,” explaining that the notarized software did not contain any known malware.
By December 2018, Apple was warning developers that starting in “Spring 2019,” (once again exposing Apple’s inability to understand that “Spring” does not mean the same thing in the Southern Hemisphere as it does in Cupertino) “macOS Mojave will more prominently highlight notarization status.” That meant it started to warn users about non-notarized software instead of congratulating them for using notarized software. Then the big shoe, confirming something said in-session at WWDC 2018: “In an upcoming macOS release, Gatekeeper will require Developer ID-signed software to be notarized by Apple.”
Four months later, with the release of macOS 10.14.5, the security policy had changed again, with Apple saying that it would now “require that all developers creating a Developer ID certificate for the first time notarize their apps.” (New and revised kernel extensions would face a similar requirement, but it is orthogonal to the Attitudinal’s upcoming point.) Then, at WWDC 2019—just one year after unveiling the entire notarization process—Apple confirmed that the “future macOS release” was Catalina, coming that same year:
Mac apps, installer packages, and kernel extensions that are signed with Developer ID must also be notarized by Apple in order to run on macOS Catalina. This will help give users more confidence that the software they download and run, no matter where they get it from, is not malware by showing a more streamlined Gatekeeper interface.
It all seems like a standard (if accelerated) rollout of a critical security feature, and it was—with one exception. You may recall that Apple tied notarization to its Hardened Runtime environment, one that disallows several low-level features unless explicitly authorized by the developer in the code-signature.
Apple did not document the Hardened Runtime until WWDC 2019. Developers were asked to use a runtime environment that was not even explained to them.
Oh, sure, Apple made allowances for that. Until early 2019, developers could notarize absolutely anything, signed or not. Then there was a period where the Developer ID signing certificate did not have to match the certificate submitted with the notarization request. Both those periods ended in 2019 before Apple published Hardened Runtime documentation. Even today, that documentation is rather sparse. It implies that apps in the Hardened Runtime can do everything as before unless they need a capability granted by one of the documented entitlements.
However, it does not state that. There are probably other restrictions in the Hardened Runtime that Apple has not documented. A more restrictive environment is good! It helps keep your system safe even if the bad guys find and attack vector! But it’s bad when programs stop working for undocumented reasons.
More resources on the Hardened Runtime might have fixed this, but one thing certainly would have. Hardened Runtime and notarization should never have been tied together. They are both security features that Apple wishes to require to protect its customers, but the binding of the two systems led to a 2019 full of “You have to notarize to launch on Catalina, but that means you have to use this other environment that we’re not really explaining yet.”
And yes, before you object, Apple did allow for this by not requiring notarized apps to adopt the Hardened Runtime in full until this very month, a full eight months after releasing the current documentation. This grace period was, in the Attitudinal’s opinion, not enough to save developers from unnecessary grief and wasted time.
No one is a bad guy in this melodrama, dear readers. No one at Apple thought, “Hey, I can save some money by tying these two requirements together! Let someone else deal with the consequences!” The Attitudinal believes that no responsible developers looked at these requirements and thought, “Screw this and screw Apple for wanting to control me.”
Everyone outside Apple wanted it to be like the Mac is supposed to be: they wanted it to just work. Dealing with shifting requirements and undocumented system features is like trying to use a new program with no menus—you poke blindly at it and you hope it works. If Apple cannot provide that kind of utility to developers, even with the talented staff it has now, then the company needs more of them.
What didn’t go wrong
The grace periods noted above can’t be praised as a good idea—they were necessary given what Apple asked of developers in a world of opaque understanding.
The Attitudinal long thought that most of notarization would be more streamlined had Apple embraced the fact that it is a Certificate Authority. When designing the Developer ID system, Apple decided to limit each developer to five macOS Developer ID certificates and five similar iOS certificates. The company told developers that typically, they would use only one certificate for all macOS deployment and another single iOS certificate for all submissions to the App Store. Apple seemed to encourage this by naming all the certificates so similarly that delineating them was annoying.
How much trouble could Apple have saved by giving a new certificate to every application on request? The Attitudinal is now convinced the answer is “not much.” First, even for companies that do nothing but issue certificates, it takes a bit of time. It may not take two days, like getting an SSL certificate in the 1990s, but it’s still asynchronous. Adding in the malware check that Apple provides with notarization would make the process take about the same amount of time.
Plus, after some digital spelunking, the Attitudinal is now convinced that notarization is not creating a new certificate for each ticket. The tickets returned by the Developer ID Notary Service seem to contain something like an application seal (MDJ 2020.02.12), a signed blob of data with enough information to tell Gatekeeper how to revoke it should that be necessary, but not a full certificate.
That’s no surprise now—with development releases (beta testing), bug fixes, and maybe separate localizations, Apple has probably issued a few million tickets by this point with trends moving upward. Responsible PKI requires more than storing those certificates in a database somewhere, and the Attitudinal believes Apple would not sign up for that problem if the company could skirt it.
Developers are still annoyed at the idea of an asynchronous step required before any distributable build, and it wasn’t hard to find early reports for notarization requests taking a long time. At WWDC 2019, Apple said it believed that most requests should return with either errors or a ticket (or both, in the case of grace periods) within 15 minutes—which implies that averages were longer than that in the prior year. Unfortunately, that would have required the full system to have been built out before introduction. The Attitudinal wishes that the industry agreed on such best practices, but it does not. Consequently, the Attitudinal finds it unreasonable to hold Apple to an expensive standard that its competitors avoid.
Will it get better?
Yes. Most things do. It’s already better—it’s faster, the Hardened Runtime is now documented. A look at Developer System Status shows the Developer ID Notary Service to be as reliable as other systems Apple provides. It also shows that there is no Web interface for it. That needs to change. It’s not feasible to ask that developers be able to find every notarization request they’ve ever made because during release cycles there can be hundreds of those things.
Yet it’s also not reasonable for any errors or warnings to be ephemeral text displayed in Xcode or returned from the command-line program altool. Just as developers can always re-download any of their certificates, they should be able to grab any output from their last two weeks (or 50 invocations) of the notary service.
The Attitudinal expects that WWDC 2020 (whenever that is) will unveil new ways to notarize programs beyond application bundles, installers, and disk images. If that is the case, then Apple must not require those new capabilities in the first release of macOS 10.16 (whatever that’s called). Also, if macOS 10.16 gives such notarized other-code any preference at all, then Apple must document all of the requirements in full at WWDC 2020. Apple must absorb the lessons of 2019 and have all the resources that developers need ready when those developers learn of new requirements.
Signing and notarization will get easier, faster, and become less of a burden. But notarization, or at least the features and services it supplies, will not go away. City dwellers cannot return to the days when it was safe to leave all the doors and windows unlocked at night; mobile phone owners find that passcodes or similar biometric security features are no longer optional, and notarization is here to stay. As the Attitudinal noted last time, it’s all but given that Apple wants malware and adware to be notarized so examples in the wild can be deactivated.
Not “stopped”—just killed when found. As long as there are strong economic incentives behind adware and nastier malware, they can’t be stopped. We can’t make cockroaches extinct or outlaw cancer either, and we’re even testing ourselves with SARS-CoV-2 (the virus that causes COVID-19). Apple can, however, make as many programs as possible go through a single gate, and staff that gate with a single Gatekeeper that has the authority to deny admission to known malefactors. Not suspected ones, not theoretical ones, but those proven to violate Apple’s published guidelines.
Welcome to the new normal.