Learning by Shipping

LinkedIn engineer Martin Kleppmann wrote a wonderful post detailing the magical and thoughtful engineering behind the new LinkedIn Intro iOS app.  I was literally verklepmpt reading the post–thinking about all those nights trying different things until he (and the team) ultimately achieved what he set out to do, what his management hoped he would do, and what folks at LinkedIn felt would be great for LinkedIn customers.

The internet has done what the internet does which is to unleash indignation upon Martin, LinkedIn, and thus the cycle begins. The post was updated with caveats and disclaimers.  It is now riding atop of techmeme.  Privacy.  Security. etc.

Whether those concerns are legitimate or not (after all this is a massive public company based on the trust of a network), the reality is this app points out a longstanding architectural challenge in API design.  The rise of modern operating systems (iOS, Android, Windows RT, and more) have inherent advantages over the PC-era operating systems (OS X, Windows, Linux) when it comes to maintaining the integrity as designed of the system overall. Yet we’re not done innovating around this challenge.

History

I remember my very first exploit.  I figured out how to use a disk sector editor on CP/M and modified the operating system to remove the file delete command, ERA.  I managed to do this by just nulling out the “ERA” string in what appeared to me to be the command table.  I was so proud of myself I (attempted) to show my father my success.

The folks that put the command table there were just solving a problem.  It was not an API to CP/M, or was it?  The sector editor was really a tool for recovering information from defective floppies, or was it?  My goal was to make a floppy with WordStar on it that I could give to my father to use but would be safe from him accidentally deleting a file.  My intention was good.  I used information and tools available to me in ways that the system architects clearly did not intend.  I stood on the top step of a ladder.  I used a screwdriver as a pry bar.  I used a wrench as a hammer.

The history of the PC architecture is filled with examples of APIs exposed for one purpose put to use for another purpose.  In fact, the power of the PC platform is a result of inventors bringing technology to market with one purpose in mind and then seeing it get used for other purposes.  Whether hardware or software, unintended uses of extensibility have come to define the flexibility, utility, and durability of the PC architecture.  There are so many examples: the first terminate and stay resident programs in MS-DOS, the Z80 softcard for the Apple ][, drawing low voltage power from USB to power a coffee warmer, all the way to that most favorite shell extension in Windows or OS X extension that adds that missing feature from Finder.

These are easily described and high-level uses of extensibility.  Your everyday computing experience is literally filled with uses of underlying extensibility that were not foreseen by the original designers. In fact, I would go as far as to say that if computers and software were only allowed to do things that the original designers intended, computing would be particularly boring.

Yet it would also be free of viruses, malware, DLL hell, system rot, and TV commercials promising to make your PC faster.

Take for example, the role of extensibility in email, Outlook even in particular.  The original design for Outlook had a wonderful API that enabled one to create an add-in that would automate routine tasks in Outlook.  You could for example have a program that would automatically send out a notification email to the appropriate contacts based on some action you would take.  You could also receive useful email attachments that could streamline tasks just by opening them (for example, before we all had a PDF reader it was very common to receive an executable that when opened would self-extract a document along with a viewer).  These became a huge part of the value of the platform and an important part of the utility of the PC in the workplace at the time.

Then one day in 1999 we all (literally) received email from our friend Melissa.  This was a virus that spread by using these same APIs for an obviously terrible usage.  What this code did was nothing different than all those add-ins did, but it did it at Internet scale to everyone in an unsuspecting way.

Thus was born the age of “consent” on PCs.  When you think about all those messages you see today (“use your location”, “change your default”, “access your address book”) you see the direct descendants of Melissa. A follow on virus professed broad love for all of us, I LOVE YOU.  From that came the (perceived) draconian steps of simply disabling much of the extensibility/utility described above.

What else could be done?  A ladder is always going to have a top step–some people will step on it.  The vast majority will get work done and be fine.

From my perspective, it doesn’t matter how one perceives something on a spectrum from good to “bad”–the challenge is APIs get used for many different things and developers are always going to push the limits of what they do.  LinkedIn Intro is not a virus.  It is not a tool to invade your privacy.  It is simply a clever (ne hack) that uses existing extensibility in new ways.  There’s no defense against this.  The system was not poorly designed.  Even though there was no intent to do what Intro did when those services were designed, there is simply no way to prevent clever uses anymore than you can prevent me from using my screwdriver as a pry bar.

Modern example

I wanted to offer a modern example that for me sums up the exploitation of APIs and also how challenging this problem is.

On Android an app can add one or more sharing targets.  In fact Android APIs were even improved to make it easier in release after release and now it is simply a declarative step of a couple of lines of XML and some code.

As a result, many Play apps add several share targets.  I installed a printing app that added 4 different ways to share (Share link, share to Chrome, share email, share over Bluetooth).  All of these seemed perfectly legitimate and I’m sure the designers thought they were just making their product easier to use.  Obviously, I must want to use the functionality since I went to the Play store, downloaded it and everything.  I bet the folks that designed this are quite proud of how many taps they saved for these key scenarios.

After 20 apps, my share list is crazy.  Of course sharing with twitter is now a lot of scrolling because the list is alphabetical.  Lucky for me the Messages app bubbles up the most recent target to a shortcut in the action bar.  But that seems a bit like a kludge.

Then along comes Andmade Share.  It is another Play app that lets me customize the share list and remove things.  Phew.  Except now I am the manager of a sharing list and every time I install an app I have to go and “fix” my share target list.

Ironically, the Andmade app uses almost precisely the same extensibility to manage the sharing list as is used to pollute it.  So hypothetically restricting/disabling the ability of apps to add share targets also prevents this utility from working.

The system could also be much more rigorous about what can be added.  For example, apps could only add a single share target (Windows 8) or the OS could just not allow apps to add more (essentially iOS).  But 99% of uses are legitimate.  All are harmless.  So even in “modern” times with modern software, the API surface area can be exploited and lead to a degraded user experience even if that experience degrades in a relatively benign way.

Anyone that ever complained about startup programs or shell extensions is just seeing the results of developers using extensibility.  Whether it is used or abused is a matter of perspective.  Whether is degrades the overall system is dependent on many factors and also on perspective (since every benefit has a potential cost, if you benefit from a feature then you’re ok with the cost).

Reality

There will be calls to remove the app from the app store. Sure that can be done. Steps will be taken to close off extensibility mechanisms that got used in ways far off the intended usage patterns. There will be cost and unintended side effects of those actions. Realistically, what was done by LinkedIn (or a myriad of examples) was done with the best of intentions (and a lot of hard work).  Realistically, what was done was exploiting the extensibility of the system in a way never considered by the designers (or most users).

This leads to 5 realities of system design:

1. Everything is an API.  Every bit of a system is an API.  From the layout of files, to the places settings are stored, to actual published APIs, everything in a system as it is released serves as an interface to people who want to extend, customize, or modify your work. Services don’t escape this because APIs are in a cloud behind REST APIs.  For example, reverse engineering packets or scraping HTML is no different — the HTML used by a site can come to be relied on essentially as an API.  The Windows registry is just a place to store stuff–the fact that people went in and modified it outside the intended parameters is what caused problems, not the existence of a place to store stuff.  Cookies?  Just a mechanism.

2. APIs can’t tell you the full intent.  APIs are simply tools.  The documentation and examples show you the mainstream or an intended use of an API.  But they don’t tell you all the intended uses or even the limits of using an API.  As a platform provider, falling back on documentation is fairly impossible considering both the history of software platforms (and most of the success of a platform coming from people using it in a creative ways) and the reality that no one could read all the documentation that would have to explain all the uses of a single API when there are literally tens of thousands of extensibility points (plus all the undocumented ones, see #1).

3. Once discovered, any clever use of an API will be replicated by many actors for good or not.  Once one developer finds a way to get something done by working through the clever mechanism of extensibility, if there’s value to it then others will follow. If one share target is good, then having 5 must be 5 times better.  The system through some means will ultimately need to find a way to control the very way extensibility or APIs are used.  Whether this is through policy or code is a matter of choice. We haven’t seen the last “Intro” at least until some action is taken for iOS.

4. Platform providers carry the burden of maintaining APIs over time.  Since the vast majority of actors are doing valuable things you maintain an API or extensibility point–that’s what constitutes a platform promise.  Some of your APIs are “undocumented” but end up being conventions or just happenstance.  When you produce a platform, try as hard as you want to define what is the official platform and what isn’t but your implied promise is ultimately to maintain the integrity of everything overall.

5. Using extensibility will produce good and bad results, but what is good and bad will depend highly on the context.  It might seem easy to judge something broadly on the internet as good or bad.  In reality, downloading an app and opt-ing in.  What should you really warn about and how?  To me this seems remarkably difficult.  I am not sure we’re in a better place because every action on my modern device has a potential warning message or a choice from a very long list I need to manage.

We’re not there yet collectively as an industry on balancing the extensibility of platforms and the desire for safety, security, performance, predictability, and more.  Modern platforms are a huge step in a better direction.

Let’s be careful collectively about how we move forward when faced with a pattern we’re all familiar with.

–Steven

28-10-13 Fixed a couple of typos.