Innovation and disruption are the hallmarks of the technology world, and hardly a moment passes when we are not thinking, doing, or talking about these topics. While I was speaking with some entrepreneurs recently on the topic, the question kept coming up: “If we’re so aware of disruption, then why do successful products (or companies) keep getting disrupted?”
Good question, and here’s how I think about answering it.
As far back as 1962, Everett Rogers began his groundbreaking work defining the process and diffusion of innovation. Rogers defined the spread of innovation in the stages of knowledge, persuasion, decision, implementation and confirmation.
Those powerful concepts, however, do not fully describe disruptive technologies and products, and the impact on the existing technology base or companies that built it. Disruption is a critical element of the evolution of technology — from the positive and negative aspects of disruption a typical pattern emerges, as new technologies come to market and subsequently take hold.
A central question to disruption is whether it is inevitable or preventable. History would tend toward inevitable, but an engineer’s optimism might describe the disruption that a new technology can bring more as a problem to be solved.
Four Stages of Disruption
For incumbents, the stages of innovation for a technology product that ultimately disrupt follow a pattern that is fairly well known. While that doesn’t grant us the predictive powers to know whether an innovation will ultimately disrupt, we can use a model to understand what design choices to prioritize, and when. In other words, the pattern is likely necessary, but not sufficient to fend off disruption. Value exists in identifying the response and emotions surrounding each stage of the innovation pattern, because, as with disruption itself, the actions/reactions of incumbents and innovators play important roles in how parties progress through innovation. In some ways, the response and emotions to undergoing disruption are analogous to the classic stages of grieving.
Rather than the five stages of grief, we can describe four stages that comprise theinnovation pattern for technology products: Disruption of incumbent; rapid and linear evolution; appealing convergence; and complete reimagination. Any product line or technology can be placed in this sequence at a given time.
The pattern of disruption can be thought of as follows, keeping in mind that at any given time for any given category, different products and companies are likely at different stages relative to some local “end point” of innovation.
Stage One: Disruption of Incumbent
A moment of disruption is where the conversation about disruption often begins, even though determining that moment is entirely hindsight. (For example, when did BlackBerry get disrupted by the iPhone, film by digital imaging or bookstores by Amazon?) A new technology, product or service is available, and it seems to some to be a limited, but different, replacement for some existing, widely used and satisfactory solution. Most everyone is familiar with this stage of innovation. In fact, it could be argued that most are so familiar with this aspect that collectively our industry cries “disruption” far more often than is actually the case.
From a product development perspective, choosing whether a technology is disruptive at a potential moment is key. If you are making a new product, then you’re “betting the business” on a new technology — and doing so will be counterintuitive to many around you. If you have already built a business around a successful existing product, then your “bet the business” choice is whether or not to redirect efforts to a new technology. While difficult to prove, most would generally assert that new technologies that are subsequently disruptive are bet on by new companies first. The very nature of disruption is such that existing enterprises see more downside risk in betting the company than they see upside return in a new technology. This is the innovator’s dilemma.
The incumbent’s reactions to potential technology disruptions are practically cliche. New technologies are inferior. New products do not do all the things existing products do, or are inefficient. New services fail to address existing needs as well as what is already in place. Disruption can seem more expensive because the technologies have not yet scaled, or can seem cheaper because they simply do less. Of course, the new products are usually viewed as minimalist or as toys, and often unrelated to the core business. Additionally, business-model disruption has similar analogues relative to margins, channels, partners, revenue approaches and more.
The primary incumbent reaction during this stage is to essentially ignore the product or technology — not every individual in an organization, but the organization as a whole often enters this state of denial. One of the historical realities of disruption is uncovering the “told you so” evidence, which is always there, because no matter what happens, someone always said it would. The larger the organization, the more individuals probably sent mail or had potential early-stage work that could have avoided disruption, at least in their views (see “Disruption and Woulda, Coulda, Shoulda” and the case of BlackBerry). One of the key roles of a company is to make choices, and choosing change to a more risky course versus defense of the current approaches are the very choices that hamstring an organization.
There are dozens of examples of disruptive technologies and products. And the reactions (or inactions) of incumbents are legendary. One example that illustrates this point would be the introduction of the “PC as a server.” This has all of the hallmarks of disruption. The first customers to begin to use PCs as servers — for application workloads such as file sharing, or early client/server development — ran into incredible challenges relative to the mini/mainframe computing model. While new PCs were far more flexible and less expensive, they lacked the reliability, horsepower and tooling to supplant existing models. Those in the mini/mainframe world could remain comfortable observing the lack of those traits, almost dismissing PC servers as not “real servers,” while they continued on their path further distancing themselves from the capabilities of PC servers, refining their products and businesses for a growing base of customers. PCs as servers were simply toys.
At the same time, PC servers began to evolve and demonstrate richer models for application development (rich client front-ends), lower cost and scalable databases, and better economics for new application development. With the rapidly increasing demand for computing solutions to business problems, this wave of PC servers fit the bill. Soon the number of new applications written in this new way began to dwarf development on “real servers,” and the once-important servers became legacy relative to PC-based servers for those making the bet or shift. PC servers would soon begin to transition from disruption to broad adoption, but first the value proposition needed to be completed.
Stage Two: Rapid Linear Evolution
Once an innovative product or technology begins rapid adoption, the focus becomes “filling out” the product. In this stage, the product creators are still disruptors, innovating along the trajectory they set for themselves, with a strong focus on early-adopter customers, themselves disruptors. The disruptors are following their vision. The incumbents continue along their existing and successful trajectory, unknowingly sealing their fate.
This stage is critically important to understand from a product-development perspective. As a disruptive force, new products bring to the table a new way of looking at things — a counterculture, a revolution, an insurgency. The heroic efforts to bring a product or service to market (and the associated business models) leave a lot of room left to improve, often referred to as “low-hanging fruit.” The path from where one is today to the next six, 12, 18 months is well understood. You draw from the cutting-room floor of ideas that got you to where you are. Moving forward might even mean fixing or redoing some of the earlier decisions made with less information, or out of urgency.
Generally, your business approach follows the initial plan, as well, and has analogous elements of insurgency. Innovation proceeds rapidly in this point. Your focus is on the adopters of your product — your fellow disruptors (disrupting within their context). You are adding features critical to completing the scenario you set out to develop.
To the incumbent leaders, you look like you are digging in your heels for a losing battle. In their view, your vector points in the wrong direction, and you’re throwing good money after bad. This only further reinforces the view of disruptors that they are heading in the right direction. The previous generals are fighting the last war, and the disruptors have opened up a new front. And yet, the traction in the disruptor camp becomes undeniable. The incumbent begins to mount a response. That response is somewhere between dismissive and negative, and focuses on comparing the products by using the existing criteria established by the incumbent. The net effect of this effort is to validate the insurgency.
Stage Three: Appealing Convergence
As the market redefinition proceeds, the category of a new product starts to undergo a subtle redefinition. No longer is it enough to do new things well; the market begins to call for the replacement of the incumbent technology with the new technology. In this stage, the entire market begins to “wake up” to the capabilities of the new product.
As the disruptive product rapidly evolves, the initial vision becomes relatively complete (realizing that nothing is ever finished, but the scenarios overall start to fill in). The treadmill of rapidly evolving features begins to feel somewhat incremental, and relatively known to the team. The business starts to feel saturated. Overall, the early adopters are now a maturing group, and a sense of stability develops.
Looking broadly at the landscape, it is clear that the next battleground is to go after the incumbent customers who have not made the move. In other words, once you’ve conquered the greenfield you created, you check your rearview mirror and look to pick up the broad base of customers who did not see your product as market-ready or scenario-complete. To accomplish this, you look differently at your own product and see what is missing relative to the competition you just left in the dust. You begin to realize that all those things your competitor had that you don’t may not be such bad ideas after all. Maybe those folks you disrupted knew something, and had some insights that your market category could benefit from putting to work.
In looking at many disruptive technologies and disruptors, the pattern of looking back to move forward is typical. One can almost think of this as a natural maturing; you promise never to do some of the things your parents did, until one day you find yourself thinking, “Oh my, I’ve become my parents.” The reason that products are destined to converge along these lines is simply practical engineering. Even when technologies are disrupted, the older technologies evolved for a reason, and those reasons are often still valid. The disruptors have the advantage of looking at those problems and solving them in their newly defined context, which can often lead to improved solutions (easier to deploy, cheaper, etc.) At the same time, there is also a risk of second-system syndrome that must be carefully monitored. It is not uncommon for the renegade disruptors, fresh off the success they have been seeing, to come to believe in broader theories of unification or architecture and simply try to get too much done, or to lose the elegance of the newly defined solution.
Stage Four: Complete Reimagination
The last stage of technology disruption is when a category or technology is reimagined from the ground up. While one can consider this just another disruption, it is a unique stage in this taxonomy because of the responses from both the legacy incumbent and the disruptor.
Reimagining a technology or product is a return to first principles. It is about looking at the underlying assumptions and essentially rethinking all of them at once. What does it mean to capture an image,provide transportation, share computation, search the Web, and more? The reimagined technology often has little resemblance to the legacy, and often has the appearance of even making the previous disruptive technology appear to be legacy. The melding of old and new into a completely different solution often creates whole new categories of products and services, built upon a base of technology that appears completely different.
To those who have been through the first disruption, their knowledge or reference frame seems dated. There is also a feeling of being unable to keep up. The words are known, but the specifics seem like rocket science. Where there was comfort in the convergence of ideas, the newly reimagined world seems like a whole new generation, and so much more than a competitor.
In software, one way to think of this is generational. The disruptors studied the incumbents in university, and then went on to use that knowledge to build a better mousetrap. Those in university while the new mousetrap was being built benefited from learning from both a legacy and new perspective, thus seeing again how to disrupt. It is often this fortuitous timing that defines generations in technologies.
Reimagining is important because the breakthroughs so clearly subsume all that came before. What characterizes a reimagination most is that it renders the criteria used to evaluate the previous products irrelevant. Often there are orders of magnitude difference in cost, performance, reliability, service and features. Things are just wildly better. That’s why some have referred to this as the innovator’s curse. There’s no time to bask in the glory of the previous success, as there’s a disruptor following right up on your heels.
A recent example is cloud computing. Cloud computing is a reimagination ofboth the mini/mainframe and PC-server models. By some accounts, it is a hybrid of those two, taking the commodity hardware of the PC world and the thin client/data center view of the mainframe world. One would really have to squint in order to claim it is just that, however, as the fundamental innovation in cloud computing delivers entirely new scale, reliability and flexibility, at a cost that upends both of those models. Literally every assumption of the mainframe and client/server computing was revisited, intentionally or not, in building modern cloud systems.
For the previous incumbent, it is too late. There’s no way to sprinkle some reimagination on your product. The logical path, and the one most frequently taken, is to “mine the installed base,” and work hard to keep those customers happy and minimize the mass defections from your product. The question then becomes one of building an entirely new product that meets these new criteria, but from within the existing enterprise. The number of times this has been successfully accomplished is diminishingly small, but there will always be exceptions to the rule.
For the previous disruptor and new leader, there is a decision point that is almost unexpected. One might consider the drastic — simply learn from what you previously did, and essentially abandon your work and start over using what you learned. Or you could be more incremental, and get straight to the convergence stage with the latest technologies. It feels like the ground is moving beneath you. Can you converge rapidly, perhaps revisiting more assumptions, and show more flexibility to abandon some things while doing new things? Will your product architecture and technologies sustain this type of rethinking? Your customer base is relatively new, and was just feeling pretty good about winning, so the pressure to keep winning will be high. Will you do more than try to recast your work in this new light?
The relentless march of technology change comes faster than you think.
So What Can You Do?
Some sincerely believe that products, and thus companies, disrupt and then are doomed to be disrupted. Like a Starship captain when the shields are down, you simply tell all hands to brace themselves, and then see what’s left after the attack. Business and product development, however, are social sciences. There are no laws of nature, and nothing is certain to happen. There are patterns, which can be helpful signposts, or can blind you to potential actions. This is what makes the technology industry, and the changes technology bring to other industries, so exciting and interesting.
The following table summarizes the stages of disruption and the typical actions and reactions at each stage:
|Disruption of Incumbent||Introduces new product with a distinct point of view, knowing it does not solve all the needs of the entire existing market, but advances the state of the art in technology and/or business.||New product or service is not relevant to existing customers or market, a.k.a. “deny.”|
|Rapid linear evolution||Proceeds to rapidly add features/capabilities, filling out the value proposition after initial traction with select early adopters.||Begins to compare full-featured product to new product and show deficiencies, a.k.a. “validate.”|
|Appealing Convergence||Sees opportunity to acquire broader customer base by appealing to slow movers. Sees limitations of own new product and learns from what was done in the past, reflected in a new way. Potential risk is being leapfrogged by even newer technologies and business models as focus turns to “installed base” of incumbent.||Considers cramming some element of disruptive features to existing product line to sufficiently demonstrate attention to future trends while minimizing interruption of existing customers, a.k.a. “compete.” Potential risk is failing to see the true value or capabilities of disruptive products relative to the limitations of existing products.|
|Complete Reimagining||Approaches a decision point because new entrants to the market can benefit from all your product has demonstrated, without embracing the legacy customers as done previously. Embrace legacy market more, or keep pushing forward?||Arguably too late to respond, and begins to define the new product as part of a new market, and existing product part of a larger, existing market, a.k.a. “retreat.”|
Considering these stages and reactions, there are really two key decision points to be tuned-in to:
When you’re the incumbent, your key decision is to choose carefully what you view as disruptive or not. It is to the benefit of every competitor to claim they are disrupting your products and business. Creating this sort of chaos is something that causes untold consternation in a large organization. Unfortunately, there are no magic answers for the incumbent.
The business team needs to develop a keen understanding of the dynamics of competitive offerings, and know when a new model can offer more to customers and partners in a different way. More importantly, it must avoid an excess attachment to today’s measures of success.
The technology and product team needs to maintain a clinical detachment from the existing body of work to evaluate if something new is better, while also avoiding the more common technology trap of being attracted to the next shiny object.
When you’re the disruptor, your key decision point is really when and if to embrace convergence. Once you make the choices — in terms of business model or product offering — to embrace the point of view of the incumbent, you stand to gain from the bridge to the existing base of customers.
Alternatively, you create the potential to lose big to the next disruptor who takes the best of what you offer and leapfrogs the convergence stage with a truly reimagined product. By bridging to the legacy, you also run the risk of focusing your business and product plans on the customers least likely to keep pushing you forward, or those least likely to be aggressive and growing organizations. You run the risk of looking backward more than forward.
For everyone, timing is everything. We often look at disruption in hindsight, and choose disruptive moments based on product availability (or lack thereof). In practice, products require time to conceive, iterate and execute, and different companies will work on these at different paces. Apple famously talked about the 10-year project that was the iPhone, with many gaps, and while the iPad appears a quick successor, it, too, was part of that odyssey. Sometimes a new product appears to be a response to a new entry, but in reality it was under development for perhaps the same amount of time as another entry.
There are many examples of this path to disruption in technology businesses. While many seem “classic” today, the players at the time more often than not exhibited the actions and reactions described here.
As a social science, business does not lend itself to provable operational rules. As appealing as disruption theory might be, the context and actions of many parties create unique circumstances each and every time. There is no guarantee that new technologies and products will disrupt incumbents, just as there is no certainty that existing companies must be disrupted. Instead, product leaders look to patterns, and model their choices in an effort to create a new path.
Stages of Disruption In Practice
Digital imaging. Mobile imaging reimagined a category that disrupted film (always available, low-quality versus film), while converging on the historic form factors and usage of film cameras. In parallel, there is a wave of reimagination of digital imaging taking place that fundamentally changes imaging using light field technology, setting the stage for a potential leapfrog scenario.
- Retail purchasing. Web retailers disrupted physical retailers with selection, convenience, community, etc., ultimatelyconverging on many elements of traditional retailers (physical retail presence, logistics, house brands).
- Travel booking. Online travel booking is disrupting travel agents, then converging on historic models of aggregation and package deals.
- Portable music. From the Sony Walkman as a disruptor to the iPod and MP3 players, to mobile phones subsuming this functionality, and now to streaming playback, portable music has seen the disruptors get disrupted and incumbents seemingly stopped in their tracks over several generations. The change in scenarios enabled by changing technology infrastructure (increased storage, increased bandwidth, mobile bandwidth and more) have made this a very dynamic space.
- Urban transport. Ride sharing, car sharing, and more disruptive traditional ownership of vehicles or taxi services are in the process of converging models (such as Uber adding UberX.
- Productivity. Tools such as Quip, Box, Haiku Deck, Lucidchart, and more are being pulled by customers beyond early adopters to be compatible with existing tools and usage patterns. In practice, these tools are currently iterating very rapidly along their self-defined disruptive path. Some might suggest that previous disruptors in the space (OpenOffice, Zoho, perhaps even Google Docs) chose to converge with the existing market too soon, as a strategic misstep.
- Movie viewing. Netflix and others, as part of cord-cutting, with disruptive, low-priced, all-you-can-consume on-demand plans and producing their own content. Previous disruptors such as HBO are working to provide streaming and similar services, while constrained by existing business models and relationships.
- Messaging/communications apps. SMS, which many consider disruptive to 2000-era IM, is being challenged by much richer interactions that disrupt the business models of carrier SMS and feature sets afforded by SMS.
- Network infrastructure. Software-defined networking and cloud computing are reimagining the category of networking infrastructure, with incumbent players attempting to benefit from these shifts in the needs of customers. Incumbents at different levels are looking to adopt the model, while some providers see it as fundamentally questioning their assumptions.
Rather than predict anything that will suddenly appear at the end of 2014, this post offers some trends that are likely to double by some measure this next year.
This will turn out to be an exponential year in many technologies and what seems far-fetched could very easily be trends that are doubling in relatively short periods of time. We humans generally have a tough time modeling things doubling (why so many companies and products did not figure out how to embrace Moore’s law or the rise of mobile).
To fully embrace exponential change means looking at the assumptions in product development and considering how optimizations for the near term might prove to be futile when facing significant change. Within each trend, design or product choices are offered that might be worth considering in light of the trend.
- Low-cost/high-function devices. The seemingly endless march of the exponential Moore’s law will continue but include more than compute. Devices will put transistors to work for sensors, rich graphics, and discrete processors. These devices will continue to drop precipitously in price to what seem today like ridiculous levels such as we’ve seen at discount super stores this holiday shopping season in the US. If automobiles are any indication, we should not assume low price is equivalent to low quality for the long-term, as manufacturing becomes more capable of delivering quality at low price. The desire to aspire an even higher level of quality will remain for many and continue to support many price points and volumes. At the same time, the usage patterns across price points will vary dramatically and we will continue to see exponential growth in-depth usage as we have this holiday season with high-value devices. This makes for a fairly dramatic split and leaves a lot open to interpretation when it comes to market share in devices. Design: First, it is worth considering target customers with more granularity when looking at share, as the pure number of devices might not determine how much your service will get used, at least in the near term. Second, don’t expect differences in capability across price points to last very long as the pace of pulling capabilities from higher price points to lower will be relentless.
- Cloud productivity. Cloud (SaaS) productivity tools will routinely see exponential growth in active users. Tools that enable continuous productivity will rapidly expand beyond tech early adopters as viral effects of collaboration kick in. Products such as Asana, Quip, Paper, Mixpanel, Lucidchart, Haikudeck or others will see viral expansion kick-in. Established tools such as Evernote, Box, Dropbox, WhatsApp, and more with high active usage will see major increases in cross-organization work as they grow to become essential tools for whole organizations. Design: Don’t assume traditional productivity tools and assume new employees, vendors, and recent grads will default to cloud-first productivity.
- Cloud first becomes cloud-only. Enterprise software in 2013 was a dialog about on-premises or cloud. In 2014, the call for on-premises will rapidly shift to a footnote in the evolution of cloud. The capabilities of cloud-based services will have grown to such a degree, particularly in terms of collaboration and sharing, that they will dwarf anything that can be done within the confines of a single enterprise. Enterprises will look at the exponential growth in scale of multi-tenant systems and see these as assets that cannot be duplicated by even the largest enterprises. Design: Don’t distract with attempting to architect or committing to on-premises.
- WWAN communication tools. WWAN/4G messaging will come to dominate in usage by direct or integrated tools (WhatsApp, WeChat, iMessage, and more) relative to email and SMS. Email will increasingly be viewed as “fax” and SMS will be used for “official” communications and “form letters” as person to person begins to use much richer and more expressive (fun) tools. This shift contributes to the ability to switch to data-only larger screen devices. Design: Skip email notifications, rely on SMS only when critical (security and verification), and assume heterogeneity for messaging choices. Expect to see more tools building in messaging capabilities with context scoped to the app.
- Cross-platform challenge. This is the year that cross-platform development for the major modern platforms will become increasingly challenging and products will need to be developed with this in mind. It will become increasingly unwieldy to develop for both iOS and Android and natively integrate effectively and competitively with the platform. Visual changes and integration functionality will be such that “cross-platform layers” might appear to be a good choice today only to prove to be short-lived and obstacles to rapid and competitive development. New apps that are cross-platform “today” will see increasing gaps between releases on each platform and will see functionality not quite “right” for platforms. Ultimately, developers will need to pick their lead platforms or have substantial code bases across platforms and face the challenge of keeping functionality in sync. Design: Avoid attempting to abstract platforms as these are moving targets, and assume dual-platform is nearly 2X the work of a single platform for any amount of user experience and platform integration.
- Small screen/big screen divergence. With increasing use of cloud productivity, more products will arrive that are designed exclusively for larger screen devices. Platform creators will increasingly face challenges of maintaining the identical user experience for “phones” and for phablets and larger. Larger screen tablets will be more able to work with keyboard accessories that will further drive a desire for apps tuned along these lines along with changes to underlying platforms to more fully leverage more screen real estate. The converse will be that scenarios around larger screen tablets will shift away from apps designed for small screen phones–thus resetting the way apps are counted and valued. Design: Productivity scenarios should be considering committing to large screen design and leave room for potential of keyboard or other input peripherals.
- Urban living is digital living. With demographic shifts in urban living and new influx of urban residents, we will see a rapid rise in digital-only lives. Mobile platforms will be part of nearly every purchase or transaction. Anything requiring reservations, tickets, physical resources, delivery, or scheduling will only win the hearts and minds of the new urban if available via mobile. While today it seems inconvenient if one needs to resort to “analog” to use a service, 2014 is a year in which every service has a choice and those that don’t exist in a mobile world won’t be picked. Design: Consumer products and services will only exist if they can be acquired via mobile.
- Sharing becomes normal. With the resources available for sharing exceeding those available in traditional ways, 2014 will be the year in which sharing becomes normal and preferred for assets that are infrequently used and/or expensive. Government and corporate structures will be re-evaluated relative to sharing from autos to office space and more. Budget pressures, rapid increase in software capabilities, and environmental impact all contribute to this change. Design: Can your business share resources? What are you using that could be shared? Is the asset you sell or rent something that runs the risk of aggregation and sharing by a new entry?
- Phablets are normal. Today’s phablets seem like a tweener or oddity to some–between a large phone and a small tablet. In practice the desire to have one device serve as both your legacy phone (voice and SMS) as well as your main “goto” device for productivity and communication will become increasingly important. The reduction in the need for legacy communication will fuel the need to pivot closer to a larger screen all the time. Improvements in voice input and collaboration tools will make this scenario even more practical. In the short-term, the ability to pair a larger screen tablet with your phone-sized device for voice or SMS may arise in an effort to always use one device, and similarly smaller tablets will be able to assume phone functionality. Design: Don’t ignore the potential of this screen size combined with full connectivity as the single device, particularly in mobile first markets where this form has early traction.
- Storage quotas go away. While for most any uses today this is true in practice, 2014 will be a year in which any individual will see alternatives for unlimited cloud storage. Email, files, photos, applications, mobile backup and more will be embedded in the price of devices or services with additional capabilities beyond gigabytes. Design: Design for disk space usage in the cloud as you do on a mobile client, which is to say worry much more about battery life and user experience than saving a megabyte.
Amara’s Law states “we tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run”. We will see 2014 not as one year of progress, but as the culmination of the past 15 years of development of the consumer internet as “it all comes together” with incredibly rapid adoption of products and technologies that at once become more affordable, more ubiquitous, and more necessary for our work and personal lives.
It looks like 2014 is shaping up to be the long-term of 2000 that we might have underestimated.
Stay tuned and Happy New Year!
I love visiting Tokyo and have been lucky enough to visit dozens of times over many years. The consumer electronics industry has certainly had ups and downs recently, but a constant has been the leading edge consumer and business adoption of new technologies. From PCs in the workplace to broadband at home and smartphones (a subject of many humorous team meetings back pre-bubble when I clearly didn’t get it and was content with the magic of my BB 850!) Japan has always had a leading adoption curve even when not necessarily producing the products used globally.
This visit was about visiting the University of Tokyo and meeting with some entrepreneurs. That, however, doesn’t stop me from spending time observing what CE is being used in the workplace, on the subway, and most importantly for sale in the big stores such as Yodobashi, Bic, and Labi and of course the traditional stalls at Akihabara. The rapid adoption, market size, and proximity to Korea and China often mean many of the products seen are not yet widely available in the US/Europe or are just making their way over. There’s a good chance what is emphasized in the (really) big retail space is often a leading indicator for what will show up at CES in January.
If you’re not familiar with Yodobashi, here’s the flagship store in Akihabara – over 250,000 sq ft and visited by 10′s of millions of people every year. I was once fortunate enough to visit the underground operations center, and as a kid who grew up in Orlando it sure feels a lot like the secret underground tunnels of the Magic Kingdom!
With that in mind here are 10 observations (all on a single page). This is not statistical in any way, just what caught my eye.
- Ishikawa Oku lab. The main focus of the trip was to visit University of Tokyo. Included in that was a wonderful visit with Professor Ishikawa-san and his lab which conducts research on exploring parallel, high-speed, and real-time operations for sensory information processing. What is so amazing about this work is that it has been going on for 20 years starting with very small and very slow digital sensors and now with Moore’s law applied to image capture along with parallel processing amazing things are possible such as can be seen in some of these Youtube videos (with > 5 million views), see http://www.youtube.com/ishikawalab. More about the lab http://www.k2.t.u-tokyo.ac.jp/index-e.html.
- 4K Displays. Upon stepping off the escalator on the video floor, one is confronted with massive numbers of massive 4K displays. Every manufacturer has displays and touts 4K resolution along with their requisite tricks at upscaling. The prices are still relatively high but the selection is much broader than readily seen in the US. Last year 4K was new at CES and it seems reasonable to suspect that the show floor will be all 4K. As a footnote relative to last year, 3D was downplayed significantly. In addition, there are numerous 4K cameras on sale now, more so than the US.
- Digital still. The Fuji X and Leica rangefinder digital cameras are getting a lot of floorspace and it was not uncommon to see tourists snapping photos (for example in Meiji Garden). The point and shoot displays feature far fewer models with an emphasis on attributes that differentiate them from phones such as waterproof or ruggedized. There’s an element of nostalgia, in Japan in particular, driving a renewed popularity in this form factor.
- Nikon Df. This is a “new” DSLR with the same sensor as the D-800/D4 that is packaged in a retro form factor. The Nikon Df is definitely only for collectors but there was a lot of excitement for the availability on November 21. It further emphasized the nostalgia elements of photography as the form factor has so dramatically shifted to mobile phones.
- Apple presence in store. The Apple presence in the main stores was almost overwhelming. Much of the first floor and the strategic main entry of Yodobashi were occupied by the Apple store-within-a-store. There were large crowds and as you often see with fans of products, they are shopping the very products they own and are holding in their hands. There has always been a fairly consistent appreciation of the Apple design aesthetic and overall quality of hardware but the widespread usage did not seem to follow. To be balanced, one would have to take note of the substantial presence of the Nexus 5 in the stores, which was substantially and well-visited.
- PCs. The size of the PC display area, relative to mobile and iOS accessories, definitely increased over the past 7 months since I last visited. There were quite a large number of All-In-One designs (which have always been popular in Japan, yet somehow could never quite leap across the Pacific until Windows 8). There were a lot of very new Ultrabooks running Haswell chips from all the major vendors in the US, Japan, and China. Surface was prominently displayed.
- iPhone popularity. There was a ubiquity of the iPhone that is new. Android had gained a very strong foothold over the national brands that came with the transition to nationwide LTE. Last year there was a large Android footprint through Samsung handsets that was fairly visible on display and in use. While the Android footprint is clearly there, the very fast rise of iPhone, particularly the easily spotted iPhone 5s was impressive. The vast expanse of iPhone accessories for sale nearly everywhere supports the opportunity. A driver for this is that the leading carrier (DoCoMo) is now an iPhone supplier. Returning from town, I saw this article speaking to the rise of iOS in Japan recently, iPhone 5S/C made up 76% of new smartphone sales in Japan this October.
- Samsung Galaxy J. Aside from the Nexus 5, the Android phone being pushed quite a bit was the Samsung Galaxy J. This is a model only in Asia right now. It was quite nice. It sports an ID more iPhone-like (squared edges), available in 5c-like colors, along with the latest QC processor, 5″ HD display, and so on. It is still not running Kitkat of course. For me in the store, it felt better than a Galaxy S. Given the intricacies of the US market, I don’t know if we’ll see this one any time soon. The Galaxy Note can be seen “in the wild” quite often and there seems to be quite a lot of interest based on what devices on display people would stop and interact with.
- Tablets. Tablets were omnipresent. They were signage in stores, menus in restaurants, in use on the subway, and in use at every place where people were sitting down and eating/drinking/talking. While in the US we are used to asking “where are all the Android tablets”, I saw a lot of 7″ Android tablets in use in all of those places. One wouldn’t expect the low-priced import models to be visible but there are many Japan OEMs selling Android tablets that could be spotted. I also saw quite a few iPad Minis in use, particularly among students on the trains.
- Digital video. As with compact digital cameras, there was a rather extreme reduction in the number of dedicated video recorders. That said, GoPro cameras had a lot of retail space and accessories were well placed. For example, there were GoPros connected to all sorts of gear/showing off all sorts of accessories at Tokyu Hand (the world’s most amazing store, imho). Professional HD and UHD cameras are on display in stores which is cool to see, for example Red and Arri. One of the neatest uses of video which is available stateside but I had not seen is the Sony DEV-50 binoculars/camera. It is pricey (USD$2000) but also pretty cool if you’ve got the need for it. They have reasonable sensors, support 3D, and more. The only challenge is stability which make sense given the equivalent focal length, but there is image stabilization which helps quite a bit in most circumstances.
There were many other exciting and interesting products one could see in this most wired and gadget friendly city. One always is on the lookout for that unique gift this holiday season, so I found my stocking-stuffer. Below you can see a very effective EMF shielding baseball hat (note, only 90% effective). As a backup stocking-stuffer, all gloves purchased in Japan appear to be designed with resistive touch screens in mind :-)
PS: Here’s me with some super fun students in a class on Entrepreneurship and Innovation at the University of Tokyo.
Much has been written recently about performance ratings and management at some large and successful companies. Amazon has surfaced as a company implementing OLRs, organization and leadership reviews, which target the least effective 10% of an organization for appropriate action. Yahoo recently implemented QPRs, quarterly performance reviews, which rates people as “misses” or “occasionally misses” among other ratings. And just so we don’t think this is something unique to tech, every year about this time Wall St firms begin the annual bonus process which is filled with any number of legendary dysfunctions given the massive sums of money in play. Even the Air Force has a legendary process for feedback and appraisal.
This essay looks at the challenges of performance review in a large organization. The primary purpose is to help share the realities that designing and implementing a system for such an incredibly sensitive topic is a monumental challenge when viewed in isolation. If you overlay the environment of an organization (stock price, public perception, revenue or profit, local competition for talent, etc.) then any system at all can seem anywhere from tyrannical to fair to kick-ass for some period of time, and then swing the other way when the context changes. For as much as we think of performance management as numeric and thus perfectly quantifiable, it is as much a product of context and social science as the products we design and develop. We want quantitative certainty and simplicity, but context is crucial and fluid, and qualitative. We desire fair, which is a relative term, but insist on the truth, which is absolute.
While there is an endless quest for simplicity, much as with airline tickets, car prices, or tax codes it is naive to believe that simplicity can truly be achieved or maintained over time. The challenge doesn’t change the universally shared goal of simplicity (believe me HR people do not like complex systems any more than everyone else does) but as a practical matter such purity is unattainable. Therefore comparing any system to some ideal (“flat tax”, “fixed pricing”) only serves to widen the gap between desire and implementation and thus increases the frustration and even fear of a system.
If this topic were simple there would not be over 25,000 books listed on Amazon’s US book site for the query “performance review”. Worse, the top selling books are about how to write your review, game the system, impress your boss, or tell employees they are doing well when they really aren’t doing well. You get pretty far down the list before you get to books that actually try to define a system and even then those books are filled with caveats. My own view is that the best book on the topic is Measuring and Managing Performance in Organizations. It is not about the perfect review system but about the traps and pitfalls of just measuring stuff in general. I love this book because it is a reminder of everything you know about measuring, from “measure twice, cut once” to “measure what you can change” to “if you measure something it goes up” and so on.
Notes. I am not an HR professional and don’t get wrapped up in the nuances of terminology between “performance review” or “performance management” or “performance rating”. I recognize the differences and respect them but will tend to intermix the terms for the purposes of discussion. I also recognize that for the most part, people executing such a system generally don’t see the subtle distinctions in these words as much as they might mean something within the academy. I am also not a lawyer, so what I say here may or may not be legally permitted in your place of doing business (geography, company size, sector). Finally, this post is not about any specific company practice past or present and any similarity is unintended coincidence.
This post will say some things that are likely controversial or appear plain wrong to some. I’ll be following this on twitter to see what transpires, @stevesi.
5 Essential Realities
There are several essential realities to performance reviews:
- Performance systems conflate performance and compensation with organizational budgets. No matter how you look at it, one person cannot be evaluated and paid in isolation of budgets. The company as a whole has a budget for how much to pay people (salary, bonus, stock, etc.) No matter what an individual’s compensation is part of a system that ultimately has a budget. The vast majority of mechanical or quantitative effort in the system is not about one person’s performance but about determining how to pay everyone within the budget. While it is desirable to distinguish between professional development and compensation, that will almost certainly get lost once a person sees their compensation or once a manager has to assign a rating. Any suggestion as to how to be more fair, allow for more flexibility, provide more absolute ratings, or otherwise separate performance from compensation must still come up with a way to stick to a budget. The presence of a budget drives the existence of a system. There is always a budget and don’t be fooled by “found money” as that’s just a budget trick.
- In a group of any size there is a distribution of performance. At some point a group of people working towards similar goals will exhibit a distribution of performance. From our earliest days in school we see this with schoolwork. In the workplace there are infinite variables that influence the performance of any individual but the variability exists. In an ideal system one could isolate all the variables from some innate notion of “pure contribution” or “pure skill” in order to evaluate someone. But that can’t be done so the distribution one sees essentially lumps together many performance related variables.
- In a system where you have m labels for performance, people who get all but the most rewarding one believe they are “so close” to the higher one. In school, teachers have letter grades or numeric ranges that break up test scores into “buckets”. In the workplace, performance systems generally implement some notion of grades or ratings and assign distributions to each of those. Much like a forced curve in a physics test, the system says that only a certain percentage of a population can get the highest performance rating and likewise a certain percentage of the team gets the lowest rating. The result is that most everyone in the organization believes they are extremely close to the next rating much like looking at a test and thinking if you could just get that one extra point you’d get the next letter grade. Because of human nature, any such system almost certain follows the corollary that managers are likely to imply or one being managed likely to hear evidence of just how close a call their review score was. There is a corollary: “everyone believes they are above average“.
- Among any set of groups, almost all the groups think their group is delivering more and other groups are delivering less. In a company with many groups, managers generally believe their group as a whole is performing better by relevant measures and thus should not be held to the same distribution or should have a larger budget. Groups tend to believe their work is harder, more strategic, or just more valuable while underestimating those contributions from other groups. Once groups realize that there is a fixed budget, some strive to solve the overall challenges by allowing for higher budgets on some teams. In this way you could either use a different distribution of people (more at the top) or just elevate the compensation for people within a group. Any suggestion to do this would need to also provide guidance as to how groups as a whole are to be measured relative to each other (which sounds an awful lot like how individuals would be measured relative to each other).
- Measurement is not an absolute but is relative. To measure performance it must be measured relative to something. Sales is the “easiest” since if you have a sales quota then your compensation is just a measure of how much you beat the quota. Such simplicity masks the knife fight that is quota settings and the process by which a comp sheet is built out, but it is still a relative measure. Most product have squishier goals such as “finish the product” or “market the product”. The larger the company the more these goals make sense but the less any individual’s day to day actions are directly related (“If I fix this bug will the sale really close?”). Thus in a large company, goal setting becomes increasingly futile as it starts to look like “get my work done” as the interconnection between other people and their work is impossibly hard to codify. Much of the writing about performance reviews focuses on goal setting and the skill in writing goals you can always brag about, unfortunately. All of this has taken a rather dramatic turn with the focus on agility where it is almost the antithesis of well-run to state months in advance what success looks like. As a result, measuring performance relative to peers “doing their work” is far more reliable, but has the downside that the big goals all fall to the top level managers. That’s why for the most part this entire topic is a big company thing—in a startup or a small company, actions translate into sales, marketing, products, and customers all very directly.
10 Real World Attributes
Once you take these realities you realize there will in fact be some sort of system. The goal of the system is to figure out how much to pay people. For all the words about career management, feedback, and so on that is not what anyone really focuses on at the moment they check their “score”. It certainly isn’t what is going on around the table of managers trying to figure out how to fit their team within the rules of the system.
Those that look to the once a year performance rating as the place for either learning how they are doing or for sharing feedback with an employee are simply doing it wrong—there simply shouldn’t be surprises during the process. If there are surprises then that’s a mistake no system can fix. There are no substitutes for concrete, actionable, and ongoing feedback about performance. If you’re not getting that then you need to ask.
At the same time, you can’t expect to have a daily/weekly rating for how you are doing. That’s because your performance is relative to something and that something isn’t determined on a daily basis. Finding that happy place is a challenge for individuals and managers, with the burden to avoid surprises falling to both equally. As much as one expects a manager to communicate well individuals must listen well.
Putting a system in place for allocating compensation is enormously challenging. There’s simply too much at stake for the company, the team, managers, and individuals. Ironically, because so much is at stake that materially impacts the lives of people, it is not unusual for the routine implementation of the process to take months of a given year and for it to occupy far more brain cycles than the actual externally facing work of the organization. Ironically, the more you try to make the process something HR worries about the even more disconnected it becomes from work and the more stress. As a result, performance management occupies a disproportionate amount of time and energy in large organizations.
Because of this, everyone in a company has enough experience to be critical of the system and has ideas how to improve it. Much like when a company does TV advertising and everyone can offer suggestions—simply because we all watch TV and buy stuff—when it comes to performance reviews since we all do work and get reviewed we all can offer insights and perspectives on the system. Designing a system from scratch is rather different than being critical of anecdotes of an existing system.
Given that so much is at stake and everyone has ideas how to improve the system, the actual implementation is enormously complex. While one can attempt to codify a set of rules, one cannot codify the way humans will implement the rules. One can keep iterating, adding more and more rules, more checks and balances, but eventually a process that already takes too much time becomes a crushing burden. Even after all that, statistically a lot of people are not going to be happy.
Therefore the best bet with any system is to define the variables and recognize that choices are being made and that people will be working within a system that by definition is not perfect. One can view this as gaming the system, if one believes the outcome is not tilted towards goodness. Alternatively, one can view this as doing the right thing, if one believes the outcome is tilted in the direction of goodness.
My own experience, is that there are so many complexities it is pointless to attempt to fully codify a system. Rather everyone just goes in with open eyes and a realistic view of the difficulty of the challenge and iterates through the variables throughout the entire dialog. Fixating on any one to the exclusion of others is when ultimately the system breaks down.
The following are ten of the most common attributes that must be considered and balanced when developing a performance review system:
- Determining team size. There is critical mass of “like” employees (job function, seniority, familiarity, responsibility) required to make any system even possible. If you have less than about 100 people no system will really work. At the same time, at about 100 people you are absolutely assured of having a sample size large enough to see a diversity in performance. There is going to be a constant tension between employees who believe the only fair way of evaluation is to have intimate knowledge of their work and a system that needs a lot of data points. In practice, somewhere between 1 and 5 people are likely to have intimate understanding of the work of an individual, but said another way any given manager is likely to have intimate knowledge of between 5 and about 50 people. At some point the system requires every level of management to honestly assess people based on a dialog of imperfect information. Team size also matters because small “rounding” efforts become enormous. Imagine something where you need to find 10% of the population and you have a team of 15 people to do that with. You obviously pick either 1 or 2 people (1 if the 10% is “bad”, 2 if it is “good”). Then imagine this rolls up to 15,000 people. Rather than 1500, you have either 1000 or 2000 people in that 10%. That’s either very depressing or very expensive relative to the budget. Best practice: Implement a system in groups of about 100 in seniority and role.
- Conflating seniority, job function, and projects does not create a peer group. Attempting to define relative contribution of a college new hire and a 10 year industry vet, or a designer and a QA engineer, manager or not, or a front-end v. ops tools are all impossibly difficult. The dysfunction is one where invariably as the process moves up the management chain there will be a bias that builds—the most visible people, the highest paid people or jobs, scarcest talent, the work that is understood and so on will become the things that get airtime in dialogs. There’s nothing inherently evil about this but it can get very tricky very quickly if those dialogs lead to higher ratings/compensation for these dimensions. This can get challenging if these groups are not sized as above and so you’ll find it a necessary balancing act. Best practice: Define peer groups based on seniority and job function within project teams as best you can.
- Measuring against goals. It is entirely possible to base a system of evaluation and compensation on pre-determined goals. Doing so will guarantee two things. First, however much time you think you save on the review process you will spend up front on an elaborate process of goal-setting. Second, in any effort of any complexity there is no way to have goals that are self-contained and so failure to meet goals becomes an exercise in documenting what went wrong. Once everyone realizes their compensation depends on others, the whole work process becomes clouded by constant discussion about accountability, expectation setting, and other efforts not directly related to actually working things out. And worse, management will always have the out of saying “you had the goal so you should have worked it out”. There’s nothing more challenging in the process of evaluation than actually setting goals and all of this is compounded enormously when the endeavor is a creative one where agility, pivots, and learning are part of the overall process. Best practice: let individuals and their manager arrive at goals that foster a sense of mastery of skills and success of the project, while focusing evaluation on the relative (and qualitative) contribution to the broader mission.
- Understanding cross-organization performance. Performance measurement is always relative, but determining performance across multiple organizations in a relative sense requires apples to oranges comparisons, even within similar job functions (i.e. engineering). If one team is winding down a release and another starting, or if one team is on an established business and another on a new business, or if one team has no competitors and another is in an intense battle, or if one team has a lot of sales support and another doesn’t, and so on are all situations which make it non-obvious how to “compare” multiple teams, yet this is what must happen at some level. Compounding this situation is that at some point in evaluation the basis for relative comparison might dramatically change—for example, at one level of management the accomplishment of multiple teams might be looked at through a lens that can be far removed from what members of those teams might be able to impact in their daily work. Best practice: do not pit organizations against each other by competing for rewards and foster cross-group collaboration via planning and execution of shared bets.
- Maintaining a system that both rates and rewards. Systems often have some sort of score or a grade and they also have compensation. Some think this is essential. Some think this is redundant. Some care deeply about one, but only when they are either very happy or very unhappy with the other. A system can be developed where these are perfectly correlated in which case one can claim they are redundant. A system where there is a loose correlation might as well have no correlation because both individuals and managers involved are hearing what they want to hear or saying one thing and doing another. At the same time, we’re all conditioned for a “score” and somehow a bonus of 9.67% doesn’t feel like a score because you don’t know what this means relative (so even though people want to be rated absolutely it doesn’t take long before they want to know where that stands relatively). Best practice: A clear rating that lets individuals know where they stand relative to their peer group along with compensation derived from that with the ability of a manager with the most intimate knowledge of the work to adjust compensation within some rating-defined range.
- Writing a performance appraisal is not making a case. Almost all of the books on Amazon about performance reviews focus on the art of writing reviews. Your performance review is not a trial and one can’t make or break the past year/month/quarter by an exercise in strong or creative writing. This holds for individuals hoping to make their performance shine and importantly for managers hoping to make up for their lack of feedback/action. The worst moments in a performance process are when an employee dissects a managers comments and attempts to refute them or when a manager pulls up a bunch of new examples of things that were not talked about when they were happening. Best practice: Lower the stakes for the document itself and make it clear that it is not the decision-making tool for the process.
- Ranking and calibrating are different. Much has been said about the notion of “stack rank” which often is used as a catch phrase for a process that assigns each member of a group a “one through n” score. This is always always a terrible process. There is simply no way to have the level of accuracy implied by such a system. What would one say to someone trying to explain the difference between being number 63 and number 64 on a 100 person team? The practice of calibrating is one of relative performance between members of peer groups as described above. The size and number of these groups is fixed and when done with adequate population size can with near certainty avoid endless discussion over boundary cases. Best practice: Define performance groups where members of a team fall but do not attempt to rank with more granularity or “proximity” to other groups.
- Encouraging excellent teams. Most managers believe their teams are excellent teams, and uniquely so. Strong performers have been hired to the team. Weak performers are naturally managed out if they somehow made it on to the team. Results show this. It becomes increasingly difficult to implement a performance review system because organizations become increasingly strong and effective. This is how it should be. At the same time this cannot possibly be a permanent state (even the sports teams get new players that don’t pan out over the course of a season). In a dynamic system there will be some years where a team is truly excellent and some years where it is not, but you can’t really know that in an absolute sense. In fact, the most ossifying element of performance appraisal is to assume that a given team or given person has reached a point where they are just excellent in an absolute sense and thus the system no longer applies. Whether the team is 100 engineers just crushing it or an executive team firing on all cylinders, it is very tempting to say the system doesn’t apply. But if the system doesn’t apply you don’t really have a system. Perhaps your organization will have a concept of “tenure” or you have a job function primarily compensated by quota based on quantitative measures—those are ways to have different systems. Best practice: Make a system that applies to everyone or have multiple systems and clear rules how membership in different systems is determined.
- Allowing for exceptions creates an exception-based process. When a team adds all of the potential constraints up and attempt to finally close in on performance of individuals, there is a tendency to “feel the pain” of all the rules and to create a model for exceptions. For example, you might have a 10% group but allow for up to 1% exceptions. Doing so will invariably create either the 9% or 11% group depending on if it is better to except up or down. If managers have the option of giving someone a low rating by extra money along with other people getting a high rating with less money, then invariably most people will get this mixed message. All of these exceptions quickly permeate an organization and individuals end up considering getting an exception a normal part of the process. Best practice: If there is going to be a system, then stick to it and don’t encourage exceptions.
- Embracing diversity in all dimensions. Far too often in performance appraisal and rewards, even within peer groups there is potential for the pull of sameness. This can manifest itself through any number of professional characteristics that can be viewed as either style or actual performance traits. One of the earliest stories I heard of this was about a manager that preferred people to set very aggressive goals for adding features. Unfortunately there was no measure for quality of the work. Other members of the team would focus on a combination of features and quality. Members of the latter group felt penalized relative to the person with the high bug count. At the same time, the team tended to be one that got a lot of features done early but had a much longer tail. Depending on when performance reviews got done, the story could be quite different. Perhaps both styles of work are acceptable, but not appreciating the “perceived slow and steady” is a failing of that manager to embrace styles. The same can be said for personal traits such as the always present quiet v. loud, or oral v. written, and so. Best practice: Any strong and sustainable team will be diverse in all dimensions.
Much more could be said about the way performance appraisal and reward can and should work in organizations. Far too much of what is said is negative and assumes a tone dominated by us v. them or worse a view that this is all a very straight forward process that management just consistently gets wrong. Like so many things in business there is no right answer or perfect approach. If there was, then there would be one performance system that everyone would use and it would work all the time. There is not.
Some suggest that the only way to solve this problem is to just have a compensation budget and let some level of management be responsible. That is a manager just determines compensation for each member of a team based on their own criteria. This too is a system—the absence of a system is itself a system. In fact this is not a single system but n systems, one for each manager. Every group will arrive at a way to distribute money and ratings that meets the needs of that team. There will be peanut butter teams, there will be teams that do the “big bonus”, and more. There will even be teams that use the system as a recruiting tool.
As much as any system is maligned, having a system that is visible, has some framework, and a level of cross-organization consistency provides many benefits to the organization as whole. These benefits accrue even with all the challenges that also exist.
To end this post here are three survival tips for everyone, individuals and managers, going through a performance process that seems unfair, opaque, or crazy:
- No one has all the data. Individuals love to remind some level of management that they do not have all the data about a given employee. Managers love to remind people that they see more data points than any one individual. HR loves to remind people that they have competitive salary data for the industry. Executives remind people they have data for a lot of teams. The bottom line is that no one person has a complete picture of the process. This means everyone is operating with imperfect information. But it does not follow that everyone is operating imperfectly.
- Share success, take responsibility. No matter what is happening and in what context, everyone benefits when successes are shared and responsibility is taken. Even with an imperfect system, if you do well be sure to consider how others contributed and thank them as publicly as you can. If you think you are getting a bad deal, don’t push accountability away or point fingers, but look to yourself to make things better.
- Things work out in the end. Since no system is perfect it is tempting to think that one data point of imperfection is a permanent problem. Things will go wrong. We don’t talk about it much, but some people will get a rating and pay much higher than they probably deserve at some point. And yes, some people will have a tough time that they might not really deserve in hindsight. In a knowledge economy, talent wins out over time. No manager will hold one datapoint against a talented person who gracefully recovers from a misstep. It takes discipline and effort to work within a complex and imperfect system—this is actually one of the skills required for anyone over the course of a career. Whether it is project planning, performance management, strategic choices, management processes and more all of these are social science and all subject to context, error rates, and most importantly learning and iteration.
—Steven Sinofsky (@stevesi)
I’ve been surprised at the “feedback” I receive when I talk about products that compete with those made by Microsoft. While I spent a lot of time there, one thing I learned was just how important it is to immerse yourself in competitive products to gain their perspective. It helps in so many ways (see http://blog.learningbyshipping.com/2013/01/14/learning-from-competition/).
Dave Winer (@davewiner) wrote a thoughtful post on How the Times reviews tech today. As I reflected on the post, it seemed worth considering why this challenge might be unique to tech and how it relates to the use of competitive products.
When considering creative works, it takes ~two hours to see a film or slightly more for other productions. Even a day or two for a book. After which you can collect your thoughts and analysis and offer a review. Your collected experience in the art form is relatively easily recalled and put to good use in a thoughtful review.
When talking about technology products, the same approach might hold for casually used services or content consumption services. In considering tools for “intellectual work” as Winer described (loved that phrase), things start to look significantly different.Software tools (for “intellectual work”) are complex because they do complex things. In order to accomplish something you need to first have something to accomplish and then accomplish it. It is akin to reviewing the latest cameras for making films or the latest cookware for making food. While you can shoot a few frames or make a single meal, tools like these require many hours and different tasks. You shouldn’t “try” them as much as “use” them for something that really matters. Only then can you collect your thoughts and analysis.Because tools of depth offer many paths and ways to use them there is an implicit “model” to how they are used. Models take a time to adapt to. A cinematographer that uses film shouldn’t judge a digital camera after a few test frames and maybe not even after the first completed work.
The tools for writing, thinking, creating that exist today present models for usage. Whether it is a smartphone, a tablet, a “word processor”, or a photo editor these devices and accompanying software define models for usage that are sophisticated in how they are approached, the flow of control, and points of entry. They are hard to use because they do hard things.
The fact that many of those that write reviews rely on an existing set of tools, software, devices to for their intellectual pursuits implies that conceptual models they know and love are baked into their perspective. It means tools that come along and present a new way of working or seeing the technology space must first find a way to get a clean perspective.
This of course is not possible. One can’t unlearn something. We all know that reviewers are professionals and just as we expect a journalist covering national policy debates must not let their bias show, tech reviewers must do the same. This implicit “model bias” is much more difficult to overcome because it simply takes longer to see and use a product than it does to learn about and understand (but not necessarily practice) a point of view in a policy debate. The tell-tale sign of “this review composed on the new…” is great, but we also know right after the review the writer has the option of returning to their favorite way of working.
As an example, I recall the tremendous difficulty in the early days of graphical user interface word processors. The incumbent WordPerfect was a character based word processor that was the very definition of a word processor. The one feature that we heard relentlessly was called reveal codes which was a way of essentially seeing the formatting of the document as codes surrounding text (well today we think of that as HTML). Word for Windows was a WYSIWYG word processor in Windows and so you just formatted things directly. If it was bold on screen then it was implicitly surrounded by <B> and </B> (not literally but conceptually those codes).
Reviewers (and customers) time and time again felt Word needed reveal codes. That was the model for usage of a “word processor”. It was an uphill battle to move the overall usage of the product to a new level of abstraction. There were things that were more difficult in Word and many things much easier, but reveal codes was simply a model and not the answer to the challenges. The tech world is seeing this again with the rise of new productivity tools such as Quip, Box Notes, Evernote, and more. They don’t do the same things and they do many things differently. They have different models for usage.
At the business level this is the chasm challenge for new products. But at the reviewer level this is a challenge because it simply takes time to either understand or appreciate a new product. Not every new product, or even most, changes the rules of the predecessor successfully. But some do. The initial reaction to the iPhone’s lack of keyboard or even de-emphasizing voice calls shows how quickly everyone jumped to the then current definition of smartphone as the evaluation criteria.Unfortunately all of this is incompatible with the news cycle for the onslaught of new products or the desire to have a collective judgement by the time the event is over (or even before it starts).This is a difficult proposition. It starts to sound like blaming politicians for not discussing the issues. Or blaming the networks for airing too much reality tv. Isn’t is just as much what peole will click through as it is what reviewers would write about. Would anyone be interested in reading a Samsung review or pulling another ios 7 review after the 8 weeks of usage that the product deserves?
The focus on youth and new users as the baseline for review is simply because they do not have the “baggage” or “legacy” when it comes to appreciating a new product. The disconnect we see in excitement and usage is because new to the category users do not need to spend time mapping their model and just dive in and start to use something for what it was supposed to do. Youth just represents a target audience for early adopters and the fastest path to crossing the chasm.
Here are a few things on my to-do list for how to evaluate a new product. The reason I use things for a long time is because I think in our world with so many different models
- Use defaults. Quite a few times when you first approach a product you want to immediately customize it to make it seem like what you’re familiar with. While many products have customization, stick with the defaults as long as possible. Don’t like where the browser launching button is, leave there anyway. There’s almost always a reason. I find the changes in the default layout of iOS 6 v. 7 interesting enough to see what the shift in priorities means for how you use the product.
- Don’t fight the system. When using a new product, if something seems hard that used to seem easy then take a deep breath and decide it probably isn’t the way the product was meant to do that thing. It might even mean that the thing you’re trying to do isn’t necessarily something you need to do with the new product. In DOS WordPerfect people would use tables to create columns of text. But in Word there was a columns feature and using a table for a newsletter layout was not the best way to do that. Sure there needed to be “Help” to do this, but then again someone had to figure that out in WordPerfect too.
- Don’t jump to doing the complex task you already figured out in the old tool. Often as a torture test, upon first look at a product you might try to do the thing you know is very difficult–that side by side chart, reducing overexposed highlights, or some complex formatting. Your natural tendency will be to use the same model and steps to figure this out. I got used to one complicated way of using levels to reduce underexposed faces in photos and completely missed out on the “fill flash” command in a photo editor.
- Don’t do things the way you are used to. Related to this is tendency to use one device the way you were used to. For example, you might be used to going to the camera app and taking a picture then choosing email. But the new phone “prefers” to be in email and insert an image (new or just taken) into a message. It might seem inconvenient (or even wrong) at first, but over time this difference will go away. This is just like learning gear shift patterns or even the layout of a new grocery store perhaps.
- Don’t assume the designers were dumb and missed the obvious. Often connected to trying to do something the way you are used to is the reality that something might just seem impossible and thus the designers obviously missed something or worse. There is always a (good) chance something is poorly done or missing, but that shouldn’t be the first conclusion.
But most of all, give it time. It often takes 4-8 weeks to really adjust to a new system and the more expert you are the more time it takes. I’ve been using Macs on and off since before the product was released to the public, but even today it has taken me the better part of six months to feel “native”. It took me about 3 months of Android usage before I stopped thinking like an iPhone user. You might say I am wired too much or you might conclude it really does take a long time to appreciate a design for what it is supposed to do. I chuckle at the things that used to frustrate me and think about how silly my concerns were at day 0, day 7, and even day 30–where the volume button was, the charger orientation, the way the PIN worked, going backwards, and more.
LinkedIn engineer Martin Kleppmann wrote a wonderful post detailing the magical and thoughtful engineering behind the new LinkedIn Intro iOS app. I was literally verklepmpt reading the post–thinking about all those nights trying different things until he (and the team) ultimately achieved what he set out to do, what his management hoped he would do, and what folks at LinkedIn felt would be great for LinkedIn customers.
The internet has done what the internet does which is to unleash indignation upon Martin, LinkedIn, and thus the cycle begins. The post was updated with caveats and disclaimers. It is now riding atop of techmeme. Privacy. Security. etc.
Whether those concerns are legitimate or not (after all this is a massive public company based on the trust of a network), the reality is this app points out a longstanding architectural challenge in API design. The rise of modern operating systems (iOS, Android, Windows RT, and more) have inherent advantages over the PC-era operating systems (OS X, Windows, Linux) when it comes to maintaining the integrity as designed of the system overall. Yet we’re not done innovating around this challenge.
I remember my very first exploit. I figured out how to use a disk sector editor on CP/M and modified the operating system to remove the file delete command, ERA. I managed to do this by just nulling out the “ERA” string in what appeared to me to be the command table. I was so proud of myself I (attempted) to show my father my success.
The folks that put the command table there were just solving a problem. It was not an API to CP/M, or was it? The sector editor was really a tool for recovering information from defective floppies, or was it? My goal was to make a floppy with WordStar on it that I could give to my father to use but would be safe from him accidentally deleting a file. My intention was good. I used information and tools available to me in ways that the system architects clearly did not intend. I stood on the top step of a ladder. I used a screwdriver as a pry bar. I used a wrench as a hammer.
The history of the PC architecture is filled with examples of APIs exposed for one purpose put to use for another purpose. In fact, the power of the PC platform is a result of inventors bringing technology to market with one purpose in mind and then seeing it get used for other purposes. Whether hardware or software, unintended uses of extensibility have come to define the flexibility, utility, and durability of the PC architecture. There are so many examples: the first terminate and stay resident programs in MS-DOS, the Z80 softcard for the Apple ][, drawing low voltage power from USB to power a coffee warmer, all the way to that most favorite shell extension in Windows or OS X extension that adds that missing feature from Finder.
These are easily described and high-level uses of extensibility. Your everyday computing experience is literally filled with uses of underlying extensibility that were not foreseen by the original designers. In fact, I would go as far as to say that if computers and software were only allowed to do things that the original designers intended, computing would be particularly boring.
Yet it would also be free of viruses, malware, DLL hell, system rot, and TV commercials promising to make your PC faster.
Take for example, the role of extensibility in email, Outlook even in particular. The original design for Outlook had a wonderful API that enabled one to create an add-in that would automate routine tasks in Outlook. You could for example have a program that would automatically send out a notification email to the appropriate contacts based on some action you would take. You could also receive useful email attachments that could streamline tasks just by opening them (for example, before we all had a PDF reader it was very common to receive an executable that when opened would self-extract a document along with a viewer). These became a huge part of the value of the platform and an important part of the utility of the PC in the workplace at the time.
Then one day in 1999 we all (literally) received email from our friend Melissa. This was a virus that spread by using these same APIs for an obviously terrible usage. What this code did was nothing different than all those add-ins did, but it did it at Internet scale to everyone in an unsuspecting way.
Thus was born the age of “consent” on PCs. When you think about all those messages you see today (“use your location”, “change your default”, “access your address book”) you see the direct descendants of Melissa. A follow on virus professed broad love for all of us, I LOVE YOU. From that came the (perceived) draconian steps of simply disabling much of the extensibility/utility described above.
What else could be done? A ladder is always going to have a top step–some people will step on it. The vast majority will get work done and be fine.
From my perspective, it doesn’t matter how one perceives something on a spectrum from good to “bad”–the challenge is APIs get used for many different things and developers are always going to push the limits of what they do. LinkedIn Intro is not a virus. It is not a tool to invade your privacy. It is simply a clever (ne hack) that uses existing extensibility in new ways. There’s no defense against this. The system was not poorly designed. Even though there was no intent to do what Intro did when those services were designed, there is simply no way to prevent clever uses anymore than you can prevent me from using my screwdriver as a pry bar.
I wanted to offer a modern example that for me sums up the exploitation of APIs and also how challenging this problem is.
On Android an app can add one or more sharing targets. In fact Android APIs were even improved to make it easier in release after release and now it is simply a declarative step of a couple of lines of XML and some code.
As a result, many Play apps add several share targets. I installed a printing app that added 4 different ways to share (Share link, share to Chrome, share email, share over Bluetooth). All of these seemed perfectly legitimate and I’m sure the designers thought they were just making their product easier to use. Obviously, I must want to use the functionality since I went to the Play store, downloaded it and everything. I bet the folks that designed this are quite proud of how many taps they saved for these key scenarios.
After 20 apps, my share list is crazy. Of course sharing with twitter is now a lot of scrolling because the list is alphabetical. Lucky for me the Messages app bubbles up the most recent target to a shortcut in the action bar. But that seems a bit like a kludge.
Then along comes Andmade Share. It is another Play app that lets me customize the share list and remove things. Phew. Except now I am the manager of a sharing list and every time I install an app I have to go and “fix” my share target list.
Ironically, the Andmade app uses almost precisely the same extensibility to manage the sharing list as is used to pollute it. So hypothetically restricting/disabling the ability of apps to add share targets also prevents this utility from working.
The system could also be much more rigorous about what can be added. For example, apps could only add a single share target (Windows 8) or the OS could just not allow apps to add more (essentially iOS). But 99% of uses are legitimate. All are harmless. So even in “modern” times with modern software, the API surface area can be exploited and lead to a degraded user experience even if that experience degrades in a relatively benign way.
Anyone that ever complained about startup programs or shell extensions is just seeing the results of developers using extensibility. Whether it is used or abused is a matter of perspective. Whether is degrades the overall system is dependent on many factors and also on perspective (since every benefit has a potential cost, if you benefit from a feature then you’re ok with the cost).
There will be calls to remove the app from the app store. Sure that can be done. Steps will be taken to close off extensibility mechanisms that got used in ways far off the intended usage patterns. There will be cost and unintended side effects of those actions. Realistically, what was done by LinkedIn (or a myriad of examples) was done with the best of intentions (and a lot of hard work). Realistically, what was done was exploiting the extensibility of the system in a way never considered by the designers (or most users).
This leads to 5 realities of system design:
Everything is an API. Every bit of a system is an API. From the layout of files, to the places settings are stored, to actual published APIs, everything in a system as it is released serves as an interface to people who want to extend, customize, or modify your work. Services don’t escape this because APIs are in a cloud behind REST APIs. For example, reverse engineering packets or scraping HTML is no different — the HTML used by a site can come to be relied on essentially as an API. The Windows registry is just a place to store stuff–the fact that people went in and modified it outside the intended parameters is what caused problems, not the existence of a place to store stuff. Cookies? Just a mechanism.
APIs can’t tell you the full intent. APIs are simply tools. The documentation and examples show you the mainstream or an intended use of an API. But they don’t tell you all the intended uses or even the limits of using an API. As a platform provider, falling back on documentation is fairly impossible considering both the history of software platforms (and most of the success of a platform coming from people using it in a creative ways) and the reality that no one could read all the documentation that would have to explain all the uses of a single API when there are literally tens of thousands of extensibility points (plus all the undocumented ones, see #1).
Once discovered, any clever use of an API will be replicated by many actors for good or not. Once one developer finds a way to get something done by working through the clever mechanism of extensibility, if there’s value to it then others will follow. If one share target is good, then having 5 must be 5 times better. The system through some means will ultimately need to find a way to control the very way extensibility or APIs are used. Whether this is through policy or code is a matter of choice. We haven’t seen the last “Intro” at least until some action is taken for iOS.
Platform providers carry the burden of maintaining APIs over time. Since the vast majority of actors are doing valuable things you maintain an API or extensibility point–that’s what constitutes a platform promise. Some of your APIs are “undocumented” but end up being conventions or just happenstance. When you produce a platform, try as hard as you want to define what is the official platform and what isn’t but your implied promise is ultimately to maintain the integrity of everything overall.
Using extensibility will produce good and bad results, but what is good and bad will depend highly on the context. It might seem easy to judge something broadly on the internet as good or bad. In reality, downloading an app and opt-ing in. What should you really warn about and how? To me this seems remarkably difficult. I am not sure we’re in a better place because every action on my modern device has a potential warning message or a choice from a very long list I need to manage.
We’re not there yet collectively as an industry on balancing the extensibility of platforms and the desire for safety, security, performance, predictability, and more. Modern platforms are a huge step in a better direction.
Let’s be careful collectively about how we move forward when faced with a pattern we’re all familiar with.
28-10-13 Fixed a couple of typos.
We are trying to change a culture of compartmentalized, start-from-scratch style development here. I’m curious if there are any good examples of Enterprise “Open Source” that we can learn from.
—Question from reader with a strong history in engineering management
When starting a new product line or dealing with multiple existing products, there’s always a question about how to share code. Even the most ardent open source developers know the challenges of sharing code—it is easy to pick up a library of “done” code, not so hard to share something that you can snapshot, but remarkably difficult to share code that is also moving at a high velocity like your work.
Developers love to talk about sharing code probably much more than they love to share code in practice. Yet, sharing code happens all the time—everyone uses an OS, web server, programming languages, and more that are all shared code. Where it gets tricky is when the shared code is an integral part of the product you’re developing. That’s when shared code goes from “fastest way to get moving” to “a potential (difficult) constraint” or to “likely a critical path”. Ironically, this is usually more true inside of a single company where one team needs to “depend” on another team for shared code than it is on developers sharing code from outside the company.
Organizationally, sharing code takes on varying degrees of difficulty depending on the “org distance” between developers. For example, two developers working for the same manager don’t even think about “sharing code” as much as they think about “working together”. At the other end of the spectrum, developers on different products with different code bases (perhaps started at different times with early thoughts that the products were unrelated or maybe one code base was acquired) think naturally about shipping their code base and working on their product first and foremost.
This latter case is often viewed as an organizational silo—a team of engineering, testing, product, operations, design, and perhaps even separate marketing or P&L responsibility. This might be the preferred org design (focus on business agility) or it might be because of intrinsic org structures (like geography, history, leadership approach). The larger these types of organizations the more the “needs of the org” tend to trump the “needs of the code”.
Let’s assume everyone is well-meaning and would share code, but it just isn’t happening organically. What are 5 things the team overall can do?
Ship together. The most straight-forward attribute two teams can modify in order to effectively share code is to have a release/ship schedule that is aligned. Sharing code is the most difficult when one team is locked down and the other team is just getting started. Things get progressively easier the closer to aligned each team becomes. Even on very short cycles of 30-60 days, the difference in mindset about what code can change and how can quickly grow to be a share-stopper. Even when creating a new product alongside an existing product, picking a scheduling milestone that is aligned can be remarkably helpful in encouraging sharing rather than a “new product silo” which only digs a future hole that will need to be filled.
Organize together to engineer together. If you’re looking at trying to share code across engineering organizations that have an org distance that involves general management, revenue or P&L, or different products, then there’s an opportunity to use organization approaches to share code. When one engineering manager can look at a shared code challenge across all of his/her responsibilities there more of a chance that an engineering leader will see this as an opportunity rather than a tax/burden. The dialog about efficacy or reality of sharing code does not span managers or importantly disciplines, and the resulting accountability rests within straight-forward engineering functions. This approach has limits (the graph theory of org size as well as the challenges of organizing substantially different products together).
Allocate resources for sharing. A large organization that has enough resources to duplicate code turns out to be the biggest barrier to sharing code. If there’s a desire to share code, especially if this means re-architecting something that works (to replace it with some shared code, presumably with a mutual benefit) then the larger team has a built-in mechanism to avoid the shared code tax. As painful as it sounds, the most straight-forward approach to addressing this challenge is to allocate resources such that a team doesn’t really have the option to just duplicate code. This approach often works best when combined with organizing together, since one engineering manager can simply load balance the projects more effectively. But even across silos, careful attention (and transparency) to how engineering resources are spent will often make this approach attainable.
Establish provider/consumer relationships. Often shared code can look like a “shared code library” that needs to be developed. It is quite common and can be quite effective to form a separate team, a provider, that exists entirely to provide code to other parts of the company, a consumer. The consumer team will tend to look at the provider team as an extension to their team and all can work well. On the other hand, there are almost always multiple consumers (otherwise the code isn’t really shared) and then the challenges of which team to serve and when (and where requirements might come from) all surface. Groups dedicated to being the producers of shared code can work, but they can quickly take on the characteristics of yet another silo in the company. Resource allocation and schedules are often quite challenging with a priori shared code groups.
Avoid the technical buzz-saw. Developers given a goal to share code and a desire to avoid doing so will often resort to a drawn-out analysis phase of the code and/or team. This will be thoughtful and high-integrity. But one person’s approach to being thorough can also look to another as a delay or avoidance tactic. No matter how genuine the analysis might be, the reality is that it can come across as a technical buzz-saw making all but the most idealized code sharing impossible. My own experience has been that simply avoiding this process is best—a bake-off or ongoing suitability-to-task discussion will only drive a wedge between teams. At some level sharing code is a leap of faith that a lot of folks need to take and when it works everyone is happy and if it doesn’t there’s a good chance someone is likely to say “told you so”. Most every bet one makes in engineering has skeptics. Spending some effort to hear out the skeptics is critical. A winners/losers process is almost always a negative for all involved.
The common thread about all of these is that they all seem impossible at first. As with any initiative, there’s a non-zero cost to obtaining goals that require behavior change. If sharing code is important and not happening, there’s a good chance you’re working against some of the existing constraints in the approach. Smart and empowered teams act with the best intentions to balance a seemingly endless set of inbound issues and constraints, and shared code might just be one of those things that doesn’t make the cut.
Keeping in mind that at any given time an engineering organization is probably overloaded and at capacity just getting stuff done, there’s not a lot of room to just overlay new goals.
Sharing code is like sharing any other aspect of a larger team—from best practices in tools, engineering approaches, team management—things don’t happen organically unless there’s a uniform benefit across teams. The role of management is to put in place the right constraints that benefit the overall goals without compromising other goals. This effort requires ongoing monitoring and feedback to make sure the right balance is achieved.
For those interested in some history, this is a Harvard Business School case on the very early Office (paid article) team and the challenges/questions around organizing around a set of related products (hint, this only seems relatively straight-forward in hindsight).