Learning by Shipping

Access to rich usage data is something that is a defining element of modern product development.  From cars, to services, to communications the presence of data showing how, why, when products are used is informing how products are built and evolve.  To those developing products, data is an essential ingredient to the process.  But sometimes, choices made that are informed by data cause a bit of an uproar when there isn’t uniform agreement.

The front page of the Financial Times juxtaposed two data-driven stories that show just how tricky the role of data can be.  For Veronica Mars fans (myself included), this past week was an incredible success as a Kickstarter project raised millions to fund a full length movie.  Talk about data speaking loud and clear.

This same week Google announced the end of life of Google Reader, and as you can see from the headline there was some controversy (it is noted with irony that the headline points out that the twitterspehere is in a meltdown).  For all the 50,000 or more folks happy with the VM movie, it seems at least that many were unhappy about Google reader

Controversy

The role of data in product development is not without controversy.  In today’s world with abundant information from product development teams and analysis of that data, there is ample room to debate and dissect choices.  A few common arguments around the use of data include:

• Representation.  No data can represent all people using (or who will use) a product. So who was represented in the data?
• Forward or backward looking.  When looking at product usage, the data looks at how the product was used but not how it will be used down the road (assuming changes).  Is the data justifying the choice or informing the choice?
• Contextual.  The data depends on context in which it is collected, so if the user interface is sub-optimal or drives a certain pattern the data does not necessarily represent a valid conclusion.  Did the data consider that the collection was itself flawed?
• Counter-intuitive. The data is viewed as counter-intuitive and does not follow the conventional wisdom, so something must be wrong.  How could the data overlook what is obvious?
• Causation or correlation.  With data you can see an end state, but it is not always clear what caused the end-state.  If something is used a lot, crashes a lot, or is avoided there might be many reasons, most not readily apparent or at least open to debate, that cause that end-state.  Is the end-state coincident with the some variables or do those variables cause the end-state?

When a product makes a seemingly unpopular change, such as Google did with reader, some of more of these or other arguments are brought forward in the discussion of the choice.

In the case of Reader, the official blog stated “usage of Google Reader has declined”.  While it does seem obvious that data informed the choice, if one does not agree with the choice there is ample opportunity to dispute the conclusion.  Was the usage in absolute or relative decline?  Were specific (presumably anonymous) users slowing their usage?  What about the usage in a particular customer segment? The counter-intuitive aspect of the data showed, as most dialog pointed out strong first party usage among tech enthusiasts and reporters.

The candid disclosure of the use of data offered some transparency to the choice, but not complete transparency.  Could more data be provided?  Would that change how the product development choice was received?

Conversely, no one is disputing the success of the VM Kickstarter project (making a movie is similar to product development).  The data is clear, there is a high level of demand for the movie.  I know that fits my intuition as a fan of the series.  The fact that this data came via a popular (and transparent) web service only seems to validate our intuition. In this case, the data is seemingly solid.

While these are just two examples, they happened in the same week and show the challenges of data, transparency, and product development choices.  While data can inform choices, no one is saying it is the only way to make a choice or that those making products should only defer to data.  Product development is a complex mix of science and intuition.  Data represents part of that mix, but not the whole of it.

Data is not strategy

Ultimately, data contributes to product development but does not replace the very unscientific notion of what to build and why.  That’s the art of product development and how it intersects with business strategy.  This follows from the fact that products are developed today for use in the future.  The future is uncertain for your own product’s evolution, but all around you is uncertainty.  Data is an input to help you define a strategy or modify it, but cannot replace what is inherently the uncertain side of innovation.

In Eric Ries’ Lean Startup book (or http://en.wikipedia.org/wiki/Lean_Startup), there is an interesting discussion on how the role of data can contribute to an early stage product.  While the anecdote and approach are described in the context of a project very close to Eric, I think we can all see parallels to other products as well.  My intent is not to restate or recast the book, but to just reflect upon it a bit.

One part of developing a new product, as described, is to develop a minimum viable product (MVP) that does not reflect the final product but is just enough of the product to collect the maximum validated data about potential customers.

An interesting point in the description is how often the people that will use this early version of the product are enthusiasts or those especially motivated and forgiving about a product while under development.  The tricky user interface, complex sign-up, or missing error conditions and things like that might not matter to these folks, for example.

Not every product ultimately targets those customers—they are not super large in number relative to the broad consumer world, for example.  As you learn and collect validated data about your product strategy you will reach a critical point where you essentially abandon or turn away from the focus on enthusiasts and tilt towards a potentially larger customer base.

This is where your strategy comes into play.  You have iterated and validated.  Presumably these early users of your product have started to use or like what you have or at least pointed you in a direction.  Then you’ll take a significant turn or change—maybe the business model will change, maybe there will be different features, maybe the user interface will be different.  This is all part of taking the learning and turning it into a product and business.  The data informed these choices, but you did not just follow it blindly. Your product will reflect but not implement your MVP, usually.

But with these choices there will probably not be universal agreement, because even with the best validated data there can be different approaches to implementing the learning.

Data transparency

The use of data is critical to modern product development.  Every product of every kind should have mechanisms in place to learn from how the product is used in the real world (note, this is assuming very appropriate policies regarding the collection and usage of this data). This is not just about initial development, but evolution and maturing of the product as well.

If you’re going to use data to design and develop your product, and also talk about how the product was designed and developed, it is worth considering how you bring transparency to the process.  Too often, both within an organization and outside, data is used conveniently to support or make a point.  Why not consider how you could provide some level of detail that could reduce the obvious ways those that disagree with your approach might better understand things, especially for those that follow along and care deeply.  Some ideas:

• Provide absolute numbers for the size of the data set to avoid discussions about sample size.
• Provide a sense of statistical significance across customer types (was the data collected in one country, one type of device, etc.).
• Provide the opportunity for additional follow up discussion or other queries based on dialog.
• Overlay the strategic or social science choices you are making in addition to the data that informed the choices.

Transparency might not remove controversies but might be a useful tool to have an open dialog with those that share your passion for the product.

Using data as part of product development will never be free of debate or even controversy.  Products today are used across millions of people worldwide in millions of different ways, yet are often used through one software experience.  What works in one context doesn’t always work in another.  What works for one type of customer does not always work for another.  Yet those that make products need to ship and need to get products to market.

As soft as software is, it is also enormously complex.  The design, creation, and maintenance preclude at today’s state of the art an everything for everyone approach.  To make those choices a combination of data an intuition make a good approach to the social science of product development.

–Steven

PS: Please click the link on the first use of data for a discussion of data versus datum. :-)