Learning by Shipping

products, development, management…

Using data to inform strategy

with 15 comments

Financial Times front page showing Veronica Mars and also Google reader headlineAccess to rich usage data is something that is a defining element of modern product development.  From cars, to services, to communications the presence of data showing how, why, when products are used is informing how products are built and evolve.  To those developing products, data is an essential ingredient to the process.  But sometimes, choices made that are informed by data cause a bit of an uproar when there isn’t uniform agreement.

The front page of the Financial Times juxtaposed two data-driven stories that show just how tricky the role of data can be.  For Veronica Mars fans (myself included), this past week was an incredible success as a Kickstarter project raised millions to fund a full length movie.  Talk about data speaking loud and clear.

This same week Google announced the end of life of Google Reader, and as you can see from the headline there was some controversy (it is noted with irony that the headline points out that the twitterspehere is in a meltdown).  For all the 50,000 or more folks happy with the VM movie, it seems at least that many were unhappy about Google reader


The role of data in product development is not without controversy.  In today’s world with abundant information from product development teams and analysis of that data, there is ample room to debate and dissect choices.  A few common arguments around the use of data include:

  • Representation.  No data can represent all people using (or who will use) a product. So who was represented in the data?
  • Forward or backward looking.  When looking at product usage, the data looks at how the product was used but not how it will be used down the road (assuming changes).  Is the data justifying the choice or informing the choice?
  • Contextual.  The data depends on context in which it is collected, so if the user interface is sub-optimal or drives a certain pattern the data does not necessarily represent a valid conclusion.  Did the data consider that the collection was itself flawed?
  • Counter-intuitive. The data is viewed as counter-intuitive and does not follow the conventional wisdom, so something must be wrong.  How could the data overlook what is obvious?
  • Causation or correlation.  With data you can see an end state, but it is not always clear what caused the end-state.  If something is used a lot, crashes a lot, or is avoided there might be many reasons, most not readily apparent or at least open to debate, that cause that end-state.  Is the end-state coincident with the some variables or do those variables cause the end-state?

When a product makes a seemingly unpopular change, such as Google did with reader, some of more of these or other arguments are brought forward in the discussion of the choice.

In the case of Reader, the official blog stated “usage of Google Reader has declined”.  While it does seem obvious that data informed the choice, if one does not agree with the choice there is ample opportunity to dispute the conclusion.  Was the usage in absolute or relative decline?  Were specific (presumably anonymous) users slowing their usage?  What about the usage in a particular customer segment? The counter-intuitive aspect of the data showed, as most dialog pointed out strong first party usage among tech enthusiasts and reporters.

The candid disclosure of the use of data offered some transparency to the choice, but not complete transparency.  Could more data be provided?  Would that change how the product development choice was received?

Conversely, no one is disputing the success of the VM Kickstarter project (making a movie is similar to product development).  The data is clear, there is a high level of demand for the movie.  I know that fits my intuition as a fan of the series.  The fact that this data came via a popular (and transparent) web service only seems to validate our intuition. In this case, the data is seemingly solid.

While these are just two examples, they happened in the same week and show the challenges of data, transparency, and product development choices.  While data can inform choices, no one is saying it is the only way to make a choice or that those making products should only defer to data.  Product development is a complex mix of science and intuition.  Data represents part of that mix, but not the whole of it.

Data is not strategy

Ultimately, data contributes to product development but does not replace the very unscientific notion of what to build and why.  That’s the art of product development and how it intersects with business strategy.  This follows from the fact that products are developed today for use in the future.  The future is uncertain for your own product’s evolution, but all around you is uncertainty.  Data is an input to help you define a strategy or modify it, but cannot replace what is inherently the uncertain side of innovation.

In Eric Ries’ Lean Startup book (or http://en.wikipedia.org/wiki/Lean_Startup), there is an interesting discussion on how the role of data can contribute to an early stage product.  While the anecdote and approach are described in the context of a project very close to Eric, I think we can all see parallels to other products as well.  My intent is not to restate or recast the book, but to just reflect upon it a bit.

One part of developing a new product, as described, is to develop a minimum viable product (MVP) that does not reflect the final product but is just enough of the product to collect the maximum validated data about potential customers.

An interesting point in the description is how often the people that will use this early version of the product are enthusiasts or those especially motivated and forgiving about a product while under development.  The tricky user interface, complex sign-up, or missing error conditions and things like that might not matter to these folks, for example.

Not every product ultimately targets those customers—they are not super large in number relative to the broad consumer world, for example.  As you learn and collect validated data about your product strategy you will reach a critical point where you essentially abandon or turn away from the focus on enthusiasts and tilt towards a potentially larger customer base.

This is where your strategy comes into play.  You have iterated and validated.  Presumably these early users of your product have started to use or like what you have or at least pointed you in a direction.  Then you’ll take a significant turn or change—maybe the business model will change, maybe there will be different features, maybe the user interface will be different.  This is all part of taking the learning and turning it into a product and business.  The data informed these choices, but you did not just follow it blindly. Your product will reflect but not implement your MVP, usually.

But with these choices there will probably not be universal agreement, because even with the best validated data there can be different approaches to implementing the learning.

Data transparency

The use of data is critical to modern product development.  Every product of every kind should have mechanisms in place to learn from how the product is used in the real world (note, this is assuming very appropriate policies regarding the collection and usage of this data). This is not just about initial development, but evolution and maturing of the product as well.

If you’re going to use data to design and develop your product, and also talk about how the product was designed and developed, it is worth considering how you bring transparency to the process.  Too often, both within an organization and outside, data is used conveniently to support or make a point.  Why not consider how you could provide some level of detail that could reduce the obvious ways those that disagree with your approach might better understand things, especially for those that follow along and care deeply.  Some ideas:

  • Provide absolute numbers for the size of the data set to avoid discussions about sample size.
  • Provide a sense of statistical significance across customer types (was the data collected in one country, one type of device, etc.).
  • Provide the opportunity for additional follow up discussion or other queries based on dialog.
  • Overlay the strategic or social science choices you are making in addition to the data that informed the choices.

Transparency might not remove controversies but might be a useful tool to have an open dialog with those that share your passion for the product.

Using data as part of product development will never be free of debate or even controversy.  Products today are used across millions of people worldwide in millions of different ways, yet are often used through one software experience.  What works in one context doesn’t always work in another.  What works for one type of customer does not always work for another.  Yet those that make products need to ship and need to get products to market.

As soft as software is, it is also enormously complex.  The design, creation, and maintenance preclude at today’s state of the art an everything for everyone approach.  To make those choices a combination of data an intuition make a good approach to the social science of product development.


PS: Please click the link on the first use of data for a discussion of data versus datum. :-)

Written by Steven Sinofsky

March 16, 2013 at 11:00 am

Posted in posts

Tagged with ,

15 Responses

Subscribe to comments with RSS.

  1. Heya i’m for the primary time here. I found this board and
    I in finding It truly helpful & it helped me out much. I am hoping to give one thing back and help others like you helped me.

    Honeywell purifier

    August 18, 2014 at 8:07 am

  2. Excellent weblog here! Additionally your website loads up very fast!

    What host are you the use of? Can I get your associate hyperlink
    in your host? I wish my web site loaded up as fast as yours lol

  3. Hello, Neat post. There is an issue with your web site in web
    explorer, may check this? IE still is the market chief and a huge part
    of other people will leave out your great writing due to this

    best wax for black car

    February 14, 2014 at 2:26 am

  4. Whats up very cool website!! Man .. Excellent ..
    Superb .. I will bookmark your site and take the feeds additionally?
    I am satisfied to seek out so many helpful info right
    here within the submit, we need develop more techniques in this regard, thanks for sharing.
    . . . . .


    January 30, 2014 at 2:58 pm

  5. This forum needed shankig up and you’ve just done that. Great post!


    October 25, 2013 at 11:11 pm

  6. 財布 アウトレット

  7. Not bad at all fellas and gaslal. Thanks.


    August 29, 2013 at 7:41 pm

  8. பீர் | Peer 11:05 PM, July 03, 2009 அதே பொருளை நம்மால் ஏன் மலிவு விலையில் உற்பத்திசெய்ய முடிவதில்லை? சீனர்களின் கடின உழைப்புதான் விலை மலிவிற்குக்கான காரணமாக இருக்கமுடியும். கடின உழைப்பிற்குத் தயங்கும் நம்மவர்கள், சீன மலிவு பொருட்களை இறக்குமதித்து இங்கு சந்தைப் படுத்துகிறார்கள். சிங்கையில் இருக்கும் தங்களுக்கு தெரியாததல்ல, சீனர்களை குறை சொல்ல ஒன்றும் இல்லை.//முதல் காரணம், அரசு அளிக்கும் மானியம் கேப்பிட்டல் முதலீடு குறைவு, உலகிலேயே மிகக்குறைந்த சம்பளம் , கமிஸ்டுகளை எதிர்த்து கமிஸ்ட்டு போராட முடியாது. யுவானின் மதிப்பு கிட்டத்தட்ட அரை அமெரிக்க டாலருக்கு சமம், ஆனால் அது கால் அமெரிக்க டாலருக்கு கீழாக ஏற்றுமதியை அதிகரிக்க கட்டுப்படுத்தி வைக்கப்பட்டுள்ளது.பி.கு:எனக்கு தெரிந்த சீன மக்கள் நல்லவர்கள். //முன்பெல்லாம் திருவிழாக்களில் ப்ளாஸ்டிக் கைக்கடிகாரம், பலூன் மற்றும் இத்யாதிகள் விற்கும், அங்கெல்லாம் தற்போது இயங்கக் கூடிய சீன கைக்கடிகாரங்களும்,விலை மலிவான சிறுவிற்பனை சீனத் தயாரிப்புகள் ஆக்கிரமித்துக் கொண்டுள்ளன. திருவிழாக்களில் சீனப் பொருள்களே மிகுதியாக வைக்கபப்ட்டு விற்பனையும் அதற்குத்தான் மிகுதியாக இருக்கிறது//மக்கள் விரும்பாத எந்தப் பொருளையும் அவர்களிடம் நாம் விற்கமுடியாது. இருபது ரூபாய் குடுத்து பொம்மை கடிகாரம் வாங்குவதை விட கூடுதலாக ஐந்து ரூபாய் குடுத்து இயங்கக் கூடிய சீனக் கடிகாரம் வாங்கவே யாரும் விரும்புவார்கள். ஒன்று நாம் சீனர்களின் பொருளின் தரத்தை விட நம் பொருளின் தரத்தை உயர்துவது அல்லது அவர்களின் தரத்திற்கே தயாரித்து அவர்களின் விலையை விட சற்று குறைவாக விற்பது என்றால் மட்டுமே நம் பொருளை நம் மக்களை வாங்கச் செய்ய முடியும், சீனர்களை குறை சொல்வதில் அர்த்தம் இல்லை.


    August 29, 2013 at 5:09 am

  9. You’ve got to be kidding me-it’s so trnlpsarentay clear now!


    August 5, 2013 at 11:47 pm

  10. While I agree that data may not be the only source to decide the next strategy, it certainly is a way to measure the success of the past strategy.


    April 4, 2013 at 10:28 pm

  11. Have you ever considered about adding a little bit more than just your articles? I mean, what you say is valuable and everything. Nevertheless think of if you added some great images or video clips to give your posts more, “pop”! Your content is excellent but with images and video clips, this website could certainly be one of the very best in its niche. Excellent blog!


    April 1, 2013 at 8:40 pm

  12. Great post. I find that one of the biggest contributors to disagreement are when numbers are used in indiscriminately – substituting percentage when absolutes would make more sense, etc. Often people don’t necessarily know why, but they walk away feeling unsettled or in disagreement. What are some of the tools you’ve used to tie data back to your overall strategy and roadmap — i.e. any dashboards type tools you’ve used?


    March 18, 2013 at 7:13 pm

  13. Steven… please use this blog post to realize that WinPho (nympho) was all about numbers and data, not real good strategy. Please build phones for Aandroid and even GOOD or Blackberry.

    Nokia is a HW manufacturer, with Wireless patentens, and should remain one. Prevent it from dying !!!!!!!!!!!!!


    March 18, 2013 at 12:24 pm

  14. What is the different between “customer insight” and “data”?


    March 17, 2013 at 8:39 pm

  15. Reblogged this on InsightsWise and commented:
    This post is pure genius! Absolute must-read.

    Gayatri Shukla

    March 16, 2013 at 8:51 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: