Learning by Shipping

At the extremes of product development methodology, characterized by waterfall and agile approaches, are different views about planning. Many today would say that planning a product “up front” is nothing more than guessing that locks you into a guess that will be wrong. At the other end, the conventional wisdom is that you should get something to the market soon for testing as a best guess and then iterate and learn to further develop a product. Since no team really practices a method precisely, let’s look at how to get the best out of the planning aspects of your own product development process.

Challenges

Developing a product, new or an update, always faces the same challenge at the start. Using software as an example, folks want to get coding as soon as possible since not coding is clearly wasting time, but the team (and management) want to know that the code will yield the right product (useful, cool, innovative, profitable, whatever). There are usually those on the team that claim instinct tells them what to go build. There are those that are certain they can work out an answer to the right product with enough up front ideation.

To compound these broad challenges, different disciplines have different perspectives. It is likely testers will want to spend more time up front on building a baseline and foundation for the work, but a baseline relative to what? Ops needs to know answers to questions in order to scale and provide the required infrastructure, but those answers require a lot of information you probably don’t know. Designers usually want time to iterate in lower fidelity media in order to hone in on the design language and overall approach. Business folks want to know that their key problems (onboarding, churn, referrals, etc.) will be addressed.

All of these lead to the natural tension between a desire to get started and a desire to pause and make sure the start heads in the right direction.

Waterfall (using this as a positive description) methods argue that an effort should begin with work to identify the problem being solved, available technologies, and proposals for how to go forward. The basic notion is that you should spend the energy considering a wide range of alternatives before you just start down a path that might be fruitless. These days you will be hard-pressed to find proponents of waterfall approaches because of the downsides often associated with waterfall execution.

The downsides of this approach, as critics claim, is that you waste a lot of time building up robust plans that are “going to be wrong (or outdated) anyway”. This criticism essentially states the reality that most plans are still guesses. This is particularly true in a fast-changing, multi-player marketplace where you can wake up one morning to a competitive product or dramatically new approach to solving the problem you identified many moons ago. The ultimate downside of the waterfall approach is that it is viewed as stubborn—so stubborn that even in the face of awareness about a changing market, the team has no choice but to move forward. Proponents will offer tools to mitigate any of these risks and say none of the risks are intrinsic to the methods.

Agile development methods respond to these criticisms by creating an approach where the primary focus is to get enough product work done so you can learn from real customers and then iterate. At the extreme, efforts should actively strip away all non-essential elements of the product development process and focus on the essence of the idea. Agile methods focus on constant iteration, learning, and responding to usage of the product (or lack thereof). Teams focused on iteration work in a tight loop to quickly adapt to the changing landscape and competitive dynamics, in addition to learning.

The downsides of agile methods are actually trickier to pin down right now. These days being critical of agile methods labels one as somewhat of a Luddite in the world of product development. That said, there are understood challenges with agile methods. The pressure to release can often result in a quality bar that is less than customers (or your own testers) would appreciate. A focus on learning might cause you to learn that your service does not scale or scales in a cost-ineffective manner. A large project (people, code, partnerships) is challenging enough to keep coherent and multiple efforts executing in different iterative loops poses a significant architectural and communication challenge. Proponents will offer tools to mitigate any of these risks and say none of the risks are intrinsic to the methods.

It is no surprise that proponents of any method can point to successes, while critics can point to failures of methods as well. In reality, the causal relationship between a project success and the method used is weak at best. Even with examples of success, we need to keep in mind that product development projects are not traditional repeated processes and as such the ability to draw scientific conclusions from them is limited. This should be apparent from the fact that successful products come from both methods, and equally true is that products can be failures from both methods.

It is not unusual for failures in waterfall-styled projects to be ascribed to the methods, but not so with successes. Whereas failures in agile projects usually ascribed to external factors, not the methods. In today’s climate, it is not unusual for agile to be viewed as a causal factor for a successful product.

That’s really why starting the project by declaring the methodology runs the risk of missing the big picture. It runs the risk of spending finite and scarce time on abstract or “meta” concepts and not the work at hand. The best thing is to avoid going down that path in the first place and focus on getting clarity on what work will get done and when that will happen (and why!).

The truth is that starting a product is always a guess. Whether you plan every detail or just start coding, beginning a new product is a leap of faith based on intuition. To those of us that build or have built new products, this leap is the most interesting, terrifying, and rewarding part of product development. It does take a special confidence to make that leap. Planning every last detail can give you a false sense of security. Coding out of the gate can give you a false sense of progress. Guessing is guessing, whether you have 1000 pages of specs and high fidelity model or a whiteboard with a sketch and a functional prototype.

These challenges—choosing between methods and the known challenges of any methods—are real. In product development we are faced with these every time we start a new project. If you have a clear starting point, clear points in time to check on your progress, and a plan so you know what you’re working against then you’ve defined a methodology that works for you.

Where to start?

In a previous post we talked about focusing on the work and not worrying about the label of the methodology. A concrete way to realize this is to take a step back and see how to combine the elements of agile and waterfall in the right amount for your project.

An approach that scales from new projects to next iterations, and small project to big, is to plan to iterate your plans. While this sounds like an obvious cop-out to pick the best of both extremes, it is what reality tends to look like in practice. If you start out knowing you are going to commit time to up front planning, but recognize you will take points in time during development to adjust and learn, then you can mitigate the challenges of both methods.

Of course agile proponents say there are always some plans. And waterfall proponents always say there is room to adjust. Let’s just say those labels and attributes don’t matter and try to arrive at an approach of where to start.

Every project can start with a plan. Legendary products start with a sketch on the back of a placemat at a diner or an all-night coding session. The original plans for some pretty big projects were conceived and documented in pretty short, but articulate, memos or detailed sketches/prototypes. The spark for a plan can come from anywhere and different people have different ways of translating that spark into something more than one person can internalize and visualize.

While a tool like PowerPoint can communicate the gist of the plans, the details will be too open to interpretation. So write down the plan in long form—writing is thinking.

The simple act of writing down a product plan in a couple of dozen pages opens you up to have useful discussions with a broad set of people. If you’ve identified the problem being solved, competitive products/services, technology bets, and overall investments you have the basis of how to talk with marketing, design, development, testing, and product management. Everyone can look at the plan from their perspective and offer insights, advice. You can even package this up as a dialog or exercise with potential customers.

Combining this overview with a functional low-fidelity prototype is a way to visualize the plan for a broader audience. It is usually a good idea for the prototype to support the written description and to lead with the written description. You’ll want to minimize the time you spend on “don’t worry this UX (user experience) isn’t final” or responding to feature suggestions without the context of where you are heading.

This is all a plan needs to be. It is a guess. You can’t prove it is right. You can’t prove it is a good business, great product, or the right thing to do. You can criticize it. You can add more to it. You can find problems. That will be true of any plan (or frankly any product). But you now have a foundation to move forward. To build something that is a “target” that is shared by a group—a vision.

With this plan comes the first chance to iterate. What’s amazing about just writing this down is how much you’ll find you’re iterating on your own thinking. You can think of this somewhat as agile planning. This shouldn’t be new though – anyone who has “told a story” of a product or an idea knows that the story improves quite a bit as you tell it more. This is basically the same thing. Any good product plan is a story—the problem you are solving is the challenge to overcome, the competitors are the antagonists, the technology bets and investments are the plot devices.

Depending on the size of the team/project, the next steps are about the specifics of what to do when. The amount of up front work and the ordering of the tasks is really a choice the team needs to make. Being economical about what you do is of course a key part of ordering. Agile methods often say to do the least possible work to express the unique value of the product. Waterfall methods are about landing all the details early. Different projects will simply have different ideas of what to do at this point and your own intuition as to what makes a good investment of time, relative to quality and time to market, will dominate.

Iteration: Local or globally optimize?

Regardless of the specifics of your development schedule, you are going to iterate. You can choose to iterate after code is in the hands of customers (in a broad or limited way) or just self-hosting until a broader release. The key though is to iterate.

Iteration is as much of an art form as deciding how much of what investments to do up front. It is super easy to fall into a trap of iterating but not making forward progress. You can find yourself rewriting the same code, circling back to previously discarded alternatives, or just changing things but not making them better. As necessary as iteration is, simply iterating does not mean you and the project are moving forward.

The lack of iteration or iteration at only the most fine-grained levels is potentially a sign of a project that won’t learn as it goes. Consider a standard kitchen remodel, something that has been done millions of times. Architects draw up plans and pass them off to contractors. Then you run into existing conditions. You find you’re missing an electrical run or there’s a support wall where one wasn’t expected. It is time to iterate on the plans. You can hack through a solution with your contractor on site. Or you can take a step back and work with the architect to reconsider the design. Either can work depending on your constraints. When you consider the time value of choices, it becomes more interesting to think about taking a step back.

In the heat of the moment with the need to get in market or respond to significant challenges with early customers, a redesign or revisiting decisions seems risky. Maybe the data is poor. Maybe the fear of discovering the need for big changes concerns you. Perhaps you just want to keep moving forward. All too often when problems arise in a project the need to iterate quickly trumps taking a step back. In today’s environment it is often viewed as a positive to iterate and try something different. Activity is not always progress. Change is not always improvement.

Of course the data you use to determine what to try and how to value the feedback is important. It is just as easy to get led astray by the wrong measures of success/failure. Regardless of the quality of data, you’re going to reach a point where you are faced with the need to change something. The question is whether the changes are the right ones.

The point of a change will be to optimize your product. You’re going to have to pick the dimension for which you want to optimize—is it for immediate mitigation or longer term success. It is easy to see mitigating the immediate challenges with some changes. Longer term might feel like another guess. On other hand immediate changes have an obvious fragility relative to broader goals and there’s clearly an appeal to being thoughtful about what to change.

Having a sense of a plan helps you to weigh the alternatives. Are you dealing with a bug or minor design nit or is there a fundamental flaw in the value proposition? There’s no mistaking the reality that you might hit a major reset, especially on a brand new product. There’s also a reality that you will have to revisit a pretty broad set of small design choices—that is steps in a flow or portions of a design, rather than the entire flow or design language.

Defining a time up front when the product is in a state to evaluate all the feedback and make choices about how to optimize is an important part of the process. This checkpoint stage can be a first self-host, private beta, partner beta, public beta or anything in between. Any product today is going to have telemetry and an understanding of how it is used and what you are measuring. This helps you to inform both what is going on and even what you failed to measure correctly.

Armed with a set of potential problems to address—optimizations yet to be done—there’s a simple question to ask of the team, which is “do we need to change/fix this or not, and if we need to take a step back and re-evaluate?”

A way to look at what to change/fix or not is to think of changes relative to the longer term goal, to go beyond the immediate. There’s no doubt the feedback about something is real. The question is really whether the cost to change (hours, risk, churn) gets you closer to the broader goals of the plan or is more reflective of iterative activity.

A checkpoint discussion where members of the team are aware of all the changes going on and what is being prioritized is a way to level-set. Some teams might have bigger challenges or more changes and other teams might be making more local changes. Calibrating these across the team is akin to making sure the whole project is thinking and acting globally, not locally. The plan that was put in place serves as a reminder of what the team was hoping to accomplish. Accountability, aka decision making, can be clear because the roles and responsibilities are clear and communication has been clear. A discussion to inform doesn’t have to be an invitation for everyone else to join in revisiting the choices made by the team.

Is the new data driving you to revisit the plan completely (whether immensely detailed or not)? That could be. For a brand new business and/or a brand new product where the effort is to grow an entirely new customer base, you could be going in a wrong direction. For a product update, there’s always going to be pressure from existing customers to innovate in a more incremental manner versus taking the product in a new direction. The presence of a plan allows you to have an informed dialog as to what went wrong. When you make choices to change things you have a shared foundation upon which to agree about what went wrong.

Is the data driving you to tweak something? That certainly is the case with some changes. The presence of the plan allows you to decide how critical it is to make changes. Too often changes to the code are made because of the presence of feedback even if the change doesn’t really alter the overall outcome. When you choose to keep things a bit more stable it doesn’t have to be viewed as blindly sticking to a plan when it can just be prudent engineering of cost-benefit. The capability to change is not the same as the value of change. Something that might be 10% better might introduce a high risk of change management or might just be 100% different–ask if something really is twice as good (or 10x better) after the change.

There’s a simple view of optimization that one can use in having discussions about changes—whether at the feature level of the whole plan. The idea is to discuss whether the change is a global optimization or a local optimization. When resources are right and time to market critical, optimizing locally is wasteful. When the plan is generally right but has some holes you discover, then making sure you optimize globally leads to an agile view of planning. The following just sketches this conceptual view–believe it or not this can often be a useful visual aid in the discussions around whether a change is needed.

Whether you label the axes performance, suitability to task, conversation rate, success rate, or transactions per second is not really as important as taking a step back and asking the question about whether the change gets you close to where you need to be over time. It is far too easy to get caught optimizing your plan relative to the nearest peak, not the best peak.

Whenever you have more than a few folks working together, having a set of tools to help you make a consistent set of choices across the team is critical. The more there is a shared view of the goals and the way to make choices the easier this becomes—a plan is a way of encapsulating the broader goals and giving you a place to both measure relative success and to decide the target was wrong. The presence of a plan does not enforce rigidity any more than the use of agility guarantees you will iterate to success.

Product development is a lot of guesswork. Planning, checkpointing, and deliberate decision making are tools to help you make the most informed guesses you can make.

This Week’s Poll

This week kicks off a new feature of Learning by Shipping — Three Quick Questions. This is a snap poll to share aggregate (non-scientific) reactions to the topic of this post, which will be reported in the next post. Take the poll – Three Quick Questions. Cameron Turner, an expert in big data and measuring how products are used in the real world is helping with these polls. No identifying information is collected or maintained.

–Steven

###