CHAPTER 13

The Learn Phase—Stepping Smarter into the Future

Having heard it is not as good as having seen it;

having seen it is not as good as knowing it;

knowing it is not as good as putting it into practice.

—Xunzi (312–230 BC)

You’ve arrived at the final step of the agile process: the Learn phase. Based on your plans and how their implementation went, you want to find out what worked well and what did not. How to tackle this is the theme of this chapter. It covers the resources available to you and how you can use them for your evaluation. And what to do when your evaluation is complete.

13.1 Evolving through Evaluation

The final step in the Think–Do–Learn process is dominated by analysis. Structured evaluation of your efforts and results enables you to learn from the past what you should do in the future. It’s here that continuous improvement is made very concrete.

Self-Evaluation: The “Retrospective”

The evaluation process was already partly setup during the review meeting of the Do phase, covered in Section 12.3. That review brings insights that are then placed on the product backlog. In addition, there is still another meeting, the two-hour-maximum retrospective. It is here, at the end of a sprint, that the team looks critically at their own approach; among other outcomes, it ensures the team avoids making the same mistakes again in the next iteration. This retrospective, therefore, takes place at a meta-level and has one simple aim, to get better. You can compare it to the cars of a roller coaster being hoisted up to the start of the ride. The process is accompanied by a loud clicking sound coming from the fuse (and possibly the more-nervous passengers). This fuse ensures that the trolleys cannot fall-back, out-of-control, should the power fail. That’s what you want to achieve with the retrospective: always moving at least one step further up, with the certainty that you will not drop back (and the fervent hope that you will not feel sick from the roller-coaster ride).

The retrospective can work quickly and visually, for example by sticking Post-its—that address a particular criterion—onto brown paper. Example issues might include job satisfaction, team atmosphere, quantity and quality of what is produced, conduct of meetings (standup planning; review), workload, story points, workflow, collaboration, flexibility of output, and so on. Next to each criterion, you draw a line with a scale of 1 to 10. Each team member can then put their own stickers on the line of their choice. Then, for each criterion, a global average is calculated so that it soon becomes clear what are the opportunities for improvement. This list is a useful starting point for a debate. However, it is key here, again and again, to ask the why question in a constructive way, in order to discover what we call the root cause. The discussion should lead to SMART formulated actions, preferably documented for the purpose of knowledge sharing.

Evaluating the Test Results

In addition to this self-evaluation, the team should also look at the effect the improvements they delivered have had on their internal or external customers. For this latter step, you return to your hypotheses from the Think phase, to determine whether, on the basis of your metrics, they have been confirmed or rejected. At the story level, look at the measurements that emerged from your tests. What do you see happening in the customer behavior? Has what you have offered the customer been used and appreciated? Does this realize your goals? You look, in other words, at to what extent the test results meet your expectations and whether the implementation went as planned. The next question: Is the latest adjustment or outcome now the new standard? If so, what is there further to improve? If not, do you need to look for an alternative?

The form in which this evaluation takes place varies greatly between and within organizations. Sometimes, there is only a small improvement or a short test period, for which a quick discussion is enough. Teams often hold a formal meeting (often called a test meeting or experiment meeting), in which multiple tests are evaluated. Obviously, meetings should be planned to fit in with the sprint-rhythm, to avoid sprint activities or tests becoming redundant. In Lean terminology, this would be seen as waste.

As we saw in Section 12.4, you need to design and evaluate your testing process to ensure you proceed as objectively as possible. The search, for relationships between effort and results is, in fact, an art in itself. Unfortunately, most people practice jumping to conclusions at an Olympic level. They sometimes confuse correlation with causality: there is a connection, but no cause–effect relationship. Or they turn this relationship around, basically arguing that the sun rises because the rooster crows. Sometimes they fit other fallacies around their “beliefs,” such as errors in logic, false assumptions, incorrect semantics, no burden of proof, circular reasoning, and so on. Therefore, someone in the meeting must take the role of critically questioning the reasoning and factual evidence. This role is most often called the challenger (comparable to De Bono’s white hat).

Naturally, visualization is again very valuable when evaluating your tests. As the English say “A picture paints a thousand words”: graphics are faster, easier, and thus, better understood than complex sets of figures and texts. Resources such as an Ishikawa diagram, decision tree, box plot, histogram, and cross-table can help you. Furthermore, you should be careful when working with averages because, sometimes, they can be misleading or even impossible (how many parents do you know with 1.6 kids?), even if you only use them for identifying trends in a time-series. A particularly easy pitfall is to look at the overall figures for the whole group, when what you need to do is look at the separate subgroups within. An example: since the launch of your new app a year ago, you can see that the cumulative number of downloads, registrations, logins, and active users is still rising rapidly. A hockey-stick curve; must be good news. Right? Closer examination reveals it is not. If you look at the number of “cohorts”¹ per month, you see that, in recent months, the number of downloads has continued to climb, but the percentage of downloaders who then go on to register is actually going down. And of those who do register, fewer and fewer login again. So all the improvements you realized are not bringing you an increased return from any of the new groups. In Lean terms, this means that you are wasting time, effort, and money: your investment in increasing the number of downloads, does not add enough value because you are decreasing the conversion rate.

In short, the better you’re able to look at the behavior of your internal and external customers, the more quickly and effectively you can customize the next improvement cycle. The following section shows you what information sources are available to you.

13.2 Sources: The “Voice of the Customer”

Do you remember those childhood puzzles where you had to connect numbered dots with a pencil line? It was only at the end, when you had connected all the dots, that you saw the whole picture. Only then did you understand what you had drawn. It works in pretty much the same way when you’re analyzing your internal or external customers. By combining information from different sources, you can get a complete picture of the needs and behaviors of your customer. Within Lean, this is known as the voice of the customer (VOC). The idea behind the VOC is that the customer is always taken as the starting point for improvement, as you saw in Chapter 10: the persona and customer journey. First, listen to the customer. Because, as Epictetus said: “Nature has given men one tongue but two ears, so we can listen twice as much as we speak.”

The VOC, in agile management, has a kind of dual role, in that it serves two purposes. When for the first time, you work through the Think–Do–Learn cycle, during the Think phase, you use the VOC to analyze your internal or external customers and to capture this analysis in a persona and customer journey. Then you can use the VOC information for your measurements and testing, which you learned how to do in Chapter 12. The insights provided by this, of course, in turn, become the basis for the Think phase in the next iteration. With the VOC, therefore, you look both backwards and forwards, and thus, it represents a fulcrum in the Think–Do–Learn cycle.

Section 12.4 also showed that not all VOC sources are suitable for all purposes. The strength of the approach lies mostly in combining different types of information. This is known as triangulation. Compare it with the triangulation of GPS satellites to determine the exact position of your car. But what sources of information are there? Figure 13.1 gives an overview.

Figure 13.1 The various types of sources within the VOC²

Figure 13.1 shows a distinction between passive and active sources. With passive sources, your internal or external client is unaware that he is being studied. He is not affected by the investigation, so you can safely assume that this is a “pure” situation that he is behaving normally. With active sources, this is not the case, and you have to look critically at the potential impact of the research itself on the customer’s behavior. Throughout there is a distinction between qualitative and quantitative information. With qualitative information, you should take care to ensure that the results are statistically reliable and suitable to be projected onto the entire population.

Each of the quadrants has its own specific added value. So, for example, actively obtained qualitative information in particular, is suitable for explorative research, in which you find eye openers and discover the key narratives of certain topics. Passively obtained quantitative information is very suitable for experiments such as A/B testing. But, as I said, the strength is in the combination of the different approaches and their sources.

Incidentally, it is always a combination of science & art, as there is always the need for a healthy dose of intuitive entrepreneurship. As sociologist William Bruce Cameron said, “Not Everything that can be counted counts, and not everything can be counted that counts.” And Einstein stated about this: “Logic will get you from A to B. Imagination will take you everywhere.” Steve Jobs also expressed this beautifully in his biography³:

Some people say ‘Give the customers what they want’. But that’s notmy approach. Our job is to figure out what they’re going to want beforethey do. I think Henry Ford once said, ‘If I’d asked customers what theywanted, they would have said a faster horse!’ People don’t know whatthey want until you show it to them. That’s why I never rely on marketresearch. Our task is to read things that are not yet on the page.

It always begins with curiosity, whether you run the analysis yourself or bring in data-mining specialists. The sources agile management uses are discussed in more detail below.

Resources for an Active Approach to Acquiring Qualitative Information

The most relevant sources here are:

• Focus group (also called a customer panel)—In these flexible group interviews, you can ask a lot of in-depth questions. So they are well suited to issues such as needs, motivations, criteria, preferences, orientation, and purchasing behavior.
• Individual depth-interview—The main difference between this and a focus group is that the individual is not affected by other participants, either negatively (group pressure) or positively (new angles and ideas). On the one hand, this means that his answers come closer to his real personal “truth”; on the other hand, it requires more creativity and skill, from the interviewer, to ask questions that are broad in range and address different angles. Moreover, you can also use customer panels and interviews to allow respondents to get acquainted with your products and services while you observe this process, possibly in combination with asking questions about your observations.
• Customer shadowing—A day “following” in the customer’s footsteps gives practical insights into their behavior in relation to your products and services, but has the great, unavoidable, disadvantage that your presence creates bias.
• Online research community—In a protected online environment, a moderator periodically invites panelists, by e-mail, to answer questions about the use of certain products and services, or to give their views on specific topics. As this concerns long-running research, it can also measure trends.

Resources for an Active Approach to Acquiring Quantitative Information

The most relevant sources here are:

• Surveys—surveys are usually completed online (and differ from an online research community in that a survey only happens once) following an e-mail invitation or a pop-up on a website. In addition, many are still contacted by telephone, both outbound and inbound (such as after a complaint call to customer service). Finally, there is also significant face-to-face work done in the field, for instance by questioning visitors to an exhibition or shop and gathering the data with the aid of a tablet.
• Bio-/neurometric research—This would include fMRI and EEG brain scans and eyetracking. All three methods involve the subjects being exposed to concrete things, such as a website or product, in order to determine which areas of their brains show activity as they react emotionally, and where their attention focuses.
• Behavioral analysis—These include viewing, listening, and surfing habits for television, radio, desktops, tablets, and smartphones, measured using set-top boxes or software.

Sources for a Passive Approach to Acquiring Qualitative Information

The most relevant sources here are:

• Desk research—Reports, reviews, trend analyses, literature etc. based on completed research around issues like trends, market developments, brand perception, needs, and competition.
• Reading reviews, test reports, and news or opinion articles—These can come from consumer associations, consumers on special interest sites, or journalists. This gives an indication of the criteria against which your products and services are being assessed and how your organization scores against those criteria, possibly in comparison with competitors.
• Listen in on call center/face-to-face conversations—You can also read the call-center logs or study call reports. This allows you to determine what support customers need, what questions or complaints they have, and what information they request.
• Internal research—Reading emails, app and text messages, tweets, and completed complaint forms make clear what are the more-extreme experiences, positive and negative, and on what these experiences are based. Exchanged or returned products and warranty claims make product defects clear and demonstrate how customers experienced products in use: what they look for, what they find important, what requirements the product must meet, and what is expected of the exchange/return process. You will gain insight into which products are exchanged or returned most often and, sometimes, why that happens.
• Walking around—If you personally, “undercover,” replace a shop-assistant or call-center employee or field sales executive, you will benefit from being in direct contact with the customer. This also provides practical insights into what can be improved in the customer experience, but from the perspective of the one providing this experience.
• Mystery shopping—Here you “get under the skin” of your internal or external customers. Again, you’ll get practical insights into what is going well in the customer experience offered within your channels, and what needs to be improved.
• Observations—These can be spontaneous observations in the form of anecdotes, personal experiences, and culture clues. This can be more targeted by using, for example, a customer “safari” in shopping centers or at trade fairs and events. This could also take place in your own or competitor’s stores. This provides general input about what customers do, how they behave, and other valuable data.

Sources for a Passive Approach to Acquiring Quantitative Data

The most relevant sources here are:

• Crowdsourcing—By actively engaging your customers in initiatives you develop and promote, you can learn a lot. Earlier, you read about the potential in crowdfunding, but you can also learn and bond with your customers through co-creation. You can organize this both offline and online. Nike does this online with NIKEiD, where customers can design their own shoes. The choices they make while doing so provide Nike with superb data about their current and future preferences, upon which it bases its future product range and designs. LEGO does something similar, letting customers design their own kits and bring in their own ideas. The latter initiative led to LEGO’s most successful product ever: LEGO Friends.
• Behavior on your organization’s website and apps—Within your own site and app is obviously the place to gather a wealth of information: landing pages, logins, click paths and patterns, reactions, calls to action, downloads, orders, filling-in forms, live chats, device type (PC, tablet, smartphone), and so on. By analyzing these data, you can learn a tremendous amount about your customers. It lends itself very well to, for example, A/B tests.
• Social-media behavior—By tracking relevant topics on social media like Facebook, Twitter, and LinkedIn, you can quickly learn what the opinion or attitude is of customers toward e.g. your brand, organization, products, services, campaigns, promotions, service, and your competitors.
• Other online behavior—Analyzing data such as the terms used in search-engine fields, home pages, click throughs on ads, and retargeting responses, can provide you with useful insights. You can also measure very specifically between online (and offline) channels using QR codes, source codes and unique URLs, phone numbers, and freepost addresses.
• Other digital and offline behavior—Systems usage can also provide important insights. For internal customers you can study intranet use, HRM administration, and other applications. For external customers, this can involve such things as digital kiosks, cash-register records, scans, debit and loyalty card use, and e-mail. You can measure the number of visitors in your stores (“footfall”) through electronic counters or cameras at the entrances. If customers use the WiFi your shops provide, via three-point measurements, you can create a heat map of their movement around the shop floor. And, of course, it is quite easy to capture their contact with the call center or field sales.

In short, no lack of potential sources. The challenge is to use them, and the information they provide, as objectively as possible. For, as Anaïs Nin said, “We do not see things as they are, we see things as we are.” The findings mean that you can adjust quickly. But how do you determine whether you need to adjust? Read below.

13.3 It’s Your Choice: Continue or Pivot

Section 13.1 talked about the test/experiment meeting, which focuses on evaluating your results. Based on the measurements you’ve made with help from the VOC sources, you determine how much you can confirm or reject your hypotheses. Depending on the outcome, there are three decisions you can make. With the first two, you continue with what you were doing but, with the last, you change course quite sharply:

• New default—If results have indeed improved, use your approach now as your new standard, your baseline. In the next iteration, you get started with refining this baseline with either additional improvements or entirely different improvements.
• Looking for alternatives—When there’s no or insufficient improvement, you go into the next iteration looking for other solutions that do produce the desired result.
• Pivoting—Sometimes you make an accidental discovery, as discussed in Section 4.4 that turns your vision or ideas upside down. Or you conclude that your approach has yielded insufficient or no improvement—or maybe even a deterioration, while there are no obvious alternatives. At that point, you have to make a more fundamental choice, which we call a Pivot.

Pivoting

Pivot, within the context of agile management, means “turning point.” If you are at a dead end, you go back to your last crossing, and take a new road, set a new course. That crossing is represented by those elements in your approach that have actually proven themselves, the elements which you are sure are successful. If you compare it with mountaineering, you can see it as your basecamp. Once you determine that the route you are taking, to establish a new camp at a higher altitude, will not succeed, you can always return to your basecamp and try a new approach or another route the next day.

Within agile management, there are multiple pivot points, for both your internal and your external customers. These pivot points are related to the components of the business model canvas, which you read in section 10.1. The most important are shown below:

• Target group—Sometimes, you find out that a different market segment makes use of your products or services than you originally imagined. You can then leave your proposition unchanged, but you will need to focus on a different audience. For example, this happened with the Peugeot 1007, which was originally intended as a trendy city car but, because of its convenient sliding doors, was mainly purchased by the elderly.
• Proposition—It may turn out that a product or service meets a need for your customers, but is less important than you expected. Or it meets a customer’s need in a way that you never unexpected: walkie-talkies and home telephone sets for example were used as baby monitors; the same happened when mobile telephony subscriptions started offering unlimited calls. Then you keep your target group, but you need to reposition or renew your product or service; for example, by making one of the characteristics of your product, service, or experience the most important, or even to ignore all the other characteristics. The converse is also possible: you just add components to the previously single part. This is called zooming-in and zooming-out. This is exactly what Amazon did when it broadened its offerings from only books and CDs to include electronics, toys, and so on.
• Channel—Section 10.4, about customer journeys, already discussed that consumers use channels when finding, buying, and using your products and services. Sometimes, they seem to prefer other channels than you expected. Or channel conflicts arise with your partners. Then you can choose to use your channels differently, or to use different channels. Long before the Internet age, the computer manufacturer Dell had already decided to deliver directly to the end customer, thus eliminating the middlemen.
• Income streams—As an organization, you might learn that your products and services do not realize the revenue streams that you had planned for. Then you need to adjust your business model. Ebay subsidiary Marktplaats, for example, did this as soon as it became clear that their model of “free ads but pay for additional services” was making a loss. Marktplaats then decided to ask €6 for advertisements for products with a value greater than €200 and also for services. And later, this was adapted into differentiated tariffs for different product-areas and advertisement duration.
• Cost—The profitability of your business can sometimes be hindered by certain activities generating excessive costs. Then you can choose to in- or outsource them, or to switch to a technology that offers a better price–performance ratio. Sometimes, it even has an impact on your channels and revenue streams, such as at Netflix. Netflix originally rented-out DVDs, which they mailed to the customer. With the advent of broadband Internet connections, they switched to a streaming model, which had much lower costs, allowing them to change their business model.
• Platform—Sometimes, companies find that the way they offer their products and services can also be used by third parties. They approach them and offer a platform: Amazon now allows third parties to sell through its webshop, and offers its proprietary technology as a cloud service called Amazon Web Services.

Through your evaluations, you have either a) chosen the approach that will be your new standard, b) to seek out alternatives, or c) to pivot. This brings you to the end of the Think–Do–Learn cycle. So it’s time, based on what you’ve learned, to start a new iteration. And when that iteration is complete, you start a new one, again and again. So, ad infinitum they follow each other until the cycle becomes second nature.

But what if you still need to start your very first iteration? If your organization has yet to make the transition to agile management? In the next and final chapter, you can read about the do’s and don’ts of implementing agile management.

By reading this chapter, you’ll have discovered the following:

•  In the Learn phase, analysis is central. This is done on two levels: first, look during the retrospective meta-level meeting at your own activities, to evaluate whether you can improve your approach. Second, look at the test results to determine if your hypotheses are confirmed or rejected. Preferably, this would also be done during the retrospective, but it can also be in a separate meeting, which is usually referred to as the test or experiment meeting.

•  The VOC gives you information for both analyzing your customers and making measurements during your testing. Thus, this is a turning point in the Think–Do–Learn cycle.

•  Within the VOC, distinction is made between passive and active sources and also quantitative and qualitative information. The four groups which arise as a result offer their own specific values, wherein the power is in the combination of the different types.

•  Based on the confirmation or rejection of your assumptions, you can take one of three decisions: choose your approach with your new standard, find alternatives, or pivot. The latter means that your approach changes, fundamentally. That can be along the lines of target groups, propositions, channels, revenue streams, costs, and platforms.

References

1.  Ries, E. (2011). The Lean Startup. New York: Crown Publishing

2.  Thanks to Robert Ossenbruggen.

3.  Isaacson, W. (2011). Steve Jobs. London: Simon & Schuster.