Measures & targets
KEY POINTS
Decide how you are going to measure the answers to the interview questions you ask
Set targets for an overall group so you know what scores constitute positive or negative evidence
In Step 2 (Identify risks) you work through your Step 1 (Write hypothesis) to discover where your assumptions are. You then grade some of the assumptions ‘risky’ as they are fundamental to the success of your overall idea and you feel that you do not have enough evidence to feel comfortable with the level of risk. In Step 3 (Create questions) you craft interview questions designed to get you evidence in the hope that you can reduce the risk level of those assumptions and feel more confident to move forward with your idea.
In Step 4 (Find interviewees) you develop some criteria to profile the customers and/or users who you think stand to benefit the most from your idea. You also work through how best to find and recruit them as you are getting ready to ask them your questions.
In this Step 5 (Measures and targets), measures are set that will help to assess and analyse the answers from each interviewee. Then group targets are set so that we know what scores across that group of interviewees constitute positive evidence. Positive evidence means that you can reduce the levels of risk in your assumptions, which will give you more confidence to make a decision whether to pursue your idea or not.
By the end of Step 5 you will be ready to move into the final stages of preparing for carrying out the research in Step 6 (Conduct interviews). If you have more than one group to interview, then each group profile will have its own set of risky assumptions, interview questions, recruitment criteria and measures and targets. If you are using the templates, you will create one CORE template for each different group.
The purpose of this step is to avoid a common pitfall of reviewing interviewees’ answers with rose-tinted glasses on. Innovators often make a decision based on gut feel of how the interview went overall. If it is a good, flowing and lively conversation it can feel like a positive response overall, when in the cold light of day, it might not have been. It is also quite natural to try and hear the answer that you want to hear and it can be difficult to stay neutral. Putting in a scoring mechanism will help to reduce your own bias and help you stay as neutral as you can.
Despite this being a relatively small exercise, it is important enough to have its very own step!
Set individual measures
These measures are for each interviewee and set against each risky assumption that you have identified in Step 2 (Identify risks). The questions you have written are designed to get you information so that you can score each measure. The measures are there in an attempt to create an unbiased scoring mechanism.
It is advisable to keep these measures as simple as possible. They should be simple enough that you can assign a score in the interview itself.
Measures for section A: Does the problem, need or desire exist?
You could use a YES/NO measure here: YES the problem, need or desire exists or NO the problem, need or desire does not exist. However, this does not really help you with the level of pain and one of the key points behind whether the problem, need or desire exists is how strong it is. It may be better to choose a metric for section A answers that show the level expressed in a scale of 0–5, where 0 is no problem, need or desire exists and 5 is a big problem, need or desire exists.
Here is an example using the indoor market project with the group profile of the potential test traders. It shows how the 0–5 scale works as a measure in interview section A.
Example: Section A measures for the indoor market project, potential test trader interviews
The risky assumption in this example is that the profiled potential test trader group actually truly desires a longer term retail space to sell their offering. The questions being asked have been designed to get them to talk about where they have sold in the past and from their answers it will be possible to tease out how strong the desire is.
So the measure here is about whether the desire exists and to what level. It will be measured on a scale of 0–5, where 0 is no desire and 5 is a big desire.
Measures for section B: Are the existing solutions good enough to solve their problem/meet their needs or desires?
This measure can be kept simple with a YES or NO. The essence behind the question is whether the interviewee feels that there is room for another solution, or whether they feel that the existing offerings give them everything they need. The answers here also help to score the level of pain in section A.
Now in some cases, people don’t know what they want until you show them an alternative. If you are showing them your idea for a solution in interview section D, you can use their answer in both interview section B and interview D to score the measure to this question.
Here is an example using the indoor market project with the group profile of the potential test traders. It shows how the scale works as a measure in interview section B.
Example: Section B measures for the indoor market project, potential test trader interviews
The risky assumption here in section B is whether there is room for another sales opportunity as it may be that the existing sales opportunities meet their needs. The questions in this case are designed to get the interviewee to talk about what is attractive or not about existing sales opportunities and to get them to describe their ideal opportunity. Through those answers, it will be possible to assess whether the shop unit offering in the indoor market can meet their desires better than what is already on offer.
One of the ways to measure the responses here is ‘YES or NO’ to whether the existing ways to sell meet their needs and desires. Now, you may think that they have already answered that question simply by having applied to take part in the research but this is an opportunity to learn more. All is never as it seems until you delve further! Remember that there is also more to learn here than just establishing the score for the measure. You will hear all about the competition – in this case, other opportunities for potential traders, that would take their attention away from this project.
Measures for section C: Are they prepared to take the necessary actions to get another solution?
In this section you are looking for evidence that they will take the action(s) you require. The strongest evidence you can get will be that they have taken this action before. Remember that you can push for parallel situations, so it is that they have taken the action, albeit not in the exact context or way that you are thinking of. For example, the action is that you want a parent to spend money on a new app to distract their child while the parent gets on with their chores. So the action you are looking for in a broad sense is the spending of money to solve that problem. It does not need to be money spent on an app, it could be on a new toy or a babysitter.
This can also be a YES or NO measure. They have either taken the action(s) before or not. If you have more than one action here, then remember to add a measure for each one.
Here is an example using the indoor market project with the group profile of the potential test traders. It shows how the YES/NO measure works in interview section C.
Example: Section C measures for the indoor market project, potential test trader interviews
In this case the risks are around whether the interviewee is prepared to invest time and a little money (not for rent, but for the reduced rates and kitting out their shop unit) and whether they are able to work within an environment that is going to be continually changing. Test traders will come and go and there will be continual experimentation of new marketing ideas, events to attract visitors, pop-up activities and so on. Some will succeed and some will fail. The place will be in flux until all the empty shop units are on permanent leases. This is not like taking a shop unit on an established high street.
The first two questions ask potential test traders to give previous examples of where they have spent time and money previously, and of having operated in this sort of environment. There are additional questions that also ask how they feel about taking these actions in this context. They already know what the context is as they have applied to be interviewed about this and have seen more information about the project in the screener. So this means that while they may not be able to give you solid examples of having taken those actions before, they are likely to have a view on whether they are prepared to take them in this case.
There are two different measures to set here as there are two risks. These measures can simply be YES or NO, interviewee is/is not prepared to invest time and a little money and YES or NO, interviewee is/is not prepared to work in that environment.
Measures for section D: (Optional) How do they feel about your idea?
In this interview section, you will be sharing your new concept in order to get some feedback. It could be just a paragraph that describes it, or something a little more elaborate. The main risks here are that they do not like the solution that is being proposed.
The main questions are most likely going to be based around The Trilogy to tease out if there is anything in particular that appeals, anything that they have questions or concerns about, anything that they hope it will do or any improvements that they would make. As with all these measures, they need to be right for your particular hypothesis and circumstances so you may have other questions that address other particulars.
With The Trilogy questions in play, the feedback will go beyond being able to score this measure and will get into all the ways that the idea could be improved upon. However, for the purposes of this step and writing this initial individual measure, you may decide to use a scale or a YES/NO answer. It depends on the detail of the idea, solution or concept that you have.
Here is an example using the indoor market project with the group profile of the potential test traders. It shows how the measures can work in interview section D.
Example: Section D measures for the indoor market project, potential test trader interviews
In this case the risks are whether any of the empty shop units actually appeal to them and whether the terms, conditions and costs are acceptable. At this point in the interview, the interviewee will be taken into the market and shown the empty available shops and the paperwork and asked appropriately tailored versions of the Trilogy question.
The measures for the questions are firstly, a scale of 0–5 as a way to measure their interest in one (or more) of the empty shop units and a YES/NO/MAYBE measure on response to the paperwork. The MAYBE option allows for questions to be asked or suggestions made of changes to the paperwork.
In the example above, there is a YES/NO/MAYBE measure. It may feel right to add in a MAYBE answer and to think through what you would do with that response. During these interviews you are going to get lots of information beyond these scores that will help you to improve upon your proposition, tighten up your target groups and so on. You might see that the MAYBEs would be a YES, if you could implement changes that the interviewee suggested. So, if they are feasible, you could turn some of the MAYBEs into a YES or conversely, into a NO if they are not feasible.
Assigning a score for the measure
There are two opportunities to actually assign the individual score. The first is going to be during the interview itself, in Step 6 (Conduct interview), while it is fresh in your mind. The second is in Step 7 (Analyse and decide) when you go back over each interview and analyse what you have learned. I strongly recommend audio recording each interview and will discuss more in the next step.
With these measures, you can use the closing section of the interview conversation to check that you have enough information to assign a score and to check directly with the interviewee whether you have assigned appropriate scores. There is more about how to do that in Step 6 (Conduct interview).
Set group targets
Now you need to set a target score for each measure across the group. Once you have completed all the interviews in a group, you will be able to do a tally and see whether they hit their group target as a group, or not. Setting this target is a judgement call that must be based on your particular idea and market.
Example: Using judgement when setting group targets for indoor market project, potential test trader interviews
In the case of the potential test traders in the indoor market project, with five interviewees in the group, the group targets are out of five. Scores of three and four out of five for these measures feel like a strong enough result. With twenty shops in total to offer up for test trading and, say, a year of rolling test trades offered up to reach full occupancy, it feels about right. If only one was positive, then perhaps this would feel a bit too low.
There is always the opportunity to go to another five potential test traders, making any improvements or tweaks that you heard about in the first round, and see how a further group scores. If the first group of five interviewees were all scored positively, then in this case, I would be unlikely to take the test to a further group.
In this example, once again, using the indoor market project, the group targets are shown against the individual measures that were discussed above.
Example: Step 5 Measures and targets for the indoor market project, potential test trader interviews
If the size of your potential audience is quite small – for example, if it is students who cannot study the subject that they want at school – then you may want a higher group target. If, on the other hand, you are looking at developing a new taxi service, then that is more of a mass market offering, so you may decide that only one out of five interviewees needs to be positive.
If you decide to add in a wildcard, then you will need to decide what to do with it regarding counting towards these scores. If your wildcard is a good fit for your idea and has shown you that your target group is different to what you thought it was, then by all means count them. If not, then leave them out of the scoring.
Avoid getting stuck! My overall advice here is to be bold. You need to just need to work out how you are going to measure the evidence you need and then what score makes it positive. Try not to overthink. Be realistic and really honest with yourself. Imagine that you are outside looking in. What sounds like a good measure to you that offers up some proof before you invest your money further developing your idea? What number makes it sound like it is a good idea to progress? I have seen people spend hours debating this. Don’t overthink it. Just call it and move on as, guess what, you can go back and iterate as you go!
It is very likely, as with every step, that when you try to work out these targets, you will find yourself wanting to make changes to earlier steps. Perhaps this will be to your risky assumptions in Step 2 (Identify risks), or the questions that you are to ask in Step 3 (Create questions) or the group profiles in Step 4 (Find interviewees), or even to the way that you have framed your overall idea in the hypothesis (Write hypothesis). So by all means, please do go back through previous steps and make changes. Just remember that at whatever step you make the changes, you need to work back through the steps chronologically to get back here.
Completing The Really Good Idea Test templates
Are you working on your own idea as you go through this book?
If so, then you should still be using the CORE template.
You are now ready to complete the box for Step 5 (Measures and targets).
You will have completed Step 1 (Write hypothesis), Step 2 (Identify risks) and Step 3 (Create questions) and Step 4 (group profile). You can see all of those are shaded.
Once you have completed the boxes for Step 5 (Measures and targets), then you have a document that you may like to share with colleagues and stakeholders to explain your next steps. It is a summary and rationale of who you are going to interview, what you are going to ask them, why and how you are going to analyse their responses to see if you have passed The Really Good Idea Test.
This CORE document is also going to become part of your full research script when you carry out your interviews in Step 6.
What do I write in these boxes?
Individual measures This box is for the individual measures for each of the risky assumptions that you entered from Step 2 (Identify risks). Remember that these measures are set against the risky assumptions and that the interview questions are there to get you enough information from the interviewees to be able to give them each a score.
When you get to Step 6, you will be able to add in the individual scores for each interviewee.
Group targets This box is for the total score of the group that are in this group profile. Remember that these are the levels you need to reach to constitute whether your evidence is enough to reduce the risky assumption or not.
When you get to Step 7 you will be able to total up the scores across the whole group and add them in.
Time to share your CORE template?
The CORE template now has something in every box, reflecting all the work that has been done in Steps 1 through to 5. Here is an example of a completed CORE template for the indoor market project, potential test traders.
Try it for yourself!
Remember that you can download these templates from productdoctor.co.uk.
TOP TIPS
Set up measures and targets before conducting interviews as this will help you to be objective when analysing the interviewees’ answers
Keep measures simple using YES, NO, MAYBE and/or simple 0–5 scales
Now you have filled out something in every box, you can use the CORE template document to communicate your journey and rationale before starting the interviews
COMMON PITFALLS
It is highly likely in carrying out this step, that you will want to make changes to previous steps. Go make them and get the most out of the test!
The measures need to be set against each risk, not each individual interview question. The interview questions are there to get you the information to assign a score for each measure.
Spending too much time over-thinking measures and targets. Just call it and move on!