๏ปฟ
For years, people have been abusing surveys. From dull subject lines to the wrong questions, from sending out questionnaires after the fact to sending them way too often, there is a lot that organizations have gotten wrong about how to design a survey.
Todayโs guest is fighting to preserve the value of surveys in the face of all this misuse. Caroline Jarrett became interested in forms around 30 years ago while delivering optical character recognition systems to the UK Inland Revenue, which included processing tax forms. It didnโt work very well, and it turned out it was because people made honest mistakes when filling in forms that the technology couldnโt read.
Ever since, sheโs become utterly fascinated by forms that are easy to fill in and the thought process that goes behind them, and on the way, she picked up an interest in surveys as well. She is the coauthor of Forms that Work: Designing Web Forms for Usability, and sheโs been working as a survey consultant for companies and government agencies that want to improve the way they design their surveys.
Creating winning forms and surveys has been top of mind for us lately. We recently launched Intercom Surveys, a new feature that empowers you to connect with customers wherever they are, capture valuable customer insights, and use collected data to drive more engaging, personalized customer experiences. But as Caroline points out, even the best tools need to be used thoughtfully in order to get meaningful results.
In todayโs episode, we chat with Caroline about her seven-step process for designing, running, and actioning considerate surveys that get results and donโt bombard customers all the time.
If youโre short on time, here are a few quick takeaways:
- Catching people in the moment and not days after the fact is the best way to get quick, accurate results. Ask people something simple and interesting that goes straight to the point.
- Be selective on the sample and make it clear why that personโs exact feedback is important that week. That way, youโll get much better response rates and data quality.
- Use the smallest sample you can that can still deliver good quality results. Itโs less work, itโs less intrusive, and it gives you the opportunity to do another one if the first one goes wrong.
- Ask one or two questions in the language, the style of speaking, and phrasing that people use rather than your corporate jargon.
- To preserve the value of surveys and not exhaust customers with constant questioning, do fewer, much smaller surveys.
If you enjoy our discussion, check out more episodes of our podcast. You can follow on iTunes, Spotify, YouTube or grab the RSS feed in your player of choice. What follows is a lightly edited transcript of the episode.
The makings of a form specialist
Liam Geraghty: Caroline, youโre very welcome. Itโs great to have you on the show.
Caroline Jarrett: Oh, itโs great to be here. Thanks so much for inviting me.
Liam: Before we get into the nuts and bolts of surveys, can you tell us a little about how you got started in the world of forms and surveys?
Caroline: Well, completely by accident. I used to tell people that when other little girls were playing with Barbies, I was cutting out forms from paper and putting them together, and I just like to gauge how long it takes the audience to realize thatโs obviously a joke. But what happened was that, back in 1990, I got the opportunity to be a project manager of a project that was using optical character recognition techniques. It was a Rank Xerox project for the European Patent Office to capture patents which is a very, very specialist area, and the idea was we would scan the patents and digitize them.
It turned out that optical character recognition was good enough, but there were some challenges around patents. We imagine that patents are interesting, but theyโre really not. I mean, a good 30% of them are obscure chemical names. So, our OCR, optical character recognition had to scan things like, oh, I donโt know, [A,C-hexifloro[, and suffice it to say, a spell checker would not work. We also had to scan patents exactly as submitted. If there were spelling mistakes or nonsense or anything like that, it had to be submitted to the patent examiner for legal reasons.
โA thing Iโm particularly fascinated by is how people answer questions and think through the process of answering questionsโ
In the European Patent Office, the patents could be submitted in English, French, or German, and they always had an abstract in each of the three languages. So, you were guaranteed to have English, French, and German in every submission. The majority of them were in English, but they could be submitted in French or German. German has longer words than English. Both languages have a lot more accents than English. So, I accidentally ended up learning quite a bit about the practicalities of implementing optical character recognition in a fairly complex environment.
Then, I got a job with a big computer systems integrator in the UK. At the time, they were called Bull Information Systems. They were delivering all of the in-office personal computers to what was called the Inland Revenue, which had struck a deal with Bull to try using optical character recognition to process some tax forms, and I got involved in project managing the delivery of these optical character recognition systems.
Long story short, they didnโt work at all, so I got permission from the revenue to go and observe the systems in use, and it turned out, for example, one of the systems was in use for processing forms that were, broadly speaking, filled in by retired people with very low incomes. Optical character recognition, to this day, cannot deal with a form where someone has written โsee the letter attached.โ There were many inventive ways in which people could mess up the form. For example, it had one line for your interest from your bank or building society account. If people had two accounts, they would write two lines of tiny little numbers one above the other in the one box, which is a perfectly reasonable thing to do, but my OCR machine had no chance of dealing with that, and that was around 1992.
โI found buckets of really interesting and useful literature in the world of surveys that helped me in my work on forms, and one thing led to anotherโ
I started working on that at the beginning of 1992, and weโre just coming up to 30 years. I just got really, really interested in how we design forms so that people can fill them in correctly, and Iโve just not lost interest in that. Ever since, Iโve continued to be absolutely fascinated by that challenge, and here we are still really interested in it, still really, really interested in it.
Iโm the form specialist, and Iโm a pragmatist about surveys. I could not describe myself as a survey methodologist. To be honest, I couldnโt even describe myself as being all that enthusiastic about surveys. What happened was that I obsessively tried to find all the academic literature and books written on forms, and to this day, I have one small bookshelf in my office, which is pretty much everything I could find. There wasnโt much out there, so I started looking for other areas where people might have written about forms and surveys came to mind because one of the really interesting challenges for me, or a thing Iโm particularly fascinated by, is how people answer questions and think through the process of answering questions. How do we write and pose questions in ways that get accurate answers? Thatโs also a problem thatโs really, really important to the survey methodologists.
And so, I found buckets of really interesting and useful literature in the world of surveys that helped me in my work on forms, and one thing led to another. I started answering questions on mailing lists about surveys because I accidentally learned about them, and then my mentor, Ginny Redish, told me that I had to write a book on surveys, and here we are.
The thought process behind a survey
Liam: For people listening, could you take us through the process of creating a survey? Iโve been reading through your presentations and everything. I know it should take a couple of days, but consider weโve about 15 or 20 minutes.
โIn terms of designing a questionnaire, you need to put that into the context of what you want to achieve with the survey overallโ
Caroline: Well, most people come to me and say, โCan you have a look at my survey and give me a questionnaire?โ Now, obviously, these two words are used very interchangeably. One of my favorite books on survey design actually has the world questionnaire in the title. That would be A.P. Oppenheimโs book. And one of my favorite books on questionnaire design, by Mick Couper, and I canโt recommend his work too highly to anyone who is interested in the literature, is called Web Survey Design, but itโs actually about questionnaires. Now, let me clarify. Iโve tried to use the word questionnaire for the question and answer thing we actually put in front of the people, and the word survey for the whole process starting from, โoh, I think we might want to do a surveyโ right through to delivering some kind of number as the result. So, thatโs how I use it. Of course, in real life, people tend to use them very much interchangeably.
In terms of designing a questionnaire, you need to put that into the context of what you want to achieve with the survey overall. And I have a seven-step process to design a survey where I start with saying to work out your goals for the survey, figure out why you want to do it, what you hope to gain by doing it, and what sort of decisions you will make based on the results of the survey, and then working through who you want to answer and how you can find a sample of them because generally, with most surveys, weโre much better off using as small a sample as possible.
And then, think about the questions you want to ask, which obviously relate to the goals but also to the people you want because you want questions that make sense to them. Then, building that stuff into a questionnaire. Of course, these days, we have many different tools to help us do that. But the hard bit is the thought process that goes into it, getting it out. I borrowed the word fieldwork from the world of market research โ they talk about putting a questionnaire into the field, and I couldnโt think of a better term to call it than fieldwork, which is getting your questionnaires out to people and getting them back.
โJust having responses isnโt good enough. Youโve got to do something with them that gets those responses to the decision-makersโ
At that point, you have some responses which hopefully people will have answered. So, you have to clean the responses, think about them, and analyze them in various ways. And then, Iโve called the final step reports because just having responses isnโt good enough. Youโve got to do something with them that gets those responses to the decision-makers.
For example, once I was wondering, โshould I write this article or this article for my next column?โ People on Twitter who follow me are predominantly my audience for what I write, so I thought, โIโll make a little Twitter poll.โ I offered them two options. It wasnโt in any way something statistical. I just needed to make a quick decision. Put it up, people clicked on the poll, and one topic was more popular than the other. That was good enough for me. I just wrote the thing. That probably took me 10 minutes to think it through and 24 hours to wait for the results, and it was better than hammering my head about which one to write.
Thatโs the tiniest end of the whole thing, and at the most elaborate end, we have things like 10-yearly censuses, and the preparation, planning, and testing for the next census starts before the existing census is finished. It takes 10 to 15 years to do a census. Thatโs the other end of it.
Asking one person the right question
Liam: So, once you come up with the questions that need answers, how do you go about targeting the correct people? And how important is it to catch them in the moment?
Caroline: Iโm really glad you mentioned catching them in the moment because I characterize three different strategies for finding people you want to answer and catching them in the moment is, I think, one of the most effective. A lot of us see these popup surveys on websites. And so, you can catch people in the moment they arrive at the website, and you have an opportunity to ask them perhaps one, maybe two interesting questions. Just as an aside, the number of websites that waste that opportunity by asking the question, โAre you willing to answer a survey?โ absolutely astounds me because we already know that everyoneโs going to click no and all weโve done is learn what we already learn.
I mean, if you are going to do that sort of catch-in-the-moment survey, my pro tip is to ask them something interesting. For example, โwhy did you come to this website today?โ is something they can answer straight away without having to distract themselves too much from what theyโre trying to do. So, catching them in the moment is a great strategy. Another example I really love is seeing people queuing at kiosks at the end of museum exhibits to give their feedback on the exhibit, unbelievably. Itโs so unusual to see people actually queuing up to answer a survey. And not just once, Iโve seen it regularly if itโs a really interesting exhibit and a well-designed survey. Thatโs one of the strategies.
โIf youโre going to use a list, try and be selective and make it clear why that personโs exact feedback this week is more importantโ
Liam: Yeah, the number of times Iโve got emails a couple of days after the fact where Iโm not really sure how Iโve felt about something or Iโm not kind of in the mood or in the zone to be answering questions, whereas I might have been if I had been offered the survey straight away.
Caroline: Absolutely. And what youโve got there, being asked after the fact, is the most typical way people try and find people, where they have some kind of list. In this case, the organization or business you interacted with has you on their system as a customer or someone whoโs contacted them. They have a list of those people and theyโre using it to generate a sample. And again, pro tip, donโt ask everybody. Weโre all bombarded with this stuff all of the time. If youโre going to use a list, try and be selective and make it clear why that personโs exact feedback this week is more important. Could you remember the title of any recent email that youโve had of that nature?
Liam: Oh god. Now youโre asking someone with a terrible memory for these things.
Caroline: It just shows how unmemorable they are, right? Another one comes in, โtell us about your recent transactionโ or something. Yawn, Iโve got dozens of those things. Now, letโs try something that says, โThank you so much for buying from us. Please can you answer our question of the week?โ Right? Weโre only asking 10 people. โOoh, Iโm one of 10. I will answer. Itโs only the question of the week. Excellent.โ You know? It would stand out. It would be a little bit different. It would be a short thing. Itโs a question of the week. Itโs not like your life history or something.
โAsking one person the right question is more useful than asking 10,000 people the wrong questionโ
These days, weโve got so many techniques for sending little tiny surveys which could be much more engaging and short and respectful of peopleโs times and could be sent to much smaller samples, and we could get much better response rates and data quality, rather than this constant bombardment. And the response rates are terrible.
Liam: I suppose that leads us to the actual questions, and as you said there, โyouโre one of 10 people weโre asking,โ is it about asking one person or a few people the right question, rather than just blanket asking a hundred people?
Caroline: I think you may have read my slides. Have you seen the slide that says that asking one person the right question is more useful than asking 10,000 people the wrong question? I see this sort of thought process in colleagues and clients all the time. Somebody in a senior exec role says, โWe need to find out about our customers.โ Great. Thatโs a great thing to do. We need to get 10,000 of them. Why? And thatโs it. Thereโs no further thought process involved in it. Itโs lovely that they are doing research, but if they knew what decisions they wanted to make based on that, they could craft the thing a lot better.
โWhy do you do the smallest sample? Because itโs less work, itโs less intrusive, and it gives you the opportunity to do another one if the first one goes wrongโ
One of the things Iโve learned from the survey methodologists is that they try and do the smallest sample they can consistent with being able to get good quality results. Why do you do the smallest sample? Because itโs less work, itโs less intrusive, and it gives you the opportunity to do another one if the first one goes wrong, which can happen to the best of us.
Iโm going to give you an example that is quite painful. Survey methodologists in Russia are finding that people just slam the door in their faces. โIโm not answering your questions. I might be thrown into jail because the Russian state is locked down because of the war theyโve inflicted on Ukraine.โ So, boom, that would ruin your fieldwork, wouldnโt it? So, having a small sample gives you the opportunity to do another one another day, and it also gives you the opportunity to learn from the first one. So, with the exception of the censuses, which obviously cover everybody, they try to keep the sample as small and tight as possible.
The point you made is absolutely so accurate. Getting the right question in front of the right people; asking questions using familiar words in familiar ways; asking them in the language, the style of speaking, and phrasing that people use rather than your corporate jargon; all of those things really, really help get good quality results.
Preserving the value of surveys
Liam: The other thing that caught my eye was reading about what you wrote on fieldwork. You had an example of invitations and how invitations are so important, and one of the examples you had was a screenshot of an email that was very leading. It was basically encouraging you to leave a review, and it had five golden stars proudly displayed at the top. And I recognized these emails. Theyโre so transparent in what theyโre trying to do that it kind of puts me off leaving a review.
โPeople think, โOh, itโs so quick and easy to do a survey,โ which is sort of true, but itโs also quick and easy to do a really bad surveyโ
Caroline: Exactly. Using a survey to manipulate customers is clearly something that I cannot possibly recommend. And then the other one is using feedback to reward or punish your staff. So, you get these unedifying stories of customer service reps saying, โPlease can you give me a 10 on the Net Promoter Score, otherwise Iโll be punished?โ And that doesnโt feel any good to your organization whatsoever. Itโs undermined the entire method. All youโre doing is measuring the ability of your staff to guilt-trip your customers. Is that something you really want to measure? I think not. Trying to get some of these routine bad behaviors eliminated would be a great thing. Donโt just routinely ask and ask and ask in ways that get gained or manipulated. Itโs an unhelpful thing to do.
Liam: Before we wrap up, do you have any particular plans or projects for the year?
Caroline: Well, secretly, Iโd love to get back to forms. I sort of accidentally refocused my work into surveys, which is fun and Iโm happy to do it, but really, I love forms the most. So, if anyone happens to have any forms things theyโd like me to do, Iโd be extremely excited. But also, having spent a lot of time writing the book, Iโm very happy to have the opportunity to talk to people about their survey questions. For example, I do form studios a lot, which is a meeting where we look at a particular form. Iโm also doing quite a lot of survey studios where I get together with an organization that has a particular survey problem, and we spend an hour or so together just looking at the ins and outs of the sort of things I say, with a focus on what that organizationโs looking at today. Theyโre a lot of fun. I certainly would love to do more of them as well.
Liam: Absolutely. And where can our listeners go to keep up with you and your work?
Caroline: Well, my website is called effortmark.co.uk, and I have quite a lot of stuff on there. Iโve been writing various articles and so on for about 20 years now. I also try, wherever I can, to release my slides for people to use under a creative commons license. So, if you want to pick them up and use them, providing you keep my strapline and my creative commons there, Iโm very happy for you to have them. Iโm also on LinkedIn and on Twitter, and I enjoy chatting with people about forms and survey questions. Those are the three places to find me, and of course, my contact details are on my website. I impress my nephews and nieces by saying, โIf you google me, you get me.โ They think thatโs amazing.
Liam: I love that. Weโll link to all of those in the post. Caroline, thank you so much for joining me today.
Caroline: Oh, thanks so much. Itโs been so fun. I really appreciate the opportunity.