Instant Census Blog
By Josh Zagorsky on February 2, 2016
Survey research professionals often ask us how to build text message surveys using questions that they’ve previously asked in web surveys or phone surveys (either CATI or IVR). Because text message surveys have some advantages but also some limitations, here are a few tips on how to adapt your survey questions to SMS:
Popular in desktop web surveys, matrix questions ask several questions in a row that all have the same answer choices. But matrix questions are often difficult for respondents to answer on mobile web browsers, because they can involve lots of scrolling back and forth and up and down to see all of the options and what they’re labeled. SurveyMonkey recommends that if you expect most of your respondents to use mobile browsers, you limit your use of matrix questions and instead ask the same questions individually.
Text message surveys take this even further: we recommend not using matrix questions at all, and instead replacing them with individual questions. So don’t use a matrix question that looks like this:
| How often do you | Daily | Weekly | Monthly | Yearly | Never |
|---|---|---|---|---|---|
| Watch a movie at home? | _ | _ | _ | _ | _ |
| Watch a movie at a cinema? | _ | _ | _ | _ | _ |
| Watch a play or opera at a theater? | _ | _ | _ | _ | _ |
Instead, ask it as a series of separate questions that look like this:
How often do you watch a movie at home: Daily, Weekly, Monthly, Yearly, or Never?
How often do you watch a movie at a cinema: Daily, Weekly, Monthly, Yearly, or Never?
How often do you watch a play or opera at a theater: Daily, Weekly, Monthly, Yearly, or Never?
This is a better user experience for the respondent, because when answering each question, they don’t have to remember what the answer choices are. It also may cause the respondent to stop and think about each question, instead of blazing through all questions in the matrix by giving the same answer to each one.
When asking multiple choice questions in text message surveys, it’s helpful to solicit only single word responses. For example:
Are you Female or Male?
Are you registered as a Democrat, Republican, Independent, or Other?
You said you have a smartphone; is it an iPhone, Android, Windows, Blackberry, or Other?
If you ask text message multiple choice questions that solicit multi-word answers, you can set Instant Census to look for keywords in answers, so that Instant Census can properly code shortened responses. For example, if you asked the question:
Compared with one year ago, is your opinion of your Congressperson more favorable, less favorable, or about the same?
you could set Instant Census’ answer parser to look for the keywords “more”, “less”, or “same”, so that whether a respondent answers “about the same,” “the same,” or “same,” that respondent gets coded as “same.” This saves your respondents the hassle of having to enter exact response phrases, and it makes a text message survey seem more like a natural, human conversation.
If you need to ask questions that solicit multi-word answers that are more complicated, or must have overlapping keywords, you can consider numbering the responses like this:
What’s the highest level of education you’ve completed?
1- Some high school
2- High school/GED
3- Some college
4- College degree
5- Graduate degree
Likert scale questions are a subset of multiple choice questions that try to classify respondent opinions on a symmetric scale. Because Likert scale answers are often multi-word with many overlapping keywords, it’s best to number the options for Likert scale questions:
What’s your opinion of the 1803 Louisiana Purchase?
1- Strongly approve
2- Approve
3- Neither approve nor disapprove
4- Disapprove
5- Strongly disapprove
Whenever possible, try to keep SMS survey questions under 160 characters, the specified maximum length of an individual text message. Most text messaging apps have built-in tools to automatically handle longer messages by breaking them up into chunks on the sending end and splicing them back together on the receiving end. Instant Census has infrastructure to work with this. Contact us if you want more technical details on how this aspect of SMS messaging works.
But because cellphone carriers don’t guarantee the order in which messages are delivered, these methods work most of the time but not always. That means that if you send messages longer than 160 characters, they may occasionally arrive out of order.
Some phone and web surveys ask long, wordy questions with many clarifications to try to be as precise as possible. Precision is important, but some surveys prioritize precision at the expense of brevity and clarity for the respondent. Make sure your questions are short and clear enough that your respondents fully read and understand them!
The 160 character limit is restrictive, but it also encourages you to use one of the most helpful characteristics of text message surveys: short questions that are easy for respondents to understand and to answer.
One of the advantages of text message surveys is that if a respondent stops answering questions partway through, Instant Census can just resend the last question some hours or days later. Phone surveys and internet surveys don’t have that advantage; if a respondent puts down the phone or closes the browser window before completing all the questions, there’s often no easy mechanism for prompting the respondent to resume later at the same point in the questionnaire. This is another reason to make each question self-contained, rather than using matrix questions: it enables questions to be re-sent to non-responders, and it doesn’t require sending them context.
Want more advice on adapting your phone or web survey questions to text message surveys? Drop us a line!