« Why Your Last Ministry Survey was Wrong! (And What to do About it!) | Home | How to Make More Impact with Your Ministry Communications »
Bogus Surveys, Bad Question Design, Web Polls and other Research Pet Peeves!
By chris | January 31, 2007
I have a few pet peeves about ministry research I’d like to vent about. I don’t like the phony surveys people use for outreach. I cringe at ill-conceived research questions. And those website polls that masquerade as research also get on my nerves. I know, I need help. Maybe getting a life would help too! Recognizing you have a problem is the first step, right? Maybe talking about it will help too.
Pet Peeve: The Bogus Survey
I once had lunch with a noted evangelism leader who engaged the server in a dialogue while we were placing our order. After we had ordered our food, he asked the waiter a question,
“[Insert Servers’ Name from Nametag] I am conducting a survey. May I ask you a question? I am asking servers all over the country what they think God requires of people for them to have eternal life. That’s a big question isn’t it [Insert Servers’ Name]? What would be your answer [Insert Servers’ Name]?”
The “survey” took me by surprise. I knew the Christian leader wasn’t telling the truth. He may have been asking questions, but he wasn’t really engaged in research. His real agenda was to talk about the gospel with the waiter, not conduct a national survey of waiters. Why not just come out and talk about the good news? The gospel is good news enough in itself to merit conversation. After all, Paul said, “I am not ashamed of the gospel” (Romans 1:6) What’s up with the fake survey?
Bogus surveys abound in the church! I have seen church planters, college student groups and community churches doing the same thing in neighborhood canvassing, at beaches, county fairs, malls, and other public places. As a ministry researcher, I have a real problem with ministries that head out to the marketplace with a survey that is designed as a bait-and-switch tool for evangelism. The kinds of questions vary, but the objectives are the same.
- What do you consider to be the greatest problem in the world today?
- What do you feel is the greatest need in our community?
- What is the greatest need in your personal life?
- Would you mind if I took just two minutes of your time to share the greatest solution to life’s problems I have found?
Often, churches take the information from these surveys and feel they have done real research. But they have not! Not only have they not designed their survey with plans for managing survey error, they have abused the trust of the public with their “end justifies the means” approach to evangelism. Unwittingly churches that do this are actually decreasing the chances they have for doing more effective evangelism, since they are increasing bias against religious research in the very community they hope to reach.
It is only a matter of time before the general public in their community gets the idea that people asking religious questions are really just fishing for prospects to evangelize. As people become more jaded to religious researchers, they will become less likely to respond candidly about their opinions. This will limit the ability of reputable researchers to learn what are the real barriers and bridges for reaching people so as to help churches improve their outreach.
The practice of selling (or evangelizing) under the guise of research is called “Sugging” and is considered unethical by professional researchers. There are many other ways to conduct evangelism than feigning you are doing research. In my opinion, churches should not do what is considered unethical.
Pet Peeve: Poorly Conceived Research Questions
Writing survey questions seems easy enough. You just ask the questions that help you find out what you want to know, right? Not exactly, survey design takes more art than many seem to understand.
Bad question design brings survey error and bias into your ministry research. Even the best survey sample methodology, high response rates, and most accurate analysis, can’t make up for poorly designed survey questions.
Before you launch into a survey, make sure you know why you are doing it. There is so much to determine! What is your research objective? Is a survey the best way to get the information you need? How will you use the data when you have it? How long will the survey be? And many more questions.
When you get to the point of designing your questions take your time and make them work hard for you. It is always a good idea to test the survey on a few people and do a dry run, not for data gathering purposes, but to make sure the survey is easy to understand and gives you the kind of information you are looking for.
Take a look at the book The Gallup Guide: Reality Check for 21st Century Churches by George Gallup and D. Michael Lindsay You’ll get a lot of good ideas about how to design your own research and you might find the questions you need answered already designed in the back of the book.
Don’t write leading questions.
Sometimes our personal biases are exposed as we design survey questions. It can happen to anyone. As a ministry communication leader, you want to get to the bottom of the problems and barriers to reaching people. We read so many books and hear what people say in creative ministry seminars that we begin to feel we have an understanding about the people in our community we do not actually have any objective basis upon which to found our beliefs. When we set out to confirm what we think we already know, through research, we end up (although unconsciously) preaching our message through the survey instead of objectively researching what people really think.
If you were looking for a church, wouldn’t you prefer to attend one that doesn’t always preach about giving money?
- Yes
- No
- Maybe
- Not sure
In this example question, the researcher wants to find out if talking about money is a turn-off for unchurched people. But the design of the question may muddy the objectivity of the results from the respondents. If you design a survey that has leading questions, you will not measure the true opinions of people.
This question makes it clear what is the “correct” answer the researcher is looking for. To make matters worse the scale used draws a line in the sand, either you are for “always preaching about money” or not. Yes or no? Consider revamping this question, making it less leading and offer a Likert Scale for respondents.
How strongly do you agree or disagree with this statement: “Churches that preach about giving money are offensive to me.”
- Strongly disagree
- Disagree
- Neither agree nor disagree
- Agree
- Strongly agree
Designed this way, the question leaves more room for subtle nuance in the answer. People may be offended, but only mildly so. This question may be better placed in a list of issues that may or may not be offensive to unchurched people (with the same scale). That way you can get to the bottom about more of the things people find offensive.
Make your scales balanced
Take a look at the options you give people for response. Give them a complete and balanced scale for their response. Otherwise you are wasting their time and yours.
How well was your child treated by the nursery workers of our church?
- Very well
Make sure you have mutually exclusive categories. What do I write if I am 28 years old?
Which age grouping contains your age?
- 18-25
- 30-45
- 50 and Up
Don’t make your questions too broad. You can’t measure “all” of anything in one question.
How strongly do you agree or disagree that all the children’s programs of our church are satisfactory?
- Strongly Agree
- Agree
- Neutral
- Disagree
- Strongly Disagree
Don’t make your questions too long, overly complicated, unclear or filled with jargon.
Our church parking lot requires a new three inch layer of asphalt. This is according to the new city ordinance # 519-A Section B-10. This was passed in the most recent referendum of the city council May 23 of last year. In order to comply with this new legislation, our church will propose a bond issue that members will have the opportunity to vote on in an up coming church vote. In order to know how much money we need for the bond issue, we will send out RFPs to various contractors for the work we need done. Today we need you to help us select the kind of asphalt we will use on our parking lot for the RFPs (This is the North parking lot facing Johnson Street, not the North one that is backed up against the Dairy Queen. Just the outer one, by the sign on down to the curb). Which kind of asphalt would you prefer the church use?
- Aggregate Gradation HMA
- Compaction HMA
- SuperPave super performing HMA
Here is a variation of that one that is less obvious to church people
Are you involved in an accountability group in the church that holds you up before Christ each week?
- I am in a Sunday school Class.
- I serve with others in the church who help me
- I have a group outside the church
- I’m not currently involved in an accountability group
Accountability groups in church are great, but are you sure everyone understands what the term means? Accountability may mean one thing to you and another to your church members. Also, what does, “hold you up before Christ mean?” Is that prayer you mean? Do they have a little voodoo doll they hold skyward in a shrine room somewhere? Ask your questions in the simplest ways you can. If you have trouble being clear, see if your average twelve year old understands what you are saying. Trust me; they won’t cut you any slack!
Lame Jokes make lame research questions
This sounds like the “pot calling the kettle black” I am sure. I love to joke, but be careful you are not also being offensive in an attempt to be funny. In fact, save the jokes for the Christmas party. You may be introducing more than just a little levity to your survey. You may be inviting people into a biased response baiting them toward their preferred punch line.
Would you be in favor of a decorating committee painting the fellowship hall in a color that looks less like a Mexican piñata?
- Yes, I support el plan
- No me gusta amigo!
- I don’t speak English
Talk to the right people.
You can’t ask one group of people what the other group thinks and know anything about the other group. You will only measure what people think the other group thinks. They don’t know any more than you do about them.
In your opinion, do guest to our church understand the preaching of the pastor?
- Strongly agree
- Agree
- Neutral
- Disagree
- Strongly Disagree
Pete Peeve: Web Polls
I alluded to this in the last post. A poll placed on your site is not scientific research. Don’t delude yourself into thinking the percentages in your poll mean anything. It is misleading to have 152 people answer your web poll and then write an article about your “findings”
Warning: If you read an article based on a poll from a website, don’t base your planning on the survey. Some of the web polls are clever looking enough, but they are only, at best, qualitative (unscientific) research.
Topics: Develop the Message |
December 26th, 2007 at 11:11 am
[...] Bogus Surveys, Bad Questions Design, Web Polls, and other Pet Peeves [...]