« What in the World is Weltanschauung? | Home | Bogus Surveys, Bad Question Design, Web Polls and other Research Pet Peeves! »
Why Your Last Ministry Survey was Wrong! (And What to do About it!)
By chris | January 28, 2007
Your last survey was all wrong–but don’t worry—all surveys are wrong! Even professionally done surveys are incorrect! Every survey ever conducted by George Barna is wrong! Thom Rainer is all wet often too! Pew Research reeks with error! Don’t get me trotting out how the folks at Gallup just aren’t correct!
How do I know all surveys are wrong? Because all surveys, no matter who conducts them, have a margin of error in them. You have no doubt seen the disclaimers in survey reports where the researchers state their survey has specific degree of error. For example, they state their research has a confidence ratio of 95% and a margin of error of +/- 5%, and so forth. What they are stating is the degree to which they are wrong.
I am not being pessimistic about surveys. I want to make a point. Good researchers (like the ones mentioned above) have a plan for managing survey error. How about you? What is your plan? If you don’t have a conscious plan for managing survey error, you are setting your survey up for failure before you even start. Managing survey error requires diligent management no matter who conducts the survey.
Many ministries (even very famous ones) do surveys very poorly. They do not manage the basic elements of survey error. They don’t have an effective strategy for scientifically sampling the opinions and preferences of people. They design their questions poorly and jump to wrong conclusions when analyzing their data. And they often give advice and believe conclusions that are based on their own faulty research.
Bad research is just wasting time, energy, and money. If your church is going to conduct a survey, make sure you have a plan for managing the amount of error in your research.
Four Ways to Manage Survey Error.
Let’s reflect a little on research design and look at methods that make sure you are not “more wrong” than the pros. You can apply the same principles used by the professionals so you can get a handle on the accuracy of your research surveys.
1. Sample design. You need a proper random sample
Many church surveys are not conducted with thought for statistical sampling. Church committees who do research usually pass out surveys and expect that the results will give them answers to their questions.
To get good research you have to ask the right people, in the right number, selected in the right manner to get a predictable level of accuracy in research.
- You need the right people. For example, you can’t ask the church members, what the first time visitor thinks—how do they know? You need to make sure the people you ask in your survey fit the description of the people you want to learn about.
- You need to ask the right number of people. You can’t hand out 50 surveys, tabulate the results from 23 people’s responses and call it scientific. Depending on the number of people you want to research and the degree of accuracy you seek, you may need to conduct a census of everyone. If the number of people is very large, you will need to select and ask enough people to form a good cross section of people that represents the whole group.
- You need to select people in the right manner. That is, you need a random selection of people using scientific sampling techniques. Don’t be intimidated, done right, you could ask just 400 people questions and get an accurate representation for millions. Imagine if you invest in learning to conduct proper research, you could study and become the expert on your whole city.
Books on survey design
- How to Conduct Your Own Survey Pricilla Salant
- Marketing Research that Won’t Break the Bank Alan Andreason
2. Question design: You can’t write the questions poorly
Anyone can write a survey question, right? Well, yes and no. Anyone who knows how to write them can, but you have to learn how to design questions properly or you will bias your survey results.
In order to write good questions, you need to think about your research objectives and the people you want to research. People can’t tell you everything you want to know. Too often, people think they can ask the collective wisdom out there and get answers to the deepest secrets of the universe—paper or plastic?
Humans are also poor at predicting how they will behave in the future. So be aware that if you ask them if they will likely come to your new Saturday evening service, they may say they will and surprise you when they don’t.
People also can’t answer (though they will try) questions they do not understand. Make sure you define your terms. For example, don’t ask people in your church something like “How missional are we?” Unless you define what “Missional” means and are sure they also understand before they answer the question.
People can best tell you who they are; what they do, what they have done and know; and what they think and feel. Beyond that, it gets more difficult.
I am going to do a post about question design since this is a larger issue. Stay tuned!
Resources for helping you write survey questions
- Take the Questionnaire Design & Use training from Anderson, Niebuhr & Associates if you are going to be a serious marketing coach, you need to understand the principles of question design. (This course is worth the investment–I know, I took the course!)
- Questionnaire Design: How to Plan, Structure and Write Survey Material for Effective Market Research (Market Research in Practice Ian Brace
3. Response error: Not enough randomly sampled people.
Much highly touted Christian research is based on so few respondents that the findings are misleading. Dare I say, wrong?
For example, a popular approach is to conduct polls and surveys through websites to find out what people think or feel about a particular subject. These polls don’t tell anything about what the general population of people thinks, they only tell what the people who took the poll think. The results are not scientific because they do not employ sampling methodology.
So the reporting of percentages is meaningless. If 60% of the people who take the survey on your website say “you rock”, don’t be surprised when you find out the general populace thinks “you reek”. Wouldn’t you rather know if you actually rock or not? Survey says?
Don’t be fooled into thinking that just because you are conducting a well-designed survey that the results are scientific too. Even if your questions are correctly designed and the people you survey are the right people–you still have to contend with the statistical monster “margin of error”. If your survey sample says you need 400 people, then 299 are not enough.
You need to keep working until you get the right number of randomly selected right people answering all of your properly designed questions.
Some surveys conducted are too long. When a ministry decides they are going to do a survey (a once in a lifetime experience for many) they often think they might as well ask everything they always wanted to know, but were afraid to ask.
If you make your survey too long, fewer people will complete your survey. (You really ought to throw out incompletes.) You will need to keep the survey short enough so people will fill it out completely.
Does all this sound like hard cheese? Hey, who said science was child’s play? It’s easy to do good research, but it can be hard work. If you hold on and keep up until you get the proper number of responses, you will be rewarded with the sweet nectar of scientific accuracy!
Read this book on getting results from minimally-sized samples.
• The Snapshot Survey: Quick, Affordable Marketing Research for Every Organization Lloyd Corder
4. Reporting error: Reporters interpret the research with a bias, they can’t manage the data, or they don’t have enough skill to interpret the data.
Sometimes we see what we want to see by the way we read data from surveys. Many church leaders fool themselves (without knowing it) by looking at data in ways that confirm their biases. If you want to see something, you will see it. That’s how conspiracy theories are possible. Science calls for objectivity!
You need to be objective when you analyze data. Sometimes that isn’t possible. You may be too close to the situation to be objective. That’s when it is time to call in a professional researcher who can help you make a dispassionate assessment. If you get professional help, make sure you include them in all aspects of your research design. A good researcher can’t take a survey conducted with bad methodology and magically make it good and scientific.
Other problems arise when researchers are not equipped to analyze the survey they have conducted. The results are accurate, but the reporter doesn’t know enough about church growth, human behavior, advertising or marketing principles to make a proper analysis.
Sometimes the culprit of faulty research reporting is the researcher’s lack of skills in handling information in a database or calculating percentages and weighting cross-tabulated data. If you want to become a skilled researcher you can. But like anything worth doing, it will take work and dedication.
Resources for analyzing research
- The survey that would help you most has already been developed by Tom Rainer; take a look at getting a professional analysis of your church with the Church Health Survey. The report will be developed by trained, unbiased researchers.
- If you are the professional help, working as a marketing coach, learn to analyze data yourself by taking the Analyzing and Reporting Questionnaire Data Anderson, Niebuhr & Associates. I took this course and highly recommend it.
- Also become familiar with SPSS survey software
- How to Lie with Statistics by Darrell Huff is a funny and helpful book on the many ways we deceive ourselves with research reporting.
- Guerrilla Marketing Research: Marketing Research Techniques That Can Help Any Business Make More Money by Robert J Kaden
Topics: Develop the Message |
December 26th, 2007 at 10:58 am
[...] Why Your Last Ministry Survey Was Wrong and What to Do About it [...]