Another post about recent evaluation work I've been doing in India. As mentioned previously, my main project here is to develop a survey to collect baseline data on a new women's economic empowerment project. We want to collect data on women's economic status, so that later on we can compare before and after to see what outcomes --if any-- did the project contribute to. It's a pretty cool hands-on experience with lots of challenges. I've developed surveys in the past, but this is the first time I'm both developing and implementing a survey. I've definitely learned in this process that survey design is not as easy or as simple as it seems.
Here are some of my learnings (and some of my favorite cartoons by Chris Lysy):
It's a misconception that surveys are simple: If it's easy, you're probably headed down the wrong path. I've learned in this process that every detail matters, especially when developing questions for projects where the interviewees are of a different culture and speak a different language. There's a lot of hurdles involved in translating concepts of empowerment from English to Hindi. What makes sense to me, quite literally does not translate. It's forced me to re-think and question my questions-- which is an odd experience in and of itself.
You can only revise a question so many times: I've come to find survey development to be a mind numbing experience. I can only look at and revise a question so many times before I can no longer comprehend the words on paper. It's been helpful for me to take a step back and take a break from the work for a while or ask for others' opinions.
Surveys are limited: This isn't a mind blowing finding, but it's a friendly reminder that there's no one panacea for all evaluation woes. As I learned with the M&E framework process discussed in my last post, surveys are another example of a tool that attempts to understand a complex reality. And like everything else there's shortcomings that need to be taken into consideration.
It's an iterative process: Just like everything else in evaluation. The survey I'm working on has been revised at least 5 times in the past week and a half. And I haven't even gone out to the field for pre-testing with project participants yet. Between working with my colleagues on question development, to working with my translator on comprehension, to repeating that cycle many times over...I think you get what I mean.
Am I alone in this mind numbing iterative never-ending process? Certainly not. Here are some resources I've found from others who have faced similar survey challenges:
How to design survey forms for quick data entry - http://www.tools4dev.org/resources/how-to-design-surveys-and-forms-for-quick-data-entry/
How to write awesome survey questions -- http://www.tools4dev.org/resources/how-to-write-awesome-survey-questions-part-1/
How to pretest and pilot a survey -- http://www.tools4dev.org/resources/how-to-pretest-and-pilot-a-survey-questionnaire/
Sheila B Robinson
Several great blog posts on the topic of survey design -- http://sheilabrobinson.com/tag/survey-design/