Over the past 3 weeks, I’ve been working on a logic model and M&E framework for the market readiness project detailed in a previous blog post. These two documents will then contribute to a baseline survey of women participating in the project. I started out thinking these tasks were pretty straightforward. For all I’ve studied and read about evaluation, I surprised myself in thinking that trying to capture the impact of a women’s empowerment-focused intervention could be so simple and straightforward. I guess I needed to make that mistake myself in order to understand what all the evaluation colleagues, professors, and authors have been telling me all along. This post includes my discoveries around how not-straightforward this kind of work is – and how we should never assume that it is.
First, let me describe how my first attempt at these tasks taught me the importance of context when it comes to M&E. Based on initial discussions with the organization I’m working for and the literature I’d reviewed, we felt that looking specifically at changes in women’s household decision-making power seemed like an appropriate indicator of empowerment. Decision-making is one of the most common areas that’s investigated in women’s empowerment and several studies have been done in India on the topic. So I felt pretty confident that we were on to something. The problem was I had clearly overlooked context. I had done my homework (AKA research), but had essentially developed an M&E system from inside an office without going out to the villages to understand from the real experts what they think might change as a result of increased financial resources in the household. And looking back now, I see that early on in the process I was already developing assumptions and a sense of attachment to this concept of decision-making; partially because the research confirmed it and it seemed so straightforward. Lesson learned: it's pretty easy to develop great M&E systems that are practically useless if they're not tracking what's actually important OR likely to change within the communities you're working with.
In recent discussions, the realities of household decision-making in the Thar Desert communities have been illuminated for me. And here’s a few things I’ve learned in that process:
For starters, it’s easy for evaluators to fall into the trap of thinking that the conditions we want to evaluate are something your average person spends their days thinking about. The reality is most households probably don’t actively think about hierarchy in decision-making and it’s unlikely that it’s as simple as, “women make decisions about ABC, and men make decisions about XYZ”. In Rajasthan, a husband and wife and their children live with the man’s parents and extended family, and there’s loads of power hierarchies not only between, but also within genders that end up shaping decision-making. There's a lot of nuance that's unlikely to a) be captured in a log frame or M&E framework, and b) be affected by an injection of financial resources as a result of this project.
I’m finding that if I put myself in the-person-to-be-evaluated's shoes, it’s easier for me to identify how ludicrous some of my ideas are. For instance, if someone were to ask me to name which areas I have a say in when it comes to household decision-making and which areas my fiancé does -- and where we make decisions jointly -- it would take me a while to sort out the complexities of our relationship; expecting others to have concise answers to these questions is unrealistic.
I am experiencing the true iterative nature of evaluation and it’s messy, uncomfortable, and overwhelming. I have no clue if what I am doing is “right”, and I’m realizing that I’m not sure I’ll ever reach a point in my career where I feel like I am. During my research I came across a quote that I think describes the unease I am feeling in doing this work. On the topic of understanding household decision-making, the author writes, “ ‘statistical’ perspectives on decision making should be remembered for what they are: simple windows into complex realities. They may provide a brief glimpse of processes of decision making, but they tell us very little about the subtle negotiations that go on between women and men in their private lives.” (Source: Kabeer, 1999) I think this concept of simple windows into complexity can be extrapolated to almost any social change condition that we attempt to evaluate.
Overtime, I've come to the conclusion that we're drawn to the work that not only interests us, but also tests our core beliefs and values we've developed as human beings. I love evaluation for its potential to be used as a tool for organizational learning, but it also really tests my beliefs around perfectionism. I'm working to accept the fact that there's nothing I can do to ever fully know or understand the impact of social change, while also embracing the opportunity to peer through that little window into these amazingly complex realities.