Thursday, December 2, 2010

Can I ask you a question?

The research projects here are run almost uniformly by economists, social scientists who see the world through the lenses of quantifiable data and predictive models.   They base their data points on questions like: How much money did you make, doing what and for how long?  And what did you spend it on? What was your child’s birth weight? How many times did you get sick and how much did you spend on it?  What price did you get for maize this month vs. last month?  They use these reams of data to explain whether or not an intervention had a “measurable impact” on lives and livelihood. 
They can thus say with a level of statistical confidence that the people in the group who got a treatment (a certain education intervention, access to credit etc…) are doing better than the group that didn’t along these dimensions: less likely to become sick, more income, greater harvest etc…  and can conclude that an intervention has been successful.
But there’s something missing in all of this, and even the economists acknowledge it.  It’s the why. 
Why is something working? Why are people deciding to open a bank account or not? Why do they choose to forgo fertilizer when it has benefited their neighbor? Why do they choose not to put water purification tablets in their water even when it’s free?  And until you know these answers, it’s difficult to say for certain that an intervention that works well in Kenya would also work well in Indonesia.   
This is where it’s helpful to have a sociologist’s perspective, that “softer social science” that I imagine normally evokes a bit of disdain from mathematically-minded economists.  That is, until they need to understand why something happens the way it does in a way that regression analyses and modeling don’t tell them. 
The researchers I’m working for now acknowledge the need to better understand the how and the why that hard data misses, and so one of my current tasks is to design and pre-test a qualitative survey asking people more directly why they make the decisions they do, what they fear and what they hope.  Of course, this is not as easy as it might seem and there are layers of cultural barriers to dismantle in order to collect information that best mirrors people’s reality.  But it’s a fascinating puzzle to solve.
Here’s an example:  When talking to people about saving for a goal we want to ask them to distinguish between their wants and needs.  We ask the question in the clearest possible language like “think of things that you are always happy to have when you have them, but that are not so necessary” without giving them actual examples.  But when we asked this question, our respondents were nearly uniformly baffled by the question.  They simply did not understand the distinction.  “If I buy something, it’s because I need it.”  Even if it looks like a “want” (like a trip to the salon), it doesn’t feel that way to me and I’ve expressed that by spending my meager income on it. 
But there’s also apparently less of a distinction between “need” and “want” linguistically, which may serve to further blur the distinction.  Yesterday, Colin was trying to tell Jane in Swahili “I need to go back to work” (nina hitaji ….) but instead said “I want to go back to work” (nina taka ….).  Being the self-appointed family Kiswahili coach, I corrected him and told him you mean “nina hitaji” (I need/must).  But Jane then duly corrected me and said “nina taka, ninahitaji.  They mean basically the same thing.”
We have another question asking respondents how likely they are to succeed at saving a specific amount of money.  (very likely, probably… all the way to doubtful)  Again, a seemingly straight-forward request.   But this assumes that the respondent has some sense of control over their future, has the confidence to make a prediction about it and is unafraid to tempt fate by voicing it.  So, we almost never got anyone to fit their answers into our neat little likely/probably/doubtful boxes.  Respondents simply said again and again, “I will try,” which really doesn’t answer the question. 
So the trick is to really disentangle what we want to know, unpackage the layers of cultural confusion and then ask it in a way that will provide a meaningful answer.  But the payoff of doing it right is finally understand the part of the story that fancy statistical analysis can’t quite capture.

2 comments:

  1. Similarly, this was the discussion at the two international design seminars I went to in Vermont and Copenhagen back in 2007. The discussion involved whether design (in all derivatives of the word) should be measured, and if so, then how? The WHY debate you discuss (battle of the "sciences") reminded me of the notion of whole-brain thinking. Right brain approach versus the left brain approach. Why do we not consider both sides, and who's to say which prevails? The emotional impact on decision-making versus the quantifiable ones. Which came first, the action or the factors that led to the actions. Chicken or the egg? May be off subject a little bit, but reminded me of this debate quite a bit. It's intriguing. One would think it so obvious...well, miss you. (JOR)

    ReplyDelete
  2. Fascinating, kimmy. It would be really interesting to present a survey to people that asked something like, "why do you think this program worked" (or didn't). I remember attending an FDA meeting on organic standards where one panel member informed the people giving public comments on the record that they didn't count: statistically, they had removed themselves from the general (presumed disinterested) population by showing up.

    Even large businesses are learning to apply programs differently in different locales (marketing efforts being a standout example).

    When everything works out to numbers, we lose the human perspective - maybe the trick is finding out where that works and where it doesn't. It seems like the only fair way to look at distributing resources, for example, but as you say, what happens after such decisions are made?

    Maybe more "soft" science would also help avoid the pitfall of coming in and describing to people what they "need".

    Love you!

    Josh

    ReplyDelete

There was an error in this gadget