David Bornstein’s excellent book How to Change the World, which is about social entrepreneurs, the citizen sector, and the work of the Ashoka Foundation, concludes with a very interesting discussion of the use of metrics and analytics to value social enterprises. He says, in part:
Citizen groups and funders should remain cautious when embracing numerical assessments. The quest for quantifiable social returns or outcomes has become an obsession in a sector that envies the efficiency of business capital markets. Given this obsession, it is important to remember that numbers have an unfortunate tendency to supersede other kinds of knowing. The human mind is a miracle of subtlety: It can assimilate thousands of pieces of information — impressions, experiences, intuition — and produce wonderfully nuanced decisions. Numbers are problematic to the extent that they give the illusion of providing more truth than they actually do. They also favor what is easiest to measure, not what is most important.
Profound.
He goes on to say that there are many areas of society in which we accept informed judgement, rather than pure analytics, as the best way to make decisions, for example, in our court system with the standard of reasonable doubt.
Rather than just using analytics to judge social enterprises, Bornstein suggests that citizen sector research analysts might be employed to assess efficacy using a variety of criteria and ultimately expert judgement.
Some particularly important points to me in this discussion include:
- By focusing on analytics, we naturally target our activities toward those numbers, not to our real goals.
- It is hard to resist gaming the analytics, again to the detriment of our real goals.
- Quality of service in the social sector is more than analytics.
This may seem counter-intuitive, but it applies to many things I am involved in right now. Online and blended learning. Assessment. Professional learning.
You can’t just boil those things down to numbers, and by trying to do so, we may be compromising our core missions.
Hi Karen,
I think numbers can be quite dangerous in the wrong hands. There is more to analysis than numbers and competent professionals analyze situations every day that do not provide numbers or ones that provide contradictory numbers. In education, we have seen how the reliance upon numbers misleads us into believing we are on the right track or uncovering the right information, but without other forms of analysis, we really cannot be sure what the numbers mean. For instance, a student who normally scores advanced suddenly drops on a given test to below basic. We panic and think the material has not been properly taught, but then the teacher adds that the student was running a 104 temperature before he left for the day. The 104 data trumps the test data. Another example that jumps out at me is that I read that HTH decided to put physics into 9th or 10th grade curriculum. This meant a severe lowering of their state test scores in science (since it is biology and chemistry that is tested in those grades). Nevertheless, a huge percentage of their graduates go on to successfully completing college in 4 years–a better stat than the test scores reveal. I hear more people discussing these sorts of issues of data reflection and I am glad to hear it. We need to be mindful of these tendencies of numbers to reduce us to lowest forms of thinking, especially in the social services fields. Thanks so much for you post!