Analytics may not be the right measure
David Bornstein’s excellent book How to Change the World, which is about social entrepreneurs, the citizen sector, and the work of the Ashoka Foundation, concludes with a very interesting discussion of the use of metrics and analytics to value social enterprises. He says, in part:
Citizen groups and funders should remain cautious when embracing numerical assessments. The quest for quantifiable social returns or outcomes has become an obsession in a sector that envies the efficiency of business capital markets. Given this obsession, it is important to remember that numbers have an unfortunate tendency to supersede other kinds of knowing. The human mind is a miracle of subtlety: It can assimilate thousands of pieces of information — impressions, experiences, intuition — and produce wonderfully nuanced decisions. Numbers are problematic to the extent that they give the illusion of providing more truth than they actually do. They also favor what is easiest to measure, not what is most important.
He goes on to say that there are many areas of society in which we accept informed judgement, rather than pure analytics, as the best way to make decisions, for example, in our court system with the standard of reasonable doubt.
Rather than just using analytics to judge social enterprises, Bornstein suggests that citizen sector research analysts might be employed to assess efficacy using a variety of criteria and ultimately expert judgement.
Some particularly important points to me in this discussion include:
- By focusing on analytics, we naturally target our activities toward those numbers, not to our real goals.
- It is hard to resist gaming the analytics, again to the detriment of our real goals.
- Quality of service in the social sector is more than analytics.
This may seem counter-intuitive, but it applies to many things I am involved in right now. Online and blended learning. Assessment. Professional learning.
You can’t just boil those things down to numbers, and by trying to do so, we may be compromising our core missions.