Tracking Poverty Change Over Time with the PPI: Lessons from VisionFund >

There are many ways organizations can use client-level poverty data to manage social performance. Generally speaking, these include assessing actual poverty outreach against goals, benchmarking against others’ poverty outreach, segmenting clients to analyze uptake of particular products by poverty level, and targeting poorer clients. Another one that is perhaps closest to our collective end goal is tracking progress out of poverty over time. It’s embedded right there in the very name of the poverty measurement tool that Grameen Foundation developed: The Progress out of Poverty Index.

Rigorously tracking changes in poverty over time is perhaps not the easiest aspect of social performance management, but we are seeing more and more PPI users begin to tackle it. We believe this trend can’t come soon enough, given the recent questions around whether MFIs and other social businesses are doing enough to measure and maximize their impact.

So we met recently with our friends at VisionFund International to learn how they are solving the challenges of tracking change in client poverty estimates over time. VisionFund has one of the most comprehensive implementations of the PPI we have yet encountered – they operate in 35 countries and 18 MFIs are using the PPI – and we believe their experience can serve as a model for others.

One initial takeaway from the learnings that VisionFund shared can be boiled down to two words. Data. Quality.

Data Quality

In 2014, VisionFund launched an effort to pilot tracking change over time with several MFIs who had become sufficiently advanced in using the PPI. The initial reports were promising, indicating that longer engagement with VisionFund correlated with greater movement out of poverty. But poor data collection in one country led to clearly erroneous results. It became obvious that they needed to institute data quality standards across the board.

Percentage of VisionFund Clients Below the Poverty Line Moving Out of Poverty
 
from VisionFund’s 2014 Social Performance Report

They asked MFIs to begin sending regular reports on data quality, centered around three components taken from the PPI Standards of Use:

Adequate Sample Size

Was the reported sample random and representative?

Spot-check Rate

Were at least 5% of surveys verified by someone other than the original enumerator?

Discrepancy Rate

Were less than 10% of spot-check responses different from the original responses?

Failing any one of these three tests can lead to erroneous results; for example, one MFI achieved a 10% spot-check rate, but still showed a discrepancy rate of 22%. Only two of the 15 MFIs who submitted reports passed all three components.

VisionFund began working with PPI champions within the MFIs to uncover ways to improve, and reached the following recommendations:

• Make one person at each branch responsible for data quality.
• Institute data-quality incentives for loan officers, monetary and/or non-monetary, such as reward badges or a monthly spotlight memo.
• Survey all new clients, not just a subset, to ensure representativeness and increase sample size.
• Train PPI enumerators regularly to reduce discrepancy rates, and require training for onboarding all new loan officers.
• Sell branch managers on the value of PPI data.

Data Analysis

In late 2014, Vision Fund hired Mark Schreiner, the developer of the PPI, to help VisionFund Ecuador (VFE) with analyzing client poverty estimates over time.  The learnings from that work were valuable:

Giving numbers context:

Rather than simply creating reports that left the reader without an understanding of what the numbers meant, the VFE team learned how to use poverty estimates as an entry point into further analysis.

Examining client characteristics:

Related to the point above, VFE learned the importance of juxtaposing changes in time with other client characteristics. For example, VFE disaggregated the results based on clients' economic activity, location, and loan methodology to identify differences in rates of change in poverty estimates.

Accounting for time:

Before the consultancy, VFE only presented the poverty reduction as xx percentage points, without mentioning the rate of change. Mark showed VFE how to select an appropriate unit of time to calculate the rate of change (i.e., poverty reduced xx percentage points per year). Given the variation in time between client interviews, this can be somewhat complicated to calculate, which highlighted the need to develop a spreadsheet tool that could calculate the rate automatically.

A deliverable from Mark’s consultancy with VisionFund Ecuador was a paper describing how to do poverty-scoring analysis, both for point-in-time and tracking through time.  In order to make it less daunting for the typical PPI manager, VisionFund and Mark continued to collaborate in the development of a spreadsheet tool that would streamline the analysis process (including sampling, data entry/storage, and computing/reporting). In November, VisionFund will pilot this poverty analysis tool with their MFIs.  It will allow the user to enter/extract data and create reports quickly and easily, both for point-in-time and tracking through time.

As VisionFund continues to expand tracking-across-time in its network, it is in parallel working with its MFIs to improve data quality practices. With better data and enhanced analysis practices, VisionFund will be empowered to better manage all aspects of their social performance, and ultimately help more people move out of poverty.  We look forward to following up with VisionFund after they have more tracking over time results and can share further lessons learned from the analysis process.  Once finalized, the analysis tool will also be shared with the broader PPI user community for everyone’s benefit.