In 2015, Room to Read launched its Girls’ Education Program evaluation in India, funded in part by the U.S. Department of Labor, Echidna Giving and the Bill & Melinda Gates Foundation.
Besides aiming to measure the effectiveness of the Girls’ Education Program in improving a range of outcomes, including academic performance and the time participants invest in education and work, the evaluation seeks to fill a critical gap in our knowledge by rigorously assessing girls’ development of life skills, which include confidence and self-esteem as well as such competencies as decision-making and perseverance.
“A focus on non-cognitive or life skills is an area that has received a lot of attention in the U.S. because we’re realizing these skills can have a significant impact on a student’s welfare,” noted one of the evaluation’s principal investigators, Williams College assistant professor of economics Jessica Leight. “But this is a relatively new concept in the developing world. There isn’t a lot of evidence beyond the anecdotal, so our aim is to contribute solid research to it.”
Using a randomized controlled trial (RCT) design, considered the gold standard in scientific research, the evaluation compares girls from two different groups of schools in India. Schools were randomly assigned to either the program or control group in order to allow for unbiased comparisons. The program group comprises 60 schools whose female students are enrolled in Room to Read’s Girls’ Education Program. The control group is a different set of 59 schools nearby whose female students are not in the program and are not receiving any kind of life skills training.
But how exactly does a scientist measure life skills? Leight’s team faced a number of challenges. “Comprehension is a problem. Different girls will perceive a particular question in different ways,” she noted. “Another challenge was to develop concrete questions that the girls can relate to their daily experiences. It’s not easy to probe their aspirations around careers, for example, when they have typically only been exposed to a small number of women working.”
The team also needed to ensure that the questions asked by the study generated a range of responses. “If we simply ask how the subject feels and everyone answers positively, then we haven’t learned much from that question,” Leight explained. “We want to structure questions in such a way that the girls feel comfortable selecting any one of a number of options to reflect their own personal experiences.”
An additional challenge was developmental. In India and most other Room to Read countries, lower secondary school (and our program) starts in the sixth grade, and abstract concepts like self-esteem and perseverance can be hard for 11- and 12-year-olds to grasp. “When you ask a girl, ‘How do you react when you encounter an obstacle?’ it may be hard for her to understand the question,” said Leight. “Questions needed to be made as concrete as possible, so that girls can relate them to specific circumstances or behaviors in their own lives.”
Finally, Leight’s team needed to ensure that they are measuring what they intended to measure. “In answering the questions, the girls may just repeat what they learned in the life skills lessons,” she explained, “while the girls in the control group didn’t hear the lessons so they would not be able to repeat them. That means we may just be picking up on exposure to life skills education.” The team’s solution was to supplement the questionnaire with more objective measures, which is how they came up with the mirror task and the scavenger hunt.
In the mirror task the girls are asked to draw a series of shapes by following their reflection in a mirror. The shapes become progressively harder to draw with the last shape being nearly impossible to reproduce accurately. “We see how long she is willing to do this, testing her perseverance,” said Leight. In the scavenger hunt the girls receive a list of things to obtain, such as an item of clothing or food. “She’s encouraged to borrow them or find them at home,” Leight explained. “The girls might ask family, friends, or others they know. It shows how well girls can advocate on their own behalf.”
A girl performing the mirror challenge in India.
These two tasks may seem like pure fun and games—and from what we’ve seen so far, girls have a good time completing them! But there is more here than may initially meet the eye. Because these measures ask girls to put key skills like perseverance and negotiation into practice, they are objective in a way that simple self-report questions can’t be. There is no way for a girl to make it up, or to simply tell her interviewers what she thinks they want to hear. And thanks to our rigorous RCT-based design, we can make a compelling case that any systematic difference we find between program participants and non-participants in the performance of these tasks is the result of our program.
In 2015 the team piloted the evaluation and selected the schools for the sample. Interviews with all 2,500 girls participating in the study began in early 2016. The evaluation will be ongoing through 2018.
Room to Read has already begun adapting the measures that Leight and her team developed in order to create a life skills assessment tool for our own ongoing use outside the evaluation and after it ends. Developing this internal tool presents additional challenges: it will need to be adjusted to meet the needs of a wider range of ages, and across a wider variety of cultural contexts. Certain particularly time-consuming or costly elements that work well in a large-scale evaluation will also need to be reduced or eliminated for use in ongoing monitoring, but this must be carefully balanced against the need to retain the reliability and objectivity of the evaluation tool. We have initiated our first phases of pilot testing for this work, and will report on our progress in next year’s Global Monitoring Report.
***
This story is part of our 2015 Global Monitoring Report. Check out the full online report here.
Download the full report (PDF, 11 MB)
Measuring the effectiveness of a life skills education
