In early 2016 the Global Innovation Fund invested in Young 1ove and Evidence Action to support the testing of a creative peer education approach to empower girls in secondary school in Botswana to reduce their risks of HIV/AIDS and unintended pregnancies. The Sugar Daddy intervention, which showed a clear effect on pregnancy in Kenya in 2005, had more mixed results in Botswana in 2016. In this guest blog, Young 1ove discuss their decision not to continue implementing this program as it was initially designed, and instead to respond to the evidence and pivot.
Imagine yourself in the mind of a teenager or young adult: what would influence you to make better choices about your sexual partners? In parts of Sub Saharan Africa, some girls are lured into relationships by “sugar daddies” – older men who offer gifts in return for sexual relationships. Many girls think that these older men are “safer” than younger men – more financially secure, wiser, more dependable. Few realise a 40-year-old is nine times as likely to have HIV as a 15-year old. Would knowing this information help influence teenage girls’ choice of sexual partner?
A Randomized Control Trial (RCT) in Kenya in 2005 found that it did. A 1-hour class that showed girls that older men were in fact riskier reduced pregnancy rates by 28%. Reduced pregnancy rates are also a proxy for unprotected sex and HIV transmission, indicating this could be a cost-effective way to reduce the spread of HIV. For example, it costs $6,700 to avert an HIV case with antiretroviral treatment (ART), $1,096 – $3,911 for male circumcision, compared to only $653 for a Sugar Daddy class.
With such promising results, the case for investing in the Sugar Daddy class seemed strong. We at Young 1ove, an NGO, decided to adapt this evidence-based program to Botswana. In doing so, we first chose to conduct a new RCT to see if the program replicated in the new context. The Global Innovation Fund provided grant funding to Young 1ove and Evidence Action to support the replication effort and an evidence-based response.
The preliminary results are in – with a working paper due to be published in June 2018 – and the Sugar Daddy intervention, which showed a clear effect on pregnancy in Kenya in 2005, had more mixed results in Botswana in 2016. Why?
First, the trial measured more things. On the main measure – whether girls had dropped out of school due to pregnancy – pregnancy decreased. This was the same measure used in the initial Kenya trial. However, a second measure of pregnancy that wasn’t used in the Kenya trial, visibly large stomachs, increased. While the second measure has not yet been validated, some academics interpret this result as meaning that the ‘sugar daddy’ message lost salience over time. As a result, some girls might have gotten pregnant a few months later once the intervention was over, and so were in an early, visible stage of pregnancy when data was collected.
Second, the trial had two groups of implementers: peer educators and teachers. This enabled the trial to test not only if the program worked, but also what the most cost-effective scale-up model would be. The results differed significantly between the two implementer groups, a clear result that shows the messenger and the model matters. While the teachers were trained to deliver messaging around the relative risks of older partners, in practice they often delivered a message that was closer to abstinence.
Third, the context had changed, and adaptations needed to be made. For example, in Kenya ten years ago, revealing to girls that older partners were more likely to have HIV was a cost they cared about since HIV was life-threatening. In contrast, in Botswana today – where ARV medication is widely available – HIV might not be as big a concern for young girls. Anecdotally, young girls claimed they were most concerned about the risks of falling pregnant, which is visible and can be debilitating. An additional lesson learned from the Botswana trial was that “sugar daddies” spanned multiple age ranges from 20 to 40, rather than the program’s initial focus, which centered on revealing the risk of the oldest age groups.
This goes to the heart of the debate on “generalisability” of evidence. Thanks to the rigorous testing in a new context, the government of Botswana and our team decided not to continue implementing this program as it was initially designed, and instead to respond to the evidence and pivot. Now, we are incorporating lessons learned into new and improved programing, such as a focus on peer-led delivery, and targeting the full age range of older sugar daddies. In the coming years, we plan to measure the program’s impact on HIV and STIs directly, moving beyond measuring pregnancy as a proxy for HIV transmission. Recently, we signed a 5-year partnership with UNICEF in Botswana, aimed at designing evidence-based sexual health programming that draws heavily on the Botswana trial.
The government is fully on board. At a recent event at the Young 1ove office in Botswana, the Permanent Secretary of the Ministry of Basic Education, Ms. Grace Muzila, emphasized: “as government, we welcome Young 1ove’s evidence-based approach.”
Our Executive Director, Noam Angrist, said: “We were founded on the notion that there is research sitting on a shelf, accumulating dust, needing to be scaled. We have realized, however, that the evidence base is often insufficient. Since we care about scale, we need to know which programs work and how they work across contexts. While there is often evidence from one place and point in time, there isn’t enough evidence across contexts. This is a crucial gap. Our RCT in Botswana proves the importance of bridging this gap; some program elements translated from Kenya, some did not.
Today, in Botswana, we are faced with a similar challenge to Kenya 10 years ago; equipped with rigorous evidence, how do we ensure it is used for maximum impact? We could scale the program up given promising results, with significant reductions in our main measure of pregnancy, ignoring the nuanced and complex ones. Alternatively, we could change course entirely, abandoning the “sugar daddy” ship given its complexities. We choose neither – both would leave evidence on the shelf and we were founded to scale evidence up. Instead, we are using the wealth of knowledge generated across Botswana and Kenya to do more of what does work and less of what does not – and are designing a program to best address the ‘sugar daddy’ challenge and prevent HIV. We are committed to bridging the ‘last mile of evidence’ – figuring out what works not only in one context, but across contexts – the youth we serve deserve it.”
Reflecting on the Young 1ove investment, the CEO of the Global Innovation Fund Alix Zwane said: “GIF takes smart bets on interventions that have the potential to be game changing, which also means discontinuing funding when evidence does not support further scale-up. The testing of the No Sugar program highlights the importance of rigorous evidence in policy making and testing interventions in multiple contexts before scaling up. The government of Botswana has saved a lot of money by waiting for RCT results before scaling a program. We commend Young 1ove for embracing the ambiguous RCT result, and look forward to seeing what new evidence-based interventions this exciting NGO comes up with.”
Karen Levy, Director of Global Innovation at Evidence Action added: “At some point, every evidence-driven practitioner is sure to face the same challenge: what do you do in the face of evaluation results that suggest that your program may not have the impact you hoped for? It’s a question that tests the fundamental character and convictions of our organizations. Young 1ove answered that question, and met that test, with tremendous courage. In the face of ambiguous results regarding the impact of No Sugar, they did something rare and remarkable: they changed course, and encouraged government partners and donors to do so as well. They pivoted away from an intervention whose impact was unclear, and in doing so displayed an incredible (and inspiring!) integrity. We could not be prouder to have partnered with Young 1ove, and we have no doubt that they will go on to innovate, pioneer and scale new interventions that transform the lives of millions of youth.”
 Lule, E., & Haacker, M. (2011). The Fiscal Dimension of HIV/AIDS in Botswana, South Africa, Swaziland, and Uganda: Experiences from Botswana, South Africa, Swaziland, and Uganda. World Bank Publications.
 Bärnighausen, T., Bloom, D. E., & Humair, S. (2012). Economics of antiretroviral treatment vs. circumcision for HIV prevention. Proceedings of the National Academy of Sciences, 109(52), 21271-21276.