IkamvaYouth’s Evaluation Launch

The Western Cape Government hosted the After-School Game Changer’s symposium on the 16th and 17th of February. The two days were packed with interesting speakers who work in different parts of the education and after-school sector. IkamvaYouth was fortunate to secure a spot at the end of the symposium to launch the findings of a quantitative assessment of the effectiveness of our programme. The assessment was conducted in 2016 by Servaas van der Berg and Lewis McLean, economists from Stellenbosch University.

In 2012, Prof van der Berg and Dr Nic Spaull produced Against the Odds, a largely qualitative study of IkamvaYouth, which provided a range of very useful insights, for example, into learners’ socio-economic situations, their performance in Maths and Science compared with the feeder schools, and the challenges they face at tertiary institutions. However, a major limitation of this study was the lack of realistic comparator groups that could serve as counterfactuals. A big challenge was finding an appropriate benchmark against which to measure the ikamvanites’ achievement. How do we know that their great results are due to IkamvaYouth, and not due to their own motivation and achievement which would have led them to perform well anyway, without being a part of IkamvaYouth?

We were thrilled to have Prof van der Berg return in 2016, together with Lewis McLean, to tackle this challenge and assess IkamvaYouth’s impact with a more rigorous approach. In Succeeding Against the Odds, the researchers have managed to control for learners’ prior performance (and thus for factors such as pre-programme motivation and ability), by matching learners with a comparison group of learners who were not part of IkamvaYouth, but achieved the same results in the grade 9 systemic assessments. They could then compare ikamvanites’ matric exam performance with this group of learners, as well as with the results of the feeder schools (in quintiles 1-3), and with quintile 5 schools.

The researchers presented their findings by taking the audience through this presentation, and demystifying the various graphs and statistics. The graphs depicting the distribution of scores for various groups of learners are especially compelling. As background for viewing these distributions, it’s helpful to have a look at Nic Spaull’s article about school inequality in South Africa, where he shows that we essentially have two public education systems operating in parallel; the distribution of scores for learners in the lower quintiles (poorest) schools, compared with those in quintile 5 (most privileged) are markedly distinct from one another. In Succeeding Against the Odds, van der Berg and McLean show that Ikamvanites’s scores bridge this gap, and, in the case of Life Sciences, even meet the distribution of quintile 5 schools.

Andrew Barret, from Olico, said that “what is especially noteworthy about this study is not just that it shows the clear impact of IkamvaYouth, but that it in all likelihood underestimates this impact. As impressive and significant as these results are, the reality is probably even better,” as the researchers included a host of control variables.

As part of the lead up to the presentation, we heard from Yanga Totyi who is the branch coordinator at our Atlantis branch, as well as Ntebaleng Morake an alumna from the Ebony Park who completed matric in 2012. IkamvaYouth Atlantis opened in June 2016 and Yanga spoke about the challenges and triumphs they have had so far. Ntebaleng Morake is a UCT graduate with an honours degree in Gender and Transformation. Ntebaleng accredits her exposure to different fields of study outside of the conventional to her time at IkamvaYouth. Through the programme she was able to meet young, black men and women who were studying and working in different fields and the poverty induced veil of ignorance was lifted.

After the presentation, Dr van der Berg mentioned that the 2012 evaluation had made two recommendations: (i) The alumni in tertiary are struggling and need additional support, and (ii) the researchers cautioned against fast organisational growth, as they felt that the dilution of the organisation’s enthusiastic leadership across a much bigger organisation may lead to the programme losing some its attractiveness to students. He said he was very glad that the organisation had heeded his first recommendation (by establishing the alumni department), and not the second (IkamvaYouth is now operating in 16 townships; a relatively quick expansion from the 5 sites in operation in 2012). When asked for his opinion on what it is that makes the programme effective, Dr van der Berg said he suspects that there is something in the model which makes motivation contagious. He advised us to look into research around Nudge theory, which we will do, as we continue to understand just what it is that makes the ikamvanites so awesome.

We were very encouraged by the participation from other practitioners in the field, who chose to spend their Friday afternoon with us engaging with statistics. Some attendees took to the Twittersphere during the launch:

IkamvaYouth sends out loud big-up thank yous to:

• DGMT, which got us onto this evaluation journey back in 2012

• The Omidyar Network, which funded the 2015 evaluation

• The Department of the Premier of the Western Cape, which accommodated us alongside the very auspicious Game-Changer symposium

• Yanga and Ntebaleng for speaking at the launch, and grounding this research and the findings within the contextual realities of daily branch operations, and #feesmustfall activism respectively

• Dr Servaas van der Berg, Lewis McLean and Nic Spaull from RESEP whose insights and perspectives on our work are greatly valued, especially as we know that they are all used to working with far larger datasets, for far more important entities (like the national Government!)