Dropout prevention in vocational education: Evidence from Finnish register data

A large-scale dropout prevention programme was implemented in Finnish vocational education in 2011–2014. We used detailed register data to evaluate the programme’s effects and to circumvent the attrition bias of pretest-posttest surveys. Our data included approximately 24,000 randomly sampled students, both from the participating schools and the non-participating schools, before and after the programme. We estimated the effect of the programme on study completion and dropping out using pooled cross sections and difference-in-differences regression methods. Both study measures have improved during the last decade, but our results suggest that the programme was not effective in increasing study completion or in reducing the number of dropouts. The likely causes for the improved levels of vocational degree completion include recent macroeconomic fluctuations and legislative changes in teenagers’ eligibility for unemployment benefits.


Introduction
The socioeconomic effects of dropping out from secondary education have long been assumed to be negative, leading to unemployment, low income, poor health, and even social exclusion. However, concrete causal evidence has been scarce. A straightforward cross-sectional comparison of economic outcomes among dropouts and among high school graduates is misleading, as dropouts are likely to come from disadvantaged backgrounds. Dropping out is not randomly allocated, and missing data on family background characteristics causes bias in cross-sectional estimates.
In a recent study, Campbell (2015) presented new evidence of dropout effects. Using sibling data to control for unobservable background characteristics, Campbell found out that dropping out of high school has an independent net effect on economic hardship. Campbell used the income-to-poverty ratio, earned income, and employment as independent variables in his sibling fixed-effect models. Results indicate that the observable differences between dropouts and high school graduates are not entirely attributable to background effects, and dropping out does appear to increase socioeconomic disadvantages (Campbell, 2015, p. 116).
Hence, there are grounds for public programmes and interventions that aim to increase the completion of secondary education. As if in anticipation of reliable scientific proof of dropout effects, numerous such measures have indeed been undertaken during the last decades, in a variety of countries. Whether these dropout prevention interventions produce desired results is another question where we are lacking an unambiguous answer, also due to missing data problems.
Recent large-scale reviews by Wilson et al. (2011) and Hahn et al. (2015) sum up our current knowledge of the effectiveness of dropout prevention programmes. The meta-analysis by Wilson et al. (2011) yields a mean odds ratio estimate of 1.72, which is interpreted as an 8 percentage point decrease in the probability of dropping out in the treated student groups (N = 317 programmes). Hahn et al. (2015) provide similar calculations for the studies published after the Wilson et al. review and produce a median difference of 6.5 percentage points in the probability of high school completion in intervention compared with control populations (N = 11 programmes).
Both reviews therefore suggest that dropout prevention programmes clearly have beneficial effects. However, a closer inspection of the reviews raises some doubts. Due to limited or non-existent register-based data, most studies included in the aforementioned reviews have used pretest-posttest survey data. In this popular study design, treated and control students complete a questionnaire before and after the treatment. 1 With a 100% response rate, such a longitudinal design yields high-quality, unbiased data. The drawback is usually that a non-negligible number of students are no longer available for the second wave survey.
The reported average attrition rate in the dropout programmes included in the review by Wilson et al. (2011) was 11%, with a standard deviation of 20%. That is, within two standard deviations around the mean there have been programmes that experienced an attrition rate of 50%. Attrition was not reported in 17% of the studied samples.
If subjects are missing at random, even an attrition rate of 50% does not cause significant bias to results according to a simulation study by Kristman, Manno, and Côté (2004). However, if subjects are missing from the post-test survey in a non-random fashion, attrition rates of 20%-30% cause seriously biased estimates (Kristman et al., 2004, p. 757). As pointed out by the authors, the literature suggests that missing not at random is the most likely mechanism for loss to followup because missing subjects tend to have different outcomes from those that remain in the programme (Kristman et al., 2004, p. 752). 2 Depending on the associations between the expected outcome and the treatment, bias can be positive or negative. If we assume that the missing students from follow-up surveys are more likely to drop out than those that remain in the programme, our pretestposttest estimates of programme effectiveness are biased upwards and vice versa. Therefore, a meta-analysis of those estimates, like Wilson et al. (2011), also produces biased results.
In a related article, Kristman, Manno, and Côté (2005) studied the merits of widely used attrition correction methods, e.g. regression imputation, weighting, and multiple imputation. Using simulation studies, they found out that none of the methods was successful in correcting the biased estimates if the subjects were missing not at random and the attrition rate was higher than 25%.
Attrition problems may be circumvented using register data, as in this study. However, register data is not a magic bullet, either. In our case, we are not able to identify individual students who were in treatment; the Finnish National Board of Education did not collect participation information on the treated students. However, we are able to identify the treated schools and compare the outcomes of students enrolled in those schools with the outcomes of students enrolled in non-treated schools using standard difference-in-differences regression methods.
The aim of this study is firstly to evaluate the effectiveness of the vocational education dropout prevention programme of 2011-2014. Previous research on vocational education dropout and dropout prevention programmes is scarce (Cerda-Navarro, Sureda-Negre & Comas-Forgas, 2017). However, Andersen et al. (2018) studied a case similar to ours using Danish register data, and we compare the respective findings in the concluding Section. Secondly, we aim to compare our results, which are based on register data, with results from pretest-posttest study designs, which have been prevalent in dropout prevention research.
Our results imply that the dropout prevention programme of 2011-2014 failed to increase study completion or decrease dropping out in Finnish vocational education. During the last decade, dropout rates have indeed decreased and study completion has improved, but likely reasons for these changes include macroeconomic fluctuations and new criteria for youth unemployment benefits.
The remainder of the paper is constructed as follows. We first give a brief overview of the Finnish education system, and then outline dropout risk factors and the dropout prevention programme of 2011-2014. We then present our data, methods, and results, followed by a concluding discussion.

Finnish education system
The education system in Finland is similar to those of the other Nordic countries, and the main parts are outlined in Figure 1. There is a mandatory nine-year basic education in comprehensive schools, starting at age 7. 3 After the basic education, over 95% of students apply for secondary education in vocational institutions or in general upper secondary schools. Secondary education has a standard duration of three years, but it is also possible to integrate vocational education with general upper secondary education for a combined degree, which may then take a longer time.
After secondary education, students may apply for tertiary education in universities or start their working careers. Most opt for the latter, although many also continue their studies. According to the latest statistics from the year 2017, in the age group of 30-34-year-olds (N = 352,269), i.e. just above the usual youth age definitions, 16% of population had completed basic education  Education, 2019). only, 45% had completed vocational qualifications or general upper secondary education, and 39% had completed tertiary education, i.e. a bachelor's degree or higher (Statistics Finland, 2019).

Dropout prevention
Dropping out of education is not a snap decision. Rather, it is an accumulation of numerous risk factors, the accumulation often starting at an early age. There is extensive theoretical and empirical literature on this disengagement process, and the relevant risk factors are reasonably well understood. For recent risk factor reviews, see Ecker-Lyster and Niileksela (2016), Ripamonti (2018), and Gubbels, van der Put, and Assink (2019).

Dropout risk factors
Broadly speaking, dropout risk factors may be divided into four categories: student-related, family-related, school-related, and school-environment-related. According to the meta-analysis by Gubbels et al. (2019), advanced age, earlier grade retention, low academic achievement, and learning difficulties are significant student-related risk factors for school dropout. The family-related risk factors include low socio-economic status and low parental involvement, while large classes contribute to dropping out as a school-related factor. Gubbels et al. (2019) also found that involvement with deviant peers -which is related to school environment -is a highly significant dropout predictor. Overall, Gubbels et al. (2019) identified 21 separate risk domains, which had a statistically significant association with school dropout.
Dropout prevention measures aim to mitigate the presence of these risk factors. Naturally, several predictors -the family's socio-economic status, for instance -are fixed and not within the reach of preventive measures. Nevertheless, schools can decrease student/teacher ratios, offer more intensive student counselling, try to improve their education methods, etc. As mentioned in the Introduction, current research consensus is that the effects of these efforts are positive on average.
How well preventive programmes reach their objectives could also depend on the implementation quality and programme fidelity. Wilson et al. (2011) concluded that programmes which had difficulties with programme fidelity tended to yield smaller effects on dropout than programmes which indicated that no implementation problems were present. More recently, Freeman et al. (2015) and Goulet et al. (2018) presented similar findings. These results may have a bearing on our study, as will be explained later.

The dropout prevention programme 2011-2014
To recapitulate, in 2017, one in six 30-34-year-olds had no qualifications or degrees above basic education in Finland. Since over 95% of each population cohort enrolls in secondary education, there are many who drop out. Dropping out has been a major problem particularly in vocational education, where yearly dropout rates have exceeded 10% in some schools. In order to alleviate this problem, the Government decided in 2010 to implement a large-scale dropout prevention programme in vocational education, and allocated 16 million euros for four-year programme grants. 4 The programme was administered by the Finnish National Board of Education, 5 and it was implemented in 2011-2014.
The participating schools were not randomly selected to the programme. Rather, the schools (or 'education providers', to use the exact term 6 ) applied for the programme, and the Finnish National Board of Education selected the participating schools based on their grant applications, which had to adhere to the general guidelines of the programme. The actual measures taken were largely decided by the participating schools: some schools aimed to enhance student counselling, while others concentrated on pedagogical improvements. As most large vocational education providers participated in the programme, the share of vocational students enrolled in the treated schools exceeded 80%.
The treated schools had to provide a final report of the programme activities and outlays, and the Finnish National Board of Education officials audited these final reports. The Finnish National Board of Education also prepared a yearly monitoring report on the programme results. Additionally, an external evaluation report on the programme was published (Ahola, Saikkonen & Valkoja-Lähteenmäki, 2015). The authors of the yearly monitoring reports and the evaluation report mainly used various qualitative methods (surveys, interviews), and statistics compiled by the Finnish National Board of Education, but from the participating schools only. The authors did not collect comparison data from nontreated schools and used only descriptive statistics.
Both the yearly monitoring reports and the external evaluation report claim that the dropout prevention programme had beneficial effects on study completion rates and that dropping out decreased during the programme. However, since no comparison data from non-treated schools was analysed or even presented, there are no reasons to attribute these changes to the dropout programme, as a number of external factors could have produced similar changes.

Data and methods
As we mentioned earlier, most dropout prevention studies have used pretestposttest survey data. However, in the Nordic Countries, the National Statistics Offices gather high quality register data from various fields, including education. Therefore, we decided to exploit register data in our approach, combined with difference-in-differences regression methods for pooled cross sections.

Data
In our study, we used detailed register data from the student registers of Statistics Finland. First, we compiled a list of schools that participated in the dropout prevention programme. We then requested Statistics Finland to collect two random samples of students from the first-year vocational student cohorts of 2002, 2007, and 2012, i.e. six random samples in total. Three samples come from the participating schools and three from non-participating schools (control schools). We restricted our sample to upper secondary vocational education carried out in vocational institutions; youth apprenticeship training and adults' competence-based qualifications were excluded.
The sample size was determined as follows. Since almost 90% of vocational students were studying in the participating schools, and the size of each enrolling cohort was approximately 50,000 students, roughly 5,000 first-year students were studying in the control schools. Therefore, we requested that the sample size of each random sample should be approximately 4,000 students to achieve a total sample size of 24,000 students. The sample sizes that Statistics Finland delivered are tabulated in Table 1. Our data include the following background characteristics of the students: • age at enrolment year • gender • grade point average in basic education certificate • special needs education status (= 1 if student is entitled to special needs education, 0 otherwise) • socio-economic status of mother • socio-economic status of father • study field. We had two dependent variables in our study. Both are indicator (dummy) variables: • whether the student completed his/her studies in three years' time (= 1 yes, = 0 no) • whether the student dropped out before the end of third study year (= 1 yes, = 0 no). The target schedule for study completion in vocational education is three years. Hence, in our data the information on study completion was collected three years after the enrolment year, i.e. from the years 2005, 2010, and 2015 (end of each year). However, some students spend a considerably longer time completing their studies. If a student has not completed his or her studies in three years' time (i.e. the completion variable above equals zero), but is still enrolled as a full-time student, the dropout variable is valued zero as well. Consequently, our dependent variables are not perfect complements. That is, it is possible that both are valued zero for some students. Table 2 presents summary statistics of the background characteristics of both treatment and control school students from the year 2012 student cohort. Note that most students are aged 16 when enrolling to vocational institutions, but in our data there are some new students aged between 20-30 or even older, and the mean age reflects this fact.
In Finnish basic education, the subject grades range from 4 ('fail') to 10 ('excellent'). Among the control school students, the mean GPA was 7.3 in the 2012 cohort, and among the treatment school students, the mean GPA was 7.2. Figure 2 depicts the distribution of grade point averages of the treatment school students. Note that data on the grade point averages of several sampled students is missing in Table 2. The reason for this is that the data include only the GPA data supplied by the students' comprehensive schools. Self-reported grades were excluded as they could be unreliable. Older vocational education applicants have occasionally used self-reported grades in their school applications. From Table 2, we observe that vocational school students usually come from family backgrounds where parents are employed in lower-level clerical or administrative occupations (typically mothers) or as manual workers (typically fathers). When we compare the backgrounds of control school students with those of treatment school students, we find only minor differences. However, the study fields are somewhat different in the two groups. We are able to control for these observable differences by including Table 2 variables in our regression equations as confounders.
Our baseline regressions include all complete observations from the 2007 and 2012 cohorts. As explained earlier, there was no 'across the board' treatment, and the participating schools largely decided themselves what they did with the programme grant. However, the outlays had to adhere to the general programme objectives outlined by the Finnish National Board of Education. As a result, there was considerable variation in the actual treatments.
In order to account for the most obvious differences in the actual treatments, we classified the treated schools into three major categories according to their final programme reports before submitting the list of participating schools to Statistics Finland: • projects enhancing student counselling (1,122) • projects aiming at pedagogical improvements (1,142) • projects improving school infrastructure (89). Since several schools were involved in multiple (two or more) projects simultaneously, we also created a fourth category: • multiple projects (1,634). The number in parenthesis is the number of treatment school students in each category from the year 2012 cohort in our data. Using this information, we may then also study the effects of each treatment category separately.

Econometric model and identification
In our econometric analysis, we use the 2007 student cohort as the 'before treatment' sample and the 2012 student cohort as the 'after treatment' sample. As explained above, the year 2007 sample also contains information on study completion and dropping out from the year 2010, and the year 2012 sample contains information on study completion and dropping out from the year 2015. The dropout programme was implemented during 2011-2014. In our analysis, we pool the 2007 and 2012 cross sections and use standard difference-in-differences regression for pooled cross sections. 7 Our estimating equation (1) may be written as: (1) The variable dropoutprog equals one if the student was enrolled in a participating school, and zero otherwise. Since we have this information from the pre-treatment period (2007) also, dropoutprog essentially controls for unobservable systematic differences between students enrolling in treatment schools and control schools. Time-invariant differences may still exist even after controlling for the observable background characteristics, namely the variables denoted X (age, gender, GPA, etc.). For instance, the participating schools may be located in larger city areas, which offer plentiful opportunities for truancy compared to quieter rural areas. 8 The variable year2012 equals one if the observation is from the latter (treatment) period and zero otherwise. This year dummy controls for nationwide fiveyear changes in legislation, economy, employment, social benefits, and so forth, which may also affect study completion and dropping out.
The interaction term dropoutprog·year2012 measures the effect of the programme. It equals one for those students enrolled in the participating schools during the latter time period (2012), and zero otherwise. Note that since our dependent variables are binary, we are estimating a linear probability model. Therefore, the coefficients measure the percentage point change in the probability of study completion or dropping out.
The key identifying assumption in difference-in-differences regression models is that of parallel trends. That is, without the treatment, we should observe no differences in response variable development over time when comparing the treated units to the control units. We may test this assumption by using data from earlier time periods, i.e. before the treatment, and running an identical regression. In this case, we should observe an insignificant coefficient for the interaction term. We test this assumption below. Note, however, that the existence of parallel trends is a necessary rather than a sufficient condition for causal inference. There might be unobserved factors that changed simultaneously with the treatment and caused changes in the outcome variables, although the pre-treatment trends were similar.
Since it was the vocational education providers -not the students -that applied for the dropout prevention programme, this exercise resembles a natural experiment where receiving the treatment is more or less random from the viewpoint of the students. It is unlikely that a student enrolling in 2012 would have applied to vocational institutions solely based on whether the schools participated in the dropout programme or not. More likely determinants for school applications are the available study fields, school locations, and friends' opinions and choices. As most large education providers participated in the programme, the odds were that the average student enrolled in a participating school.
All student-level data in this study were stored in Statistics Finland's server and analysed using the Fiona remote access system. The statistics package Stata 15 was used in the estimations.

Results
As a first step in our analysis, we tabulate the outcome variables for each enrolment year and student group. Combining these tabulations yields the following Table 3. The numerical values of the treated (2012) student cohort are in italics. By design, our data include random samples of students from the treated and control schools, both before and after the treatment. Note that the percentages do not add up to 100%. There were some students who failed to graduate in three years but did not drop out, i.e. they were still full-time students.  Table 3 indicates that over the 10-year period 2002-2012, study completion has increased and dropping out has decreased in vocational education. We observe similar development in both student groups. From the year 2002 cohort, 28% of the control school students dropped out. Ten years later, the dropout rate was 16% -an improvement of 12 percentage points. In the treated schools, the year 2002 cohort had a dropout rate of 27%. A decade later, the dropout rate was 18%. There could be several reasons for these observations. During the years 2002-2005, when the year 2002 cohort was in vocational education, Finland's economy and employment were growing rapidly. In these circumstances, there were plenty of alternatives for studying, for instance taking on a full-time job. After the 2008 global financial crisis, Finland's economy experienced a deep and prolonged recession. Especially youth unemployment soared, which is depicted in Figure 3. The three vertical lines indicate the years (2005, 2010, and 2015) when the students in our samples should have graduated; i.e. three years after enrolling. Additionally, the criteria for youth unemployment benefits were tightened in 2013. Under 18-year-olds without a completed secondary education were no longer eligible for unemployment benefits. In this case, continuing one's studies despite minor school difficulties could well be worthwhile.  Table 3, the effect of the dropout prevention programme 2011-2014 is not immediately obvious -in fact, the completion rate increased faster among the control school students than in the treatment schools in the 2007-2012 student cohorts, and the percentage point decrease in dropout rates was equal. However, the simple comparison in Table 3 does not take into account the fact that there were observable differences between the control school students and the treatment school students (Table 2). There might also be unobservable systematic differences across time between the two groups of students, for instance related to school locations.

Baseline results
In Table 4, we report the results from our baseline regressions, which include the total available sample from the 2007 and 2012 student cohorts. For brevity, the coefficients on parents' socioeconomic status and study field indicators are omitted and are available on request. As explained above, those enrolled in 2007 belong to the 'before the treatment' cohort and those enrolled in 2012 belong to the 'after the treatment' cohort. Note that the number of observations is 14,175, which is less than the aggregate N of our pooled samples (3,978 + 3,993 + 3,775 + 3,987 = 15,733). As mentioned earlier, this is due to missing GPA data on some (mostly older) students, who have used self-reported grades in their applications. The estimates in Table 4 indicate that both study completion and dropping out depend on several observable factors. The student's advanced age at enrolment has a negative effect on study completion probabilities and a positive effect on dropout probabilities. Female students are more likely to drop out than male students. Special needs students, who may experience various learning difficulties, exhibit a similar pattern. A high grade point average decreases the probability of dropping out. If the GPA increases by one unit (i.e. from 7.0 to 8.0, for instance), the probability of dropping out decreases by 10 percentage points, ceteris paribus. The GPA in basic education certificate is the most significant independent variable in both equations, with t-values exceeding 10. The socioeconomic status of parents and the study field also have an effect on study completion and dropping out. The lower the socioeconomic status of parents, the higher the probability to drop out.

Variable
We find that the observed characteristics mentioned above account for most of the variation between the control and treatment school students. After controlling for observable factors, the treatment school students are not systematically different from the control school students. That is, the dropoutprog indicator is not statistically significant. Note that dropoutprog equals one for the participating schools and zero for the control schools, both before and after the treatment.
The year 2012 indicator is highly significant. This variable controls for various nationwide factors that may affect study completion, such as economic fluctuations, unemployment, and social benefits. We find that the three-year dropout probability decreased by roughly 6 percentage points from the 2007 enrolment year to the 2012 enrolment year.
We measure the effect of the programme with the coefficient on the interaction term dropoutprog·year2012. This term equals one for the treatment school students enrolled in 2012, and zero otherwise. Overall, the results do not seem very encouraging: both coefficients are statistically insignificant, by a large margin. Hence, we find zero programme effects on study completion and dropping out in our baseline regressions.

Alternative specifications
The next three Subsections contain statistical specification and identification tests. For a more general discussion, see the concluding Section.
In order to test the robustness of our baseline results, we experimented with various alternative specifications. In our basic equation, there is a single indicator, dropoutprog, which controls for unobservable differences between the treatment school students (dropoutprog = 1) and the control school students (dropoutprog = 0). As an alternative, we can use school indicators for each school separately. Although the identities of schools are withheld in our sample, the data include encrypted school codes, which allows us to estimate school fixed effects. In this case, it is necessary to omit dropoutprog in order to avoid perfect collinearity. Our difference-in-differences estimates with school fixed effects are presented in Table 5, Panel A.
As mentioned above, most vocational school students were enrolled in treatment schools. Therefore, the probability of being sampled in a 50/50 random sample was somewhat different for the treated students than with the control students. We test whether this could have an effect on our baseline results by using fixed probability weights for both groups in the regression equations. The probability weight for the control school students is 10.5, and the probability weight for the treated students is 1.6. The weights are based on treatment school student enrolment, as reported in the official programme monitoring reports, and on vocational education statistics. The weighted estimates are reported in Table  5, Panel B.
As a third alternative, we restrict our analysis only to those first-year students who were sixteen years old when enrolling to vocational education. This is by far the largest age cohort in our sample and also the most vulnerable group in many ways. 16-year olds have the least experience in life in general, they may still be in the middle of teenage turmoil, and they are easily led to other pursuits besides studying. Age-restricted estimates are presented in Table 5, Panel C.  Table 4 included, except for age included, except for age N 7,543 7,543 R-sq. 0.099 0.057 Regression results in Table 5 align with our baseline estimates, albeit with minor differences in the estimated coefficients. However, the main conclusions remain the same: the programme has had no effects on study completion or dropping out in Finnish vocational education, if we pool the treatment measures and analyse the average effectiveness of the programme.

Effects by treatment category
Next, we further elaborate our programme evaluation and study the programme effects by treatment category. As mentioned above, we classified the treatment measures into three categories, plus one extra category for multiple projects. Results by treatment category are presented in Table 6, Panels A-D. Each sample contains only those treatment school students that were included in a particular treatment category; other treatment school students are omitted. The group of control school students is the same in each regression.   Table 4 included  Table 4 included included N 8,763 8,763 R-sq.  Table 4 included included N 6,893 6,893 R-sq.  Table 4 included included N 9,849 9,849 R-sq. 0.092 0.058 Notes: School-clustered standard errors in parenthesis. * p ≤ 0.05; ** p ≤ 0.01; *** p ≤ 0.001.
Examining the results in Table 6, we observe that disaggregating the treatment measures and analysing the treatment categories separately unmasks several interesting findings. However, the evidence on the effectiveness of the dropout prevention programme seems to be far from clear-cut. The results suggest that when we analyse the probability of study completion, school infrastructure projects appear to be effective -the coefficient on the interaction term is +0.15, i.e. an increase of 15 percentage points, with a p-value smaller than 0.001. The drawback is that in our sample, only 89 students were enrolled in schools that participated in infrastructure projects. Hence, this positive result hinges on a relatively small number of observations. On the other hand, when we study the effects on dropout probabilities, treatments aiming to pedagogical improvements seem to fare well. The coefficient on the interaction term is -0.054, and the coefficient is statistically significant at the 5% level. Teaching innovations and improvements therefore seem to have a beneficial effect on dropping out, decreasing the dropout probability by 5.4 percentage points. In addition, the treatment sample in this case is considerably larger (N = 1,142) than in the aforementioned infrastructure projects (N = 89).
However, we obtain contrary evidence when we analyse student counselling treatments (Panel A in Table 6). The results seem to indicate that projects aiming to improve student counselling in fact decreased the probability of study completion by 7.5 percentage points, and the estimate is significant at the 5% level. This result is rather counterintuitive, as we would expect at least a zero effect when doing nothing.

Identification tests
Since we observe beneficial effects among those dropout prevention projects that targeted pedagogical efforts, we should also test whether our identifying assumption of parallel trends holds. This would require that we find no differences in trends before the treatment. As our sample contains data from three student cohorts, 2002, 2007, and 2012, we are able to test the existence of parallel trends by using the 2002 and 2007 cohorts in a pooled difference-in-differences regression identical to equation (1). 9 The regression is restricted to those schools that undertook pedagogical improvement projects during the dropout prevention programme.
This identification test yields a coefficient of 0.054 for the interaction term dropoutprog*year2007. 10 The p-value, 0.074, is quite close to the conventional risk level of 5%. That is, the treated and control schools already had diverging trends prior to the treatment. It is instructive to graph the time series of the three-year dropout rates in the control schools vis-à-vis those treatment schools aiming at pedagogical improvements. These time series are presented in Figure 4, where the vertical axis measures the share of students who dropped out before the end of the third study year. Three-year dropout rates in the control schools (N = 3,775-3,993) and in the treatment schools (N = 1,105-1,321) in the 2002, 2007, and 2012 first-year student cohorts, treatment: pedagogical improvements, implemented in 2011-2014. In Figure 4, we find unambiguous evidence of diverging trends before the treatment, i.e. from the year 2002 to the year 2007. In the control schools, the dropout rate decreased from 0.28 to 0.22 -an improvement of six percentage points. In those vocational schools that were treatment schools in 2011-2014, the dropout rate was approximately constant during 2002-2007. Therefore, we do not have a credible counterfactual for causal inference in our analysis. This would require that the trends were parallel ex ante.
We may apply a similar test to those schools that undertook student-counselling projects during the 2011-2014 dropout programme. This had a seemingly peculiar effect on study completion according to the results in Panel A of Table  6. That is, we use the year 2002 and 2007 student cohorts, and test whether the coefficient on the interaction term dropoutprog*year2007 is different from zero. If our assumption of parallel trends holds, the coefficient should not diverge from zero. In this case, our interaction coefficient is negative, but statistically insignificant, by a large margin. Hence, we observe parallel trends ex ante. Taken at face value, this result implies that the dropout programme caused a decrease in study completion rates in those treated schools that concentrated on student counselling projects.

Discussion
We studied the effects of a large-scale dropout prevention programme in Finnish vocational education. Instead of usual pretest-posttest survey data, we utilized highly detailed register data, and standard difference-in-differences estimators. Overall, our results suggest that the dropout prevention programme implemented in 2011-2014 did not succeed in reducing dropping out or increasing completion rates in Finnish vocational education. Our finding is in stark contrast to earlier, positive dropout programme evaluation results reported in the Introduction.
Otherwise our results are well in line with earlier research on school dropouts. We find that the student's advanced age, poor academic achievement, learning difficulties, and low socioeconomic status of parents are statistically significant dropout risk factors. We also observe that nationwide changes in legislation and youth labour demand have had an effect on school dropout. These factors may broadly be classified as school-environment-related risk factors.
However, our focus is on the dropout prevention programme 2011-2014, and according to our results, the programme failed to reach its objectives. There could be several explanations for this unfortunate outcome. Firstly, the dropout prevention programme in question may have been poorly planned and executed, and therefore produced insignificant results. Secondly, it may also be true that the dropout programme was in fact superbly managed and implemented by the Finnish Board of Education, but our result is just a random statistical coincidence. Thirdly, our disappointing result could be an outcome of using detailed register data, which has zero attrition rate. As mentioned in the Introduction, there have been dropout prevention programmes where approximately 50% of study subjects were unavailable for the posttest survey. As a result, the evaluation estimates of these programmes may have been seriously biased in favour of producing desirable results.
Which of the aforementioned explanations is true? Answering this question is plain guesswork at this stage. Poor programme implementation and intervention fidelity might be reasons for indiscernible effects in our case. As mentioned earlier, the participating schools were only obliged to follow the general outlines of the programme ('aiming to increase study completion'), and the implementation details were largely left to the participating schools to decide. As a result, there was considerable variation in the actual treatments. This may not have been the best policy, since finding out ex post what was being implemented is tricky on the basis of haphazard final reports.
We might assume that a random statistical coincidence is unlikely due to our large sample size (over 14,000 in baseline regressions). In any case, our results could be interpreted as a warning sign when pondering the reliability of pretestposttest study designs, the mainstay in dropout programme research. One could argue that we definitely need more register-based studies on dropout prevention in order to mitigate attrition bias.
Our methodological approach may be compared with a recent study by Andersen et al. (2018), who evaluated a vocational education dropout prevention programme aimed at improving participating schools' social environment in Denmark. 190) and logistic regression to estimate the effect on school dropout. Adjusting for age, sex, ethnicity, parental income, prior school dropout, and type of basic course, they find that after two years, the dropout rate was 36% in intervention schools and 40% in control schools. The difference of four percentage points was significant at the 5% level. Hence, they find desired effects for the dropout prevention programme.
However, the drawback in the research by Andersen et al. (2018) is that the participating schools were not randomly assigned. Rather, the same four vocational schools that had originally developed the intervention were chosen as intervention schools, while the control schools were chosen by the researchers to match the intervention schools in size, location, and basic courses. Thus, there might be significant self-selection bias in the estimates, since Andersen et al. (2018) are unable to control for unobservable factors due to cross-sectional data. That is, they have no observations from the intervention and control schools prior to the programme.
However, our study is not totally without good news. During our 10-year study period, dropout rates have decreased, and study completion rates have increased in Finnish vocational education -by several percentage points. Probable factors and causes for these beneficial changes include the prolonged economic recession after the 2008 global financial crisis, and the tightened criteria for youth unemployment benefits, which were implemented in 2013. After the year 2013, teenagers without a secondary education could no longer apply for unemployment benefits. Both occurrences have created economic incentives to stay in vocational education despite minor school difficulties.
Our study also demonstrates the importance of rigorous evaluation practices. In the official evaluation report of the programme (Ahola et al., 2015), only data from intervention schools were analysed. Improvements in study completion rates during the programme were then erroneously attributed to the dropout prevention programme. Since similar development occurred in the non-intervention schools as well, the probable causes were the nationwide economic fluctuations and the legislative changes in unemployment benefits. Counterfactual designs effectively unmask these and other similar confounders.
Our recommendations concern mainly future dropout programme implementation. The participating classes or schools should collect lists of participants, preferably students' social security numbers. This would enable evaluators to combine participants and detailed register data, avoiding the attrition problem of pretest-posttest designs. The drawback of this study is that we were unable to identify individual students who actually participated in the dropout prevention projects, as this information was not collected by the intervention schools. The implementation of school programmes should be as uniform as possible; in the present case, the schools largely decided themselves what they did with the grant. In retrospect, it may be difficult to decipher what the treatment actually entailed.
Endnotes 1 A simpler design follows the treated group only, but single-group designs were excluded from the Wilson et al. (2011) andHahn et al. (2015) review studies. 2 For an illustration related to school programmes, see for instance Weisman and Gottfredson (2001). 3 Note that the cutoff point is the end of each calendar year, so that those born in September-December start their school at age 6. 4 The programme's official name in Finnish was Ammatillisen koulutuksen läpäisyn tehostamisohjelma. 5 The current name of the organisation is the Finnish National Agency for Education. 6 Unfortunately, the Finnish education system is plagued with professional jargon. In this paper, we mainly use the old-fashioned term 'school' instead of 'education provider' or 'institution'. 7 For details, see for example Wooldridge (2010), Chapter 6. 9 The only difference is that the special needs education status dummy is not available for 2002. This has no effect on the results, since the dropout prevention estimate in Table  6, Panel B is nearly identical if special needs education status is omitted from the model. 10 Full results are available on request.

Note on contributor
Ville Vehkasalo is a Principal Performance Auditor in the National Audit Office of Finland, specialising in regional policies, employment policies, and education policies.