Evaluation Summary of the Atlantic Youth Inclusion Program
Table of Contents
- Author’s Notes
- Evaluation Objectives
- Evaluation Methodology and Limitations
- Short-term Outcomes
- Intermediate Outcomes: School Performance, Absenteeism and Suspension
- Long-term Outcomes: Reduction of Criminal Offending
by Danièle Laliberté
Developed in 2000 by the Youth Justice Board for tackling youth crime in England and Wales, the neighbourhood-based Youth Inclusion Program (YIP) aims to reduce youth crime and anti-social behaviour by creating a safe place where youth can learn new skills, participate in activities with others, and receive educational support. More evidence was needed to determine YIP’s effectiveness in a variety of socio-cultural contexts. Public Safety Canada (PS) contracted NRG Research GroupFootnote1 to conduct a multi-site impact evaluation of three projects implemented in the Atlantic region with funding from PS: Northside YIP in North Sydney, Nova Scotia (January 2010 - June 2013); Seeds of Change YIP in Spryfield, Nova Scotia (September 2010 - November 2012) and ONE Change YIP in St. John, New Brunswick (April 2010 - January 2014)Footnote2. This final evaluation summary provides an overview of the evaluation study valued at $427,534.00, which began in August 2010 and ended in March 2014.
The views expressed are those of the evaluators and do not necessarily reflect those of Public Safety Canada. Correspondence concerning this report should be addressed to:
Public Safety Canada
340 Laurier Avenue West
The author of this research report, Danièle Laliberté, would like to acknowledge the evaluators Nathalie Gagnon and Lesley Duncan for their excellent work; the host implementation agencies for their strong support in all aspects of the evaluation (Chebucto Connections, ONE Change and Island Community Justice Society); and the police and schools who provided their data and played a key role in the evaluation (Cape Breton Regional Police Service, Halifax Regional Police and Saint John Police Force, Cape Breton-Victoria Regional School Board, J.L. Ilsley High School and Lorne Middle School). Thanks also to Steve Turgeon (Public Safety Regional Programs Officer) who managed the contribution agreements for the implementation of the YIPs, as well as Lucy Burke (2009-2011) and Mary Peters (2011-2012) who had been technical authorities for the evaluation contract before Danièle Laliberté played this role. Special thanks to the youth, parents, guardians and stakeholders who agreed to participate in the evaluation.
The Youth Inclusion Program (YIP) is a geographically based program aimed at reducing crime in specific neighbourhoods with high rates of crime by targeting 50 youth aged 13-16 who are most at risk of offending. This complex and intensive program is based on individualized action plans developed for each youth and targets specific risk factors. Youth are expected to receive approximately 10 hours of intervention per week (500 hours per year), including a combination of one-on-one case management sessions, and group activities such as life skills training, mentoring, recreation, tutoring, youth support and parent/guardian support. In addition, the program connects participants to external community resources. Specifically, YIP aims to increase protective factors, school attendance and school performance, while decreasing risk factors, youth offending and the number of youth in the criminal justice system. Three YIP projects implemented in the Atlantic region with funding from PS participated in an impact evaluation conducted by a contractor hired by Public Safety. These include: Northside YIP in North Sydney, Nova Scotia (January 2010 - June 2013); Seeds of Change YIP in Spryfield, Nova Scotia (September2010- November 2012) and ONE Change YIP in St. John, New Brunswick (April 2010 - January 2014)Footnote3. Public Safety funded this evaluation study for the amount of $427,534.00 (between August 2010 and March 2014).A total of 257 youth were admitted. The planned number of 60 youth was reached at the Seeds of Change site. The Northside YIP and the ONE Change YIP sites which expected to recruit 150 youth each recruited fewer youth than anticipated (79 in Northside and 118 in ONE Change), due to a lack of after-school services in these neighbourhoods and because many participants required more than one year of intervention before their transfer to mainstream services. Northside also faced a lack of community referrals. Approximately 98% (n= 251) of participants were in the target age rangeFootnote4 including 65% males in Northside and 53% for the other sites. To establish if a referred youth was sufficiently at risk to be suitable for the program, structured professional judgment on the part of the program staff and the advisory committee was provided, taking into consideration the risk scores measured with the ONSET toolFootnote5. In general, the three YIP sites were very successful in reaching their intended targeted groups. The mean total ONSET score (out of a total possible of 48) for the 230Footnote6 participants was 23.93 (Northside : 26.38; ONE Change : 23.68 and Seeds of Change: 20.66)Footnote7. About 98 % (n= 230) of all participants scored high (score = 3 or 4) on at least one risk factorFootnote8 (Northside : 99%; ONE Change: 99% and Seeds of Change: 94%) and 81% also scored high (score = 3 or 4) on at least two risk factors (Northside : 87%; ONE Change : 79% and Seeds of Change: 75%). Consistent with the idea that a YIP should be placed in a high risk area, Neighbourhood was among the top two risk factors across all three sites. Lifestyle, Thinking and behavior, School and education, Family and Personal Relationships were among the top factors in at least one site. In-depth interviews with school staff, police and representatives from various social service organizations confirmed that the YIPs were reaching high risk and appropriate youth; these stakeholders believed that the program was better received by those youth whose risk level was somewhat more moderate, and should not target youth who were too entrenched in the criminal justice system. Youth were expected to spend approximately half of the program hours in direct YIP activities and half of the hours in services offered by community partners. A minimum target of five hours of intervention per week was planned (between 5 to 10 hours). The actual weekly dosage and the average total number of hours were below target for all three sites (see table 1). The ONE Change program had the advantage of operating within the school which increased engagement and the likelihood of participation in the program. Program participation may have also been higher as the program had a vibrant lunch program in collaboration with several community churches. Seeds of Change also operated in a school environment but almost exclusively within school hours. This limited the number of human resources which subsequently reduced the number of hours youth could participate in the program.
|Northside||ONE Change||Seeds of Change|
|Mean weekly dosage per youth (# hours)||2.28 (n= 74)||3.90 (n= 113)||0.86 (n= 43)|
|Total dosage per youth (# hours)||118.77||202.75||44.62|
The main objectives of this impact evaluation were to assess whether the intended outcomes were achieved and if there were any unintended outcomes; to identify lessons learned, exploring what worked well in the program and what did not work as well; to make recommendations to strengthen the program; to assess the extent to which the program was implemented as planned; to achieve a descriptive cost analysis and conduct a cost-effectiveness study if feasibleFootnote9.
Evaluation Methodology and Limitations
A single-group repeated measures design was implemented for each site as a result of difficulties in identifying a suitable comparison group in terms of size and youth characteristics. The quantitative component of the evaluation was based on data collected before (pre-program), during (program), one year (post-year 1) and two years (post-year 2) after the intervention. The evaluators used project data collected by staff with the ONSET tool, official police data and school recordsFootnote10. They also conducted 83 semi-structured interviewsFootnote11 with 38 participants, 19 parents and 26 other stakeholders in order to enhance the interpretation of quantitative data with qualitative data. In addition, a short survey of 25 local community stakeholders was conducted in January 2013 at the Northside siteFootnote12 to gather information on incidents experienced by business institutions in the past three years.
Inferential statistics planned for this evaluation were not ultimately conducted as a result of small sample sizes and/or the failure to meet the necessary statistical assumptions. Furthermore, the assumptions for combining and comparing data between the sites were not met. As a result, any difference observed could be due to chance. Quantitative analyses were descriptive, consisting of frequencies, proportions and testing differences between means. As for any volunteer program where consent is required, the YIP evaluation faced a selection bias as well as bias related to attrition. Information from secondary data sources was sometimes impossible to validate due to staff turn-over and changes in protocol. The ONSET framework was not used consistently by every site. Despite best efforts, all three sites found it too difficult to re-assess all of the participants as planned. Some youth participated in YIP activities before their formal admission to the program, thus we cannot be sure that their baseline measures are the same as they would have been had they not engaged in the YIP. Also, some changes in outcomes may have been the result of a combination of the YIP and several external interventions provided by other community organizations in the area (interactive effects). Official police and school data were not collected for the purpose of research but rather for administrative and decision making reasons, and were therefore recorded, stored and delivered in various formats and timeframes making it a challenge to use for this evaluation study. Also, differences in concepts for police and school data may lead to biases.
Table 2 shows that 67% of all participants decreased their total risk between the pre and post-program according to ONSET measures. For each risk factor, there were always more participants who made positive changes than youth with unfavorable changes; moreover, for 8 risk factorsFootnote13, the percentage of participants with a favorable change was at least 40%. The risk factors that had the largest number of youth improving were Lifestyle (53%), Thinking and behaviour (49%), School and education (48%), and Family and personal relationships (46%).
|Factors||Northside *||ONE Change**||Seeds of Change***||Total ****|
|Total ONSET score||50%||38%||89%||10%||0%||75%||67%||26%|
|Thinking and behaviour||49%||14%||57%||14%||8%||46%||49%||17%|
|School and education||44%||23%||60%||10%||7%||13%||48%||15%|
|Family and personal relationships||35%||17%||63%||5%||7%||71%||46%||17%|
|Perception of self and others||35%||31%||52%||11%||9%||36%||41%||21%|
|Emotional and mental health||27%||21%||60%||10%||0%||43%||40%||18%|
|Motivation to change||37%||31%||48%||8%||7%||60%||39%||23%|
|Attitudes to offending||33%||27%||44%||10%||0%||73%||35%||24%|
Decrease in risk factors: ↓
Increase in risk factors: ↑
* 51 ≤ n ≤ 52
** n = 63
*** 8 ≤ n ≤ 15
**** 123 ≤ n ≤ 130
Participants with no change in the ONSET risk factors are not included in this table.
Numbers were very small in the Seeds of Change site.
For more details on these risk factors see: Public Safety Canada, Youth Inclusion Program in Canada ONSET: A Risk Assessment Tool.
For more details on these risk factors see: Public Safety Canada, Youth Inclusion Program in Canada ONSET: A Risk Assessment Tool.
About 75% of youth at the Seeds of Change site increased their total risk level and 25% did not change (n=8)Footnote14, however, during in-depth interviews staff members said they strongly felt that most youth improved while participating in the program. The vast majority of participants reduced their risk factors at the ONE Change site (89%, n= 63) and at the Northside site (50%, n= 52). Those who had a reduction in risk had a reduction of more than 10 points out of 48 (total ONSET scale). The areas where Northside had the largest number of youth improving were Thinking and behavior (49%), School and education (44%) and Lifestyle (42%). The areas with the largest percent of improvements amongst the youth at the ONE Change site included: Lifestyle (70%), Family and personal relationships (63%), and Living arrangements (62%). Factors related to School and education and Emotional and mental health also had a high percentage (60%) of participants decreasing their level of risk in the ONE Change site. The analysis of risk in relation to dosage suggests that youth who participated more intensely in the Northside and ONE Change sites derived greater benefit from the YIP interventions. In the Northside site, 60% of high dosage youth (n=10) decreased their ONSET risk score compared to 47% of Low dosage participants (n= 38). In the ONE Change site, 94% of high dosage participants (n= 17) decreased their score compared to 87% of low dosage youth (n= 45). These results should be interpreted with caution since participants who had higher levels of engagement might have been at lower risk at the pre-program period or some risk factors might have been excluded during the referral phase making it difficult to account for all of the potential factors that might have contributed to positive changes in the outcomes.
Intermediate Outcomes: School Performance, Absenteeism and Suspension
Overall, the ONE Change program showed positive changes in youth grades. About 55% (n=44) of the ONE Change site participants increased their GPA by 9% during the YIP, and 47% (n= 17) improved their grades by 5% in the year following the program. Youth in the ONE Change program were in a lower grade than the participants at the other two sites who all attended the same small school. At the Northside site, 40% (n= 10) of participants showed a 25% increase in GPA in the year after the program. These youth might have benefitted from having an academic component to the program. Approximately 28% (n= 32) of the Seeds of Change participants improved their GPA by 12% a year after the YIP. Seeds of Change youth were starting high school and generally older than those in the other sites; their characteristics might have been in place for a longer period of time and therefore, more difficult to change. The fact that such high risk participants were remaining in school indicated that the YIP may have contributed to enhancing their attachment to school.Across all sites and all comparison periods, at least one quarter (27%) of the youth reduced their absenteeism from the start and end of the comparison period. Attendance was higher for those in the high dosage group compared to those in the low dosage group at the Northside and ONE Change sites. Approximately 29% (n= 14) of the participants decreased their rate of absenteeism during the YIP and 61% (n= 18) in the year following the program at the Northside site.
Absenteeism decreased for 47% (n= 49) of the ONE Change youth during the intervention and for 48% (n= 23) one year after the program. Focusing on changes in absenteeism from the year while the youth participated in the YIP at the Seeds of Change site to the year after the program, the outcomes showed that 27% (n= 33) of them decreased their rate of absenteeism. Finally, 48% (n= 21) of youth at the Seeds of Change decreased their absenteeism two years after the program.
For all three sites, frequency and length of suspensions increased from the program year to the year following the YIP, although there were some youth with decreases in frequency of suspensions (25% at Northside, 40% at ONE Change and 17% at Seeds of Change)Footnote15 and decreases in length of suspensions (31% at Northside, 40% at ONE Change and 17% at Seeds of Change). Sample sizes were very small for the analysis of suspensions so these results could be due to chance. Also, at ONE Change suspension length was determined both by the behavior that incited the suspension and the number of previous suspensions (subsequent suspensions are longer than previous ones by matter of policy).
Long-term Outcomes: Reduction of Criminal Offending
According to police records (see table 3), the percentage of youth with a decrease in the number of times they were suspected to be involved in criminal incidents and charged (or not) by the police (suspected/charged contacts)Footnote16 was always equal to or higher than the proportion of youth with an increase at the Seeds of Change site where 50% of participants saw a decrease two years after the YIP. At the Northside site, 60% youth saw a decrease a year after the YIP and 59% two years after. There was a general increase at the ONE Change site.
|Northside||ONE Change||Seeds of Change|
|↓ *||↑ **||NC***||↓ *||↑ **||NC***||↓ *||↑ **||NC***|
|Pre-program to program||29%
|Pre-program to post-year 1||29%
|Program to post-year 1||60%
|Program year to post-year 2||59%
Official police records.
The participants in the ONE Change YIP saw a general increase in suspected/charged police contacts; that is, 63% of youth showed an increase in suspected/charged police contacts one year after their participation in the program. Their exposure to YIP programming before the official start date makes it difficult to determine the causal relationship between the program and the number of police contactsFootnote17. As well, these youth were too young to have an official record of contact with police in the pre-program time frameFootnote18. Given their age, they might have been experimenting with new peers and illegal substances for the first time during the program period. The increase in antisocial behaviours leading to the suspected/charged contacts with police in the year following the program might be attributable to the fact that the youth were no longer in the program and had greater opportunity to reengage in negative activities. Once the protective nature of the intervention was removed, the youth might have relapsed by making poor choices. The Northside youth who engaged with the YIP had fewer suspected/charged contacts per youth across all periods of measurement than those who did not engage to the same extent. The youth in the ONE Change site who received at least the recommended program hours also had fewer suspected/charged contacts than those with less exposure to the YIP intervention.
The average number of suspected/charged incidents per youth was also analyzed (see table 4). Based on this indicator, which is used as a proxy of criminal offending, there has been a reduction of criminal offending in the Northside and Seeds of Change sites between the pre-program year and 2-year post-program periods. It should be noted that the average number of suspected/charged incidents per youth was lower during the YIP and the 1-year post-program period for Seeds of Change. The mean number of suspected/charged incidents per youth was lower during the YIP implementation than before the program for the ONE Change site.
|Period||Northside||ONE Change||Seeds of Change|
|# Suspected/ Charged incidents||Average # of contacts per youth||# Suspected/ Charged incidents||Average # of contacts per youth||# Suspected/ Charged incidents||Average # of contacts per youth|
|Pre -program year||69 (n= 15)||4.6||34 (n= 15)||2.3||16 (n= 4)||4.0|
|Program year||97 (n= 16)||6.1||21 (n= 11)||1.9||8 (n= 5)||1.6|
|Post-year 1||71 (n= 13)||5.5||27 (n= 7)||3.9||8 (n= 3)||2.6|
|Post-year 2||18 (n= 8)||2.3||NA||NA||0 (n= 0)||0.0|
Official police records.
Community Level Outcomes (Northside site only)
Eight of the 25 businesses (32%) who completed the Stakeholders Survey in 2013 did not experience any incidents of the types listed in table 5 in the three years during the YIP. When a similar survey was conducted in 2010, all of the businesses reported experiencing some type of incident; for example, at least 40% of businesses had experienced incidents such as theft (52%), graffiti (48%), broken windows and glass (40%). Most of the 2013 respondents (between 60% and 96% depending on the type of incident) did not perceive a change in the frequency of an incident. However, in most of the cases, in 2013 more respondents perceived a decrease in frequency than an increase. Also, according to the 2013 survey, there were not as many youth hanging around with nothing to do compared to the baseline information collected prior to the intervention to before. Sixty percent of the businesses who completed the survey were aware of the YIP and 86% felt the program was somewhat or entirely responsible for decreasing the amount of youth antisocial activity in the area.
|Types of incidents||% of businesses with an incident (n=25)||Perceived change in incident frequency (n=25)|
|Motor vehicle vandalism||16%||4%||4%||92%|
|Motor vehicle theft||8%||4%||0%||96%|
Stakeholders Survey data (January 2013). This survey was conducted by the end of the YIP in the Northside site only since baseline data was not available for the other sites.
The descriptive cost analysisFootnote19 was completed only in two sites and not in the Seeds of Change site as a result of incomplete data and also because the project was used to support the YIP as well as a school based credit recovery program which was not limited to the YIP participants, nor was it used by all YIP participants. The Northside site reported $1,100,251.11 in expenses and the ONE Change site $1,223,812.56. The Northside site had a total of 34 months of programming hours, 99 youth with dosage recorded. The average cost per Northside participant was $11,755.45 and the average cost per dosage hour was $84.72. The ONE Change site had a total of 35 months of programming hours and 209 youth had dosage recorded. Excluding evaluation costs, the average cost per ONE Change participant was $5,855.56 and the average cost per hour of programming was $23.65. The expenses recorded at the Seeds of Change site reached $817,538.05.
The fact that some youth were involved in the YIP intervention before being officially admitted in the ONE Change site had made it difficult to determine the extent to which the outcomes could be attributed to the intervention. This challenge illustrates how important it is to appropriately align program recruitment, implementation and the outcomes being measured. Another key finding is that youth who had participated more intensely in the program derived greater benefit from the YIP intervention. An indication that the expected short-term outcomes of the YIP intervention were reached was that 67% of all participants decreased their total risk level based on the ONSET score. It is notable that some progress was also made regarding intermediary outcomes related to school related indicators. We also saw a decrease in criminal offending for several YIP participants, particularly in the Northside and the Seeds of Change sites where respectively 59% and 50% of the youth decreased suspected/charged contacts with the police between the end of the YIP and the end of year 2 post-program. It is of interest to note that 25% of the Northside businesses (n= 25) saw a decrease in criminal incidents during program implementation. The majority of Northside business representatives expressed the belief that the program was somewhat or entirely responsible for decreasing youth antisocial behaviour.
Gagnon, Nathalie and Lesley Duncan (2014). Youth Inclusion Program Evaluation: Final Report. Includes Stakeholder Survey Results, submitted in March 2014 to Danièle Laliberté, Technical Contract Authority, Public Safety Canada. Email addresses: firstname.lastname@example.org, email@example.com
Public Safety Canada, Youth Inclusion Program in Canada ONSET: A Risk Assessment Tool.
Another impact evaluation of YIP is being conducted by the Social Research Demonstration Center (SRDC) in Montréal and Salaberry-de-Valleyfield and its final findings will be presented in a separate evaluation summary in 2017. Also, a YIP implemented in the Péninsule acadienne was evaluated by an evaluator hired by the community organization through Public Safety funding.
The agencies that implemented the program were: Island Community Justice Society, Chebucto Communities Development Association and O.N.E. Change Inc.
The agencies that implemented the programs were: Island Community Justice Society, Chebucto Communities Development Association and O.N.E. Change Inc.
The target age was 12-17 years in Northside while the actual mean age at program start was 14.46. For ONE Change , the target age was 11-15 years while the actual mean age at program start was 13.02. The target age was 15-20 years for Seeds of Change while the actual mean age at program start was 15.74.
The ONSET referral and assessment framework was designed by the Centre for Criminology at the University of Oxford for the UK Youth Justice Board. This structured instrument consists of 12 different sections which aim to facilitate the identification of youth appropriate for YIP. Risk and protective factors specific to individual youth are measured and twelve domains are targeted. For more details on these domains see: Public Safety Canada, Youth Inclusion Program in Canada ONSET: A Risk Assessment Tool. Document adapted for the Canadian context by Public Safety Canada with permission from the Youth Justice Board, United Kingdom.
Complete ONSET data were available for 230 youth.
The ONSET scores vary from 0 (not associated) to 4 (very strongly associated and believed to significantly contribute to the youth’s risk of criminal behaviour).
For more details on the risk domains see: Public Safety Canada, Youth Inclusion Program in Canada ONSET: A Risk Assessment Tool. Document adapted for the Canadian context by Public Safety Canada with permission from the Youth Justice Board, United Kingdom.
A descriptive cost analysis was not completed for the Seeds of Change site as a result of incomplete data. Cost-effectiveness studies were not feasible in any of the sites.
The consent rate for police data was 33% (n= 79) in Northside , 44% (n=118) in ONE Change and 55% (n= 60) in Seeds of Change. In Northside , 41% (n= 79) provided consent for school records, compared to 58% (n= 118) in ONE Change and 52% (n= 60) in Seeds of Change.
Northside interviews: 32; ONE Change interviews: 36; Seeds of Change interviews: 15.
Since a similar survey was conducted with local businesses by the Northside Business Association in Northside prior to the implementation of the program, it was decided to conduct such a survey at this site only.
For more details on the 12 risk factors see: Public Safety Canada, Youth Inclusion Program in Canada ONSET: A Risk Assessment Tool.
This result may be due to the fact that different staff members completed the ONSET pre and post assessments at Seeds of Change. Also, the small sample may be biased: only 8 participants had sufficiently complete pre and post assessments to allow a comparison over time. For those reasons no further interpretations of the ONSET data for the Seeds of Change site were conducted.
Northside : n= 16; ONE Change : n= 10; Seeds of Change: n= 6.
The term « suspected/charged » will be used when referring to youth who were suspected to be involved in criminal incidents and charged or not by the police.
Seeds of Change did not have any youth who met the recommended weekly dosage over the course of the program year, so it has been excluded from this analysis as no dosage comparison could be made.
As a result, the number of contacts before the YIP might be underestimated and the increase in contacts overestimated.
Evaluation expenses were excluded from this cost descriptive analysis to provide a clear picture of the cost of the program.
- Date modified: