Final Program Evaluation – Towards No Drug Abuse

PDF (265 KB)

Table of Contents

Introduction

Towards No Drug Abuse (TND) is an evidence-based program designed to help youth reduce their tobacco, alcohol or drug use, and violent behaviourFootnote 1. TND has been tested in seven experimental field trialsFootnote 2 in the United States. The most consistent finding (in 7 out of 7 trials) was a favourable reduction in the consumption of hard drugs. Favourable effects on alcohol use were found in 4 trials, on marijuana use in 3 trials and on cigarette smoking in 2 trials. Favourable reductions in violence victimization and weapons carrying were found in 3 trials.

The program was implemented in Canada to determine if this intervention can contribute to changes in drug abuse patterns in Canada.

The John Howard Society of Hamilton implemented TND between July 2009 and September 2014 in schools within the Hamilton-Wentworth District School Board and the Hamilton-Wentworth District Catholic School Board, in Hamilton, Ontario. The National Crime Prevention Strategy (NCPS) provided $1,556,000 in funding for the program.

Program Description

TND was originally developed as a classroom-based drug abuse prevention program to be used among alternative high school students, aged 14–19 (Sussman, 1996). However, the program has also been found to be effective with a general high school sample (Dent, Sussman & Stacey, 2001).

TND focuses on three factors that predict tobacco, alcohol, and other drug use, violence-related behaviours, and other problem behaviours among youth, including:

The model developer, Sussman, indicates that at the completion of the TND program it is expected that students will be able to:

The American TND design consists of 12 in-class interactive sessions. Each session should last from 40 to 60 minutes. Sessions address active listening, stereotyping, myths and denials, chemical dependency, the impacts of drug abuse, marijuana, tobacco use cessation, stress, health and goals, self-control, positive and negative thought and behaviour loops, perspectives and decision-making, and commitment.

To accommodate the program in the Ontario school system, the same content and duration was implemented; however, the number of weekly sessions was reduced to eight. To compensate for this change, session length was extended to 90 minutes and the frequency of sessions per week was increased. If this transformation of the project was not considered, physical education departments in Ontario who only have 5 of the 16 days devoted to substance abuse would not be able to accommodate the program in Ontario schools.

Target Group

The project’s aim was to target youth in schools and in the community who are at a high risk of committing crimes related to drug and substance abuse. The target group for the school-based program consisted of the general population of high school students in the Hamilton-Wentworth area, with a focus on grade 9 students. In defining the school-based target group, an assumption was made that all students in the Hamilton-Wentworth area have some degree of risk for drug-use. The target group for the community-based program consisted of higher-risk youth involved with community organizations including youth that have offended, and were being referred for extra-judicial measures. These participants were higher risk as a result of being part of regular or alternative education, expulsion and suspension programs.

The experimental group sample (TND participants) included both the school-based component (n=1,836) and the community-based component (n=81). The samples from the school-based and community-based components were pooled in order to enhance the sample size so that multivariate analyses could be undertaken. The TND group had a slightly higher proportion of females and the comparison group had a slightly higher proportion of males. The average age for the TND participants at program start was 14.24 years (SD=1.01), and the average age for the comparison group participants at the pre-test was 14.29 years (SD=.57). With respect to ethnic background, for both groups, the majority of participants is Caucasian and speak and write English only.

Based on socio-demographics measured, there were no statistically significant differences between the TND treatment group and comparison group. Based on a risk level categorization developed by the evaluation team, the majority of the TND participants were considered ‘low risk’(68%), while 21% were ‘medium risk’, and a smaller proportion were categorized as ‘high risk’ (11%).

The comparison group had a similar risk profile with the majority ‘low risk’ (69%), 17% were ‘medium risk’, and 14% were categorized as ‘high risk’ (see Table 1). Chi-square test results confirmed that the experimental and comparison group were similar indicating that any differences in results between the TND intervention and the group without the program could likely be attributed to the intervention and not to the predisposition of the groups.

Table 1: Characteristics of TND Participants and Comparison Group
Characteristic TND Participants (n=1,917) Comparison Group (n=309)
Age at start (mean) 14.2 years 14.3 years
Gender
Male 48% 53%
Female 52% 47%
Program Component
School 96% 100%
Community 4% 0%
Risk Profile*
Low Risk 68% 69%
Medium Risk 21% 17%
High Risk 11% 14%

*Low risk = used no alcohol nor drugs, no prior weapon carrying nor involvement with the legal system
Medium risk = occasional to moderate use of marijuana or alcohol
High risk = high use of marijuana or alcohol, hard drug use, sometimes carries weapon or gun, and has prior involvement with the criminal justice system

Evaluation Objectives

The NCPS contracted Goss Gilroy to conduct the impact evaluation of TND. The impact evaluation, valued at $197,412 started in 2010 and ended in 2014.

The objectives of this evaluation were as follows:

Evaluation Methodology

The school based component used a quasi-experimental, repeated measures between-group design. The comparison group used classes from schools that were not participating in TND during the semester. The within-group repeated measures involved the administration of self-reported questionnaires at four stages: pre-intervention (T1); immediately post-intervention (T2); 6 months post-intervention (T3); and 12 months post-intervention (T4). Due to the repeated efforts of the Senior Evaluation Advisor in the NCPS and evaluation team, innovative strategies were developed to ensure that the comparison group would remain long enough for  between measures to be taken at the 6 and 12 months follow-up stagesFootnote 4. The interactions between time and group (Time x Group) were the key results for determining whether the program made a difference in the outcome measures. These will be illustrated in tables in this report.

The evaluation design for the community-component was a within-group repeated measures design similar to the school-based component, but with no comparison group. This type of design tests changes in the target group before the program and at various follow-up points after the completion of the intervention.

The quantitative analyses included using descriptive frequencies, cross-tabulations (chi-square), used multivariate analyses (ANOVAs)Footnote 5, to identify significant differences between the intervention and the comparison group,  subgroups (e.g., gender, risk level) and time (e.g., pre, post, follow-up measures) on key outcome variables.

Qualitative analyses of focus group and key informant interview data were conducted using a content analysis approach to identify common trends in responses.

When reviewing the tables in this summary, review the interaction effects (farthest column to the right) to determine if there were statistically significant differences in changes within time periods between the TND participants and the comparison group. The columns to the left labeled TND Participants identifies changes within the TND participant group as a wholeFootnote 6 and also delineates change by risk and gender levels (subgroups).

Research Limitations

The main limitations identified for this evaluation were the smaller number of TND participants than intended, the limited proportion of TND participants assenting/consenting to the evaluation component, and, consequently, the limitation in follow-up data because participants had not obtained parental consent for the evaluation follow-up surveys. For example, while approximately 1,800 youth participated in the pre-post sessions conducted in class, less than half (n=765) provided parental consent for the evaluation team to follow-up with participating youth at the 6- and 12-month periods. This likely has resulted in some biases in results when combined with response rates to follow-up surveys in the range of 35%.

In addition, the recruitment of participants for the comparison group was initially quite slow, with few classes participating; however improvements were made in the last two years of the evaluation study and the total numbers of comparison group participants increased. Having much fewer comparison cases in relation to TND participants had an impact on the reliability of the analyses in terms of statistical power.

A further limitation related to the assessment of participants’ risk level and subsequent categorization. Due to issues regarding the amount of time required to administer the risk assessment tool that was initially selected for use by TND, an alternative risk assessment approach was developed and implemented. Once these data were analyzed, however, there were limited participants deemed high risk and little variance on the items in order to conduct a factor analysis for construct validity purposes.

There were challenges in collecting individual level information on attendance, suspensions, expulsions and records pertaining to involvement with the law (e.g., police records, court records), therefore, self-reports were the only source of data. Not having data from these external sources limited the extent to which some of the longer-term outcomes such as decreased anti-social behaviour leading to suspension and expulsions, and reduction of criminal behaviour could be assessed.

Content fidelity was high throughout the program in that all content was delivered in all courses with the same required content and duration; however, fidelity with respect to the number of classes and frequency of delivery did not adhere to the model developer’s requirementsFootnote 7. Given the Canadian educational context with longer instructional periods (90 minutes vs. 45–50 minutes) and block teaching (arranging for one subject to be taught intensely within a short period of time — five classes per week vs. one per week), the program was adapted in Year 2 and Year 3 to meet these structural differences. The evaluation was able to test whether the change in fidelity requirements had an effect on the outcomes being measured. The test revealed that there were no differences in outcomes between the group that used the developer’s fidelity requirements and the group that adapted the fidelity to fit within the Canadian school structureFootnote 8.

Data Collection Method

The data sources used in this evaluation were the model developer’s self-administered and online questionnaires. Complementary methods included focus groups with school-based program participants, and key informant interviews with TND program stakeholders (i.e., teachers, principals, a representative from each school board, and community partners).

Outcomes Measured

The evaluators assessed the program’s ability to increase participants’ knowledge of the consequences of tobacco and substance abuse, and to increase participants’ social skills, decision-making, self-control and coping ability. They also evaluated the program’s ability to reduce the risk factors associated with drug-related crimes including cigarette use, hard drug/substance use, weapons carrying, and victimization.

Outcome Evaluation Findings

The overall summary of findings for anticipated outcomes is that approximately six months after participating in the program, TND participants have similar results to the comparison group on many of the outcomes. This indicates that participation in TND is having a limited impact on the anticipated skills and behaviour outcomes at six months post-program. One exception is that on the knowledge outcome, TND participants continued to have higher test scores than the comparison group indicating that TND participation had a positive impact on knowledge. Unfortunately, this increased level of knowledge does not appear to have been translated into similar positive changes in skills or behaviour.

Table 2: Overall Summary of Findings for Anticipated Outcomes
Anticipated Outcome Results after comparing the TND participants with the Comparison GroupFootnote 9 (pre-post 6 monthsFootnote 10)
Mean test scores of knowledge related questions Favourable
Stay away from, or not increase, use of tobacco, alcohol or other drugs No change
Quit or reduce use of tobacco, alcohol, or other drugs No change
Know about other things you can do with your friends besides using drugs No change
Make your household become or stay drug free No change
Reduction in cigarette use No change
Reduction in alcohol use No change
Reduction in marijuana use No change
Reduction in cocaine use No change
Reduction in hallucinogens use No change
Reduction in inhalants use No change
Reduction in stimulants use Unfavourable
Reduction in painkiller use No change
Reduction in other drug use No change
Reduction in weapon carrying No change

Favourable Change: The results showed a statistically significant change indicating that there were favourable outcomes for the TND participants.

No Change: There were no changes in the outcomes being measured before and after the program, or after calculating between group differences.

Unfavourable: The results indicate that the change was more favorable for the comparison groupFootnote 11.

Note: Additional tables in this report may show different levels of change as these other tables show results for all stages of measurement and provide disaggregated results by gender and risk.

Knowledge

Results indicate that TND participants’ knowledge of the consequences of alcohol, tobacco and drug use was significantly increased as a result of participation in the program. The pattern of increased knowledge for TND participants was significantly greater compared to the comparison group of non-participantsFootnote 12. Furthermore, this increase in knowledge was sustained over 6Footnote 13- and 12Footnote 14-month follow-up periods after the intervention was completed (see Table 3). These findings were corroborated by focus group discussions with TND participants, and key informant interviews with program stakeholders indicating that participants’ knowledge about drugs and drug-related statistics had increased as a result of their participation in the program. While the pattern of increased knowledge was found across all three risk profiles of TND participants, the actual amount of knowledge as reflected in test scores was lower among the high risk participants and males.

Table 3: Summary of Findings regarding Knowledge OutcomesFootnote 15
Immediate Outcomes TND Participants Changes between the TND and Comparison Group (Interaction Effects)
Within comparison of TND participants Subgroup Time X GroupFootnote 16
Outcome
Indicator
Pre vs. Post Pre vs. 6-mths post Pre vs. 12-mths post Risk Levels Gender Pre vs. Post Pre vs. 6-month Post Pre vs. 12-month Post
Increased knowledge
Mean knowledge test score

X
Increased Scores

X
Increased Scores

X
Increased Scores

X
High Risk: Lower Scores

X
Females: Higher Scores

X
TND Group: scores increased more pre-post

X
TND Group: scores increased more pre-post

X
TND Group: scores increased more pre-post

X denotes a significant main effect between the TND and comparison group at the significance level of p<0.05.

Attitudes and Skills

In the pre-post period, TND participants reported short-term increased likelihood of using what they learned in school to assist them in the area of social situations, decision making, self-control, and coping. These short-term increases were significantly greater compared to the comparison group of non-participants (see Table 4). TND participants’ self-reported increase in skills was not maintained at the 6-month and 12-month follow-up levels. All four measures related to skill indicate a decreased likelihood that the student would be able to use what they learned in school to: 1. stay away or not increase their use of alcohol, tobacco or other drugs; 2. quit or reduce use; 3. know about other things they can do with their friends besides using drugs; or 4. make their household become or stay drug-free. These decreases were not significantly different from the changes in the skills reported by the comparison group.

Across the four skill measures, the participants with high-risk profiles reported a lower likelihood of being able to use what they learned in school to assist them in the various areas of decision making, social situations, self-control and coping. For three of the skill areas, female participants reported more confidence than males in being able to apply what they learned to various situations including: 1. quitting or reducing use; 2. knowing other things to do with friends other than using; and  3.making their household become or stay drug free.

Table 4: Summary of Findings regarding Attitude and Skills Outcomes
Immediate Outcomes TND Participants Changes between the TND and Comparison Group (Interaction Effects)
Within comparison of TND participants Subgroup Time X Group
Outcome
Indicator
Pre vs. Post Pre vs. 6-mths post Pre vs. 12-mths post Risk Levels Gender Pre vs. Post Pre vs. 6-month Post Pre vs. 12-month Post
Increased skills
Likelihood of what they learned in school will help them to:

Stay away from, or not increase, use of tobacco, alcohol or other drugs

X
Increased Skills

X
Decreased Skills
(not expected)

X
Decreased Skills
(not expected)

X
High Risk: Lower skill had  greatest positive change

 

X
TND Group: Larger improvement pre-postFootnote 17

   

Quit or reduce use of tobacco, alcohol, or other drugs

X
Increased Skills

X
Decreased Skills
(not expected

X
Decreased Skills
(not expected

X
Higher Risk: Lower Skill than those reported by low and medium risk participants

X
Females: Higher Skill

X
TND Group:
Larger improvement pre-postFootnote 18

   

Know about other things you can do with your friends besides using drugs

 

X
Decreased Skills
(not expected)

X
Decreased Skills
(not expected)

X
Higher Risk: Lower Skill than those reported by low and medium risk participants

X
Females: Higher Skill

     

Make your household become or stay drug free

X
Increased Skills

X
Decreased Skills
(not expected

X
Decreased Skills
(not expected

X
Higher Risk: Lower Skill than those reported by low and medium risk participants

X
Females: Higher Skill

X
TND Group:
Larger improvement pre-post
Footnote 19

   

X denotes a significant main effect between the TND and comparison group at the significance level of p<0.05.

Note: The areas with grey shading indicate that there was a positive change likely attributable to the TND Program. The other columns provide information on changes reported by TND participants, but that may be similar to changes observed in the comparison group.

Behaviours

Cigarette Use

TND participants reported increased 30-day cigarette use from the pre-to-post-test period (see Table 5). This increase was not statistically significantly different from the changes in the cigarette use reported by the comparison group. There were no statistically significant differences between the TND participants and comparison group observed at the 6 month or 12 month follow-up periods. The participants with a medium or high risk profile were more likely to use cigarettes, and showed greater increases in cigarette use over the pre-post periodFootnote 20. Males were also more likely to use cigarettes and showed greater increases over this periodFootnote 21.

It should be noted that both the TND participants and comparison group participants moved to a new environment between grades 8 and 9. It is possible that the change from junior high to high school contributed to an increased use due to increased exposure and access to cigarettes.

Table 5: Summary of Findings regarding Cigarette Use
Intermediate Outcomes TND Participants Changes between the TND and Comparison Group (Interaction Effects)
Within comparison of TND participants Subgroup Time X Group
Outcome
Indicator
Pre vs. Post Pre vs. 6-mths post Pre vs. 12-mths post Risk Levels Gender Pre vs. Post Pre vs. 6-month Post Pre vs. 12-month Post
Reduction in cigarette use

30-day cigarette use

X
Increased Use
(not expected)

   

X
Higher risk: greater and increased use
(not expected)

X
Males had greater and increased use

     

Alcohol, Marijuana and Other Drug Use

There were some unintended negative outcomes in the area of alcohol, marijuana, and stimulant use. In the pre-post period, TND participants reported increased 30-day alcohol and marijuana use. This short-term increase in alcohol and marijuana use was not statistically significantly different from the 30-day use of alcohol and marijuana by the non-participants in the comparison group. There was, however, higher marijuana use among TND participants’ when compared to the decrease in use by the comparison group at the 12 month measurement.Footnote 22 However, the 12 month data is based on a relatively small comparison group sample.

There was also statistically significant higher stimulant use by the TND participants when compared to the comparison group at the 6 month and 12 month measurements. Males were more likely to use marijuana and show greater increases over this period. Participants with a high risk profile were more likely to use all types of substances, and showed greater increases on all substances over the pre-post period.

Table 6: Summary of Findings for Alcohol, Marijuana and Other Drug Use
Intermediate Outcomes TND Participants Comparison Group Interaction
Within comparison of TND participants Subgroup Time X Group
Outcome
Indicator
Pre vs. Post Pre vs. 6-mths post Pre vs. 12-mths post Risk Levels Gender Pre vs. Post Pre vs. 6-month Post Pre vs. 12-month Post
Reduction in hard drug or substance use

30-day alcohol use

X
Increased Use
(not expected)

(use is reduced but not statistically significant)

 

X
Higher risk: greater and increased use
Medium risk: greater and increased use
(not expected)

       

30-day marijuana use

X
Increased Use
(not expected)

   

X
Higher risk: greater and increased use
(not expected)

X
Males had greater and increased use

   

X
TND Group: larger increase than comparison (not expected)

30-day cocaine use

     

X
Higher risk: greater and increased use
(not expected)

       

30-day hallucinogens use

     

X
Higher risk: greater and increased use
(not expected)

       

30-day inhalants use

     

X
Higher risk: greater and increased use
(not expected)

       

30-day stimulants use

     

X
Higher risk: greater and increased use
(not expected)

   

X
TND Group: smaller decrease than comparison (not expected)

X
TND Group: smaller decrease than comparison (not expected)

30-day painkillers use
(**caution in interpreting results; item likely misinterpreted)

     

X
Higher risk: greater and increased use
(not expected)

X
Females had greater use
Males had increased use
*item may have been misinterpreted

     

30-day other drugs use

     

X
Higher risk: greater and increased use
(not expected)

       

In focus groups, several TND participants stated that their behaviours did not change following TND because they did not take or consider taking drugs before their participation in the program. Most participants said that, if offered drugs such as marijuana, they would have said no even without the TND sessions. 

Reduced Weapon Carrying and Victimization

Among both the TND participants and the comparison group, self-reported incidence of weapon carrying was overall very low. For the initial pre-post-test analysis, the TND participants did not show the same increase in knife carrying as the comparison group.Footnote 23 For the 6 month follow up data, there were no significant differences between the experimental and comparison groups. At the 12 month measurement, the comparison group had a larger decrease in knife carrying.Footnote 24 As noted previously, the 12 month data is based on a relatively small comparison group sample.

Males and TND participants with high risk profiles were more likely to report carrying either a knife or a gun. As well, high risk participants were more likely to report a decrease in knife carrying during this period.

Among TND participants, there was a significant reduction in victimization with respect to “being injured on purpose without using a weapon” for all three post-intervention periods. TND participants also showed a reduction in the 6-month and 12-month data for “having had property deliberately stolen or damaged.” None of these reductions were significantly different from the changes observed for the comparison group. At the 12 month follow up, the TND participants had no change in reporting being “threatened with a weapon but not actually injured” while the comparison group had a decrease in being “threatened with a weapon but not actually injured”Footnote 25.

Medium risk participants reported the greatest decrease from pre to post program on “being injured on purpose without using a weapon”. High risk participants reported “being injured with a weapon” significantly more often after the program. Higher risk participants were more likely to report all four aspects of victimization including injury, threats and stolen property. Male participants were more likely than female participants to have experienced “being injured with a weapon” as well as “threatened with a weapon but not injured”.

Table 7: Summary of Findings for Weapon Carrying and Victimization
Intermediate Outcomes TND Participants Comparison Group Interaction
Within comparison of TND participants Subgroup Time X Group
Outcome
Indicator
Pre vs. Post Pre vs. 6-mths post Pre vs. 12-mths post Risk Levels Gender Pre vs. Post Pre vs. 6-month Post Pre vs. 12-month Post
Reduction in weapon carrying

Carry a knife in past 12-months

     

X
Higher risk: more likely and greater decrease

X
Males more likely

X
TND Group: smaller increase than the comparison

 

X
TND Group: The comparison group had a larger decrease
(Not expected)

Carry a gun in past 12-months

     

X
Higher risk: more likely

X
Males more likely

     
Reduction in victimization

Been injured on purpose without using a weapon in past 12-months

X
Reduction

X
Reduction

X
Reduction

X
Higher risk: greater and increased use (not expected)

       

Threatened with a weapon, but not actually injured in past 12-months

     

X
Higher risk: more likely

X
Males more likely

X
TND Group: smaller decrease than comparison group
(not expected)

 

X
TND Group: Comparison group had a larger decrease
(not expected)

Been injured with a weapon in past 12-months

     

X
Higher risk: more likely and greater increase

X
Males more likely

     

Had property deliberately stolen or damaged in past 12-months

 

X
Reduction

X
Reduction

X
Higher risk: more likely

       

Cost Effectiveness Analysis (CEA)

During the interim evaluation of the TND program, there were some statistically significant differences between the TND participants and comparison group that allowed for the calculation of the CEA.

The average cost per participant for the TND program was $544, taking into account NCPS contributions and JHS in-kind contributions. Some in-kind contributions were not included in these costs such as the in-kind contributions from schools to host the TND program such as classroom use and school staff time to make arrangements for TND facilitators to work with their classes. It is important to note that the cost per participant includes the school and community participants as the financial data were not separated for each group, therefore it was not possible to conduct the cost analysis for each program component separately.

The main significant incremental outcome is increased knowledge regarding the consequences of tobacco use and substance abuse. The cost-effectiveness analyses determined that a 25% incremental gain in knowledge costs $544 per student and $135 per knowledge point gain. Theoretically, if one considers the findings from published studies of TND and the program logic, one could speculate that this incremental knowledge should contribute with time to decreases in alcohol, tobacco, marijuana, and hard drug use in the range of 7 to 50%, depending on substance and study, when compared to non-participants. Given the findings in the final analysis, it is questionable how much this increase in knowledge has had an impact on substance abuse. Based on these estimates and the risk profile of youth participating in the TND Project, the lifetime benefits of the reduction in alcohol and drug use would have to exceed $54K for the conservative impact estimate and approximately $11K for the upper impact estimateFootnote 26.

The present study was unable to identify any sustained incremental changes in the outcome measures beyond knowledge scores. There were no demonstrated long term (at 6 months and beyond) impacts of TND on increased skills, reduction in substance use, weapon carrying or victimization. Consequently, the overall conclusion from the cost analysis is that the incremental costs in gains in knowledge do not appear to translate into incremental gains in other outcomes for TND participants.

Challenges and Lessons Learned

While overall there were high levels of satisfaction with the TND program among student participants, some students did make some suggestions for improvements in the curriculum such as: 1. providing more information about certain topics such as alcohol use, weapon carrying, peer pressure, and how drugs cause stress; 2. emphasizing the interactive nature of the instruction, and 3. focusing on the types of drugs and substances that they felt were more relevant such as alcohol and marijuana. The adjustment in focus towards alcohol and marijuana was supported by the stakeholders consulted during the key informant interviews. Stakeholders also indicated that the reduction from 12 to 8 sessions to adapt to the longer periods in the Ontario school system was appropriate, especially since the quality and quantity of the course was maintained.

The main challenges encountered with the evaluation revolved around the recruitment of a suitable comparison group and the rates of agreement by TND participants to be involved with the evaluation. As a result, the main lessons learned with respect to developing and implementing an evaluation of this type are: 1. to highlight the importance of having suitable comparison groups in the implementation of a rigorous evaluation; 2. to consider having participation in the intervention contingent upon participation in the evaluation; 3.to conduct a pre-implementation study to assess the demand for a specific intervention; and 4. to obtain support within the contribution agreementFootnote 27 to recruit an adequate number of TND participants and comparison group participants for the evaluation study.

Given the lack of demonstrated incremental impacts on identified outcomes for TND within the context provided, expanded or additional implementation of the TND Program within the Canadian context should be undertaken with caution. The results of the impact evaluation were unable to demonstrate that participation in TND leads to significant incremental impacts on identified key anticipated outcomes. Although there were high levels of satisfaction with the program among students, facilitators, and stakeholders, the program would likely need to be further adapted and/or changed to obtain the intended impact among Canadian youth.

Since addressing substance abuse is a key factor in reducing crime, some adaptations to the program may include focusing the intervention on the high risk youth group not currently in school or participating in alternative schoolsFootnote 28. Focusing on secondary prevention ensures that resources are being used by those that need to make changes in their substance abuse patterns. In the present study, given the low risk nature of the school community, even if the intervention were effective, substantial change would not be visible within this low risk population. Modification of the curriculum to address the specific needs of the high risk user combined with revisions in the model developer’s survey instruments would enhance the evidence-based nature of substance abuse interventions in Canada.

References

Dent, C. W., Sussman, S., & Stacey, A. W. (2001). Project towards no drug abuse: Generalizability to a general high school sample. Preventive Medicine, 32, 514-520.

Sussman, S. (1996). Development of a school-based drug abuse prevention curriculum for high-risk youths. Journal of Psychoactive Drugs, 28, 169-182.

GGI.Footnote 29 (2014). Towards No Drug Abuse (TND) Impact Evaluation: Final Report.

Smith-Moncrieffe (2010). Using a Fidelity Index to Increase Program Attribution, Canadian Evaluation Society

Smith-Moncrieffe, D.  (2014). Interim Results from the Towards No Drug Abuse (TND) Project. Retrieved from: http://www.publicsafety.gc.ca/cnt/rsrcs/pblctns/rslts-twrds-ndrg-bs/index-eng.aspx

End Notes

  1. 1

    See: http://tnd.usc.edu/about.php

  2. 2

    For a summary, see: http://tnd.usc.edu/files/TND_TRIALS_SUMMARY_FOR_WEBSITE_Revised_LR.pdf

  3. 3

    Personal commitments regarding drug use are measured using questions q. 40-44 in the survey; these are reported as "skills".

  4. 4

    Due to the nature of the contribution agreement in the federal funding context, the agreement is positioned to focus primarily on participation in the intervention which often competes with efforts to recruit youth with similar risk and need levels to participate in a comparison group for research purposes.

  5. 5

    Analysis of variance (ANOVA) is a statistical test used to analyze the differences between group means and their associated procedures (such as "variation" among and between groups).

  6. 6

    This technique in ANOVA is still important to have an overall sense of change for the TND group.

  7. 7

    The program developer states that TND should include "12 classroom-based sessions, each of which is 40 to 50 minutes in length" and that "the program was designed for implementation over a four-week period (i.e., 3 sessions per week)." For more information, see: http://tnd.usc.edu/about.php.

  8. 8

    This test was possible as data was available for one year where participants followed the original fidelity requirements.

  9. 9

    If TND participation had a positive impact on anticipated outcomes when compared with comparison group, we should have "favorable" changes in this column. For example, even though the skills for TND participants reportedly decreased (unfavorable in first column), we saw this same decrease among the comparison group over the same period ("no change" in second column). From this we can infer that TND participation did not impact skills (either negatively or positively).

  10. 10

    The 12-month data are based on a relatively small comparison group sample; therefore, these results are omitted in this table.

  11. 11

    In this particular case, TND participants had a smaller decrease than the comparison group indicating that the intervention was not contributing to a reduction in stimulant use amongst TND participants.

  12. 12

    F=323.52, p<.001

  13. 13

    F=8.64, p<.01

  14. 14

    F=5.95, p<.05

  15. 15

    "X" denotes a significant main effect at least to the significance of p<0.05 level.

  16. 16

    Time x Group interaction means that there was a significant difference in the rate of change from pre to post between the TND and comparison group participants. Having no statistically significant Time x Group interaction indicates that the TND Program has not demonstrated an incremental impact on participants.

  17. 17

    F=17.32, p<.001

  18. 18

    F=7.97, p<.01

  19. 19

    F=5.83, p<.05

  20. 20

    F=12.37, p<.001

  21. 21

    F=17.90, p<.001

  22. 22

    F=6.83, p<.01

  23. 23

    F=4.74, p<.05

  24. 24

    F=6.31, p<.05

  25. 25

    F=4.71, p<.05

  26. 26

    These calculations do not factor in the potential unintended negative outcomes that were found in this study (i.e., the higher marijuana use among TND participants' when compared to the decrease in use by the comparison group at the 12 month measurement).

  27. 27

    The contribution agreement is the vehicle used to fund the intervention. The impact evaluation is funded separately and often has little influence on the evaluation requirements regarding the comparison group.

  28. 28

    Alternative schools tend to have a higher proportion of at risk youth.

  29. 29

    Technical Authority and Contract Manager: Donna Smith-Moncrieffe, NCPS, Policy Research & Evaluation, managed the evaluation. Goss Gilroy Inc. (GGI) was hired by Public Safety to conduct the evaluation.

Date modified: