Wraparound Model
Evaluations' Lessons Learned
Table of contents
Introduction
Public Safety Canada's (PS) National Crime Prevention Strategy (NCPS) funds the implementation and impact evaluations of select crime prevention projects. These impact evaluations are conducted by external, third party evaluators, who are hired by the project administrators. For projects that choose to undertake an impact evaluation, funding is provided for this work via the project's grant and contribution agreement. Results and learnings from these project evaluations are often synthesized and shared with stakeholders to develop the knowledge base about what works in crime prevention in Canada.
Wraparound Model
The Wraparound model is an intensive care management program for youth with serious or complex emotional, behavioural, and/or mental health conditions, whose needs have not been adequately addressed by other available services aloneFootnote 1. Instead of leaving youth and their families to deal with multiple service providers on their own, the Wraparound model provides a coordinated approach to the needs of the youth, their family, and other involved parties (e.g., court counsellor, teacher, social worker, physician, support networks, etc.).
Although Wraparound's principles and phases should remain central and consistent, the program is not "one size fits all" or manualized. Implementation can be adapted to reflect local contexts, resources, and specific participant or community needs.
This product presents learnings from 11 Wraparound project impact evaluations funded by the NCPS between 2008 and 2019.
Methodology
Content from the impact evaluations was coded thematically using qualitative analysis software, MAXQDA, to identify key evaluation challenges and recommendationsFootnote 2.
Results — Key evaluation challenges
Methodology Challenges
Challenges surrounding the evaluation's methodology (referenced in 8 of 11 evaluations).
- Inappropriate evaluation design, indicators, questions and/or tools were the most frequently cited challenges (mentioned in 7 evaluations).
Examples:
- "… a conventional pre-post [evaluation] design, does not work well with the vulnerable youth population served by [the project]. Such 'point in time' measurement does not reflect the fluctuating nature of youth's lives."
- "…our interview guides and surveys had questions relating to guns, gangs, and extreme forms of delinquent behaviour. However, the … program had difficulty recruiting and retaining higher risk youth (e.g., those involved in gangs, serious crime, etc.)…"
Recommendations:
The evaluations offered strategies to help mitigate these challenges:
- Consider the use of emergent approaches to evaluations (e.g., developmental evaluation);
- Integrate data collection into project planning and delivery;
- Consider GBA+ and cultural competency when planning the evaluation;
- Identify comparison groups from the beginning.
Data Challenges
Challenges surrounding data collection and accessibility (referenced in 7 of 11 evaluations).
- Inconsistencies in data collection was the most frequently cited challenge (mentioned in 6 evaluations).
Examples:
- "Youth Facilitators [project staff] said that when youth know they are getting close to the end of the program, they stop connecting with the Youth Facilitator because they believe they do not need any more support. That makes it hard (if not impossible) for Youth Facilitators to collect end-program data."
- "Despite excellent working relationships and protocols for sharing information, police data were inconsistent, and were only provided for a minority of project youth. Many data were missing, and the time periods upon which the offending behaviours were reported varied widely."
Recommendations:
The evaluations offered strategies to help mitigate these challenges:
- Use large sample sizes;
- Provide adequate evaluation-related training to project staff and partners;
- Use mixed methodologies, multiple sources and triangulation;
- Invest resources into establishing trusting relationships between youth and data collectors;
- Identify partnership/information-sharing agreements from project's start;
- Develop data collection tools that are easy, clear and accessible.
Collaboration Challenges
Challenges surrounding collaboration between stakeholders (e.g., project staff, evaluators and partners; referenced by 4 of 11 evaluations).
- Mistrust in the evaluation was the most frequently cited challenge (mentioned in 3 projects).
Examples:
- "… getting participants to provide their name and signature on an official form proved very difficult due to a general lack of trust for evaluators and suspicion about how their data would be used."
- "Working within a social science positivist prescribed paradigm resulted in a culturally unsafe atmosphere for both the evaluation team and [project] staff. The academic nature of the evaluation design was not aligned with community-based research and resulted in feeling as though we were 'helicopter researchers'."
Recommendations:
The evaluations offered strategies to help mitigate these challenges:
- Facilitate consistent contact among evaluation staff, project staff, participants and partners;
- Foster a positive working relationship between evaluation staff and project staff;
- Have the same evaluation staff member(s) meet with the same participants, whenever possible;
- Increase time to build rapport with the community prior to data collection.
Project Staff-related Challenges
Challenges surrounding project staff (e.g., staff overburdening, staff turnover; referenced in 4 of 11 evaluations)
- Staff overburdening was the most frequently cited challenge (mentioned in 3 evaluations).
Examples:
- "… frontline staff continue to mention how onerous and difficult the evaluation data collection process has been, sometimes to the detriment of service delivery."
- "…evaluation can put a substantial burden on program staff if the evaluator has limited access to program participants and/or information about them."
Recommendations:
The evaluations offered strategies to help mitigate these challenges:
- Set realistic data collection requirements from project staff;
- Hire individuals to be responsible for data collection/management specifically;
- Provide sufficient administrative supports to help project staff with data collection and management;
- Develop data collection tools that do not require too many resources from project staff.
Resource-related Challenges
Challenges surrounding evaluation resources (e.g., time and budget; referenced in 4 of 11 evaluations).
Examples:
- "In the end, the [project] team could only provide the evaluation team with limited detailed data about participant progress and outcomes. Neither of the [projects] had administrative support staff to assist the Wraparound facilitators…"
- "Due to resource constraints, the youth received only one six-month follow-up interview. This is unlikely to be sufficient to demonstrate changes in attitudes."
Recommendations:
The evaluations offered strategies to help mitigate these challenges:
- Provide adequate guidance and resources to project staff to ensure appropriate data collection;
- Keep project timelines in mind when designing the evaluation;
- Conduct an evaluability assessment prior to evaluating a project;
- Consider using more emergent and less resource intensive approaches to evaluations when necessary;
- Integrate data collection methods in project activities.
- Date modified: