Skip to main content

File audit to assess sustained fidelity to a recovery and wellbeing oriented mental health service model: an Australian case study

Abstract

Background

Over the past decade there has been increasing attention to implementing recovery-oriented approaches within mental health service practice and enhancing fidelity to such approaches. However, as is often the case with evidence-based practices, less attention has been paid to the sustainability of recovery-oriented approaches over time. This study sought to investigate whether fidelity to a recovery-oriented practice framework – the Collaborative Recovery Model could be sustained over time.

Method

The study setting was an Australian community managed mental health organisation. A file audit of consumer support plans was undertaken using the Goal and Action Plan Instrument for Quality audit tool (GAP-IQ). The audit tool assessed 17 areas for quality. Consumers (n = 116) from a large community managed mental health organisation participated in the study. Sustained fidelity to the Collaborative Recovery Model (CRM) was determined by comparing results from the file audit to a similar audit conducted 3 years earlier.

Results

The file audit revealed a significant increase in fidelity to CRM practices between 2011 and 2014. Fidelity to individual audit items that comprise the GAP-IQ was also found to significantly increase across 16 of the 17 GAP-IQ audit items, with the exception of the ‘Action Plan Review’ audit item.

Conclusions

A comparison of file audit data across different time points within the same setting can provide useful feedback about whether or not a practice is being sustained over time. Although fidelity increased overtime the study design does not allow conclusions that training and coaching practices implemented by the organisation were responsible.

Peer Review reports

Background

Recovery-oriented practice: the collaborative recovery model (CRM)

The promotion of recovery-oriented approaches and their provision in mental health services has increased during the past decade for consumers experiencing and living with mental illness [1]. Recovery-oriented approaches involve fostering personal growth, encouraging active participation in one’s broader community, and empowering consumers to live a valued and meaningful life [2, 3]. Consumers have identified that desired outcomes are not limited to symptom reduction or “getting better” [1], but are more about learning how to live a personally fulfilling life even if symptoms continue to persist [3]. However, challenges are routinely experienced by services in the mental health sector when they attempt to implement recovery-oriented approaches within service practice [4, 5]. Evidence suggests that barriers are influenced by the extent of orientation at the organisational systems and procedural levels [4, 5].

Internationally governments have encouraged or compelled service providers to operate in recovery enhancing ways by linking provision of funding for community-based organisation to this objective [6, 7]. In Australia, the strength of commitment to recovery-oriented practice is easily recognisable in health policy documents [8]. Several Australian National Mental Health Plans have recommended that a recovery orientation should drive service delivery (e.g., Australian Health Ministers, 2009). One of the five priority areas includes addressing “Social Inclusion and Recovery”, and stipulates that mental health providers develop cultures that are founded on and reflective of a recovery orientation [9]. To assist translation at the service and practice level a National Framework for recovery-oriented services was published in 2013 to provide guidance for all people working in the mental health sector for what a recovery-orientation is and how it can be implemented [10]. However, high-level policies and guidance documents are important but not sufficient to ensure recovery-oriented care is delivered. Even when these recovery values are espoused at the level of individual organisations these values may not be translated into practice.

The Collaborative Recovery Model (CRM) is a well-established recovery-oriented approach to working with consumers experiencing a severe mental illness (SMI) [11, 12]. It employs the use of a coaching-style relationship that encourages individual growth, hope and supports people to, “move beyond illness towards one’s best possible self” [13]. The CRM empowers consumers to clarify their values, establish goals and complete action plans that provide them with the direction needed to pursue their vision of a valued life. Progress is reviewed collaboratively between the consumer and their mental health practitioner throughout the course of support. Any barriers to completing an action plan or achieving set goals are identified and modified accordingly [11].

Past studies examining the efficacy of the CRM approach for consumers experiencing a SMI have identified a range of positive outcomes. The approach has been found to be positively regarded by both consumers and mental health practitioners [13, 14] and perceived as being implemented at a high level [14]. Consumers receiving services from CRM trained workers identify components of the approach being delivered (e.g., action planning) at a higher frequency than consumers whose support workers have not been trained in the model [13]. Workers trained in CRM report significantly more positive recovery-oriented attitudes following training [15]. The action planning (therapeutic homework) components of CRM have been associated with more positive service user outcomes [14, 16]. A number of mental health organisations in Australia have adopted the CRM as their primary service delivery model. However, as is common with the implementation of evidence-based practices (EBP’s) in general, barriers have been encountered when implementing the CRM [17].

Studies examining the key factors that facilitate and hinder the implementation of EBP’s have consistently found that staff training is crucial to the successful implementation of EBP [15, 18]. Providing staff with training in the CRM has not only been found to facilitate the implementation of this approach in mental health settings, but also increases the skill and overall confidence of workers in their engagement with consumers [15]. However, staff training in isolation has not resulted in mental health practitioners sustaining their use of recovery-oriented approaches over time [17]. Recent studies indicate that the likelihood of staff transferring newly acquired therapeutic skills, and continuing to use these skills over time, is improved when it is supplemented with regular coaching [11]. Specifically, it has been found that providing staff with training in the CRM that is supplemented by regular supervision or coaching: (i) promotes the transfer of newly acquired skills into everyday clinical practice with consumers, and (ii) is associated with increases in the quality of documented CRM care planning over time [11]. Therefore, the provision of coaching for staff not only has important implications for the successful implementation of CRM practices in mental health settings, but also for the sustainability of these practices over time.

Sustainability to evidence-based practice (EBP) in mental health settings

While the implementation of EBP in mental health settings has been examined extensively, there has been less of a focus on what happens beyond that point [19]. Sustainability as it relates to EBP is defined as the successful continuation of practice beyond the implementation phase [20, 21]. More recently, the need to examine the sustainability of programs past the implementation stage has emerged as an important research priority [21].

Inadequate funding, staff turnover, and staff resistance to delivering a new treatment approach are factors that have been identified as hindering the sustainability of EBP’s [20]. Facilitators that increase the likelihood that an EBP will be sustained include: staff training that is supplemented with additional training over time, organisations prioritising routine utilisation of an EBP, and funding [20, 22].

Assessing Fidelity to EBP in mental health settings

Fidelity to EBP refers to how closely services adhere to delivering a particular treatment approach when working with consumers [23, 24]. Assessing fidelity is valuable for organisations as it serves as a quality improvement mechanism for enhancing service provision over time [25]. Research examining fidelity to evidence-based approaches in mental health settings has relied heavily on staff self-reports. Few studies have examined fidelity to EBP by conducting consumer file audits. File audits are advantageous as they provide an alternative and arguably more objective assessment of fidelity [26]. Fidelity to the CRM following staff training has been previously assessed using the Goal and Action Plan Instrument for Quality (GAP-IQ) audit tool [11]. The GAP-IQ measures the quality of CRM care planning within the domains of vision, goal setting, motivational enhancement, action planning and review, as documented in consumer files [11, 13].

The current study

This paper presents file audit findings from a larger program of evaluation aimed at exploring implementation fidelity to the Collaborative Recovery Model within an Australian community managed mental health service organisation. The organisation is one of the many community managed organisations (also referred to as non-government organisations or NGO’s) in Australia that provide support services to people living with and/or experiencing mental illness and distress. In 2009, the organisation adopted the CRM as a ‘whole of service’ framework to guide service policy and practice. There was high-level leadership support (CEO and Board) for implementation of the program along with the establishment of a mandatory learning and development program to embed the approach across all areas of service practice. All service delivery staff receive 3 days of initial training, a 6 month booster and ongoing annual boosters. Accompanying formal training, staff undertake ongoing parallel process coaching to enhance practice skills, maximise confidence and increase fidelity to the approach.

The broad program evaluation involved three components: 1) a cross-sectional survey with consumer and staff participants assessing perceptions regarding the importance of the model’s key practice elements for assisting recovery and the frequency by which these are applied, 2) a file audit of consumer support plans, and, 3) interviews with staff across the organisation to assess implementation at the policy and process level. Findings from the cross-sectional survey have been presented in an earlier article [14] and the focus of the current study is on the results of the file audit review. Few studies have examined fidelity to a recovery-oriented model of mental health service provision over time. The aim of the current study is to address this research gap and assess whether fidelity to the CRM was sustained over time following its implementation in the study settings. The results provide an example of how audit review methods can be useful to assess implementation fidelity and sustainability over time.

Method

Design

The study involved an audit review of consumers’ files. The results of the file audit from the current 2014 study were compared to an identical file audit that was conducted 3 years earlier in 2011 at the same community managed mental health service. Details regarding file selection procedures are provided below.

Participants

Participants of the current study were active consumers (n = 116) of an Australian community managed mental health service. ‘Active consumer’ refers to people who are currently accessing support as opposed to those on a waiting list. At the time of this study, the service provided support to over 3000 consumers from 29 service site locations across 5 states of Australia.

Recruitment involved a proportional random selection process to select 12 sites from the 29. This resulted in recruitment activities that were distributed across the 5 states as follows: Western Australia (1), Queensland (1), South Australia, (2), New South Wales (4) and Victoria (4). Secondly, lists of active consumers from the 12 sites were generated and a randomisation procedure applied to invite participants. Participation in the file audit review involved consent for researchers to access and review participant files.

Five researchers with a lived experience of recovery (consumer researchers) undertook recruitment and data collection activities. Researchers started at the top of the randomised lists and contacted potential participants initially by phone to inform them about the study and invite participation. To optimise participant representativeness exclusion criteria was limited to age parameters (18-65) and a person’s hospital status and perceived capacity to participate. That is, if a person was at the time of the study an inpatient in hospital and it was deemed by treating staff that they were to unwell to participate they were omitted from the list.

For those participants who were approached to participate, researchers introduced themselves as having a lived experience of mental illness and recovery and provided preliminary verbal information about the study, their role and what participation would involve.

For potential participants who indicated interest in knowing more, a meeting time and place was set with the researcher to discuss the project more fully. Participants were provided with full information (including in written form), invited to discuss and ask questions, and offered time to consider and talk to significant others (e.g. family) about participating. In all communications the voluntary nature of participation was clearly communicated along with options for withdrawing consent. Participants were informed of the protocols to protect participant confidentiality including the use of participant ID numbers on all paper based and electronic data with only researcher access to the ID key. All paper-based data stored in locked filing cabinets with only researcher access and signed consent forms stored separately to paper-based data responses.

Researchers stopped recruitment once participation numbers reached up to 10 per site. Altogether, 263 participants were invited to participate and 117 (44%) agreed to participate after being taken through the informed consent process. One participant subsequently withdrew due to a change in circumstances.

Procedure

To assess fidelity to the implementation of key practices that form the therapeutic structure of the CRM, the files of the 116 participants were audited. Prior to commencing the file audit, two consumer researchers were trained in how to use the audit tool and then independently audited five files and discussed their ratings to increase consistency. Following this, each researcher audited 58 files each across the twelve selected sites. The two consumer researchers then independently rated n = 12 of the other’s files (blind to others ratings) in order to assess interrater reliability. The audit results were then compared to data obtained in a previous study undertaken in several mental health organisations in 2011 [11], however only data from the same organisation was extracted for the purposes of direct comparison in the present study.

Measures

Fidelity to the CRM in the current study was examined using the Goal and Action Plan Instrument for Quality (GAP-IQ) audit tool. This measure was utilised in a similar study that also examined fidelity to the CRM [11] and is provided as an Additional file 1.

Data analysis

A series of independent sample t tests (Bonferroni adjusted p < .003) were conducted to determine whether fidelity to the implementation of practices that form the therapeutic structure of the CRM had increased, decreased or remained the same between 2011 and 2014. To verify these effects and assess rates of “high fidelity” a series of Chi Square test of contingencies were also conducted (p < .001). This analysis allowed us to determine whether the proportion of “Yes” responses for individual GAP-IQ audit items were significantly different between 2011 and 2014 (see Table 1). We were particularly interested in the proportion of “Yes” ratings since this indicates the highest level of fidelity.

Table 1 Mean ratings, Standard Deviations and ‘Yes’ percentages for single item GAP-IQ Fidelity ratings

Results

Of the 116 participants, 55% were male and 45% were female. The mean age of participants was 42.9 years (SD = 11 years; range 20 to 69 years). Participants were identified as holding primary diagnosis categories of; schizophrenia (41%), depression (20%), bipolar disorder (14.3%), schizo-affective disorder (11.4%), anxiety (2.9%), and other (10.7%). Length of time receiving support from the service was; less than 1 year (10%), 1–2 years (46%), 2–3 years (26%), 3–4 years (10%), 5+ years (7%). Fourteen percent of participants were of Non-English speaking background and 6% identified as Aboriginal or Torres Strait Islander. Participants as a group had a total of 62 different ‘key workers’ who had primary responsibility for the coordination and delivery of their mental health support.

Interrater reliability on the subset of 12 independently rated files was calculated using Kappa. Coefficients ranged from a low of .39 to 1.00 with a median of .85. Using the descriptors provided by [27], 1 of the 17 codes had “Fair” interrater reliability (0.21–0.40), 3 codes were “Moderate” (0.41–0.60), 3 codes were “Substantial” (0.61–0.80) and the remaining 10 codes had “Almost perfect agreement” (0.81–0.99).

An independent sample t-test of the total GAP-IQ score revealed a significant increase in the overall fidelity to the CRM between 2011 and 2014. Table 1 outlines the mean and standard deviation ratings across 2011 and 2014 for the 17 GAP-IQ CRM items. A series of independent samples t-tests were conducted for individual items using a Bonferroni adjusted p-value of .003.

Table 1 below outlines the mean ratings, standard deviations and ‘Yes’ percentages for single item GAP-IQ fidelity ratings.

With the exception of the ‘Action Plan Review’ item on the GAP-IQ audit tool, the results from a series of Chi Square contingencies were all significant indicating a significantly higher proportion of “Yes” ratings in 2014 than in 2011 for 16 of the 17 items (all p < .001). The percentage “yes” responses in 2011 revealed four items that were particularly low (Time frame for goals, 3.4% to Recovery vision, 11%). All showed improvements by 2014 but, the provision of “Timeframe for goals” remained low (11.4%).

Variation between service sites

Since there were only an average of 10 files selected from each service site only descriptive data is provided to portray the variation in GAP-IQ ratings. There was considerable variability between the 12 sites with the best performing site having a mean of 31.00 (SD = 2.08) on the GAP-IQ and the poorest performing site having a mean of 17.10 (SD = 7.55). When sites were categorised into High (31–27), Medium (26-22) or Low (21-17) groupings based on mean scores, there was only one site in the High group, with 6 in the Medium group and 5 in the Low group.

Discussion

The aim of this study was to determine whether fidelity to CRM practices had remained the same, increased or decreased between 2011 and 2014. Overall, an increase in fidelity to CRM practices was observed for 16 of 17 GAP-IQ items, with the exception of Action Plan Review item. Fidelity to Action Plan Review did increase between 2011 and 2014 (19.2 to 25%) but this increase was not significant and reflected poor fidelity to this component of the CRM. In addition, there were several components that had particularly low fidelity ratings in 2011 but all had subsequently improved in 2014 (e.g., Recovery vision, Time frame for goals, Goal confidence, Social support). It is possible that feedback from the 2011 audit to trainers within the organisation contributed to greater emphasis on these components of the CRM model in training activities. However, specification of time frames for goals remained low in 2014. It is possible that this aspect of the goal planning process was infrequently specified since there is no specific prompt on care planning forms to specify the time frame. Thus, revision of the goal plan forms to include a specific time frame prompt along with a focus during training activities is warranted.

Although there were some components of CRM practice that could clearly be improved, overall fidelity to CRM practices increased over time in the current study and was on average high in this organisation. However, currently no benchmarks exists for the fidelity of organisations to CRM practices. Without a benchmark, it is unclear whether fidelity ratings in the current study are acceptable. To determine whether fidelity ratings in the current study were practically meaningful, we compared the study results to a similar study which examined fidelity to therapeutic homework implementation and completion within a CRM framework based on 122 case manager self-reports [28]. Fifty-eight percent of case managers reported that they specified the frequency of homework practice “often” or “almost always” and 49% similarly specified the location, with 84% indicating they reviewed the homework at the beginning of the next session [28]. When compared to equivalent components of the 2014 audit, ratings of how often (frequency specified) an action plan was completed was higher (69%), where (location) an action should be completed was higher (70%) but the frequency of evidence that homework has been reviewed was lower (25%). Caution is needed in comparing these rates of action plan components due to the difference in methods used between the studies (self-report versus file audit review). The difference in rates that mental health workers indicate they review the homework is likely due to methodological differences where self-report has clearly produced higher rates than in the current study where documented evidence of the review process was required.

Although there may be no external organisations that currently have data to support benchmarking of CRM, there is the potential to use GAP-IQ scores to benchmark between different service units within the organisation. There was variability between sites on overall GAP-IQ mean scores. Although the number of files available for each site did not allow statistical analysis of this variability, descriptively it was clear that some sites performed substantially better than others with one site in particular a stand out. Such site-specific data has the potential to be used to guide service level training in order to be able to improve overall adherence to treatment and quality protocols.

To our knowledge, the current study one of the first to examine sustained fidelity to recovery oriented practices over time. One prior study [20] that examined sustainability of evidence-based practices (EBP) in community mental health agencies included a recovery-oriented EBP. This EBP was Illness Management and Recovery [29] and for the 12 agencies who implemented this approach 8 (66.7%) indicated they had sustained it 2 years later and 3 (25%) at 6 year follow-up [20]. Further, the IMR was described as the EBP that had the most significant adaptations out of the five EBPs reviewed. Judgements of sustained practices were based on interviews with site and state agency leaders. Thus, the differences in methods for assessing sustained practices and varying follow-up time frames make comparisons between studies problematic. In addition, although IMR might have been judged by site leaders to be continuing, the adaptations described raise questions about the fidelity to the original IMR program.

Given different methodology between the studies it is difficult to determine whether discrepancies in fidelity to particular components are a function of self-report versus file audit methods of data collection. Where appropriate to we would encourage more studies to utilise independent rater audits of files using the GAP-IQ or relevant fidelity tools even if these are via interviews with key informants. Such methods are likely to reduce potential bias (e.g., social desirability) that may inadvertently influence perceptions of current practice.

Irrespective of the availability of benchmarking reference points, repeat measures of file audit data provide important feedback about areas of focus for improving fidelity in the future. Overall, an acceptable level of fidelity to the action planning components of CRM practices was broadly suggested by the results in the current study, however, review of action plans needs improvement. If action plans are not consistently reviewed this can potentially be demotivating for consumers and diminish the perceived importance of the planned therapeutic homework activities. In ongoing coaching and booster training highlighting the importance and skills associated with reviewing progress on prior action plans could be emphasised. This might involve clarifying the importance of these reviews to provide; encouragement and reinforcement for consumer effort and work done, clear connections for the links between action plans, goal attainment and valued life directions, and revising and planning future therapeutic homework activities. Repeat file audits will provide the opportunity to determine whether changes in coaching and training result in improvements in particular domains.

Strengths and limitations

In the current study, is it is not possible to make causal statements about the effects of coaching and training effects on fidelity and sustainability of CRM in this organisation. Although the same audit tool (GAP-IQ) was used at both time points, it is possible that different file selection processes and different audit raters may also have affected the results. However, the use of the same randomisation process used in the current study could be replicated in future studies in order to reduce these potential sources of variability.

The response rate (44%) for participation in this study was relatively low and may influence the reliability of the findings presented. However, the response rates may also be the result of steps taken in the study design to limit bias and potential perceptions of coercion. For example, recruitment activities were undertaken by consumer researchers who had no prior relationship with participants. Recruitment rates may have been higher if consumers had been recruited by staff they already knew. Moreover, participation in mental health research is often influenced by clinician decision making about whether a person is suitable or likely to want to be involved [30]. To improve opt-in, opt-out choice the study limited exclusion criteria to that of age and current hospital usage. Given that participation in this study was a component of a larger evaluation process that included surveys, it is possible that consumers decided not to participate because of other elements also being requested.

Conclusion

The current study is noteworthy as it is one of the first studies to examine fidelity to a recovery-oriented model of mental health service provision and sustainability over time. Providing clinicians with training and ongoing coaching in evidence-based practices (EBP’s) is thought to not only increase fidelity to EBP’s within organisations, but also the frequency that clinicians will deliver, and consumers will receive the specific components of an EBP. While the extent that recovery oriented approaches can be sustained over time in mental health settings is not known, results from the current study suggest that the community managed mental health service successfully: (i) sustained their use of CRM practices over time, and (ii) increased their fidelity to CRM practices across a 3-year period. It remains for future research to determine whether these sustained and relatively high fidelity practices are associated with staff coaching, booster training and/or high-level organisational support.

Availability of data and materials

The datasets generated and/or analysed during the current study are not publicly available to comply with University of Wollongong Ethics approvals but are available from the corresponding author on reasonable request and HREC application. The measure used in this study is included in the Additional file 1.

Abbreviations

CRM:

Collaborative Recovery Model

EBP:

Evidence-based practice

Gap IQ:

Goal and Action Plan Instrument for Quality

SMI:

Serious mental illness

References

  1. Roberts G, Wolfson P. The rediscovery of recovery: open to all. Adv Psychiatr Treat. 2004;10(1):37–48.

    Article  Google Scholar 

  2. Kidd SA, George L, O'Connell M, Sylvestre J, Kirkpatrick H, Browne G, et al. Fidelity and recovery-orientation in assertive community treatment. Community Ment Health J. 2010;46(4):342–50.

    Article  PubMed  Google Scholar 

  3. Piat M, Lal S. Service providers' experiences and perspectives on recovery-oriented mental health system reform. Psychiatr Rehabil J. 2012;35(4):289–96.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Deane FP, Crowe TP, King R, Kavanagh DJ, Oades LG. Challenges in implementing evidence-based practice into mental health services. Aust Health Rev. 2006;30(3):305–9.

    Article  PubMed  Google Scholar 

  5. Drake RE, Goldman HE, Leff HS, Lehman AF, Dixon L, Mueser KT, et al. Implementing evidence-based practices in routine mental health service settings. Psychiatr Serv. 2001;52(2):179–82.

    Article  CAS  PubMed  Google Scholar 

  6. Slade M, Amering M, Oades L. Recovery: an international perspective. Epidemiol Psichiatr Soc. 2008;17(2):128–37.

    Article  PubMed  Google Scholar 

  7. Le Boutillier C, Leamy M, Bird VJ, Davidson L, Willams J, Slade M. What does recovery mean in practice? A qualitative analysis of international recovery-oriented practice guidance. Psychiatr Serv. 2011;62(12):1470–6.

    Article  PubMed  Google Scholar 

  8. Oades L, Anderson J. Recovery in Australia: Marshalling strengths and living values. Int Rev Psychiatry. 2012;24(1):5–10.

    Article  PubMed  Google Scholar 

  9. Commonwealth of Australia. Fourth National Mental Health Plan - An agenda for collaborative government action in mental health 2009-2014. In: Health, editor ACT2009.

  10. Australian Health Ministers Advisory Council. A national framework for recovery-oriented mental health services: Guide for practitioners and providers. In: DoHa A, editor. . Canberra: ACT: Commonwealth of Australia; 2013. p. 1–88.

    Google Scholar 

  11. Deane FP, Andresen R, Crowe TP, Oades LG, Ciarrochi J, Williams V. A comparison of two coaching approaches to enhance implementation of a recovery-oriented mental health service model. Adm Policy Ment Health Ment Health Serv Res. 2014;41(5):660–7.

    Article  Google Scholar 

  12. Oades LG, Deane FP, Crowe TP. The collaborative recovery model: positive organisations enabling the use of strengths and values to serve recovery in enduring mental illness. In: Boniwel I, David S, editors. Oxford handbook of happiness. Oxford: Oxford University Press; 2012. p. 1050–66.

    Google Scholar 

  13. Marshall SL, Oades LG, Crowe TP. Mental health consumers' perceptions of receiving recovery-focused services. J Eval Clin Pract. 2009;15(4):654–9.

    Article  PubMed  Google Scholar 

  14. Wolstencroft KE, Deane FP, Jones CL, Zimmermann A, Cox M. Consumer and staff perspectives of the implementation frequency and value of recovery and wellbeing oriented practices. Int J Ment Health Syst. 2018;12(1):60.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Crowe TP, Deane FP, Oades LG, Caputi P, Morland K. Effectiveness of a collaborative recovery training program in Australia in promoting positive views about recovery. Psychiatr Serv. 2006;57(10):1497–500.

    Article  PubMed  Google Scholar 

  16. Kelly PJ, Deane FD. Relationship between therapeutic homework and clinical outcomes for individuals with severe mental illness. Aust N Z J Psychiatry. 2009;43(10):968–975.

    Article  Google Scholar 

  17. Uppal S, Oades LG, Crowe TP, Deane FP. Barriers to transfer of collaborative recovery training into Australian mental health services: implications for the development of evidence-based services. J Eval Clin Pract. 2010;16(3):451–5.

    PubMed  Google Scholar 

  18. Tsai J, Salyers MP, McGuire AB. A cross-sectional study of recovery training and staff attitudes in four community mental health centres. Psychiatr Rehabil J. 2011;34(3):186–93.

    Article  PubMed  Google Scholar 

  19. Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7(17):1–19.

    Google Scholar 

  20. Bond GR, Drake RE, McHugo GJ, Peterson AE, Jones AM, Williams J. Long-term sustainability of evidence-based practices in community mental health agencies. Adm Policy Ment Health Ment Health Serv Res. 2014;41(2):228–36.

    Article  Google Scholar 

  21. Peterson AE, Bond GR, Drake RE, McHugo GJ, Jones AM, Williams JR. Predicting the long-term sustainability of evidence-based practices in mental health care: an 8-year longitudinal analysis. J Behav Health Serv Res. 2014;41(3):337–46.

    Article  PubMed  Google Scholar 

  22. Swain K, Whitley R, Mchugo GJ, Drake RE. The sustainability of evidence-based practices in routine mental health agencies. J Commun Ment Health. 2010;46(2):119–29.

    Article  Google Scholar 

  23. Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2003;18(2):237–56.

    Article  PubMed  Google Scholar 

  24. Schoenwald SK, Garland AF, Chapman JE, Frazier SL, Sheidow AJ, Southam-Gerow MA. Toward the effective and efficient measurement of implementation fidelity. Adm Policy Ment Health Ment Health Serv Res. 2011;38(1):32–43.

    Article  Google Scholar 

  25. Cuddeback GS, Morrissey JP, Domino ME, Monroe-DeVita M, Teague GB, Moser LL. Fidelity to recovery-oriented ACT practices and consumer outcomes. Psychiatr Serv. 2013;64(4):312–23.

    Article  PubMed  Google Scholar 

  26. Ritter PL, Stewart AL, Kaymaz H, Sobel DS, Block DA, Lorig KR. Self-reports of health care utilization compared to provider records. J Clin Epidemiol. 2001;54(2):136–41.

    Article  CAS  PubMed  Google Scholar 

  27. Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360–3.

    PubMed  Google Scholar 

  28. Kelly P, Deane FP, Kazantzis N, Crowe TP, Oades L. Use of homework by mental health case managers in the rehabilitation of persistent and recurrent psychiatric disability. J Ment Health. 2006;15(1):95–101.

    Article  Google Scholar 

  29. Mueser KT, Meyer PS, Penn DL, Clancy R, Clancy DM, Salyers MP. The illness management and recovery program: rationale, development, and preliminary findings. Schizophr Bull. 2006;32(1):S32–43.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Pinfold V, Cotney J, Hamilton S, Weeks C, Corker E, Evans-Lacko S, et al. Improving recruitment to healthcare research studies: clinician judgements explored for opting mental health service users out of the time to change viewpoint survey. J Ment Health. 2017:1–7. https://doi.org/10.1080/09638237.2017.1340598.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

This study was made possible by the interest and diligence of the lived experience researchers who undertook recruitment, data collection and file audit activities.

Funding

This study was funded by Neami National. The funder had no involvement in the analyses and did not have the right to disapprove publication of any of the results.

Author information

Authors and Affiliations

Authors

Contributions

FD was responsible for the conceptualisation and design of this study. AZ and KW recruited and trained the lived experience researchers and assisted with implementation of research activities in the study context. FD guided and together with CJ undertook the statistical analysis and interpretation. All authors contributed to the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Frank P. Deane.

Ethics declarations

Ethics approval and consent to participate

This research was granted human research ethics approval by the University of Wollongong Human Research Ethics Committee - reference number HE13/335. All participants involved in this study provided informed consent to participate.

Consent for publication

All participants provided informed consent for publication. No individual patient data is published in this manuscript.

Competing interests

At the time that this research was conducted KW and AZ were employed by Neami National which funded this study. FD was employed by the University of Wollongong which provided an exclusive license for Neami National to deliver training in CRM. KW is currently employed by Neami National. All analyses reported in this paper were undertaken by FD and CJ independently.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jones, C.L., Deane, F.P., Wolstencroft, K. et al. File audit to assess sustained fidelity to a recovery and wellbeing oriented mental health service model: an Australian case study. Arch Public Health 77, 50 (2019). https://doi.org/10.1186/s13690-019-0377-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13690-019-0377-6

Keywords