Comorbidity treatment service model evaluation

Evaluation approach

Page last updated: August 2009

A program logic approach was adopted for the evaluation of the Comorbidity Treatment Service Model Evaluation project. The program logic proposes a theoretical causal pathway where desired outcomes such as improved client health and wellbeing are presumed to depend on the generation of certain impacts such as achievement of treatment goals. These impacts are presumed to be caused by certain processes or structures being in place within treatment services, such as clear policies, processes and practices for intake, treatment, and referral. In turn, the development of improved policies, processes and practices for intake are enabled by inputs, such as funding, workforce, and service links. These chains of inputs and effects occur within a wider geographical, social and political context in which treatment services are located.

The next chapter outlines the methodology for this project. This is followed by a chapter concerned with the project's findings and a discussion chapter that also includes recommendations.

Development of program logic for the comorbidity treatment service model evaluation

Commonly, the first step in developing any program logic framework is to describe the strategic program or initiative and identify key activities, aims and objectives. For the present evaluation, there is no single policy or program document that applies to the provision of services to people with comorbid disorders. The treatment services that were evaluated are largely funded under different state/territory and/or Commonwealth programs. However, relevant Australian Government policies such as the National Drug Strategy (Australian Government, 2008a) and the National Mental Health Strategy (Australian Government, 2008b) were reviewed, a literature review was undertaken, and key informants (i.e. individuals with high levels of expertise in the areas of comorbidity, mental health, substance misuse, rural/metropolitan health care settings, and service delivery design) were consulted to confirm the findings of the literature review and address any gaps that were identified in the review.

Based on the analysis of the policy documents and the literature review a list of domains for measurement was developed. The Comorbidity Treatment Service Model Evaluation program logic map includes six domains:
  • Context
  • Inputs
  • Service system I (policies and procedures)
  • Service system II (practices)
  • Client impact
  • Outcomes.
The domains and sub-domains are outlined in figure 2 below.

From the program logic map, a list of broad questions was developed. These are outlined in figure 3 below.
Top of page

Figure 2: Comorbidity treatment service model evaluation program logic map

What is good practice in comorbility treatment service provision?
Text equivalent below for Figure 2: Comorbidity Treatment Service Model Evaluation program logic map
Larger image of figure 2 (GIF 262 KB)

Text version of figure 2

What is good practice in comorbility treatment service provision?

The map flows sequentially along 6 key areas:
  1. Context
    1. urban, rural, demographics1
    2. density of service system & workforce
    3. State regulatory or other issues with funding

  2. Inputs
    1. service structure, e.g. stand-alone, or part of larger organisation, management structure
      • determines procedures
      • consumer participation
      • organisational philosophy
    2. funding sources/issues
    3. workforce
      • composition, qualifications
      • volunteers - skills & roles
    4. service promotion - service links & types of relationships & types of organisations & purpose
      • ways of advertising
      • cost
      • funding sources

  3. Service system elements I (policies & procedures)
    1. intake processes2
      • appropriate assessment of client need
      • comprehensive, incl. screening for comorbidity
    2. clear treatment protocols
      • ext. guidelines re treatment of specific condition, or
      • multiple internal guidelines
      • culturally & gender sensitive guidelines
    3. processes/procedures/protocols for referring on & communication with other providers
    4. staff are trained in service procedures
    5. staff have adequate & appropriate training, skills & supervision in assessment, treatment & specialisation

  4. Service system elements II (practices)
    1. 'appropriate' treatment is provided (evidence-based). timing of delivery:3
      • sequential
      • parallel
      • integrated
    2. clear care/treatment plans (communicating treatment)3
      • involves consumer/carer
      • involves all providers (internal & external)
    3. use of referral/communication/feedback for others involved in clients' care (or who should be)3 4
    4. service supports client self-management post treatment5
    5. cost - client episode of treatment

  5. Client impact
    1. completion of treatment3 6
    2. achievement of treatment goals3
    3. 'continuity of care discharge'/post-service planning (for episodic care)7
    4. client self-care - knowledge of
      • early intervention
      • services
    5. service - referring health professional aware of client status

  6. Outcomes
    1. improved client health and wellbeing
    2. improved social functioning
    3. reduced MH symptoms
    4. less/less harmful AOD use
    5. less need for services (incl. readmission rates)Top of page

Footnotes

11.1 relates to 1.2
2 3.1 relates to 3.2
3 4.1, 4.2, 4.3, 5.1 and 5.2 all relate to each other
4 4.3 relates to 5.5
5 4.4 relates to 4.5 and 5.4
6 5.1 relates to 5.2
7 5.3 relates to 5.4

Figure 3: Comorbidity treatment service model evaluation program logic questions

Broad evaluation questions
Text equivalent below for Figure 3: Comorbidity Treatment Service Model Evaluation program logic questions
Larger image of figure 3 (GIF 383 KB)
Top of page

Text version of Figure 3

Broad evaluation questions
  1. Context
    • What is the context in regard to location (i.e. urban, rural, remote) and client demographics?
    • How does location influence service density and work-force?
    • What are the state regulatory requirements or other issues with funding sources?

  2. Inputs
    • What are the service and management structures? To what extent do these determine program procedures, consumer participation, and organisational philosophy/guiding priciples?
    • What are the processes for monitoring of performance and cost? What are the funding sources and any issues relating to funding?
    • What is the composition and qualification of the workforce, including employed staff and volunteers? What are their skill, their roles?
    • How is the service promoted (to other organisations and to potential clients)? How does the service link to other relevant organisations? What is the purpose and nature of these relationships? What does it cost and how is it funded?

  3. Service system elements I (policies & procedures)
    • How do services recruit clients? What are the intake processes? Are they appropriate and comprehensive? Do they include screening for comorbidity? Who does what?
    • Are there clear treatment protocols? What care is provided and how? Who does what? How extensive are the guidelines re treatment and treating clinician? Are the guidelines sensitive to gender, ATSI and CALD clients?
    • Are there clear processes/procedures/protocols for referring on and communicating with other providers? How are referrals made and what are the criteria? Are the staff trained in these processes?
    • Do staff have adequate and appropriate training, skills, supervision in assessment and treatment? What does it cost and who pays for it? Do staff have additional specialised skills and experience?
    • What is the service's capacity to meet the needs of clients with comorbid problems?

  4. Service system elements II (practices)
    • Can potential clients access the service? Do clients receive appropriate and evidence-based treatment? Who provides it? What is the timing of treatment (i.e sequential, parallel, integrated)?
    • Do all clients have clear treatment plans, and are these communicated to the client/carer? Are clients/carers invovled in the development of plans? Are other providers involved?
    • Do referral processes include feedback to referring clinician or anyone else who should be involved?
    • Are clients supported in self-management post treatment?
    • What does an episode of treatment cost? Is there a cost to the client?

  5. Impacts
    • What proportion of clients complete treatment? How is this related to the achievement of treatment goals?
    • Does discharge planning for episodic care lead to continuity of care post-discharge?
    • In regard to post-discharge planning, are clients made aware of early intervention options and relevant services?
    • Are the referring health and other professionals aware of client status and discharge planning?

  6. Outcomes
    • Does the service/program:
    • improve over-all client health and wellbeing?
    • improve social functioning?
    • reduce mental health symptoms?
    • reduced/less harmful use of AOD?
    • What is the evidence and how is it collected? Does it lead to less need for services? Does it reduce readmission rates?

Measurement of the program logic domains

The program logic map was used to guide the determination of the kind of information that needed to be gathered in order to reflect on the hypothesised effects of treatment service provision on the impacts and outcomes for clients. Subsequently, a tool (i.e. a survey) was developed to measure the domains.