Evaluation of the mental health nurse incentive program
2 Evaluation: approach
Table of contents
2.1 Evaluation scope
2.2 Project methodology
2.3 Data limitations
2.4 MHNIP program logic
2.5 Relationship to key evaluation areas
2.1 Evaluation scope
Application of the terms appropriateness, effectiveness and efficiency were guided by definitions described by the Secretary, Department of Finance and Deregulation4:- Appropriateness: the continued relevance and priority of program objectives in the light of current circumstances such as government policy context, including the suitability of program design in response to identified needs;
- Effectiveness: whether program outcomes have achieved stated objectives, and to what extent outputs have contributed to outcomes; and
- Efficiency: whether there are better ways of achieving these objectives, including consideration of expenditure and cost per output, project governance arrangements, and implementation processes.
2.2 Project methodology
To complete the evaluation HMA undertook the following steps:- Project initiation – a project plan was developed and this identified key stakeholders, documents, and data sources.
- Prepared a situation analysis - including a review of key MHNIP documentation and had preliminary discussions with key stakeholders to establish a comprehensive understanding of MHNIP operations and its environment.
- Evaluation framework developed - a detailed evaluation framework was developed to inform conduct of the evaluation. The framework contained criteria for determining achievement of objectives of the MHNIP. This guided stakeholder consultations and supported our application for ethics approval for relevant consultations and surveys.
- Modelled demand - the evaluation team built a spreadsheet model that can project future demand for the program. The model can assess the activity and cost impacts of different demand scenarios including adjustments to the program guidelines. Data was obtained from the Department of Human Services (DHS) for this purpose.
- Conducted provider surveys - medical practitioners and mental health nurses participating in MHNIP received an online survey about the program operations. The number of survey responses by provider category is shown in Table 2.1.
- Conducted case studies - 18 case study visits were conducted at a range of different service provider type including AMSs, Medicare Locals (formerly Divisions of General Practice), and general and psychiatry practices in metropolitan, regional and rural locations. The team undertook structured interviews at each case study site with medical practitioners, mental health nurses, and consumers using MHNIP (see table 2.2 for numbers involved).. We also spoke to the CEO and practice or finance manager where they were available.
- Undertook a cost analysis - a cost analysis of the program was prepared using data on program usage and patient outcomes obtained from case study sites. Information on the number of patients considered in the cost analysis are shown in table 2.3.
- Prepared the final report - the draft final report (this document) assesses the appropriateness, effectiveness, and efficiency of MHNIP using the information collected from the preceding project stages.
Top of page
Table 2.1: Survey responses and responses removed at each stage of the cleansing process
| Survey stage | No. of responses - mental health nurse | No. of responses - medical practitioners |
|---|---|---|
| Total respondents who began the survey | 355 | 278 |
| Excluding - answered 'other' to the employment type and were subsequently excluded from the survey | 27 | |
| Excluding - answered the section on demographic questions only | 58 | 20 |
| Excluding - answered a small number of questions and appeared to be a duplicate | 8 | 2 |
| Total responses analysed | 289 | 229 |
Table 2.2: Number of case study participants interviewed by organisation type
| Participant category | Medicare Local / Division of General Practice | General Practice | Private Psychiatry Practice | Aboriginal Medical Service | Totala |
|---|---|---|---|---|---|
| Number of sites | 4 | 8 | 4 | 2 | 18 |
| Mental health nurses | 10 | 9 | 10 | 1 | 30 |
| General practitioners | 3 | 10 | 1 | 1 | 15 |
| Psychiatrists | 1 | 1 | 4 | 0 | 6 |
| Clients | 15 | 34 | 14 | 3 | 66 |
| Carers | 3 | 1 | 1 | 1 | 6 |
| CEO / Practice Manager / Finance Manager | 7 | 13 | 5 | 0 | 25 |
| Totalb | 39 | 58 | 39 | 6 | 142 |
a The number of case study participants (respondents) may vary from the total number of responses (consultations) conducted, as some stakeholders were consulted together.
b The sum of the columns may be greater than the total, as some participants fell into more than one category, ie GP that was also the principal / CEO.
Table 2.3 Patient details received from case study sites (n=15), including details of those included, and number of patients removed from the analysis
Table 2.3 is presented as a list in this HTML version for accessibility reasons. It is presented as a table in the PDF version.Measures of analysis stages:
- Total patient details received - 464
- Patients excluded
- Patients where hospitalisations not limited to 12 months prior to MHNIP - 112
- Patient entered after 1/9/2011, and had not exited the program - 84
- Patient did not have a MHNIP entry/exit date - 1
- Total excluded - 197
- Patients included:
- Patient entered before 1/9/2011 and had exited the program - 92
- Patient entered after 1/9/2011 and had exited the program - 31
- Patient entered before 1/9/2011 were receiving support from MHNIP - 144
- Total included - 267
- Hospitalisations:
- Number of hospital admissions 12 months prior to joining MHNIP - 34
- Number of hospital admissions 12 months after joining MHNIP - 30
Source: Patient Impact Templates completed by case study sites.
2.3 Data limitations
The quantitative data used in the analysis was from a range of sources. Sources included DHS (Medicare) data, provider surveys, case study site visits and a consumer template completed by case study organisations.The DHS data used in the demand profiling was a full set of data, and therefore representative of MHNIP. However, all other data sample sizes used in this evaluation varied and the project did not test whether they were representative samples. In addition, selection techniques used to obtain data could not be described as random as participating organisations were requested to select the people to be interviewed, including consumers.
Caution should therefore be taken when interpreting the findings in this report. Observations and trend are indicative of the samples used in the analysis in this evaluation only. Further detailed analysis would be required on a representative sample before the evaluation findings could be described as statistically representative.
2.4 MHNIP program logic
The logic for a program explains in summary form how a public policy intervention is expected to work and the underlying cause and effect relationships between program inputs and outputs. Figure 2.1 presents the program logic for MHNIP. This shows the relationship between the following program components:- Policy context: MHNIP was announced in July 2006 as part of the Council of Australian Government's National Action Plan on Mental Health;
- Program objectives: the aims of MHNIP are to:
- Improve levels of care for people with severe and persistent mental disorders;
- Reduce the likelihood of unnecessary hospital admissions and readmissions;
- Assist in keeping people with severe disorders feeling well and connected within the community; and
- Alleviate pressure from privately practicing psychiatrists and GPs.
- Program scope: the key program design features of MHNIP, including the financial, operation and service delivery characteristics are;
- MHNIP is delivered by community based primary health services, including GPs, private psychiatrists and Aboriginal Medical Services funded by the Office of Aboriginal and Torres Strait Islander Health;
- Eligible organisations receive an establishment grant and payments for sessions of care provided to patients within the program target group;
- Program requirements: the requirements for eligible organisations to implement MHNIP, include:
- Development of patient management protocols;
- Recruitment of a mental health nurse credentialed with the Australian College of Mental Health Nurses (ACMHN); and
- Reimbursement via submission of claim forms to the DHS;
- Implementation and service delivery: the journey for patients receiving support under the program includes: assessing patient eligibility, development of a mental health plan, and implementation of the treatment and support plan; and
- Outcomes: the expected overall outcome as a result of the intervention is increased health and wellbeing of people with severe and persistent mental illness.
Figure 2.1: MHNIP program logic
Large image of figure 2.1 (GIF 265 KB)
Top of page
Text version of figure 2.1
Figure 2.1 shows the Program Logic for the Mental Health Nurse Incentive Program (MHNIP). The logic for a program explains how a public policy intervention is expected to work and the underlying relationship between program inputs and outputs.The Program Logic diagram starts with the policy context for the MHNIP which is followed by a summary of the program objectives, then the program scope, the steps for implementing an individual patient mental health plan and outcomes from the program.
Policy context:
- The Australian government Mental Health Nurse Incentive Program is an initiative under the COAG: National Plan on Mental Health 2006-2011, which forms part of the National Mental Health reform agenda and associated initiatives.
- MHNIP Program objectives are to:
- Improve levels of care for people with severe and persistent mental disorders;
- Reduce likelihood of unnecessary hospital admissions and readmissions;
- Assist in keeping people with severe disorders feeling well and connected within the community; and
- Alleviate pressure from privately practicing psychiatrists and GPs.
- The program scope is defined in the MHNIP Program Guidelines. To be eligible to participate in the program, organisations must be community based primary health services. These include:
- General practice services;
- Private psychiatry services;
- OATSIH funded Aboriginal Medical Services; and
- Divisions of General Practice / Medicare Locals.
- Eligible organisations can receive payments of up to $10,000 for establishment costs and $240 per clinical session, with a 25% loading for sessions provided in rural areas.
- To establish the site, eligible organisations must meet Site Level Program Requirements. Organisations must develop formal patient management protocols with input from:
- GPs;
- Mental health nurses;
- Psychiatrists; and
- Patient/carers.
- Organisations are also required to:
- Complete an application form;
- Recruit mental health nurses (credentialed with ACMHN & trained in the use of HoNOS); and
- Submit claim forms to Medicare for reimbursement for clinical services.
- Implement Individual Patient Mental Health Plans:
- Patient eligibility for the MHNIP involves a diagnosis of severe mental illness under DSM-IV or ICD-10 and an assessment of whether the individual is at risk of future hospital admissions, requires treatment over the next 2 years, needs GP or psychiatrist involvement, and whether the patient consents to mental health nurse involvement.
- Following this a mental health care plan is developed which includes the roles and responsibilities of the medical practitioner and mental health nurse.
- Following the development of a mental health care plan, if there is an improved diagnosis, the patient may be discharged.
- A baseline HoNOS measurement should be conducted for the patient if necessary.
- After a mental health care plan has been developed and a baseline HoNOS measurement conducted, where necessary, the patient's mental health care plan should be implemented. As part of implementing the plan there should be a review of the patient's mental state and physical care and a mental health nurse should administer, monitor and ensure compliance with medication and other treatments, improve links to other health professionals services, and liaise with carers.
- Periodic measurement using the HoNOS should be conducted for the patient, followed by a review of the mental health care plan.
- Upon completion of this work, organisations can lodge a Medicare claim to a maximum of 10 sessions per week per mental health nurse.
- Increased levels of care; more effective care for people with severe mental illness;
- Reduction in the number of unnecessary hospital admissions and readmissions for people with severe mental illness;
- People with severe mental illness report increased feeling of connectedness to the community and increased assistance to keep well;
- Psychiatrists and GPs have increased capacity to see more patients and deal with more complex health care needs; and
- Expanded roles and responsibilities of mental health nurses, including the creation of alternative career structure in the community sector.
- Increased health and wellbeing of people with severe and persistent mental illness.
2.5 Relationship to key evaluation areas
Clarification of the program logic demonstrates how the key evaluation areas relate to this conceptualisation of MHNIP, as illustrated in figure 2.2. This framework guided the development of detailed questions and data collection processes used in the evaluation.Appendix A documents the relationship between detailed evaluation topics specified in the RFQ and the remaining contents of the report.
Figure 2.2: Relationship of key evaluation areas to the MHNIP program logic
Large image of figure 2.2 (GIF 256 KB)
Top of page
Text version of figure 2.2
In Figure 2.2 the MHNIP Program Logic as described in Figure 2.1 has been overlayed with the key evaluation areas. The key evaluation areas were:- Program Structure, which includes Program Take-up, Demand Profile and Compliance; and
- Patient Outcomes
- Program Structure:
- Program Take-up: encompasses the community based primary health services under the Program Scope.
- Demand Profile: encompasses patient eligibility, under Implement individual patient mental health plans.
- Compliance: encompasses the implementation of the mental health care plan, the HoNOS periodic measurement and review of the mental health care plan, under Implement individual patient mental health plans.
- Patient Outcomes:
- Includes outcomes following the implementation of an individual patient mental health plan.
- Hinderances/Barriers/Enablers/Alternate Approaches, which applied to the following MHNIP Program Logic components as defined in figure 2.1:
- Program Objectives;
- Program Scope;
- Implement Individual Patient Mental Health Plans;
- Outcomes; and
- Overall Outcomes.
- Effectiveness, which applied to the following MHNIP Program Logic components as defined in figure 2.1, with a stronger focus towards outcomes:
- Program Objectives;
- Program Scope;
- Implement Individual Patient Mental Health Plans;
- Outcomes; and
- Overall Outcomes
- Efficiency, which applied to the following MHNIP Program Logic component as defined in figure 2.1:
- Program Scope; (The consideration of costs and benefits was undertaken with a cost analysis, and examined as part of the efficiency examination.)
- Outcomes, which applied to the following MHNIP Program Logic components as defined in figure 2.1:
- Outcomes; and
- Overall Outcomes.
- Appropriateness, which applied to the following MHNIP Program Logic components as defined in figure 2.1, with a stronger focus towards Program Objectives:
- Program Objectives;
- Program Scope; and
- Implement Individual Mental Health Plans.