Fourth national mental health plan: measurement strategy

Section 2 Approach to indicator and target development

Page last updated: May 2011

2.1 Indicator development
2.2 Target development
2.3 Defining mental illness for measurement purposes

2.1 Indicator development

Fourth Plan indicator development was underpinned by the following principles:
  • Be inclusive of all components of the mental health sector - Under the National Mental Health Strategy there has been significant expansion in mental health services available to the community from public, private and non-government agencies in both primary and specialist sectors. The scope of the Fourth Plan indicators includes all components of the mental health sector. Notwithstanding the broad scope of the Fourth Plan, it is recognised that people experiencing severe and enduring mental illnesses are among the most disadvantaged in our community, so in addition to reporting on the general population, two social inclusion and recovery indicators will also be reported for consumers of state and territory public mental health services.
  • Utilise existing national data and specifications where available - There has been significant investment in the development of a nationally agreed and endorsed mental health performance framework and a set of key performance indicators. Wherever possible, the Fourth Plan indicators will utilise existing national data sets and indicator specifications to build on the capacity of existing frameworks, provide consistency, allow comparison and reduce collection burden.
  • Indicator specifications should meet recognised quality selection criteria - The National Health Performance Framework provides a set of desirable criteria for health sector performance indicators. Details of the criteria are contained in Appendix 3.
  • Use of interim indicators where new data sources need to be developed - Under the Fourth Plan, governments committed to public reporting of the 25 indicators. However, it is acknowledged that there is a need for extensive data development to populate some indicators. For those indicators that do not currently have a suitable data source, interim indicators have been identified for reporting during the period in which the required data and reporting arrangements are being developed. Top of page
To measure the influence of cross-sectoral reform, key intersections are targeted. For example, Proportion of primary and secondary schools with mental health literacy component included in curriculum is an indicator of whether health promotion strategies are reaching into the education sector while Proportion of front-line workers within given sectors who have been exposed to relevant education and training indicates whether a desirable increase in early intervention capacity within the community has been achieved.

Performance measurement generally uses four main types of indicators (Table 1), described as input, process, output, and outcome measures. Historically the health sector has focused on measures of efficiency - inputs and outputs. The mental health sector is now placing a greater emphasis on outcome measures - what impact on consumers' health and wellbeing occurred as a result of receiving a mental health service? Population-based indicators are considered broad outcome measures—what impact on the broader community has occurred? All four types of performance indicators are included in the Fourth Plan, providing a broad range of information on the sector.

Table 1 Types of performance indicators

Table 1 is presented as a list in this HTML version for accessibility reasons. It is presented as a table in the PDF version.

Types of performance indicators:
  • Input - Measurement of the resources used to create a service (such as funding, capital works or human resources).
  • Process - Actions that convert inputs into outputs.
  • Output - Measurement of services provided (such as the number of consumers seen by a service).
  • Outcome - Measurement of the impact as a result of the service or intervention (such as reduction of symptoms or engaging in employment). Top of page

2.2 Target development

The Fourth Plan commits governments to agreeing to national targets where appropriate, against the indicators. Whereas performance indicators are the tools used to measure and gauge the extent to which a goal is met, targets represent 'markers' on the measurement scale that define desired levels of performance.

Target setting is an evolving process and typically must be based on imperfect evidence. It also needs to be approached with reference to the political and economic contexts, recognising that not all jurisdictions are at the same point nor have the same capacity to achieve reform goals.

The targets proposed in this document have been informed by international and local research where available, and/or consultation with relevant experts and stakeholders. Targets have not been set for those indicators requiring new data or further specification, and other indicators for which there is no evidence basis or body of opinion available on which to base credible targets. Where possible, interim targets have been set where additional work is planned. Targets will be the subject of periodic review, informed by ongoing analysis of relevant data.

A systematic process was applied to evaluate and select potential targets, following three steps:

  1. Determining the best evidence base that is available to set a target.
  2. Determining the type of target.
  3. Assessing any proposed target against the 'SMART' criteria. Top of page
The target evaluation process is summarised below:
  • Step 1 - Evidence base (on what is the target based?)
    • Based on evidence.
    • Based on consensus/opinion.
    • Based on community values/aspirations.
  • Step 2 - Type of target (how should the target be expressed?)
    • Rate.
    • Range.
    • Movement or shift.
    • Difference.
  • Step 3 - Overall assessment of suitability 'SMART'
    • Specific.
    • Measurable.
    • Achievable.
    • Realistic.
    • Timely.
Steps 1 and 2 are hierarchical, with the most desirable criteria highest on the list. This process allowed for ranking of potential targets when there were competing candidates. All targets were required to meet the five SMART criteria.

For those indicators identified by the NMHPSC as suitable for target development, the approach used was to propose targets that will present a challenge rather than simply reflecting a level of performance that is easily achievable within the life of the Fourth Plan. In this way, the proposed targets have been developed to assist in driving reform by reflecting the sectors' desire to improve in agreed directions, and at an agreed rate or level of performance. This level of performance may not be achieved by the end of the Fourth Plan, but the targets are aimed at providing an objective reference point by which to judge progress to date and the extent to which further reform is required. Top of page

2.3 Defining mental illness for measurement purposes

Developed separately and for different requirements, the data sources for the indicators utilise different definitions, parameters, terminologies and tools with which to identify and describe mental ill-health. This variation poses a challenge for effective reporting, analysis and interpretation. However, even with this limitation, the available data does provide considerable information relevant to the reform agenda set out in the Fourth Plan.

Regardless of the exact terminology of each data source, this document utilises the following terms adopted in the Fourth National Mental Health Plan 2 to describe mental ill-health:
  • Mental health problem:"Diminished cognitive, emotional or social abilities but not to the extent that the criteria for a mental illness are met."
  • Mental illness:"A clinically diagnosable disorder that significantly interferes with an individual's cognitive, emotional or social abilities. The diagnosis of mental illness is generally made according to the classification systems of the Diagnostic and Statistical Manual of Mental Disorders (DSM) or the International Classification of Diseases (ICD)."

Footnotes

2 Australian Health Ministers' Conference (2009) Fourth National Mental Health Plan: An agenda for collaborative government action in mental health 2009–2014, Commonwealth of Australia, p. 84.