Fourth national mental health plan: measurement strategy

NMHP PI 23 Mental health outcomes for people who receive treatment from state and territory services and the private hospital system

Page last updated: May 2011

  • Strategic issue - Quality Improvement and Innovation.
  • Rationale
    • Improvement in clinical outcomes, measured by a reduction in the severity of symptoms and improvements in functioning, is a core objective of mental health services.
    • The implementation of routine mental health outcome measurement in Australia provides the opportunity to monitor the effectiveness of mental health services across services and jurisdictions.
    • Identifying the comparative effectiveness of mental health services informs benchmarking between services and related service quality improvement activities.
  • Endorsement status - Endorsed by AHMAC Mental Health Standing Committee, February 2011.
  • Date last updated - January 2011.
Indicator details
National Mental Health Performance Framework
Data collection details

Indicator details

  • Description - The proportion of episodes of care, or partial episodes, where either significant improvement, significant deterioration, or no significant change was identified between baseline and follow-up of completed outcome measures. Top of page
  • Numerator - Number of episodes or partial episodes with completed outcome measures, partitioned by mental health setting, where either significant change/significant deterioration/no significant change was identified between baseline and follow-up within the reference period.
  • Denominator - Number of episodes or partial episodes of care with completed outcome measures, partitioned by mental health setting within the reference period.
  • Computation - (Numerator Denominator) x 100.Calculated separately for each group.
  • Calculation conditions
    • Coverage/Scope - All public mental health services organisations, and private psychiatric hospitals, with the following exclusion.Public community residential mental health services.
    • Methodology
      • Only the HoNOS family of measures is considered in the calculation of this indicator.
      • Outcomes are to be calculated for the following three cohorts of consumers.
        • Group A: Consumers discharged from hospitalAll people who were discharged from an acute psychiatric inpatient unit within the reference period. Scores should be calculated as the difference between the total score recorded at admission (the 'baseline') and discharge (the 'follow-up').
        • Group B: Consumers discharged from ambulatory care.All people who were discharged from an ambulatory care episode within the reference period. Scores should be calculated as the difference between the total score recorded at admission to the episode (the 'baseline'), and discharge from the episode (the 'follow-up'). Ambulatory episodes that are completed because the consumer was admitted to hospital must be excluded from the analysis (that is, where the National Outcomes Casemix Collection (NOCC) 'reason for collection' equals change of setting). Top of page
        • Group C: Consumers in ongoing ambulatory care.All people who have an 'open' ambulatory episode of care at the end of reference period – that is, the person commenced the ambulatory episode some time either during or prior to the reference period and has not been discharged from that episode at the end of the reference period. Outcome scores should be calculated as the difference between the total score recorded on the first occasion rated which will be either admission or review, (the 'baseline') and the last occasion rated which will be a review (the 'follow-up') in the reference period.
      • Group change analyses can only be determined for episodes of care where both baseline and follow-up ratings are present. This excludes specific episodes defined by the NOCC data collection protocol as not requiring follow-up as well as episodes or partial episodes where either the baseline or follow-up measure is not available.
      • The total score is determined for each individual baseline and follow-up score. This is the sum total of the 12 HoNOS/65+ scales or the first 13 scales of the 15 HoNOSCA. Where one or more of the HoNOS/65+ or HoNOSCA scales has not been completed correctly, the collection occasion should only be regarded as valid and complete if
        • For the HoNOS and HoNOS65+: A minimum of 10 of the 12 scales have a valid severity rating (ie a rating of either 0, 1, 2, 3 or 4).
        • the HoNOSCA: A minimum of 11 of the first 13 items have a valid severity rating.
      • Outcome scores are classified as either 'significant improvement', 'significant deterioration or 'no significant change', based on Effect Size.
      • The group 'baseline' standard deviation score is calculated separately on the reference period for each age group and service setting stratification using the national data set. The group baseline standard deviation includes all valid clinical ratings (ie any valid baseline rating even although at an individual episode of care level it may not form a matched pair), and will be recalibrated periodically.
      • The reference period for this indicator (including calculation of the effect size) is typically a single financial year, and the result of modifying the reference period is unknown. Top of page
  • Definitions
    • HoNOS family includes HoNOS, HoNOS 65+ and HoNOSCA.
    • As defined by the NOCC Specifications Version 1.6, an Episode of Care is defined as a more or less continuous period of contact between a consumer and a Mental Health Service Organisation that occurs within the one Mental Health Service Setting.
    • Episodes of Care may be brief or prolonged, and may be provided in three settings – inpatient, ambulatory or residential. Under the NOCC protocol, a consumer may be in only one episode of mental health care at any one time.
    • The term 'partial episode' is used here to refer to the period between baseline and follow-up measurement for those consumers who are in ongoing ambulatory episodes, as discussed above (See Group C).
    • Effect size is a statistic used to assess the magnitude of a treatment effect. It is based on the ratio of the difference between the baseline and follow-up scores to the standard deviation of the baseline score. As a rule of thumb, effect sizes of 0.2 are considered small, 0.5 considered medium and 0.8 considered large. Based on this rule, a medium effect size of 0.5 is used to assign outcome scores to the three outcome categories. Thus individual episodes are classified as either: 'significant improvement' if the Effect Size index is greater than or equal to positive 0.5; 'significant deterioration' if the Effect Size index is less than or equal to negative 0.5; or 'no change' if the index is between -0.5 and 0.
  • Presentation - Percentage by group. Top of page
  • Disaggregation - State/Territory (for public mental health services), by HoNOS measure.
  • Notes
    • A specific issue in the interpretation of 'change' scores is how they relate to 'expectations of change' for a given consumer within a given mental health service setting. For consumers who have episodes of care in acute inpatient settings, it is generally accepted that there would be positive significant change as measured by the HoNOS family. In ambulatory settings, the expected outcome for some people may be improvement, but for others might be prevention of relapse (ie no statistical change. The thresholds for change need to be specific to mental health service settings and programs.
    • This indicator is only indicative of a single type of effectiveness and outcome for mental health consumers. Where possible, NOCC-based consumer outcome measures should be complemented by one or more other measures of consumer outcomes (eg social outcomes – housing tenure, employment etc) that demonstrate the different perspectives on, and dimensions of, mental health consumer outcomes.
    • This indicator addresses the sub-domain of consumer outcomes, and assesses severity of symptoms from the clinician's perspective. Improvements on other measures that assess other dimensions from both clinician and consumer perspectives should be considered for future development of performance indicators.
    • The national data set does note currently allow episodes of care to be connected across financial years. This limitation does not exist for states and territories own data sets.
    • This indicator was designed as a measure of aggregate group change.
  • Is specification interim or long-term? - Long-term.
  • Reported in - COAG National Action Plan Progress Report (Indicator 6 Mental health outcomes for people who receive treatment from State and Territory services and the private hospital system). Top of page

National Mental Health Performance Framework

  • Tier - Tier III - Health Service Performance.
  • Primary domain - Effective.
  • Mental health sub-domain - Consumer outcomes.
  • Type of measure - Outcome.
  • Level at which indicator can be useful for benchmarking
    • Service Unit.
    • Regional group of services.
    • Mental Health Service Organisation.
    • State/Territory.

Data collection details

  • Data source(s)
    • Numerator - National Outcomes and Casemix Collection.
    • Denominator - National Outcomes and Casemix Collection.
  • Data source(s) type
    • Numerator - Outcome data.
    • Denominator - Outcome data.
  • Frequency of data source(s) collection
    • Numerator - Annually.
    • Denominator - Annually.