Evaluation questions
Data sources
Relationship between evaluation questions and data sources

Evaluation questions

The summative evaluation addresses a series of evaluation questions about the appropriateness, effectiveness and impact of Better Access. There are three primary questions that relate to whether Better Access has: improved access to mental health care, been effective in producing positive consumer outcomes, and had any impact on the distribution of the mental health workforce between public and private sectors. Nested under each primary question is a series of secondary questions that explore the achievements of Better Access in more detail. The questions are as follows:
  1. Has Better Access improved access to mental health care?
    1. What is the overall level of uptake of Better Access services?
    2. Has Better Access increased the treatment rates of people with mental disorders?
    3. Has Better Access reached groups who are traditionally disadvantaged in their access to mental health care (e.g., people in rural areas, people in areas of socio-economic disadvantage, young people), and if so, has the rate of Better Access uptake improved over time for these groups?
    4. Has Better Access reached new consumers?
    5. Has Better Access reached consumers with moderate to severe disorders, or has it predominantly provided care to those with mild symptoms?

  2. Is Better Access an effective (and cost-effective) model of service delivery?
    1. Is Better Access achieving positive outcomes for consumers?
    2. Do some consumers experience better outcomes than others?
    3. Is Better Access a cost-effective way of delivering primary mental health care?

  3. Has Better Access had an impact on the profile and operation of Australia's mental health workforce?
    1. To what extent has Better Access had an impact on the distribution of mental health care providers in the public and private sectors?
    2. Has Better Access improved multi-disciplinary collaboration between mental health care providers?
The current questions have been modified from those in the original evaluation framework to reflect some of the public debate about the implementation and impact of Better Access. The current evaluation questions overlap substantially with those in the original evaluation framework2 (see Appendix 1), but a small number are not considered in the summative evaluation.

Data sources

The current report relied on a number of data sources. They included the evaluation components articulated in the original evaluation framework (see Chapter 1), as well as several additional data sources. The full range of data sources is listed in Table 1. Each source is described, and its strengths and weaknesses are outlined. Further detail on each of these data sources is available in the cited reports and journal articles.

Top of page

Table 1: Data sources

Table 1 is presented as a list in this HTML version for accessibility reasons. It is presented as a table in the PDF version.

Part of original evaluation framework

  • Component A: A study of consumers and their outcomes3 - In this study Better Access providers recruited 5-10 consumers to participate, according to a protocol. Forty one clinical psychologists recruited 289 consumers, 49 registered sychologists recruited 317 consumers, and 39 GPs recruited 277 consumers. Consumers were profiled on a range of sociodemographic, clinical and treatment variables, and their outcomes were monitored. Consumers and providers were also interviewed/surveyed about their respective experiences with receiving and delivering Better Access care.

    Strengths - Large and reasonably representative samples of providers and consumers; data collected that weren't available from any other source (e.g., clinical and treatment data, outcome data) for these consumers.

    Limitations - Self-selection of providers and consumers which may have introduced biases; potential differential sample attrition of those who did not improve; no potential to determine whether improvements in outcomes were sustained over time; no capacity to include non-English speaking consumers; reliance on self-report.

  • Component A.2: A study of consumers and their outcomes (focusing on the occupational therapy and social work sectors)4 - In this study Better Access providers invited 2-5 consumers to participate, according to a protocol. One hundred and ninety one social workers recruited 458 consumers, and 35 occupational therapists recruited 72 consumers. Consumers and providers were interviewed/surveyed about their respective experiences with receiving and delivering Better Access care.

    Strengths - Large and reasonably representative samples of providers and consumers; systematic collection of data on consumers' views.

    Limitations - Self-selection of providers and consumers which may have introduced biases; no capacity to include non-English speaking consumers; reliance on self-report.

  • Component B: Analysis of Medicare Benefits Schedule and Pharmaceutical Benefits Scheme administrative data5 - This involved analysis of routinely-collected data from a range of sources (e.g., person- and service-level Medicare Benefits Schedule mental health item data and Pharmaceutical Benefits Scheme data, 2007 National Survey of Mental Health and Wellbeing, and the Access to Allied Psychological Services projects' minimum dataset). It addressed questions relating to the access, affordability, equity, and interdisciplinary nature of Better Access services, and the potential impact of Better Access on service use in other parts of mental health system.

    Strengths - Use of Medicare Benefits Schedule data to provide person-level profiles of Better Access item uptake according to available consumer characteristics; use of ecological analyses combining Medicare Benefits Schedule and other data to explore issues beyond the scope of Medicare Benefits Schedule data alone (e.g., equity, impact on other mental health programs).

    Limitations - Limited socio-demographic data and lack of clinical information available in Medicare Benefits Schedule (and Pharmaceutical Benefits Scheme) data; difficulties in making inferences about individuals from aggregated data (e.g., Medicare Benefits Schedule data could not be linked to other mental health program data to explore the impact of Better Access).Top of page

  • Component C: An analysis of allied mental health workforce supply and distribution6 - This involved the analysis of routinely-collected data from a range of sources (e.g., Medicare Benefits Schedule and Medicare provider data, Australian Bureau of Statistics 2006 Census of Population and Housing, Mental Health Establishments – National Minimum Data Set, Registration Board and other administrative data, and stakeholder consultations). It addressed issues related to identifying and characterising the Consolidation and interpretation of a large amount of data. Difficulties associated with quantifying the workforce (e.g., determining workforce denominators, converting headcounts to full time equivalents). Difficulties in making inferences about individuals from actual and potential Better Access workforce, the impact of Better Access on the distribution of the allied mental health workforce, and future workforce needs.

    Strengths - Consolidation and interpretation of a large amount of data.

    Limitations - Difficulties associated with quantifying the workforce (e.g., determining workforce denominators, converting headcounts to full time equivalents). Difficulties in making inferences about individuals from aggregated data (e.g., no potential for linking public and private sector workforce data to determine how many people may be working part-time in both sectors).

  • Component D: Stakeholder consultations7 - This involved consultations with 1,200 stakeholders (e.g., service providers from the private, public and non-government sectors, representatives of professional bodies, health insurers, consumers and carers) regarding their views about Better Access.

    Strengths - Large number of stakeholders from diverse settings and locations.

    Limitations - Self-selection of stakeholders which may have introduced biases; reliance on self-report; some difficulties in interpreting findings (e.g., deciding what weight to give particular opinions; determining how representative particular views were of specific stakeholder groups).

  • Component E: Evaluation of main education and training projects8 9 - The most prominent education and training initiative was the Mental Health Professionals Network (MHPN), which undertook a large-scale project to promote interdisciplinary networking. The evaluation of this initiative drew on data from nearly 20 sources, some internal and some external, some routinely-collected (e.g., workshop and network attendance data) and some purpose-designed (e.g., surveys).

    Strengths - Evaluation developed alongside the MHPN project, in collaboration with MHPN; drew on information from a number of evaluation components, to allow triangulation of the findings.

    Limitations - Some data sources more reliable than others (e.g., some surveys had sub-optimal response rates); in some cases, certain biases may have been introduced (e.g., because of self-selection of participants); some data were collected before an appropriate establishment period for the relevant activity had elapsed.

  • Component F: An analysis of the second National Survey of Mental Health and Wellbeing, completed in 200710 - The 2007 National Survey of Mental Health and Wellbeing was a nationally representative household survey of 8,841 individuals aged 16-85. The survey instrument was based on the World Mental Health Survey Initiative version of the Composite International Diagnostic Interview. It also collected information on the respondents' mental health status to assess whether or not they had experienced an affective disorder, an anxiety disorder and/or a substance use disorder during their lifetime, the past 12 months or the past 30 days. The survey also collected information on respondents' health service use and perceived needs for care, and a range of related variables (e.g., respondents' levels of functioning and disability, levels of psychological distress, quality of life) and socio-demographic variables.

    Strengths - Large and representative sample of Australian adults.

    Limitations - Excluded children and young adolescents, elderly adults and those who were not fluent in English; response rate was 60%; relied on self-report; covers only early period of Better Access implementation.
Top of page

Additional data sources

  • Independent analyses of the 2007 National Survey of Mental Health and Wellbeing, including comparisons with the 1997 National Survey of Mental Health and Wellbeing11-16 - Several members of our team and others have also conducted independent analyses of National Survey of Mental Health and Wellbeing data. Harris et al explored the use of Medicare-subsidised allied health services among people with a need for mental health treatment and Burgess et al considered service use for mental health problems more generally. The summative evaluation also makes reference to other more general publications on the 2007 National Survey of Mental Health and Wellbeing, and comparisons with the 1997 National Survey of Mental Health and Wellbeing, conducted before the introduction of Better Access.

    Strengths - As above.

    Limitations - As above. In addition, some caution should be exercised in interpreting the comparisons between the 2007 and 1997 National Surveys of Mental Health and Wellbeing because of differences in sampling, survey instruments and response rates.

  • Relevant Department of Health and Ageing documentation17 - The Department of Health and Ageing has undertaken work to calculate current treatment rates for people with common mental disorders for each of the four financial years since Better Access was implemented.

    Strengths - Clearly articulated steps in calculating treatment rates; recourse to a range of appropriate data sources to calculate prevalence, service use etc., and correction for data gaps in the National Surveys of Mental Health and Wellbeing.

    Limitations - Reliance on various assumptions in the absence of data for some key parameters in the process.

  • Post-implementation review of Better Access18 - This was conducted by the Department of Health and Ageing to assess the uptake, implementation and appropriate use of the Better Access item numbers. It involved analysis of service-level, patient-level and provider level Medicare data, as well as consultations with stakeholders.

    Strengths - Combination of quantitative data from Medicare (service-, patient- and provider-level) and qualitative data from stakeholders.

    Limitations - Limited socio-demographic data and lack of clinical information available in Medicare Benefits Schedule data; potential for selection and response bias in stakeholder consultations.

  • Australian Institute of Health and Welfare (AIHW) analyses of service-level and patient-level Medicare data19 20 - As part of its routine reporting on mental health services in Australia, the AIHW has analysed the number of Medicare funded services provided and patients served by GPs, psychiatrists, psychologist and other allied mental health professionals.

    Strengths - Use of Medicare Benefits Schedule data to provide patient-level and service-level profiles of Better Access item uptake according to available consumer characteristics.

    Limitations - Limited socio-demographic data and lack of clinical information available in Medicare Benefits Schedule data.

  • Independent analyses of service level Medicare data21-26 - Several independent analyses of service-level Medicare data have been conducted, including ones by: Russell; Bassilios et al; Fletcher et al; and Whiteford et al.

    Strengths - Use of Medicare Benefits Schedule data to provide service-level profiles of Better Access item uptake according to available consumer characteristics.

    Limitations - Reliance on service-level data to draw person-level conclusions; limited socio-demographic data and lack of clinical information available in Medicare Benefits Schedule data.Top of page

  • The Bettering the Evaluation and Care of Health (BEACH) program27 28 - BEACH is a continuous national study of general practice activity. Each year, BEACH recruits 1,000 GPs who provide data on 100 consecutive encounters in their practices. Analyses have examined trends in GP encounters for psychological problems, and the use of Medicare Benefits Schedule items for these encounters.

    Strengths - Provides details on GP encounters that are not available from any other source; large and reasonably representative sample of GPs.

    Limitations - Potential for selection bias; reliance on self-report; very broad listing of eligible "psychological problems".

  • A study of uptake of Better Access item numbers by women29 - This study examined the uptake of Better Access item numbers among women with depression and anxiety, linking survey data from 14,911 participants in the Australian Longitudinal Study on Women's Health (ALSWH) to their Medicare records for the period November 2006-December 2007.

    Strengths - Provides accurate information on use of Better Access items for a large number of women, through linkage to Medicare data.

    Limitations - Identifies women as having a mental disorder if they report having received such a diagnosis from a doctor (with supplementary indicators like the SF-36); involves univariate (as opposed to multivariate) analysis of demographic and socio-economic factors; covers only early period of Better Access implementation.

  • Australian Psychological Society surveys of members30 31 - The Australian Psychological Society conducts ad hoc surveys of its embers. The surveys examined participants' experiences with delivering care under Better Access.

    Strengths - Surveys have the imprimatur of the professional body which represents psychologists.

    Limitations - Numbers of respondents and response rates are not always clear; restricted to Australian Psychological Society members.

  • Independent surveys of occupational therapists32 - This involved surveys of occupational therapists conducted six months (n=21) and 12 months (n=44) after the introduction of Better Access. The surveys examined participants' experiences with delivering care under Better Access.

    Strengths - Providers sampled from up-to-date, comprehensive lists of occupational therapists (although a significant proportion could not be contact because they withheld their details); provides unique information on the views of occupational therapists.

    Limitations - Variable response rates (36% in first survey; 80% in second); reliance on self-report.

  • Independent survey of psychologists33 - This involved a survey of 73 psychologists conducted in the year after Better Access was introduced. The surveys examined participants' experiences with delivering care under Better Access.

    Strengths - Providers sampled from up-to-date, comprehensive lists of occupational; provides unique information on the views of occupational therapists.

    Limitations - Sub-optimal response rate (36%); reliance on self-report.

  • Australian Psychological Society survey of consumers34 - The Australian Psychological Society conducted a one-off survey of consumers receiving Medicare-funded psychological services under Better Access. Equal numbers of clinical and registered psychologists with a Medicare provider number recruited consumers. In total, 2,223 consumers were recruited. These consumers provided socio-demographic, clinical and treatment information, and offered their perceptions of treatment effectiveness.

    Strengths - Large and reasonably representative samples of providers and consumers; systematic collection of data on consumers' views.

    Limitations - Self-selection of providers and consumers which may have introduced biases; no capacity to include non-English speaking consumers; reliance on self-report.

  • A study of outcomes for consumers seen by psychologists through Better Access35 36 - This study examined outcomes for 525 consumers seen by psychologists in a single private practice.

    Strengths - Provides routinely-collected information on outcomes for consumers seen by psychologists.

    Limitations - Generalisability of findings; potential selection biases regarding the inclusion of consumers; potential differential sample attrition of those who did not improve; no potential to determine whether improvements in outcomes were sustained over time.

  • A study of outcomes for consumers seen by occupational therapists through Better Access37

    This independent study examined outcomes for 31 consumers recruited by a single occupational therapist.

    Strengths - Provides unique information on outcomes for consumers seen by occupational therapists.

    Limitations - Generalisability of findings; potential selection biases regarding the inclusion of consumers; potential differential sample attrition of those who did not improve; no potential to determine whether improvements in outcomes were sustained over time.
Top of page

Relationship between evaluation questions and data sources

Table 2 indicates the data sources used to answer each of the evaluation questions.

Table 2: Key questions and data sources

Table 2 is presented as a list in this HTML version for accessibility reasons. It is presented as a table in the PDF version.

1. Has Better Access improved access to mental health care?

1a. What is the overall level of uptake of Better Access services?
  • Component B5
  • Post-implementation review of Better Access18
  • Australian Institute of Health and Welfare (AIHW) analyses of servicelevel and patient-level Medicare data19 20
  • Independent analyses of service-level Medicare data21-26
1b. Has Better Access increased the treatment rates of people with mental disorders?
  • Independent analyses of 2007 and 1997 National Surveys of Mental Health and Wellbeing11 13 15
  • Component F10
  • Relevant Department of Health and Ageing documentation17
1c. Has Better Access reached groups who are traditionally disadvantaged in terms of access to dental health care and, if so, has the rate of Better Access uptake improved over time for these groups?
  • Component B5
  • Post-implementation review of Better Access18
  • Australian Institute of Health and Welfare (AIHW) analyses of service level and patient-level Medicare data19 20
  • Independent analyses of service-level Medicare data21-23
  • Independent analyses of 2007 National Survey of Mental Health and Wellbeing14
  • Bettering the Evaluation and Care of Health (BEACH) program27 28
  • A study of uptake of Better Access item numbers by women29
1d. Has Better Access reached new consumers?
  • Component A3
  • Component B5
  • Independent analyses of 2007 National Survey of Mental Health and Wellbeing14
  • Australian Psychological Society surveys of members 30 31
  • Bettering the Evaluation and Care of Health (BEACH) program27 28
  • A study of uptake of Better Access item numbers by women29
1e. Has Better Access reached consumers with moderate to severe disorders, or has it predominantly provided care to those with mild symptoms?
  • Component A3
  • Component B5
  • Independent analyses of 2007 National Survey of Mental Health and Wellbeing14
  • A study of uptake of Better Access item numbers by women29

2. Is Better Access an effective (and cost-effective) model of service delivery?

2a. Is Better Access achieving positive outcomes for consumers?
  • Component A3
  • Component A24
  • Australian Psychological Society survey of consumers34
  • A study of outcomes for consumers seen by psychologists through Better Access35 36
  • A study of outcomes for consumers seen by occupational therapists through Better Access37
  • Post-implementation review of Better Access18
  • Component D7
2b. Do some consumers experience better outcomes than others?
  • Component A3
2c. Is Better Access a cost-effective way of delivering primary mental health care?
  • Component A3
  • Component B5

3. Has Better Access had an impact on the profile and operation of Australia's mental health workforce?

3a. To what extent has Better Access had an impact on the distribution of allied health professionals in the public and private mental health sectors?
  • Component C6
  • Component D7
  • Australian Psychological Society surveys of members30 31
  • Independent surveys of occupational therapists32
  • Independent survey of psychologists33
3b. Has Better Access improved multi-disciplinary collaboration between mental health care providers?
  • Component A3
  • Component A.24
  • Australian Psychological Society surveys of members30 31
  • Independent surveys of occupational therapists32
  • Independent survey of psychologists33
  • Component E8 9
Top of page