Evaluation of the Better Access to Psychiatrists, Psychologists and General Practitioners through the Medicare Benefits Schedule initiative: summative evaluation

Appendix 2: Assessing the effects of uncertainty in the Department of Health and Ageing's estimate of the proportion of Australians with a mental disorder who received mental health care each year between 2006-07 and 2009-10

Page last updated: 22 February 2011

This appendix describes the results of modelling that was undertaken by the evaluation team to assess the effects of uncertainty in key parameters used in the Department of Health and Ageing model to estimate the proportion of Australians with a mental disorder who received mental health treatment in each year over the period 2006-07 to 2009-10. The model is described in more detail below and is elaborated in full in papers made available by the Department to the evaluators.

The most recent estimate of treatment rates for mental disorder in Australia was provided by the 2007 National Survey of Mental Health and Well Being (NSMHWB, 2007). In principle, the Survey provides estimates of both (1) the number of persons with a mental disorder in Australia in the past year and (2) the proportion of these persons who reported receiving some type of mental health treatment in that year. This survey had two major limitations for the purposes of estimating any impact that the Better Access Initiative has had on treatment rates for mental disorder.

The first limitation was that it was conducted in 2007 and asked about access to treatment in the one year period before the date of the survey. The Survey therefore only covered the first year's operation of the Better Access Initiative so these data could not be used to assess the impact of Better Access on rates of mental health treatment for any of the subsequent years. The second limitation was that the survey data did not provide complete coverage of the Australian population. Specifically, it did not sample Australians who were younger than 16 years of age or older than 85 years. In addition its coverage was restricted to persons living in private households. This excluded elderly people in hospitals and aged care facilities with mental disorders. For these reasons, estimates of the number of persons in the younger and older age groups who had a mental disorder in the past year had to be derived from other sources.

Why model the effects of uncertainty?
The Department of Health and Ageing model
Modelling the effects of uncertainty in the Department's model
Acknowledgments
References

Why model the effects of uncertainty?

In order to produce the desired estimate, the Department needed to calculate (1) the number of persons with a mental disorder treated in each year as a proportion of (2) the number of Australians who had a mental disorder in that year. There is uncertainty in the estimates of both (1) and (2) that arises for a number of reasons.

One cause of uncertainty is sampling error in some estimates e.g. of the proportion of persons aged 16 to 64 years estimated to have had a mental disorder in the past year by the NationalSurvey of Mental Health and Well Being, 2007, a population survey of mental disorders in a representative sample of Australians between 16 and 85 years.

A different type of uncertainty arises, for example, because of the lack of key data. For example, there are no data on the proportion of persons who received mental health care in any year from both state mental health services and Commonwealth funded MBS services (e.g. those provided by general practitioners or other health professionals under Better Access). In this case, the Department's model had to make assumptions about the likely extent of such double treatment.

By modelling the effects of uncertainty in key parameters of the Department's model we were able assess the extent to which changes in the estimated proportion of persons with mental disorders who receive mental health treatment may have been due to sampling error or uncertainty in key parameters.
Top of page

The Department of Health and Ageing model

The Department's model estimated the proportion of persons with a mental disorder who received mental health treatment in the past year using the following data sources and steps.

All population data on the estimated number of Australians in each age group in each year were obtained from the Australian Bureau of Statistics projections from census data of the number in the Australian population in each year by age as at June for that year. We treated the ABS estimates as being without error for the purpose of this exercise.

1. Estimating of number of Australians with mental disorders

The number of Australians with mental disorders in the past year was estimated by adding together estimates in each of the following three age groups (see table 1).

1.1 Mental disorders in persons aged 0-15 years:

Age specific estimates on the proportion of persons in this age group with a mental disorder were obtained from a New South Wales Department of Health report 'Mental Health Clinical Care and Prevention Model (MH-CCP): A Population Mental Health Model', first released in 2001 and currently being revised (New South Wales Department of Health, 2001) . This model used estimates from the Australian Burden of Disease and Injury study (Begg et al, 2007) that were, in turn, derived from multiple epidemiological sources.

1.2 Mental disorders among persons aged 16-64 year olds:

These estimates were based on the estimated prevalence of mental disorders in this age group in the NSMHWB, 2007 (Australian Bureau of Statistics, 2009).

1.3 Mental disorders in persons aged 65 years and older

The prevalence of mental disorders in this age group was also estimated from the New South Wales MH-CCP estimates for 2 age groups: 65-74 and 75 and over. These were obtained by the same type of modelling used to estimate the prevalence of mental disorders among person 15 years and younger.

In the opinion of the evaluators these data sources were the best available for the purpose for estimating number of persons in each age group with a mental disorder in the past year. Each of these prevalence estimates was multiplied by the number of persons in each age and sex grouping in the Australian population (see table 1).
Table 1: The estimated prevalence and number of persons with a mental disorder
Age groupPrevalenceNumber of people with 12 month disorders

2006-07
Number of people with 12 month disorders

2007-08
Number of people with 12 month disorders

2008-09
Number of people with 12 month disorders

2009-10
0-15
15.4%
674,141
681,546
690,366
697,657
16-64
22.2%
3,089,046
3,158,081
3,230,351
3,282,449
65-74
13.6%
197,087
202,750
210,740
219,523
75+
16.1%
210,359
214,342
218,280
223,092
Total
20.1%
4,170,634
4,256,720
4,349,738
4,422,721

2. Estimating the number of Australians using mental health services

In the Department's model an estimate of the number of Australians who received treatment for a mental disorder in each year was obtained by combining a series of separate estimates of the number of persons receiving different types of mental health care in each year (see table 2). These numbers were derived as follows.

2.1 Persons receiving identified mental health services funded by the Commonwealth

Data on these persons was derived from Medicare claims data on mental health services subsidised by the Department of Health and Ageing through the Medical Benefits Schedule. These services could be provided by general practitioners, specialist psychiatrists and other health professionals under the Better Access Initiative and other Commonwealth schemes that allow mental health specific consultations to be identified. These data provide the best available data source for these services.

2.2 Persons who receive mental health treatment in state and territory funded mental health services

This comprised data on persons receiving treatment in state- and territory-funded community mental health services which are believed to provide a reasonably comprehensive count of the total population treated. The coverage of these data varied between states and methods of collection may have differed between states Adjustments were made to account for known under-counting by Victoria but specific studies are needed to improve the consistency and quality of these data in all jurisdictions.

2.3 Persons treated for mental disorders with the Department of Veterans' Affairs.

This estimate was provided by the Department Veterans 'Affairs based on their records of the number of veterans and their dependents who received mental health services funded by the Department in 1997-98, projected forward in subsequent years as a proportion of the declining DVA treatment population. More recent data would be preferable but were unavailable. Top of page

2.4 Other persons treated by a general practitioner for a mental disorder

The NSMHWB in 1997 and 2007 showed that general practitioners are the health professional most often consulted by persons with a mental disorder. Surveys of general practice consultations (such as BEACH) also show that GPs do not always bill these consultations as mental health consultations (Australian Institute of Health and Welfare, 2010). BEACH data were used to estimate the rate of growth in GP mental health consultations since Better Access was implemented. The proportion of persons with a mental disorder who consulted a GP for a mental health problem in the 2007 NSMHWB was used to estimate the number of persons who consulted a GP for a mental disorder in each year (assuming that this proportion remained constant over the period). Based on this analysis, the estimated number of people with mental disorders treated by GPs who were not billed under a mental health item declined steeply between 2006-07 and 2009-10, probably as more GPs used MBS mental health under Better Access to bill for such consultations.

2.5 Persons receiving other mental health services not included in any of the above

The 2007 NSMHWB estimated that 6.6% of persons with a mental disorder in the past year had sought treatment for that disorder from a practitioner, other than a GP, specialist psychiatrist or psychologist. This may have included: specialist medical practitioners funded through the MBS (e.g. paediatricians who treated children with mental disorders); general public hospitals and health centres without a mental health unit (and hence not included in state and territory client counts); public alcohol and drug treatment services; state and territory community health services; aged care residential centres; and general hospital emergency departments.

When adjusted to remove people who also received treatment from specialist mental health provider the NSMHWB estimate was reduced to 1.6%. This was probably an underestimate because it excluded treatment providers who were not counted elsewhere in the survey. In the absence of better information, the number of people treated by 'other health services' was estimated as the midpoint (4.1%) of the minimum (1.6%) and maximum (6.6%) estimates derived from the NSMHWB.

2.6 Commonwealth and State and Territory Mental health Services not separately included

The Department's model did not include counts of the number of persons receiving mental health treatment from some Commonwealth and State and territory-funded services. In the case of the Commonwealth-funded services, these included: persons seen by allied health professionals under the Access to Allied Psychological Services (ATAPS) program; and persons seen by services funded through the National Youth mental health initiative (headspace). It was assumed in each case that persons treated through these programs were likely to be counted elsewhere, for example, under one of the MBS mental health service streams or in state and territory counts.

People treated in psychiatric units of private hospitals number (around 24,000 pa) were not included because it was assumed that almost all would already be included in the MBS data among people seen by Consultant Psychiatrists.

In the case of states and territory-funded services, the counts excluded: a small number of persons treated in hospital psychiatric units and community residential services who do not receive clinical ambulatory care services; and persons treated by primary mental health care teams based in state funded community health centres. For some jurisdictions the latter services may include significant numbers of clients but no national data were available to estimate their number.

The evaluators accepted that it was reasonable not to include separate estimates of the number of persons treated through these mental health services. They agreed that in many cases these persons would already be included under other headings. In those cases, where they may not be already included, the model errs appropriately in the direction of conservatism. The consequence is that the estimates from the Commonwealth model are more likely to underestimate the number of persons with mental disorders who are receiving treatment for those disorders.

2.7 Adjusting for overlap between Persons Treated in Commonwealth and State and Territory mental health services

A person with a mental disorder in any one year could receive treatment for that disorder in both a state/territory mental health service and from a Commonwealth-funded health practitioner. An accurate estimate of the overlap between the clientele of the two types of service would require the linkage of records for individuals in order to provide a count of the number of unique persons who were treated in each system.
Top of page
In the absence of such data, the number of persons treated in state/territory services was discounted by 15% in the Department's model. This figure was derived from the proportion of persons in state mental health services who were reported to have had 'one contact only' with state/territory mental health services. The view was that these persons were unlikely to have the type of severe mental illnesses that state services primarily cater for (namely, a psychosis, severe depression, severe personality disorder, etc). It was more likely that they had a common mental disorder, such as, anxiety or depression, and that they would obtain treatment from a practitioner funded by MBS.

The evaluators could not identify any better data on the extent overlap between persons receiving mental health services provided by the Commonwealth and State and territory governments. They thought it reasonable to assume that state/territory and Commonwealth mental health services treat different patient populations. Since this was the least certain data source contributing to the Department's estimate we modelled the effects of substantial uncertainty around this figure in sensitivity analyses by varying this assumed proportion between 10% and 20%.

Table 2: Estimated numbers of persons treated for a mental health problem

2006-072007-082008-092009-10
State and Territory MHS
300,108
299,530
308,722
312,689
MBS funded mental health services - GP only
235,285
386,885
485,056
533,261
MBS funded services - other providers +/- GP
413,990
620,519
740,455
833,519
DVA mental health care
63,415
60,815
58,151
55,628
MBS funded GP services not billed as mental health items
373,518
222,816
188,100
123,240
Other health services
170,996
174,526
178,339
181,332
Total
1,557,313
1,765,091
1,958,824
2,039,668

3. The percentage of Australians with mental disorders who received treatment

The estimated percentages of Australians with a mental disorder in the past year who received mental health treatment in that year was produced by dividing the estimated number of persons treated for a mental disorder in each year by the estimated number of persons with a mental disorder in that year (see table 3).

Table 3: Estimated percentages of Australians with a mental disorder in the past year who received mental health treatment in that year

2006-072007-082008-092009-10
State and Territory MHS
7.2%
7.0%
7.1%
7.1%
MBS funded mental health services - GP only
5.6%
9.1%
11.2%
12.1%
MBS funded mental health services - other +/- GP
9.9%
14.6%
17.0%
18.1%
DVA mental health care
1.5%
1.4%
1.3%
1.3%
MBS funded GP services not billed as mental health items
9.0%
5.2%
4.3%
2.8%
Other health services
4.1%
4.1%
4.1%
4.1%
Total
37.3%
41.5%
45.0%
46.1%

Modelling the effects of uncertainty in the Department's model

We used the Monte Carlo simulation program Ersatz (Barendregt, 2010) to assess the effects of uncertainty on the estimated proportion of persons with a mental disorder who were treated for a mental health problem in each year. This numerical method was used because uncertainty originated from several sources, which made it not feasible to use an analytical approach.

The sources of uncertainty can be grouped into two categories: sampling uncertainty, and other uncertainty. First, sampling uncertainty affected the estimated prevalence of people with any mental disorder in the 16-64 year age group that was based on the NSMHWB, 2007. The standard assumption is that prevalence estimates follow a Binomial distribution. We used the Beta distribution as a continuous approximation of the Binomial (Gelman, 2004). The parameters of the Beta distribution were the number of cases and number of non-cases that were obtained by multiplying the observed proportion by the sample size of the NSMHWB.

For age groups 0-15, 65-74, and 75+ the estimated proportions were based on various sources, and no sample sizes were available. We assumed that the standard error in these age groups was the same as in the 16-64 age group and derived parameters for Beta distributions for these age groups, using the method of moments (Briggs, 2006).

Other data inputs are based on population counts instead of a sample, and therefore have no sampling error. But they can be affected by uncertainty because of misclassification, incompleteness, and double counting .The uncertainty in each case was modeled by assuming triangular distributions, which took as parameters a lowest, most likely, and highest value according to the informed judgments of the evaluators. Go to table 4 for the variables.

To calculate the uncertainty in the proportion receiving mental health services, we replaced the central estimates of the variables listed in table 4 with the corresponding random Ersatz functions, and recalculated the spreadsheet 2000 times. At each recalculation the Ersatz functions drew a random value from each of the random functions. From the resulting distribution in outcomes, Ersatz calculated 95% uncertainty intervals based on percentiles.
Top of page

Table 4: Variables with uncertainty, assumed distributions and parameter values

Variable Distribution1Parameter 1Parameter 2Parameter 3
Prevalence disorder 0-15
Beta
809
4436
n/a
Prevalence disorder 16-64
Beta
1537
3399
n/a
Prevalence disorder 65-74
Beta
638
4072
n/a
Prevalence disorder 75+
Beta
870
4549
n/a
MBS funded GP services not billed as mental health items
Triangular
0.9
1
1.1
Other health services (non mental health)
Triangular
0.016
0.041
0.066
Double count state/territory and Commonwealth
Triangular
0.1
0.15
0.2

1: Beta distribution: parameter 1 is α1, parameter 2 is α2, parameter 3 is not applicable; Triangular distribution: parameter 1 is lowest, parameter 2 is most likely, parameter 3 is highest.

Results

The results of modeling the uncertainty in the Department's model are summarized in table 5 below for each year. These include in each case the best or mean estimate of the percentage of persons with a mental disorder in the past year who received mental health treatment for that disorder and the upper and lower limits of a 95% uncertainty interval around each of these estimates (derived from the distribution generated from 2000 iterations). The key results are as follows.

First, the estimated proportion of persons treated for a mental health problem increased steadily each year from 37.4% in 2006-07 to 46.1% in 2009-2010. There was an overall increase of 8.7% in the proportion of persons with a mental disorder who were estimated to have received mental health treatment.

Second, there was considerable uncertainty around each of these estimates. They generally fell within plus or minus 3.9% of the best estimate. Thus, for example, in 2006-07, the 95% uncertainty interval around the estimate of 37.4% ranged between 33.9% and 40.8% (a range of 6.9%).

Third, the intervals around the best estimate for each year did not overlap for 2007-8 and 2008-9 with the best estimate in the year before. This was not the case for 2009-10. This pattern of results indicates that the increase between 2006-7 and 2008-9 in the percentage persons with a mental disorder who received treatment was unlikely to be due to chance variations in the data. We can be less confident about the smaller increase between the last two years of the period, suggesting a possible slowing in the percentage treated in the most recent period. With this exception, overall, there has been a significant increase in the proportion of persons with a mental disorder who receive mental health treatment for that disorder in the past year.

Table 5: Estimated percentages of persons with a mental disorder treated in each year with 95% uncertainty intervals

2006-072007-082008-092009-10
Mean
37.4
41.5
45.0
46.1
95%CI - LL
35.0
39.7
42.7
43.8
95%CI - UL
39.6
43.7
47.3
48.4

Discussion

The uncertainty analysis indicated that the proportion of persons with a mental disorder in the past year who received treatment for that disorder in the year has increased steadily since 2007. Uncertainty in the estimates is unlikely to explain this trend, with the exception of the last two years in the data series.

The only external source of comparison with these estimates is that derived from the NSMHWB in 2007. The survey can be used to estimate the proportion of persons with a mental disorder in the past year who received treatment for that disorder (see table 6). According to the Survey, 34.9% of persons with a disorder in that year received mental health treatment, reducing to 32.9% when adjustments are made for the younger and older population groups. The estimate for 2007 derived from the Department's model (as indicated above) was 38.7%. These estimates show reasonable convergence for the single year in which estimates can be derived by the two methods. There may have been some underestimation in the 2006-07 administrative data because this was the commencement year for Better Access and the data used to derive the estimate were only available for 8 months of the year (Nov 06 to Dec 07).
Top of page

Table 6: Estimated number of persons with a mental disorder receiving treatment for that disorder in 2007 (from NSMHWB)

Age group Persons with any 12 month disorder using health services for a mental health problem
%
Persons with any 12 month disorder using health services for a mental health problem
N
0-15
25.0
168,535
16-64
35.0
1,081,166
65-74
38.9
76,667
75+
22.6
47,541
Total
32.9
1,373,909

Limitations of model estimates

The data sources used to produce the Department's estimates are the best available but they have their limitations.

The Survey data are the best available on the number of Australians with a mental disorder in the past year but they are known to under-enumerate Australians with a mental disorder in the past year in persons under 16 years and over 65 years. The revised estimates supplement the survey estimates to the degree that is possible for persons under 16 years and over 65 years, using the best vailable international data.

The estimates of numbers of persons treated for a mental disorder come from a combination of sources. Some are administrative data such as that from the MBS based on claims made for specific mental health services, or state/territory records of the number of persons treated for a mental disorder. The former data has probably been more consistently collected than the latter, where methods for the collection vary between states and territories. Some of the estimates are derived from NSMHWB data because there were no accessible data on the numbers of persons treated for a mental disorder by GPs who do not bill this as a mental health consultation or services provided by other practitioners. Nor were there good estimates of the proportion of persons with disorders who receive mental health care from other health care practitioners. There was considerable uncertainty around the nearest estimates of this proportion that were derived from the 2007 NSMHWB (namely, 1.6% to 6.6%).

Another major source of uncertainty was the extent of overlap in number of persons who receive treatment in both state/territory mental health services and from private practitioners under MBS billing was unknown. Such data could be obtained in future by record linkage between MBS and state mental health data sets. The overlap assumed in the Department's model was plausible, given what is known about the type of mental disorders in persons seen by the two types of service. The sensitivity analysis included an estimate of the effect of 33% variations in this assumed proportion.

In the absence of such data we think it unlikely that this source of uncertainty explains the increase in the proportion of persons with mental disorders treated for those disorders between 2006-07 and 2008-09. The extent of overlap would need to be much larger than we have assumed to eliminate this trend. This proportion would also need to increase steeply over time and even then it could not wholly explain the trend observed because many fewer persons received care in state mental health services than persons funded under the MBS. If policy makers wish to reduce this source of uncertainty then they need to fund and approve record linkage studies to estimate the number of patients who are treated in both treatment systems.

Acknowledgments

The consultants gratefully acknowledge the expert technical advice of Jan Barendregt (Associate Professor of Epidemiological Modelling, School of Population Health, The University of Queensland), Philip Burgess (Professor, Mental Health Services Research, School of Population Health, The University of Queensland), Amanda Baxter (Project Manager, Burden of Mental Disorders Research Group, Queensland Centre for Mental Health Research) and Bill Buckingham (Technical Advisor to Mental Health Reform Branch, Department of Health and Ageing). Top of page

References

Australian Bureau of Statistics. 2007 National Survey of Mental Health and Wellbeing: Users' Guide (Cat. No. 4327.0). Canberra: Australian Bureau of Statistics, 2009.

Australian Institute of Health and Welfare. Mental health services in Australia 2007–08. Mental health series no. 12. Cat. no. HSE 88. Canberra: AIHW, 2010.

Barendregt JJ. Ersatz. 1.1 ed. Brisbane: EpiGear International (www.epigear.com); 2010.

Begg S, Vos T, Barker B, Stevenson C, Stanley L, Lopez AD. The burden of disease and injury in Australia 2003. PHE 82. Canberra: AIHW 2007.

Briggs A, Sculpher M, Claxton K. Decision Modelling for Health Economic Evaluation. Oxford: Oxford University Press 2006.

Gelman A, Carlin JB, Stern HS, Rubin DB. Bayesian data analysis. 2nd ed. Boca Raton: Chapman & Hall/CRC 2004.

New South Wales Department of Health. Mental Health Clinical Care and Prevention Model: A Population Mental Health Model (MH-CCP Version 1.11). Sydney: New South Wales Department of Health, 2001. PDF available on the NSW Ministry of Health website (www.health.nsw.gov.au)