The eHealth Readiness of Australia's Allied Health Sector - Final Report

Appendix 2: Research Methodology

Page last updated: 30 May 2011

Summary of overall approach

In forming our perspectives on the eHealth readiness of the allied health sector, we utilised both quantitative and qualitative primary research sources, supplemented by secondary research as appropriate.

By way of overview, our approach followed a four step process:

Step One: Define macro segmentation

We adopted an initial segmentation based on profession, geography and key demographic lenses for the purposes of conducting our primary research, as follows:

  1. Professions (75 respondents required for each)
    • Dental allied health practitioners
    • Dietitians
    • Exercise physiologists
    • Occupational therapists
    • Osteopaths
    • Optometrists
    • Physiotherapists
    • Podiatrists
    • Psychologists
    • Radiographers and sonographers
    • Social workers
    • Speech pathologists
  1. Geography (minimum of 75 respondents required for each)
    • Major city
    • Inner regional
    • Outer regional
    • Remote and very remote
  1. Other lenses (Soft targets for each based on population distributions)
    • Share of public and private income
    • Practitioner age and gender
    • Proportion of patients aged over 65 years
    • Practitioner State or Territory

Step Two: Develop hypotheses and survey:

An effective survey requires both a clear understanding of hypotheses, as well as a set of questions that address – and can support or disprove - these hypotheses. To inform our hypotheses, we conducted an extensive scan of international examples of eHealth initiatives and stakeholder challenges encountered. We then built issue trees to ensure we had a complete landscape of potential areas to test, and enable us to then focus on those we felt were of critical importance. Further detail on these issue trees is contained in Exhibits 34-36.

Exhibit 34 – Hypothesis development


Exhibit 34 – Hypothesis developmentD

Exhibit 35 – Hypothesis development


Exhibit 35 – Hypothesis developmentD

Exhibit 36 – Hypothesis development


Exhibit 36 – Hypothesis developmentD

These initial hypotheses were then translated into survey questions, and tested in approximately 15 allied health practitioner interviews. These interviews served the dual purpose of both refining the initial hypotheses and ensuring the survey questions were interpreted by the audience as intended.

Step Three: Finalise and conduct survey

In conducting the survey, we used a combination of approaches to ensure sample biases were avoided, and the results for each profession type would be representative of the each profession. Further detail on our statistical approach is found in the conducting the allied health survey section below.

Step Four: Analyse results and test conclusions

The insights and recommendations are being developed from analysis of the raw survey data (a very preliminary work in progress draft is included in this interim report), and will be refined with and enriched with an additional approximately 25 deep-structured interviews with practitioners once the survey has closed. The analysis and interpretation of the research results will also be syndicated and refined via engagement of key stakeholders (including DoHA and NeHTA decision makers and clinical leads with a direct interest in the survey results), either individually or in workshops. This will allow us to probe and refine initial conclusions, and define and test potential engagement strategy options given those conclusions. Finally, a literature search of available secondary data will be used to support or challenge hypotheses and assumptions arising from the research.

Below is a more detailed description of the statistical methodology used in conducting the survey.

Primary and secondary data sources

To ensure this report serves as an independent assessment of the eHealth readiness of allied health practitioners, we employed a quantitative and qualitative survey of professionals in the industry as primary data source for this report (as opposed to relying on anecdotal evidence or literature reviews).

The primary data item is an assessment of the overall readiness of Australian allied health to adopt and use eHealth technologies and solutions, including the primary drivers of, and barriers to, adoption.

Our primary research approach was informed by the experience of several national peak bodies, special interest groups and other organisations with a professional interest in the content of this report. Where appropriate, the input of these bodies was considered in the formation of this report, with an understanding of the experience each organization or group brought to bear in providing their input.

Secondary data sources for this research include background data from other institutions including but not limited to Australian Bureau of Statistics, Australian Health Workforce Advisory Committee, Australian Institute of Health and Welfare, Department of Health and Ageing, Medicare Australia, National eHealth Transition Authority, National Health and Hospitals Reform Commission.

Key questions of Research

The research aims to address the following key questions:

  1. Are Australian allied health practitioners ready to adopt and use eHealth technologies and solutions, both today and in a way consistent with future policy direction?
  2. What are the barriers impacting eHealth readiness and adoption and how can we minimise them?
  3. What are the eHealth enablers and how can we apply them to drive adoption and effective usage?

Conducting the allied health survey

The survey target population is the entire Australian allied health workforce. According to the ABS Labour Force Survey 2008, this comprises approximately 127,200 individuals within the 15 practitioner specialties: Aboriginal and Torres Strait Islander health workers, Audiologists, Chiropractors, Dental health professionals, Dietetics, Exercise Physiologists, Occupational therapists, Optometrists, Osteopaths Physiotherapists, Podiatrists, Psychologists, Radiographers, Social workers, and Speech pathologists.

The allied health survey frame aims for as complete a coverage of the target population as possible. Hence we will use a variety of sources to obtain a large and representative frame, including:
  • Directories (e.g. business and specialist directories)
  • Professional organizations and associations, peak national bodies
  • Purchased practitioner lists

The survey involved creating a list of possible respondents with the primary information as practitioner name, specialty, gender and contact details. Typically each practitioner will also have a postcode, although many work across multiple postcodes, with their appropriate region determined during the survey.

The frame aims for as complete a coverage of the target population as possible. Based on the contracted survey provider’s past experience, there are expected to be fewer than 5% of the respondents out of business, and fewer than 5% will be out of scope.

Approximately 10% of the total population are expected to have previously opted out of participating in market research. However, the support of the allied health and various sector peak national bodies is expected to reduce this incidence.

The primary limitations of sampling will be (i) practitioners who do not have their contact details listed through directories or peak national bodies, and (ii) practitioners who specifically opt out of survey participation.

There are no efficient actions to fully counteract (i) although their prevalence is considered low for allied health practitioners. Typically peak national bodies comprise 70-100% of the practitioner workforce, with many specialties requiring registration. Independent research by several allied health peak national bodies has indicated that their membership base is highly representative of the broader population. Improving (ii) relies on bolstering survey legitimacy and personal engagement, for which we have engaged the peak national allied health bodies.

To ensure participation of the allied health sector, McKinsey entered into an agreement with AHPA on 31 March 2011 for the duration of the contract for AHPA to provide an Independent Expert Reference Group to:
  • Provide input on survey interpretation, insight generation and guidance on approach at times requested by McKinsey
  • Provision of each profession’s support of the survey by encouraging member participation with the express objective of increasing response rates to the survey and accuracy of response
  • Provision of currently held demographic information for each profession plus a statement containing contextual aspects for that profession, and facilitate access to a structured sample of members and enabling linking to any survey proposed by McKinsey.

Determining the appropriate sample size

As a methodological note, the statistic error (i.e., generalising from the survey results to the whole population of allied health practitioners – 127,200) of a representative sample (approximately 1000) is +/- 3.1 % at the 95% level of confidence. This means that if 50% of professionals in the sample agree with a particular proposition, it can be assumed with 95% confidence that, had the whole population of professionals been interviewed, between 46.9% - 53.1% would also have agreed with the proposition at the time of the survey.

When the survey results are broken down into subgroups of allied health practitioners (e.g. audiologists), the error of estimation will be higher for smaller sample sizes, in the order of 1 divided by the square root of the sample size. Table 7 below shows the error of estimation at a 95% confidence interval based on specific sample sizes:

Table 7: Error of estimation at 95% confidence interval

Total population size
Sample size
100
200
400
800
1000
1500
1500+
25
17.1
18.4
19.0
19.3
19.4
19.4
19.6
50
9.8
12.0
13.0
13.4
13.5
13.6
13.9
75
5.7
9.0
10.2
10.8
10.9
11.0
11.3
100
0.0
6.9
8.5
9.2
9.3
9.5
9.8
125
5.4
7.3
8.1
8.2
8.4
8.8
150
4.0
6.3
7.2
7.4
7.6
8.0
175
2.6
5.6
6.6
6.2
6.5
7.4
200
0.0
4.9
6.0
6.2
6.5
6.9
250
3.8
5.1
5.4
5.7
6.2
300
2.8
4.5
4.7
5.1
5.7
400
0.0
3.5
3.8
4.2
4.9
500
2.7
3.1
3.6
4.4
600
2.0
2.5
3.1
4.0
700
1.3
2.0
2.7
3.7
800
0.0
1.6
2.4
3.5
900
1.0
2.1
3.3
1000
0.0
1.8
3.1
As Table 7 illustrates, the maximum error of estimation when comparing any two allied health practitioners using a sample size of 75 for each profession is 11.3%. What this means is that regardless of whether a population of a given profession is 500, 5,000 or 50,000, the maximum error of estimation is approximately 11.3%.

To help illustrate the implication of this approach and to interpret the above table, two short case examples are helpful.

Case example 1: Single practitioner group confidence

To determine error for a single population, we read the relevant cell in the above table for both population and sample. For example if Profession A had a population of 400, a sample size of 75, and a score of 50% eHealth ready, the result would show 95% confidence that 39.8% - 60.2% (i.e., 50% +/- 10.2%) of the population is eHealth ready.

Case example 2: Comparing two practitioner groups using confidence

Assume Profession A had a population size of 400, and Profession B had a population size of 5,000. Assume also that the objective is to determine whether the mean eHealth readiness of each profession is statistically different, with 40% of Profession A responding they are eHealth ready and 65% of Profession B responding they are eHealth ready. Assume a sample size of 75 across each profession.

Using the above table and given the assumptions described, we can see that a sample of 75 of the 400 Profession A results in an error of estimation of 10.2%. A sample of 75 of the 5,000 Profession B (i.e., 1500+) results in an error of estimation of 11.3% - the theoretical maximum error of estimation when surveying 75 professionals from a very large population. So, what we can say is that it can be assumed with 95% confidence that, had the whole population of:
  • Profession A been interviewed, between 29.8% - 50.2% (i.e., 40% +/- 10.2%) would be eHealth ready
  • Profession B been interviewed, between 53.7% - 76.3% (i.e., 65% +/- 11.3%) would be eHealth ready.

Given these two ranges do not overlap, we would conclude that there was a statistically significant difference between these two professions.

For this survey, we selected a sample size of 75 responses for each of the 15 categories of allied health practitioner. The selection of 75 responses represents a balance between a desire to minimise the error of estimation, the likely variance between each allied health profession, and the financial resources available by the Department for this work. While surveying 1,000 allied health practitioners in each profession would have reduced the error of estimation from 11.3% to 3.1%, this would have resulted in an inefficient use of resources given the specificity required for the hypotheses in this project and been impractical given the relative size of some of these professions and anticipated yield rates.

This constraint is meaningful. By way of illustration, there are approximately 22 000+ physiotherapists in Australia. Even if responses had been in the order of 150 for this group (approximately 0.7% response rate), we can see from the above table the error of estimation would still have been in the order of +/- 8 %.

Therefore, although the general maxim of ‘bigger is better’ for surveys such as the one in this project is true, we feel confident that 75 allied health practitioners per profession provides an acceptable level of sampling error to identify outlier professions. This is particularly so given the focus of this effort is to identify directional trends (rather than a precise point estimate).

Caveat: Appropriate use of data

Given the nature of the underlying hypotheses, and the desire to identify outliers, the primary research approach was calibrated to an acceptable level of residual error as described above (maximum of 11.3%, depending on the type of analysis being undertaken). The output of the research identified directional differences between clusters and specialties. The nature of this approach means that future research studies cannot be directly compared to the outcome of the primary research in this report without replicating the research methodology.

Sampling, biases and response rate

Research based on sampling from a population necessarily inherits a level of uncertainty in drawing conclusions from incomplete data. Statistical analyses are based on the assumption of unbiased responses – therefore maximising the usefulness of a survey dataset requires the survey design to address potential biases that may be introduced through sampling method. The two key potential biases for many surveys are selection biases (the possible exclusion of a relevant strata, e.g., physiotherapists from the Northern Territory) and non-response biases (e.g., the data collection requirement is so onerous that only a few practitioners complete the survey). Both can be countered through careful sampling design.

We applied several survey techniques to control for selection biases in this survey. For example, we:
  • Collected approximately 75 respondents across per category selected with representation across Australia’s regional breakdown
  • Ensured representativeness of the sample by replicating the demographic profile of each allied health profession in the sample

To ensure all strata of interest were adequately represented, the allied health survey used a simple stratified random sample. We stratified invitations by two primary segmentations - profession and geographic location. The survey used a single-phase sample of 75 respondents per specialty strata, as the balance point between required accuracy and survey expense. Further, we required a minimum representation of 75 respondents per location type strata. For private/public practice, gender, state/territory, and age, we set soft quotas based on population distribution statistics from the Australian Institute of Health and Welfare, the ABS 2006 Census and registration data for those allied health professions requiring registration.

The survey was designed to collect a sufficiently detailed data set without over-imposing on practitioner time. For example, we:
  • Allowed survey participants to respond to the survey by either completing an online form or undertaking a telephone survey
  • Ensured surveys were 15 minutes in length to minimise imposition on respondents

Additionally, we used two approaches to address potential non-response bias. Firstly, we engaged the AHPA to reach out to practitioner members in support of the survey, for those professions where their membership is comprehensive. In these cases (chiropractors, dental allied health professionals, dietitians, exercise physiologists, osteopaths, physiotherapists, podiatrists and psychologists) we designed a specific target population (of 5-600 practitioners) from de-identified membership lists. Secondly, where peak body membership was either unavailable or less than comprehensive (Aboriginal and Torres Strait Islander health workers, audiologists, occupational therapists, optometrists, radiographers and sonographers, social workers and speech pathologists) our survey vendor, Sexton Market Research, compiled a comprehensive list of allied health practitioners from business directories, professional associations and lists available for purchase. Again, a target population was designed and contacted directly by phone.

In total, the quantitative survey contacted approximately 6,500 allied health practitioners in order to return 1,125 responses, for a response rate of 17.3%.

Data weighting methodology

Any survey relying on a limited sample size may cover segments of the target population in proportions that do not match their proportions in the population itself. In this case clear differences arise from our deliberate sample design, intended to capture the full spectrum of perspectives across a diverse population. In such situations we can improve the relation between the sample and the population by weighting sample responses such that each response is attributed its appropriate level of emphasis as determined by the overall population distribution. This operation is known as sample-balancing or raking, and the population totals are usually referred to as control totals.

The adjustment to control totals is sometimes achieved by creating a cross-classification of the control variables (e.g. age categories x gender x remoteness x % private income) and then matching the total of the weights in each cell to the control totals. This approach, however, can spread the sample thinly over a large number of cells. It also requires that the control totals exist for all cells of the cross-classification. Often this is not feasible (e.g. control totals may be available by age x gender x remoteness but not when those cells are subdivided by income source). The use of marginal control totals for single variables (i.e., each margin involves only one control variable) often avoids many of these difficulties. In return, of course, the two-variable (and higher order) distributions of the sample are not required to mimic those of the population.

The procedure known as raking adjusts a set of data so that its marginal totals match specified control totals on a specified set of variables. (The term “raking” is employed in analogy with the process of smoothing the soil in a garden plot by alternately working it back and forth with a rake in two perpendicular directions.) In a simple 2-variable example the marginal totals in various categories for the two variables are known from the entire population, but the joint distribution of the two variables is known only from a sample (such as a 5% sample). In the cross-classification of the sample, arranged in rows and columns, one might begin with the rows, taking each row in turn and multiplying each entry in the row by the ratio of the population total to the sample total for that category, so that the row totals of the adjusted data agree with the population totals for that variable.

The column totals of the adjusted data, however, may not yet agree with the population totals for the column variable. Thus the next step, taking each column in turn, multiplies each entry in the column by the ratio of the population total to the current total for that category. Now the column totals of the adjusted data agree with the population totals for that variable, but the new row totals may no longer match the corresponding population totals. The process continues, alternating between the rows and the columns, and agreement on both rows and columns is usually achieved after a few iterations. The result is a tabulation for the population that reflects the relation of the two variables in the sample.

Case example 3: Iterative rim weighting

To determine the appropriate weighting for a simplified 2-variable case, we first find the sum of the sample size across each dimension (numbers for illustration purposes only):
StateATSIH workerAudiologistChiropractorDentistDietitianSum of sample size
NSW30012006030301620
Vic15010809030301380
Sum of sample size450228015060603000

We then compare the sample size with the known distribution in the population. To develop weights for each of the categories required (e.g. Audiologists in Victoria), we alternately rake by each dimension, until sample weights converge to the actual population size.

For example, commencing with the population by state:
StateSum of sample sizeKnown populationRake factor
NSW162015101510/1620
Vic138014901490/1380

Multiplying each row by its rake factor gives the correct distribution by state, but fails to account for known population by profession:
StateATSIH workerAudiologistChiropractorDentistDietitianSum of weights
NSW279.631118.5255.9327.9627.961620
Vic161.961166.0997.1732.3932.391380
Sum of weights441.592284.61153.1060.3560.353000

Alternately raking by profession, we determine rake factors across columns using the same method:
ATSIH workerAudiologistChiropractorDentistDietitian
Sum of weights441.592284.61153.1060.3560.35
Known population600212015010030
Rake factor600/441.592120/2284.61150/153.10100/60.3530/60.35

Now, multiplying each column by its rake factor gives the correct distribution by profession, but unbalances the distribution by state:
StateATSIH workerAudiologistChiropractorDentistDietitianSum of weights
NSW379.941037.9354.7946.3313.901532.90
Vic220.061082.0795.2153.6716.101467.10
Sum of weights6002120150100303000

Iterating the raking process until convergence gives a series of weights to be allocated to each population category (e.g. Audiologists in Victoria), such that their emphasis within the survey analysis matches their representation in the population as a whole.

Weighting variables used

As described, the allied health sample was iteratively weighted across four variables:

1. Practitioner type

Practitioner typeSample distributionPopulation distribution
Aboriginal Health Worker7%3%
Audiologist7%2%
Chiropractor7%3%
Dentist7%13%
Dietitian7%2%
Exercise physiologist7%1%
Occupational therapist7%5%
Optometrist7%3%
Osteopath7%1%
Physiotherapist7%16%
Podiatrist7%2%
Psychologist7%20%
Radiographer/Sonographer7%12%
Social worker7%13%
Speech pathologist7%5%

2. State

StateSample distributionPopulation distribution
NSW28%33%
VIC27%25%
QLD19%19%
WA9%11%
SA9%7%
TAS3%2%
NT1%1%
ACT4%2%

3. Region

RegionSample distributionPopulation distribution
Major city62%77%
Inner regional22%16%
Outer regional12%6%
Remote4%1%

4. Gender

GenderSample distributionPopulation distribution
Male41%33%
Female59%67%

Clustering analysis

Purpose

We used cluster analysis to group practitioners with similar needs and attitudes. This allowed us to develop tailored interventions based on each cluster’s distinct motivations.

Through this process, we sought to define clusters that are:
  1. Reachable – i.e., we can target specific interventions to each group
  2. Interpretable – i.e., we can understand who they are and what matters most to each
  3. Distinct – i.e., different from each other on key attitudes and behaviours.

We can easily define reachable clusters based on demographic criteria. However, by adhering to a strictly demographic approach, we lose the ability to incorporate the needs-based insights required to develop targeted interventions. In contrast, while a needs-only approach provides rich and robust insights on attitudes and behaviours, it falls short on the reachability dimension. Therefore, we applied a hybrid “needs-plus” clustering approach, which combines needs-based and demographic dimensions so that we can identify targeted interventions that can be effectively delivered to the relevant practitioners: Exhibit 37.

Exhibit 37 – Needs-plus clustering


Exhibit 37 – Needs-plus clusteringD

We need strong variation in at least some attributes or variables to define groups that are distinct. A ‘needs plus’ clustering approach enables us to identify much greater variation in responses, especially when compared with basic demographic measures such as age, gender and region.

Process to develop ‘needs-plus’ clusters

The ‘needs plus’ approach follows a two-step process, described below.

Step 1: Select variables and create clusters based on needs dimensions:

]Identify attitude “themes” – We used a factor analysis across all eHealth attitudinal variables to identify themes across attitudes Identify & retain relevant “themes” – We identified a set of themes (approximately 7-8) that were the most representative of eHealth attitudinal dimensions Select final list of variables – We identified 1 or 2 variables that were strongly related to each theme and that also exhibited sufficient variability in the sample Conduct 1st stage segmentation – We used hierarchical clustering (Ward’s method[9) to fold similar respondents into increasingly larger groups – this approach yielded an initial “coarse” solution to assess
  • Select ideal cluster solution – We ran multiple iterations of the hierarchical clustering analysis and identified the cluster solution that offered the most effective target groups

Step 2: Refine needs-based clustering by adding behavioural and profile variables to create ‘needs plus’ clusters:

  • Improve needs-based cluster solution – We used k-means clustering on existing hierarchically derived clusters to improve on our solution by adding % private as a variable to enhance reachability
  • Profile the final solution – We profiled the final solution on all variables to make a comprehensive, final assessment of its quality, relative to the segmentation objectives
  • Refine and ‘bring to life’ the profiling description – We used deep-structured interviews and stakeholder discussions and workshops to test and refine the profiles and ensure that they were both insightful and actionable (i.e., helpful to define relevant intervention strategies, meaningful and reachable).

Compliance and regulation

The research survey and all related materials were reviewed and approved by the Australian Government Statistical Clearing House. The approval number is 02172-01.

9 For more detail, refer to Joe H Ward’s “Hierarchical Grouping to Optimise an Objective Function”, Journal of the American Statistical Association, Volume 58, Issue 301 (Mar 1963), 236-244[


Top of page