Evaluation of the consumer - directed care initiative - Final Report

3.6 Challenges and limitations

Up to Consumer Directed Care

prev page| TOC |next page

There were a number of limitations to the evaluation and challenges in conducting the evaluation:

• Timing of the evaluation, bearing in mind the length of time the initiative had been in operation when the evaluation was being conducted
• The extent to which the evaluation could attribute change or improvement for participants (and carers of participants) to the initiative as distinct from change or improvement as a result of the receipt of care and support
• Utilising data collection methods appropriate for older people and their carers within the scope, timeframes and resources available for the evaluation
• Identifying the perspectives and experiences of participants in some special needs groups (as identified in the Aged Care Act), given their low representation in the CDC and CDRC participant groups.

Timing of the evaluation and evaluation data collection

The evaluation was undertaken less than 12 months after implementation of the CDC initiative.

The operationalisation of the initiative by providers was still evolving during this period, and providers were learning from their own experiences as well as the experiences of other providers to further develop their approach. A number of providers also experienced some delays in allocating packages to participants.

For many participants, their involvement in and experience of CDC and CDRC was limited and influenced by providers’ evolving approach. Given most participants had been involved in the initiative for less than six months and a number less than three months at the time of the evaluation data collection activities, the benefits or impacts for participants from the initiative were unlikely to have been fully evident.

As a consequence of this, the evaluation and associated data collection placed more emphasis on the ‘process’ components of the evaluation – that is, how the CDC initiative had been implemented and operationalised, and the nature of participants’ involvement in CDC – rather than on the outcomes of the initiative.

Attributing change or improvement to the initiative

One of the main methodological issues for this evaluation relates to the extent to which the evaluation could attribute change or improvement for participants to the initiative itself as distinct from change or improvement as a result of the receipt of care and support (or more care and support than they received previously). This is particularly relevant for participant outcomes, but also for their experiences and perspectives of the CDC initiative.

This is particularly relevant for CDRC participants, the majority of whom were not accessing supports or were utilising only low-level supports prior to commencing their CDC package. It is also relevant for CDC participants who had not received standard packaged care (CACP, EACH or EACHD) prior to CDC.

The evaluation approach attempted to explore this through participant interviews, as well as by undertaking comparisons of participant survey responses on three levels:

1. Comparing participants’ perspectives, experiences and outcomes of CDC/CDRC with the perspectives, experiences and outcomes of a similar group of care recipients and carers receiving standard, non-CDC/CDRC packages (the comparison group).
2. Comparing participants’ perspectives, experiences and outcomes at points in time when they were in receipt of a CDC/CDRC package compared with six months prior (when the majority did not have a CDC/CDRC package).
3. Comparing the perspectives, experiences and outcomes of CDC participants who had received a standard package prior to commencing a CDC package with those participants who did not receive a standard package⁶.Top of page

Data collection methods

The ‘burden’ of data collection on the CDC/CDRC participant groups and comparison groups was considered during the design phase of the evaluation, bearing in mind the fact that this group is older and frail. As noted above, the evaluation encompassed two different participant data collection methods: a paper-based survey and participant interviews.

The participant surveys – which were able to be completed by respondents in their own time – allowed the same data items to be collected from a large group of participants. However, detailed data was not able to be collected, and some groups were less likely to complete surveys than others (for example, people from an Aboriginal or Torres Strait Islander background, people who were homeless or at risk of homelessness, people living with dementia or cognitive impairment). The survey tools were designed to be simple, easy to read and understand, and of an appropriate length to gather the required data while minimising the burden on the respondent from completing the tool.

A number of interviews were also conducted, which allowed a much richer set of qualitative data to be collected (albeit from a small group of people). Conducting interviews also meant that those groups under-represented in the survey responses could be specifically targeted. The interviews were conducted in a semi-formal and semi-conversational way, using set questions as a guide.

Identifying the experiences and impacts for people in special needs groups

One of the challenges faced in conducting the evaluation was collecting sufficient data (through surveys or interviews) from participants from specific special needs groups (as defined in the Aged Care Act) to identify their perspectives and experiences of CDC and CDRC.

There were very small numbers of participants in the following special needs groups, which meant that no findings could be made in relation to these groups: people from an Aboriginal or Torres Strait Islander background, care-leavers, veterans and to some extent those who were previously homeless or at risk of being homeless.

While not a special needs group, it was also difficult to collect data from people living with dementia separately from carers of people living with dementia.

---

6. This comparison was not undertaken for CDRC participants given only a small number of participants were receiving NRCP services prior to commencing a CDRC package, and numbers were too small to do any meaningful analysis.

Top of page

prev page| TOC |next page