Evaluation of the Bringing them home and Indigenous mental health programs

2. Methodology

Up to Bringing Them Home

prev page| TOC |next page

This chapter outlines the components of the methodology used for the evaluation. These components included:

• fieldwork
• key informant telephone interviews
• written submissions
• a survey in relation to the Mental Health Service Delivery Projects
• literature review
• program data
• a communications strategy

Also outlined are issues for future evaluations.

2.1 Fieldwork

Two rounds of fieldwork were conducted in 15 locations in six States and Territories (see table 2.1). These rounds of fieldwork were conducted during April-June and September-December 2006 respectively. During the first round of fieldwork the consultants also conducted consultations at three national forums convened and/or funded by OATSIH: Sydney – Link-Up and BTH Counsellors, Brisbane – SEWB RC Forum, and the Adelaide – National Sorry Day Committee (NSDC).

In some locations, staff and stakeholders from outside the nominated consultation location travelled some distance to participate in the consultations.

Initially one round of fieldwork was planned, including consultation with clients of the programs. However early in this process, a decision was made by OATSIH to seek ethics approval to incorporate client consultations into the evaluation. Therefore the first round of fieldwork proceeded without seeking input from clients (other than in two locations already visited prior to this decision ie Adelaide and Brisbane). An ethics application was then submitted to the DoHA Ethics Committee to allow consultations in a second round of fieldwork with client and Stolen Generations members (both clients and non-clients of the programs, to provide a source of clients outside of the services themselves). Approval for the application was granted subject to various conditions, for example, to ensure that participation in field visits was voluntary and undertaken with informed consent, and participants had access to support afterwards if required (primarily a BTH Counsellor).Top of page

The fieldwork locations were selected in consultation with the Reference Group for the evaluation and (for the second round of fieldwork) the NSDC.

The fieldwork was organised by the consultants with the assistance of the Link-Up and BTH services, and in some instances State OATSIH offices and local Stolen Generations groups.

Consultations were conducted with the following categories of stakeholders:

• staff of the services – in all locations
• external stakeholders (Aboriginal and non-Aboriginal) – in all locations
• clients of the services – a total of 49 clients in Mt Druitt (Sydney), Brisbane, Albany, Adelaide, Melbourne and Darwin
• Stolen Generations members, including both clients and non-clients of the services – a total of 40 in Darwin, Albany, Kununurra, Melbourne and Redfern (Sydney). In addition consultations were conducted with the NSDC as part of the Adelaide fieldwork.

In practice the number of clients and Stolen Generations members consulted is somewhat higher than indicated above, since some people interviewed as staff and external stakeholders were also Stolen Generations members and/or had previously been clients of the programs.

Most of the consultations were conducted by way of small group discussions. Separate groups were conducted with Aboriginal and non-Aboriginal stakeholders where this was regarded as preferable by the services assisting with fieldwork organisation.

All field visits were conducted jointly by an Aboriginal and non-Aboriginal team member. In Melbourne, Adelaide, Brisbane and Darwin one of the Aboriginal team members conducted some of the consultations alone. The Aboriginal team members also provided assistance and advice in relation to organisation of the fieldwork.Top of page

Detailed question guides were developed for the consultations with each of the above stakeholder groups and approved by OATSIH.

Small financial allowances (of $30 each) were paid to all clients and Stolen Generations members who were consulted on the fieldwork. The purpose of these allowances was to encourage voluntary participation and cover any associated costs such as travel and childcare expenses.

The staff and stakeholders consulted, and the numbers of clients and Stolen Generations members consulted, are provided by location in appendix A.

Overall, in the consultants' view, the fieldwork locations provided representation across an informative mix of metropolitan and regional locations with a range of different service delivery issues, and good national coverage (particularly in combination with the telephone interviews). The number of locations visited is larger than would typically be the case for a national program evaluation.

Some challenges were experienced in organising the fieldwork. There was less than optimal cooperation by a number of the funded services (and in some instances the State OATSIH offices) in assisting with this task. Some of the reasons for this appeared to include:

• services having scarce resources to conduct activities relating to evaluation
• services not regarding evaluation activities as a major priority of benefit to them (an issue discussed further in chapter 7)
• disputes or lack of clarity within the service about whose responsibility it was to assist with fieldwork organisation.

A four-day field visit to Hobart and Launceston in Tasmania had also been planned in the second round of fieldwork. However, this did not occur due to non-engagement by the service in those locations. (This was also the case with the prior ATSIS evaluation: ATSIS 2003b, p1). Additional attempts were made to interview the BTH service and other Aboriginal organisations in Launceston and elsewhere in Tasmania by telephone instead, but all the agencies declined to participate. Telephone interviews with some State agencies were organised but fell through. Therefore, in Tasmania it was only possible to consult with OATSIH. Top of page

Another challenge for organisation and conducting the fieldwork was that, although concerted attempts were made to consult with clients and Stolen Generations members in all of the locations for the second round of fieldwork, this was not always possible for a range of reasons. These include:

• In locations without official Stolen Generations groups, Stolen Generations members could not be recruited. Even in some locations with these groups, the groups were not available to assist with fieldwork organisation due to limited resources.
• In some locations services reported that it was difficult to recruit clients for the consultations due to:
  • discomfort about participating in an ‘evaluation’ (even when the process and purpose of this was explained to them)
  • only being interested in talking about their experiences as Stolen Generations members and not service delivery issues
  • not wanting to discuss their experiences as Stolen Generations members (even when advised that this would not be required and the matter of interest was the efficiency and effectiveness of the service delivery programs)
• In a small number of locations consultations with clients and/or Stolen Generations members had been arranged but no-one turned up (in some locations it was known that this was due to a funeral or other competing priorities on the day).

Nonetheless, in the consultants' view, the number of clients and Stolen Generations members consulted, and the issues raised by them, provided a reasonable level of coverage of the views and key priorities of these groups in relation to the delivery of the four programs. It should be noted that this is the first national evaluation of Indigenous health programs by OATSIH to include direct consultations with clients. It is also the first time that the Australian Government has directly consulted with Stolen Generations members and organisations about the BTH Programs. Participation was actively encouraged by OATSIH and the NSDC.Top of page

Table 2.1: Locations Visited in Rounds 1 and 2 of Fieldwork, by State/Territory

State/territory	Locations visited in round 1 of fieldwork	Locations visited in round 2 of fieldwork
NSW	Redfern and Mt Druitt, Sydney Taree National Link-Up Forum, Sydney (28 March)	Mt Druitt, Sydney
Queensland	Brisbane National SEWB RC Forum (10 May)	Rockhampton
WA	Perth Broome	Albany Kununurra
NT	Darwin Alice Springs Katherine	Darwin
SA	Adelaide NSDC Forum, Adelaide (28 April)	Port Augusta
Victoria	Melbourne	Melbourne Shepparton

Top of page

2.2 Key informant telephone interviews

A total of 33 telephone interviews were conducted with key informants around Australia to supplement the fieldwork.

The informants interviewed by telephone were identified through various means, including:

• identification of key stakeholders by OATSIH, the Reference Group, the NSDC and its Chairperson
• suggestions by other key informants interviewed
• people identified by other stakeholders in locations visited who were unavailable to participate in the consultations.

A key priority for the telephone consultations was stakeholders located in places which were not visited for the fieldwork.

As with the fieldwork, a detailed question guide was developed for the telephone interviews and approved by OATSIH.

2.3 Written submissions

Stakeholders were able to provide written submissions to the consultants directly or via the website established for the evaluation (see below). Stakeholders were invited to provide written submissions through a number of approaches: the Communications Strategy (see below); informing the staff and stakeholders consulted on the fieldwork of this option; and a letter written directly to key national and State record-keeping/searching agencies by OATSIH in November 2006. A total of 19 submissions were received, including one from a client of one of the programs. Most of the submissions were from record-keeping/searching agencies. A list of those who provided submissions is provided in appendix A.

2.4 Survey in relation to Mental Health Service Delivery Projects

OATSIH distributed a short pro-forma to its State offices to gather information about the Mental Health Service Delivery Projects. A response was received in relation to 11 of the 19 projects.Top of page

2.5 Literature review

The aim of the literature review was to identify current and emerging issues, policies and approaches to meeting the mental health and SEWB needs of Indigenous peoples who have been affected by dispossession, forced removal from families and associated grief, trauma and loss. This was to include identifying best practice models and possible alternative service delivery models for consideration.

The literature review sought to identify:

• current thinking about best practice strategies for meeting the SEWB needs of Stolen Generations groups, both within Australia and internationally – including aspects ranging from counselling approaches to organisational and locational arrangements
• examples of good practice in meeting those needs
• the current situation regarding Stolen Generations people in Australia and any future trends.

Material produced within the last five years (1999-2004) was collected, as well as some seminal references from before this time. The primary emphasis was on Australian material, but a small amount of overseas literature was also collected where it was directly relevant to the Australian context.

Various sources were used to identify resources for the literature review:

• searches of a range of Australian and overseas literature databases concerning health, psychology and Indigenous issues
• internet searches
• sources identified by OATSIH, the Reference Group for the evaluation and staff and external stakeholders consulted on the fieldwork.

Some limitations of the literature review are discussed in appendix B.Top of page

2.6 Program data

The consultants analysed program data for the Link-Up and BTH services provided by OATSIH. These data were provided by the services to OATSIH as part of their annual reporting requirements. The Link-Up data covers the period 1998-1999 through to 2005-2006, and comes from the Foxtrot database used by the Link-Up services. The BTH data covers the period 2001-2002 through to 2004-2005, and comes from the annual BTH Questionnaire completed by ACCHSs funded to employ counsellors.

There are reportedly a range of issues concerning the reliability of both the Link-Up and BTH data. These issues are discussed in more detail in chapter 5.

2.7 Communications strategy

A Communications Strategy was developed and implemented by specialist Aboriginal communications firm Gavin Jones Communications (GJC) in consultation with the evaluation team and OATSIH. The aim of the Strategy was to inform key stakeholders about the research and how they could contribute to it, either via participating in the fieldwork or making a written submission.

The activities conducted for Part A of the Communication Strategy (at the beginning of the evaluation) are set out in table 2.2 below. These activities were conducted in March and April 2006.

A website was also established for the evaluation: bringing them home.

Part B of the Communications Strategy will be conducted after submission of this report to DoHA to publicise the findings of the evaluation. This will include a similar range of activities to the above. The website will include a short community summary report of the key findings of the evaluation, and will be transferred to OATSIH for ongoing management and updating of any key developments. Top of page

Table 2.2: Activities for Part A of the Communications Strategy

Title	Topic
Koori Mail	Article on evaluation
National Indigenous Times	Link Up Video release
Deadly Vibe	Issues you should consider if planning to meet your natural family
National Health Workers Journal	25 years of Link-Up
Various stations	Community Service Announcement
Deadly Sounds	Interview questions for nominated representative
eVibe/eLetter	Article on evaluation

2.8 Issues for future evaluations

There were some lessons for this evaluation which may be useful to consider for the purpose of future evaluations of the programs. In light of the challenges experienced in organising the fieldwork, it would be beneficial for:

• the State OATSIH offices to take a more proactive role in assisting with fieldwork organisation
• the national OATSIH office to encourage the services to value and prioritise evaluation activities to a greater degree (some recommendations to this report in Chapter 11 address this issue)
• the funded services to be offered small financial allowances for every client recruited who turns up for the consultations (say $30), to compensate in part for their time
• a fairly long lead time (at least around six weeks) be allowed for organisation of the fieldwork, given that it may be difficult to contact and/or engage services

prev page| TOC |next page