Evaluation of the Bringing them home and Indigenous mental health programs
This chapter outlines the components of the methodology used for the evaluation. These components included:
- key informant telephone interviews
- written submissions
- a survey in relation to the Mental Health Service Delivery Projects
- literature review
- program data
- a communications strategy
2.1 FieldworkTwo rounds of fieldwork were conducted in 15 locations in six States and Territories (see table 2.1). These rounds of fieldwork were conducted during April-June and September-December 2006 respectively. During the first round of fieldwork the consultants also conducted consultations at three national forums convened and/or funded by OATSIH: Sydney – Link-Up and BTH Counsellors, Brisbane – SEWB RC Forum, and the Adelaide – National Sorry Day Committee (NSDC).
In some locations, staff and stakeholders from outside the nominated consultation location travelled some distance to participate in the consultations.
Initially one round of fieldwork was planned, including consultation with clients of the programs. However early in this process, a decision was made by OATSIH to seek ethics approval to incorporate client consultations into the evaluation. Therefore the first round of fieldwork proceeded without seeking input from clients (other than in two locations already visited prior to this decision ie Adelaide and Brisbane). An ethics application was then submitted to the DoHA Ethics Committee to allow consultations in a second round of fieldwork with client and Stolen Generations members (both clients and non-clients of the programs, to provide a source of clients outside of the services themselves). Approval for the application was granted subject to various conditions, for example, to ensure that participation in field visits was voluntary and undertaken with informed consent, and participants had access to support afterwards if required (primarily a BTH Counsellor).Top of page
The fieldwork locations were selected in consultation with the Reference Group for the evaluation and (for the second round of fieldwork) the NSDC.
The fieldwork was organised by the consultants with the assistance of the Link-Up and BTH services, and in some instances State OATSIH offices and local Stolen Generations groups.
Consultations were conducted with the following categories of stakeholders:
- staff of the services – in all locations
- external stakeholders (Aboriginal and non-Aboriginal) – in all locations
- clients of the services – a total of 49 clients in Mt Druitt (Sydney), Brisbane, Albany, Adelaide, Melbourne and Darwin
- Stolen Generations members, including both clients and non-clients of the services – a total of 40 in Darwin, Albany, Kununurra, Melbourne and Redfern (Sydney). In addition consultations were conducted with the NSDC as part of the Adelaide fieldwork.
Most of the consultations were conducted by way of small group discussions. Separate groups were conducted with Aboriginal and non-Aboriginal stakeholders where this was regarded as preferable by the services assisting with fieldwork organisation.
All field visits were conducted jointly by an Aboriginal and non-Aboriginal team member. In Melbourne, Adelaide, Brisbane and Darwin one of the Aboriginal team members conducted some of the consultations alone. The Aboriginal team members also provided assistance and advice in relation to organisation of the fieldwork.Top of page
Detailed question guides were developed for the consultations with each of the above stakeholder groups and approved by OATSIH.
Small financial allowances (of $30 each) were paid to all clients and Stolen Generations members who were consulted on the fieldwork. The purpose of these allowances was to encourage voluntary participation and cover any associated costs such as travel and childcare expenses.
The staff and stakeholders consulted, and the numbers of clients and Stolen Generations members consulted, are provided by location in appendix A.
Overall, in the consultants' view, the fieldwork locations provided representation across an informative mix of metropolitan and regional locations with a range of different service delivery issues, and good national coverage (particularly in combination with the telephone interviews). The number of locations visited is larger than would typically be the case for a national program evaluation.
Some challenges were experienced in organising the fieldwork. There was less than optimal cooperation by a number of the funded services (and in some instances the State OATSIH offices) in assisting with this task. Some of the reasons for this appeared to include:
- services having scarce resources to conduct activities relating to evaluation
- services not regarding evaluation activities as a major priority of benefit to them (an issue discussed further in chapter 7)
- disputes or lack of clarity within the service about whose responsibility it was to assist with fieldwork organisation.
Another challenge for organisation and conducting the fieldwork was that, although concerted attempts were made to consult with clients and Stolen Generations members in all of the locations for the second round of fieldwork, this was not always possible for a range of reasons. These include:
- In locations without official Stolen Generations groups, Stolen Generations members could not be recruited. Even in some locations with these groups, the groups were not available to assist with fieldwork organisation due to limited resources.
- In some locations services reported that it was difficult to recruit clients for the consultations due to:
- discomfort about participating in an ‘evaluation’ (even when the process and purpose of this was explained to them)
- only being interested in talking about their experiences as Stolen Generations members and not service delivery issues
- not wanting to discuss their experiences as Stolen Generations members (even when advised that this would not be required and the matter of interest was the efficiency and effectiveness of the service delivery programs)
- In a small number of locations consultations with clients and/or Stolen Generations members had been arranged but no-one turned up (in some locations it was known that this was due to a funeral or other competing priorities on the day).
Table 2.1: Locations Visited in Rounds 1 and 2 of Fieldwork, by State/Territory
|State/territory||Locations visited in round 1 of fieldwork||Locations visited in round 2 of fieldwork|
|NSW||Redfern and Mt Druitt, Sydney |
National Link-Up Forum, Sydney (28 March)
|Mt Druitt, Sydney|
National SEWB RC Forum (10 May)
NSDC Forum, Adelaide (28 April)
2.2 Key informant telephone interviewsA total of 33 telephone interviews were conducted with key informants around Australia to supplement the fieldwork.
The informants interviewed by telephone were identified through various means, including:
- identification of key stakeholders by OATSIH, the Reference Group, the NSDC and its Chairperson
- suggestions by other key informants interviewed
- people identified by other stakeholders in locations visited who were unavailable to participate in the consultations.
As with the fieldwork, a detailed question guide was developed for the telephone interviews and approved by OATSIH.
2.3 Written submissionsStakeholders were able to provide written submissions to the consultants directly or via the website established for the evaluation (see below). Stakeholders were invited to provide written submissions through a number of approaches: the Communications Strategy (see below); informing the staff and stakeholders consulted on the fieldwork of this option; and a letter written directly to key national and State record-keeping/searching agencies by OATSIH in November 2006. A total of 19 submissions were received, including one from a client of one of the programs. Most of the submissions were from record-keeping/searching agencies. A list of those who provided submissions is provided in appendix A.
2.4 Survey in relation to Mental Health Service Delivery ProjectsOATSIH distributed a short pro-forma to its State offices to gather information about the Mental Health Service Delivery Projects. A response was received in relation to 11 of the 19 projects.Top of page
2.5 Literature reviewThe aim of the literature review was to identify current and emerging issues, policies and approaches to meeting the mental health and SEWB needs of Indigenous peoples who have been affected by dispossession, forced removal from families and associated grief, trauma and loss. This was to include identifying best practice models and possible alternative service delivery models for consideration.
The literature review sought to identify:
- current thinking about best practice strategies for meeting the SEWB needs of Stolen Generations groups, both within Australia and internationally – including aspects ranging from counselling approaches to organisational and locational arrangements
- examples of good practice in meeting those needs
- the current situation regarding Stolen Generations people in Australia and any future trends.
Various sources were used to identify resources for the literature review:
- searches of a range of Australian and overseas literature databases concerning health, psychology and Indigenous issues
- internet searches
- sources identified by OATSIH, the Reference Group for the evaluation and staff and external stakeholders consulted on the fieldwork.
2.6 Program dataThe consultants analysed program data for the Link-Up and BTH services provided by OATSIH. These data were provided by the services to OATSIH as part of their annual reporting requirements. The Link-Up data covers the period 1998-1999 through to 2005-2006, and comes from the Foxtrot database used by the Link-Up services. The BTH data covers the period 2001-2002 through to 2004-2005, and comes from the annual BTH Questionnaire completed by ACCHSs funded to employ counsellors.
There are reportedly a range of issues concerning the reliability of both the Link-Up and BTH data. These issues are discussed in more detail in chapter 5.
2.7 Communications strategyA Communications Strategy was developed and implemented by specialist Aboriginal communications firm Gavin Jones Communications (GJC) in consultation with the evaluation team and OATSIH. The aim of the Strategy was to inform key stakeholders about the research and how they could contribute to it, either via participating in the fieldwork or making a written submission.
The activities conducted for Part A of the Communication Strategy (at the beginning of the evaluation) are set out in table 2.2 below. These activities were conducted in March and April 2006.
A website was also established for the evaluation: bringing them home.
Part B of the Communications Strategy will be conducted after submission of this report to DoHA to publicise the findings of the evaluation. This will include a similar range of activities to the above. The website will include a short community summary report of the key findings of the evaluation, and will be transferred to OATSIH for ongoing management and updating of any key developments. Top of page
Table 2.2: Activities for Part A of the Communications Strategy
|Koori Mail||Article on evaluation|
|National Indigenous Times||Link Up Video release|
|Deadly Vibe||Issues you should consider if planning to meet your natural family|
|National Health Workers Journal||25 years of Link-Up|
|Various stations||Community Service Announcement|
|Deadly Sounds||Interview questions for nominated representative|
|eVibe/eLetter||Article on evaluation|
2.8 Issues for future evaluationsThere were some lessons for this evaluation which may be useful to consider for the purpose of future evaluations of the programs. In light of the challenges experienced in organising the fieldwork, it would be beneficial for:
- the State OATSIH offices to take a more proactive role in assisting with fieldwork organisation
- the national OATSIH office to encourage the services to value and prioritise evaluation activities to a greater degree (some recommendations to this report in Chapter 11 address this issue)
- the funded services to be offered small financial allowances for every client recruited who turns up for the consultations (say $30), to compensate in part for their time
- a fairly long lead time (at least around six weeks) be allowed for organisation of the fieldwork, given that it may be difficult to contact and/or engage services