The approach taken in evaluating the two programs differed. The CHCI was a summative evaluation while the EHSDI evaluation was formative.
Summative evaluations generally take place at the conclusion of a program and describe what happened as a result of the initiative—the changes, impacts or outcomes. This approach was chosen for the CHCI as it recognises that the initiative is nearing completion. The evaluation documents and reports on the impacts and lessons from the program and can be used to support future improvements to wellness checks and child health in general.
A formative assessment occurs relatively early in the life of a project and is intended to help shape the program by engaging with the development of the initiative—its implementation, inputs, and procedures. The EHSDI is very much a ‘work in progress’ and the measures implemented are part of a long-term reform process. There is an expectation that the shape and structure of the programs under the EHSDI will continue to evolve. The formative evaluation approach provides information that can support ongoing improvements to the EHSDI. The evaluation team has engaged with those responsible for implementing the program at regular intervals to identify and discuss current issues affecting implementation and to share interim evaluation findings.top of page
A number of agencies and inter-agency groups have a key role in the CHCI and EHSDI programs and this evaluation. This includes the three main evaluation partner agencies:
- the Australian Government Department of Health and Ageing (DoHA), through the Office for Aboriginal and Torres Strait Islander Health (OATSIH), has overall responsibility for funding, accountability and evaluation of the CHCI and EHSDI
- the NT Department of Health and Families (DHF)1 has responsibility for planning and delivering PHC services, implementing change management strategies in support of expanding and reforming services and monitoring the health status of the NT population
- the Aboriginal Medical Services Alliance of the Northern Territory (AMSANT) represents the Aboriginal community controlled health sector and provides support to Aboriginal communities in working towards regional health services.
- the Memorandum of Understanding (MoU) Management Committee, which has representatives from the three partners above, provides advice on the implementation of the CHCI and EHSDI evaluation design, consultation and communication with key stakeholders, and on the preparation of evaluation reports
- an Indigenous Advisory Group (IAG) provides independent advice to the MoU Management Committee and to Allen and Clarke as the independent evaluators of the CHCI and EHSDI
- the Australian Institute of Health and Welfare (AIHW) is the data custodian for the CHCI data collections, and is also a member of the MoU.
- the Northern Territory Aboriginal Health Forum (NT AHF), which has representatives from the three partners above, provides strategic direction for progressing the NT PHC reforms
- the Primary Health Reform Group (PHRG), which has representatives from the three partners above, has been established under the NT AHF to support the NT PHC reforms.
3.2.1 MethodsThe evaluation was conducted from June 2009 to March 2011 and involved four key phases of work:
- Phase 1: Design—this involved developing an evaluation design to address each of the evaluation objectives. The design is outlined in the Evaluation Design Report and is available at www.allenandclarke.co.nz
- Phase 2: Implementation—the evaluation objectives were addressed through a mixed–method design that included key informant interviews, case studies, workshops on EHSDI-related issues and analyses of health datasets.
- Phase 3: Consultation, communication and dissemination; and Phase 4: Analysis and reporting; overlapped and involved feedback and consultation with a range of audiences to check for consistency of findings, increase the utility of findings, improve the accuracy of the report and review and refine any future evaluation plans. The feedback and consultation has informed the drafting of the final evaluation report.
3.2.2 Data sourcesThe evaluation used a mixed-method design, drawing on multiple sources of information. The methods used in the evaluation included:
- key informant interviews with people who played a key role in the NTER, the CHCI, the EHSDI or the wider PHC system reform
- case studies of five communities within four regions (Barkly, Central Australia, East Arnhem and Katherine East), involving interviews, observation, questionnaires and document review
- workshops on EHSDI-related issues with the program partners (AMSANT, DHF and DoHA). The workshops identified and discussed issues and ideas relating to ‘hot topics’ to provide feedback to those working on the EHSDI programs to help improve its ongoing implementation
- analysis of existing population health datasets with regard to the CHCI. This included analysis of whether the population of children who received a child health check was similar to the population eligible for a health check but who did not have one
- review and analysis of program and program-related data such as financial data on the CHCI and EHSDI, deployment data collected on the RAHC, workforce data relating to the EHSDI and health workforce and population data more generally
- review of program documents and research literature including the Little Children are Sacred report, submissions on the 2008 review of the NTER (FaHCSIA 2009), program documents, policy papers and research literature on PHC in remote communities and Indigenous health.
3.2.3 Data analysisThe evaluation analysis focused on bringing together qualitative information from interviews and literature, validating findings from the various sources and evaluation methods used, and corroborating the quantitative analysis (particularly on the CHCI) with the qualitative information such as case studies and interview findings.
The body of evidence—the story of the CHCI and the EHSDI—grew progressively from single sources and methods (individual interviews or case studies) to multiple sources and methods (multiple interviews and cross-case analysis). The process included corroborating interview findings with program data, workshop discussions and research literature and, for the CHCI, the information from the analysis of health datasets. We continually revisited our findings to check whether and how the supporting and relevant evidence fitted with the emerging findings.
Generally, we considered data or evidence to be more valid, and therefore gave it more importance, when we could validate it with data collected from other sources and/or by other methods; however, we remained conscious of the importance of context and the risks of generalising. We looked for areas of convergence (that is, does it fit the story?) but also for areas of divergence, and then we asked whether differences could be explained by different contexts.
1 From 1 January 2011 the Department of Health and Families became known as the Department of Health.top of page