Evaluation of the Child Health Check Initiative and the Expanding Health Service Delivery Initiative - Final Report

2.6 Strengths and limitations

Evaluation of the Child Health Check Initiative and the Expanding Health Service Delivery Initiative - Final Report

Page last updated: 17 April 2012

There are three main strengths to the evaluation approach and methodology:

  • It accounts for context by:
    • collecting context-rich information through case studies
    • assessing the CHCI and the EHSDI within the context of the NTER
    • looking at what was happening in child health before the CHCI
    • looking at the characteristics of children who did not have a child health check
    • considering the EHSDI program of expansion and reform within the wider context of the NT PHC system.
  • The formative design supports the continuous improvement of the EHSDI. It is responsive to the emerging nature of the program and engaged with key participants during the evaluation process.
  • The evaluation design included the collection of, and reporting on, the views and experiences of 175 participants from all levels within the NT PHC system, with an emphasis on the intended users of the system.
The main limitations to this evaluation approach and methodology are:
  • Limited ability to generalise—the case studies provided context-rich data that allowed us to interpret stakeholders’ experiences of the progress and achievements of the CHCI and the EHSDI. The case studies were not intended to produce generalised statistical measures. Such data was inaccessible as no benchmark indicators had been established (see Section 2.3.4).
  • Time lapse since the CHCI—many community members found it difficult to recall how they felt about the child health checks, given that the checks were completed by early 2008 in most communities. A number of health centre staff and government officials who had major roles in the program have moved on. Not all of them participated in the evaluation.
  • Timing of the EHSDI—the components of the EHSDI program are part of an ongoing process of reform and continued to develop throughout the course of the evaluation. This is not a limitation of the formative evaluation approach; however, it does limit the ability to draw conclusions about criteria relating to the evaluation objectives concerning impact, effectiveness and efficiency.
  • Causal linkages—the context for these programs is very crowded. The NTER comprises seven measures made up of 36 sub-measures, most of which have been in effect in remote Aboriginal communities in the NT since July 2007. There have also been other initiatives that have had considerable impact on these communities, such as the local government reforms, making it difficult to ascribe specific impacts or outcomes to single programs.
  • Relationship bias—there is a long history of relationships between the community controlled and government sector in the NT health system. The nature of these historical relationships may have influenced some of the feedback provided.
  • Theory-building—we have built an emergent program theory to inform future implementation, evaluation and monitoring activities associated with the EHSDI. The process of building a full program theory for evaluation of the EHSDI would have required a higher level of engagement with key people involved in the program and this was not possible within the scope of this evaluation.
  • Children’s voices—the evaluation design meant that others spoke for children, particularly the very young. No participants in the evaluation were under 18 years of age.
Specific limitations and statistical boundaries for quantitative data analyses are included in this report where such data is used.Top of page