Evaluation of the Child Health Check Initiative and the Expanding Health Service Delivery Initiative - Final Report

2.2 Evaluation objectives and organisational framework

Evaluation of the Child Health Check Initiative and the Expanding Health Service Delivery Initiative - Final Report

Page last updated: 17 April 2012

In designing the evaluation, Allen and Clarke identified a set of evaluation questions based on the four evaluation objectives for the CHCI and the six evaluation objectives for the EHSDI. Table 1 lists the evaluation objectives for the CHCI and explains how we have organised the findings in this report on these objectives.

Table 1: The CHCI—approach to evaluation objectives
Evaluation objectiveExplanation
1. Assess the extent to which the child health checks reached the target population.This objective is addressed in Section 4.2. Our focus has been on comparing the group of children who had a child health check with the group of children who were eligible for a child health check but who did not have one.
2. Identify the prevalence and, if possible, the severity of the health conditions found through the child health check and validate these findings with data from other sources.This objective has been largely addressed by the AIHW in the process of monitoring the progress of the CHCI. We considered validating the child health check data against data from additional data sources. This, as well as the results of the AIHW’s analysis, is discussed in Section 2.3.4. The severity of conditions has not specifically been assessed by the AIHW or this evaluation.
3. Assess the extent to which requested primary care, allied health and specialist follow-up services have been received, gaps in existing health service delivery, and barriers to the completion of follow-up treatment.This objective is addressed in Section 4.3 where we discuss the referral and follow-up process, report on what services were requested and received, and the appropriateness of referrals.
4.1 Assess whether or not the CHCI has led to improvements in health service delivery for Aboriginal and Torres Strait Islander children.This objective is addressed in Section 4.4 which looks at the impact of the CHCI on health staff, services and the wider PHC system in the NT.
4.2 Assess the health status of children in relation to the social determinants of health and access to comprehensive primary health careThese objectives are addressed in Section 4.5 where we discuss what treatment was provided as part of the CHCI, any change in health status that has occurred since the CHCI, and provide some commentary and baseline data on the social determinants of health.
4.3 Assess treatment outcomes.
The CHCI evaluation objectives have largely been met, including analysis of the reach of the program, provision of follow-up services and impacts of the CHCI on the health system and health status. Where the CHCI evaluation objectives had already been significantly addressed by the AIHW (for example, objective 2), we have provided additional commentary.

The organisation of the evaluation findings for the formative evaluation of the EHSDI is more complex. In addition to considering the evaluation objectives as an organising framework, we considered:
  • the EHSDI funding components—expanded PHC services, development of regions and the move towards Aboriginal community control, the RAHC, capital and infrastructure and evaluation
  • the five systems-wide policy issues identified in the first of our evaluation workshops as pillars of the NT PHC reform program—leadership/governance/regionalisation, core PHC services, workforce/infrastructure, NT AHKPIs and CQI
  • the PHRG’s health systems planning framework based on the World Health Organization’s (WHO) framework which encompasses six building blocks (WHO 2007a)—leadership and governance, service delivery, access to medicines, vaccines and technology, workforce, financing and information
  • other, less tangible elements of a health system discussed at our two evaluation workshops— partnership, capacity, communication and sustainability
  • the overarching evaluation objectives of effectiveness, efficiency and appropriateness..
There is some alignment in the areas, in particular the funding components, systems pillars and building blocks. There is not always a clear link though from these to the six evaluation objectives for the EHSDI. As previously discussed, the EHSDI evaluation is inherently formative and there was a strong message from the evaluation partners that EHSDI should be considered as part of a wider program of PHC system reform. As a result we have taken a pragmatic and flexible approach to organising the EHSDI evaluation findings and recommendations, closely aligning them to the pillars and WHO building blocks rather than aligning them to the evaluation objectives.

Section 7 of this report, which covers key findings and recommendations, fully adopts the PHRG’s health systems planning framework (based on the WHO’s building blocks), but with the omission of ‘access to medicines, vaccines and technology,’ which was not covered by this evaluation.
Top of page
Table 2 shows how we have organised the findings in this report to align with the evaluation objectives for the EHSDI.


Table 2: The EHSDI—approach to evaluation objectives
Evaluation objectiveExplanation
1. Assess the impact and sustainability of the EHSDI on primary health care service delivery and equitable distribution of resources. This will include measurement against indicators of the number, range, and accessibility of services compared with agreed standards for primary care across the NT.This objective is principally addressed in Section 5.3 which looks at how the EHSDI has affected health services and remote health infrastructure and the impact of the EHSDI on the equitable distribution of resources. The sustainability of the EHSDI reforms is considered throughout the report, with the information drawn together in Section 6.4.
2. Assess the extent to which Aboriginal and Torres Strait Islander people were engaged and empowered to contribute to health service planning, governance and responsiveness of services.This objective is addressed in two sections. Section 5.4 discusses leadership and governance in the NT PHC system. Section 5.5 looks at Aboriginal engagement through the process of regionalisation and Aboriginal community control.
3. Assess the impact and sustainability of the RAHC on health workforce availability and flexibility in the NT. This will include measurement against indicators of workforce supply in all locations and the effectiveness of clinical governance structures.This objective is addressed in Section 5.6 which considers the performance of the RAHC workforce model and its impact on meeting local needs, quality and clinical governance, workforce capacity and the overall sustainability of the model.
4. Assess the efficiency of the EHSDI and how well it has maximised health service delivery with the available funds.Efficiency is an overarching evaluation objective. Discussion on efficiency in health service delivery is in Section 5.3.
5. Assess the effectiveness of the EHSDI in achieving change in health status. This will include measurement against primary care related health indicators as developed through the NT key performance indicators (NT AHKPIs) project and the analysis of the NTER child health check program.Effectiveness is also an overarching evaluation question. It was premature to fully address the question of the effectiveness of the EHSDI in changing health status, so the discussion focuses on the likelihood of the program contributing to improved health over the longer-term. Discussion of the NT AHKPIs is included in Section 5.7.
6.1 Assess the impact of the regional reform process on the efficient and effective operation of health services.The program of regional reform had not progressed sufficiently to fully address these objectives. However, discussion on clinical governance (objective 6.2) and information systems (objective 6.3) is included in Section 5.7. The impact of the regional reform process more generally, is discussed in Section 5.5.
6.2 Assess the impact of the regional reform process on clinical governance, including quality of health services delivery.
6.3 Assess the impact of the regional reform process on information systems and planning capacity.
The discrete elements of the EHSDI are intended to work together as part of a wider process of reform in the NT PHC system. We have addressed the evaluation objectives of the impact and sustainability of the EHSDI on health service delivery, distribution of resources, the PHC workforce, and engaging Aboriginal people in health service planning and governance as part of the wider reform process. As noted in Table 2, it was premature to address the effectiveness of the EHSDI in achieving change in health status (objective 5) and the regional reform process had not yet progressed sufficiently to assess its impact on the areas outlined in objective 6.

Findings relating to the three overarching evaluation objectives of effectiveness, efficiency and appropriateness are incorporated, where relevant, in the sections that address each of the CHCI and EHSDI evaluation objectives. The findings on each of these overarching objectives are brought together in Section 6, along with issues relating to sustainability which emerged as a fourth overarching issue.

In addition to the evaluation objectives, DoHA asked us to consider the longer-term monitoring and evaluation approach for PHC system reforms and for future child wellness checks and services. Section 7.3 of this report includes theoretical program models for PHC system reform and child wellness checks. It includes important issues and questions to inform both future monitoring and evaluation approaches and the ongoing reform activity. Top of page