Evaluation of the Child Health Check Initiative and the Expanding Health Service Delivery Initiative Summary Report

4.4 Longer-term monitoring and evaluation approach

The Child Health Check Initiative (CHCI) and the Expanding Health Service Delivery Initiative (EHSDI) were designed to address the health needs of people living in remote Aboriginal communities in the Northern Territory (NT).

Page last updated: 10 June 2011

We have developed theoretical program models for child wellness checks and the PHC system reform to inform major issues and questions for future monitoring and evaluation and the ongoing implementation of the reforms. While the models have limitations (such as the risk of over-simplifying and de-contextualising the programs), they are a useful way of illustrating relationships between different parts of the health system and focusing on the key issues underpinning these relationships.

4.4.1 Child wellness checks

In constructing a theoretical model and identifying issues and questions for future monitoring and evaluation of child wellness checks, we have assumed a programmatic response. There are other ways of delivering child wellness services and the majority of PHC is likely to be provided outside formal screening programs, such as when a child presents at a clinic with an illness. Children’s journeys through the system should be the same and access to services and outcomes need to be monitored for children who receive services through a wellness check program and children who receive services outside the program. The child wellness check model, therefore, sits within this broader context of child wellness services and the health system.

The theoretical model for future child wellness checks is shown in Figure 1. The child wellness check, as the principal activity, is at the centre of the model and involves a cyclic process of checking, treatment, referral and follow-up treatment. In this model the wellness check includes the whole treatment pathway and not just the screening element. Importantly, there is an element of prevention and early detection in child wellness checks (that is, identifying conditions before they are symptomatic and preventing illnesses from happening) and not all children will require treatment and referral. The model then sets out the main elements of the development process (inputs) and the hierarchy of anticipated outputs and outcomes, from both a health service perspective (left side of model) and population health perspective (right side).

There is a focus in the model on the development process for wellness checks, reflecting a number of the central themes from the CHCI evaluation. This is depicted by the two central (and closely related) ideas (and their component parts) shown within the central and corresponding outer circles on either side of the model. The collection and analysis of information should address the following:top of page
  • the nature of the health problem
  • the population group that needs to be targeted for the health problem
  • the determinants of health associated with the health problem and the target population and which need to be addressed
  • the nature of any inequalities that exist in the target population and the types of intervention that will reduce these inequalities
  • the types of intervention regarded as effective in addressing the health issue and which of these are acceptable to the target population
  • how considerations of cost, cost-effectiveness, workforce, infrastructure and the operating environment will affect any interventions that might be selected
  • the sustainability of any interventions that might be selected.
Many of the monitoring and evaluation questions we have identified for child wellness checks test some of the assumptions that need to be addressed at this developmental stage and which have been identified by others (such as Wilson and Jungner 1968). Table 7 lists these assumptions and questions.

Table 7: Monitoring and evaluation questions for child wellness checks

Assumptions

Key monitoring and evaluation questions

The wellness check focuses on condition(s) which are important for the target population (i.e. the population health need is defined)
  • Are the conditions that are checked for related to the gap in health status experienced by the target population?
  • What outcomes were planned and were these achieved?
  • Were the services planned as a result of the check coordinated with services aimed at addressing health determinants?
  • How does the check complement and add value to existing health services, including existing wellness checks for children and other population groups?
The target population is defined
  • Does the check reduce any inequalities that exist in the target population?
  • How do the outcomes for the checked population compare with outcomes for the non-checked population?
  • What are the barriers to accessing the check?
There is a suitable test specific to the condition(s)
  • Are the health professionals undertaking the tests experienced in the specific tests and in Indigenous child health?
  • What in the design of the check adds value to existing services and interventions (e.g. does it add more specialist skills)?
There is an acceptable treatment/intervention for the condition(s)
  • Are guidelines for the condition(s) relevant to the target population?
  • How has the check supported the patient journey through health services?
  • How have/are local communities and service providers involved in planning and implementing the check?
  • How satisfied are local communities with the check?
Early treatment is better than later
  • Are guidelines specific for each individual condition?
  • Are guidelines relevant to the target population?
Facilities for diagnosis and treatment are available
  • Are referral and recall processes clear, understood and used consistently?
  • Are the referral and follow-up processes efficient and is the length of time between referral and follow-up acceptable for the condition(s)?
  • Have system interfaces and the coordination of care across the health care continuum (e.g. between primary care and secondary/specialist/allied follow-up) improved over time?
  • Are there any barriers in the patient journey through the system?
  • Are follow-up services sustainable?
The resources are economically balanced in relation to possible expenditure on medical care as a whole
  • Is the check cost-effective?
  • Is the resourcing sustainable?
  • What alternative approaches might deliver the same result?
Data is collected and used to improve service delivery
  • How does data collected in the wellness check combine with other patient information to support health care management?
  • How is data collection and analysis shared across the sector and with other sectors to improve service delivery?
  • How do health service use and outcomes compare between the checked and non-checked populations?
The wellness check is part of a continuous process and not a one-off project
  • Is the availability of the check and treatment services suitable for the condition(s) being checked?
  • Is the approach sustainable?
top of pageFigure 1: Model of program theory for a child wellness check
Figure 1: Model of program theory for a child wellness check

4.4.2 NT PHC system reform

The theoretical model for the NT PHC system reform is shown in Figure 2. The model reads from bottom to top, beginning with identifying assumptions and context, and then the system level ‘enablers’, which are the inputs and resources required for each of the health systems building blocks. The next level requires appropriateness, effectiveness and efficiency in planning, managing and delivering services.

Shifting to the service level, the model details the types of outcomes that might be expected in the short term (1–3 years). The next level considers the changes that might be expected in community development and capability over the medium term (3–5 years). Beyond this the model moves to longer term (5–10 years) outcomes for community members or individuals, with the Council of Australian Governments (COAG) outcomes for Aboriginal and Torres Strait Islander health as the ultimate outcome.

Down the right-hand side of the model are four deeper, cross-cutting themes that apply across all levels of the system:
  1. partnership, capacity and communication
  2. improved partnerships/relationships with Aboriginal communities
  3. improved coordination of services
  4. sustainability of inputs, activities and outcomes.
We have developed questions for future monitoring and evaluation of each of the building blocks considered in this evaluation. These questions and associated information sources are detailed in the full evaluation report. The following summary outlines the key issues for monitoring and evaluation: top of page

Service delivery

  • service availability and use against the core PHC service framework
  • equity in access to core PHC services
  • coordination of health care across the health system (for example between primary and specialist services) and of health care with other social services such as child protection, corrections, housing and special education.

Leadership and governance

  • the effect of regionalisation models on: health service utilisation and acceptability/ satisfaction; the engagement of Aboriginal communities in health service governance, planning and delivery; and efficiency
  • the capacity for systems-wide approaches to health strategy, policy and data collection, analysis and use
  • the effectiveness of partnerships and engagement with other sectors and departments.

Workforce

  • progress against any agreed workforce strategies
  • the impact of additional positions created under the EHSDI on the scope of PHC services and service utilisation
  • the effectiveness of systems and processes for recruiting, training and supporting AHWs and for supporting Aboriginal employment and career pathways within the PHC system more generally.

Financing

  • progress against any agreed funding targets, funding equity between regions and the ability of the system to spend resources effectively.

Information

  • the effectiveness of CQI activities and of the collection and use of information and data to support health service planning and delivery at a local level.

Long-term health and wellbeing outcomes

  • To what extent has the gap in life expectancy between NT Aboriginal and non-Aboriginal people been reduced?
  • To what extent has the gap in mortality rates for NT Aboriginal and non-Aboriginal children under five years of age been reduced?
  • To what extent have health and wellbeing improved, for whom, how and why, and what else happened?
  • To what extent have social determinants improved, for whom, how and why, and what else happened? top of page

4.4.3 Further evaluation considerations

Evaluation, like other aspects of service delivery, can become something that is ‘done to’ communities rather than a tool in which they are actively engaged and empowered. Future evaluation approaches need to see the regionalised community controlled organisations as a main reference point and as active participants in the evaluation process. A formative approach, and one in which the participants are active players in all aspects of the process, is recommended.

The need for the Australian Government to determine whether it is getting value for money from its investment in NT remote PHC services has been a major driver of this evaluation. Future evaluation must not only meet these needs, but give a more predominant role to the questions coming from remote communities.

Our conclusion from this evaluation of the CHCI and the EHSDI is that formative evaluation approaches, which are longer-term and more interactive with the participants, are likely to be most valuable.
Figure 2: Model of program theory for NT PHC system reform
Figure 2 is a model of program theory for NT PHC system reform
top of page

2 This percentage is based on the total number of resident children measured as part of the GAA program, as a proportion of the total number of resident children.
3 Eligibility based on residence in DHF’s Client Master Index and ABS definition of identifying population in prescribed areas.
4 Estimate based on child health check dataset and pro-rata of the ABS population estimate of Indigenous children aged 0–5 years living in prescribed areas.
5 This data is based on participation in the GAA program delivered by DHF service providers between 1992–93 and 2008. Data from all NT service providers for the period 2003–06 shows a similar result.
6 We looked at low birth weight because it is an important indicator of both the immediate health of the newborn and of the long-term risk of adult chronic disease.
7 These figures refer to the number of referrals and not the number of children. A child may have multiple referrals.
8 This includes funding from OATSIH, DoHA Regional Health Service funding, and NT Government funding.