Evaluation of the Child Health Check Initiative and the Expanding Health Service Delivery Initiative - Final Report

7.3 Longer-term monitoring and evaluation approach

Evaluation of the Child Health Check Initiative and the Expanding Health Service Delivery Initiative - Final Report

Page last updated: 17 April 2012

7.3.1 Child wellness checks
7.3.2 NT PHC system reforms
7.3.3 Further recommendations on future monitoring and evaluation

As part of the evaluation we have been asked by DoHA to recommend an appropriate longer term monitoring and evaluation approach for the EHSDI and for child wellness check programs. We have developed theoretical program models for PHC system reform and child wellness checks, and key issues and questions to inform both future monitoring and evaluation approaches and ongoing reform activity. The models have limitations: they risk oversimplifying and decontextualising complex programs; and they are static whereas programs evolve, so the logic or theory that is relevant at the start of a program may no longer be relevant at later stages. Nevertheless, the models are a useful way of illustrating relationships between different parts of the health system and of focusing on key issues underpinning these relationships.


7.3.1 Child wellness checks

In constructing a theoretical model and identifying issues and questions for future monitoring and evaluation of child wellness checks, we have assumed a programmatic response. There are other ways of delivering child wellness services, and most PHC is likely to be provided outside formal screening programs, such as when a child presents at a clinic with an illness. A child’s journey through the system should be the same whether it starts with a wellness check or not, and access to services and outcomes needs to be monitored both for children who receive services through a wellness check program and for children who receive services outside the program. The child wellness check model therefore sits within this broader context of child wellness services and the health system.

The program logic model for future child wellness checks is shown in Figure 16. This is a theoretical model for a child wellness check initiative and is not necessarily based on the logic of the CHCI.

The child wellness check, as the principal activity, is at the centre of the model and involves a cyclic process of checking, treatment, referral and follow-up treatment. In this model the wellness check includes the whole treatment pathway, not just the screening element. Importantly, there is an element of prevention and early detection in child wellness checks (that is, identifying conditions before they are symptomatic and preventing illnesses from happening) and not all children will require treatment and referral.

The model then sets out the key elements of the development process for a child wellness check, from both a health service perspective (on the left-hand side of Figure 16) and a population health perspective (on the right-hand side). The hierarchy of outputs and outcomes from the child wellness check is fairly straightforward. From a population health perspective children are checked and treated and this leads to improved community health outcomes for the targeted medical and social conditions, which leads to improved health outcomes. From a health service perspective the screening and treatment pathway contributes to seamless people-centred service delivery and improved equity, effectiveness and efficiency, which in turn leads to the ultimate outcome of improved health service delivery. The other key activity on the health service side is the need to regularly monitor service access and outcomes—not only for children who participate in the wellness check—to inform ongoing planning and decision making. This monitoring needs to be appropriate for different levels of the health system, with some reporting aimed at policy makers and funders and other reporting aimed at PHC providers to support their planning and quality improvement processes.

The remaining parts of the model are perhaps the most critical, at least to central themes from the evaluation of the CHCI. These are the two closely related central ideas shown in the central and corresponding outer circles on either side of the model, at the very beginning of the program logic (that is, the inputs):
  • develop a program logic and implementation plan
  • establish links between the health problem, the target population and the social determinants 
of health.
The collection and analysis of these information inputs should address:
  • the nature of the health problem
  • the population groups that need to be targeted in relation to the health problem
  • the determinants of health that are associated with the health problem and the target population and need to be addressed
  • any inequalities that exist among the target population and the types of intervention that will reduce these inequalities
  • the types of intervention that are regarded as effective in addressing the health issue and are acceptable within the target populations
  • how considerations of cost, cost-effectiveness, workforce, infrastructure and the operating environment will affect any intervention that might be selected
  • the sustainability of any intervention that might be selected.
The other central element of the model—and this is evident from the CHCI evaluation findings—is the importance of context. This is shown in Figure 16 as underpinning the entire conceptualisation of a child wellness check, and includes elements such as existing health services and programs, other (non health sector) initiatives, and characteristics of the target populations/communities.


Figure 16: Model of program theory for a child wellness check

Figure 16: Model of program theory for a child wellness check
[D]

Table 63 includes a list of key questions to guide future monitoring and evaluation of child wellness checks. The questions are organised around a set of key assumptions underpinning child health screening programs, many of which are based on issues recommended for consideration in the design and operation of a screening program (Wilson and Jungner 1968). The questions are also informed by lessons from the CHCI evaluation. The assumptions and questions focus on issues relating to the central ideas in the theoretical model (Figure 16). The list of questions is not complete but is a starting point based on issues identified in this evaluation.
Top of page

Table 63: Monitoring and evaluation questions for child wellness checks
AssumptionsLessons from CHCI evaluationKey monitoring and evaluation questions
The wellness check focuses on condition(s) which are important for the target population (i.e. the population health need is defined)The initial focus of the policy response was to target child sexual abuse, but the focus of the CHCI was broadened to include the wider health needs of children. These needs were important for the target population.

The CHCI focused primarily on medical conditions. There is also a need for a stronger focus on the socio-economic and environmental conditions that affect the target population.

The outcomes sought for the conditions were not specified.

There were existing wellness checks operating in the NT that many of the target population were already accessing.

  • Are the conditions that are checked for related to the gap in health status experienced by the target population?
  • What outcomes were planned and were these achieved?
  • Were the services planned as a result of the check coordinated with services aimed at addressing health determinants?
  • How does the check complement and add value to existing health services, including existing wellness checks for children and other population groups (e.g. adult health checks)?
The target population 
is definedThe target population was defined as all Indigenous children of eligible age living in prescribed areas.

There was no policy on targeting specific groups within this population to address issues of health equity.

The group of children who had a child health check had a similar health status to the group of children who did not.

The group of children who had a child health check were more likely to already be accessing screening programs.

  • Does the check reduce any inequalities that exist in the target population?
  • How do the outcomes for the checked population compare with outcomes for the non-checked population?
  • What are the barriers to accessing the check?
There is a suitable test specific to the conditions(s)There is no suitable screening test for child sexual abuse.

There are suitable screening tests for most of the conditions covered in the child health check, although not all practitioners were experienced in screening/testing for these conditions in a remote Indigenous child population.

  • Are the health professionals undertaking the tests experienced in the specific tests and in Indigenous child health?
  • What in the design of the check adds value to existing services and interventions (e.g. does it add more specialist skills)?
There is an acceptable treatment/intervention for the condition(s)The evidence relating to treatment for some conditions in a remote Indigenous primary health context is limited or conflicting.

There is a need for further research evidence and updated treatment guidelines for some conditions.

Many conditions were considered acute and managed/treated as such; however, many conditions are chronic within the target population and would benefit from a population health approach that, for example, tracks the patient journey through an ongoing care management type approach.

The management of chronic conditions is not well-suited to a ‘fly-in/fly-out’ model of care in which there is generally a lack of continuity.

There was little engagement with the NT remote health system and Indigenous communities during policy development and planning to test the acceptability of the CHCI.

  • Are guidelines for condition(s) relevant to the target population?
  • How has the check supported patient access through the health services?
  • How have/are local communities and service providers been/being involved in planning and implementing the check?
  • How satisfied are local communities with the check?
Early treatment is better than laterThis needs to be judged against the natural history of the specific condition(s) and the positive and negative impacts of intervening early.

There is a need for further research evidence and updated treatment guidelines for some conditions.

  • Are guidelines specific to each individual condition?
  • Are guidelines relevant to the target population?
Facilities for diagnosis and treatment are availableFull assessment of follow-up services and implementation planning was not completed until after the CHCI was launched.

Initially there were problems with referral processes and bottlenecks between primary and secondary care and other referred services.

Referral processes and follow-up services improved over time and additional facilities were funded.

The sustainability of follow-up services is a major concern.

  • Are referral and recall processes clear, understood and used consistently?
  • Are referral and follow-up processes efficient and is the length of time between referral and follow-up acceptable for the condition(s)?
  • Have system interfaces and the coordination of care across the health care continuum (e.g. between primary care and secondary/specialist/allied follow-up) improved over time?
  • Are there any barriers in the patient journey through the system?
  • Are follow-up services sustainable?
The resources are economically balanced in relation to possible expenditure on medical care as a wholeThe child health checks cost between $1,600 and $3,200 per child.

There were considerable opportunity costs for existing services and staff.

Research evidence suggests that the costs of delivering health services in remote areas of the NT is around $168 for a 20-minute consultation with a doctor (Zhao et al 2006).

More information is required on the costs of delivering health services in remote areas.

  • Is the check cost-effective?
  • Is the resourcing sustainable?
  • What alternative approaches might deliver the same result?
Data is collected and used to improve service deliveryExtensive data was collected.

Data was used with good effect to create and sustain a wider understanding of the problem.

Data was less effective in informing 
PHC service delivery.

Data outside the more traditionally recognised primary and secondary care services (e.g. on social support services) was not used effectively.

Data for the checked population was 
not analysed against data for the unchecked population.

  • How does data collected in the wellness check combine with other patient information to support health care management?
  • How is data collection and analysis shared across the sector and with other sectors to improve service delivery?
  • How do health service utilisation and outcomes compare between the checked and unchecked populations?
The wellness check is part of a continuous process and not a one-off projectThe use of fly-in/fly-out teams made the child health check a one-off or episodic process for many communities.

Other health services were able to operate continuous checking over the program lifespan by using the MBS 
Item No. 708 check.

The child health checks had a two-year life span.

The follow-up services will run for up 
to five years.

  • Is the availability of the check and treatment services suitable for the condition(s) being checked?
  • Is the approach sustainable?
Suggestions on the overall approach for future monitoring and evaluation are discussed in Section 7.3.3.


7.3.2 NT PHC system reforms

We developed two basic program logic models for the EHSDI in our evaluation design report (Allen and Clarke 2009a): one that described the EHSDI in terms of its context, inputs/activities, outputs and outcomes; and another that provided a sequence map for the planned activities and outcomes. In further developing these models we have aimed to:
  • allow for flexibility that fits with the emergent nature of the program
  • expand the model to the level of the PHC system rather than restricting it to the component parts 
of the EHSDI
  • include a framework consistent with the PHRG’s use of the WHO health systems building blocks
  • allow for other ‘deeper’ elements of a health system, such as sustainability, partnerships, communication and coordination
  • identify the key issues or questions underpinning the model and the relationships between 
the different levels of the PHC system included in it: health system, health service, community 
and individual.
The program logic model is shown in Figure 17. The model reads from bottom to top, beginning with identifying assumptions and context and then the system level ‘enablers’, which are essentially the inputs and resources required for each of the health systems building blocks. The next level requires appropriateness, effectiveness and efficiency in planning, managing and delivering services. Shifting to the service level, the model details the types of intermediate outcomes that might be expected in the short term (one to three years) for the five building blocks we looked at.

The next level considers ‘outcomes’ at the community level, which are better described as the changes that might be expected in community development and capability over the medium term (three to five years). Beyond this the model moves to longer term (five to ten year) outcomes for community members or individuals, with the COAG outcomes for Indigenous health as the ultimate outcome.

Down the right-hand side of the model are four deeper, cross-cutting themes or ‘pillars’ that apply across 
all levels of the system: partnership, capacity and communication; improved partnerships/relationships 
with Aboriginal communities; improved coordination of services; and sustainability (of inputs, activities 
and outcomes).
Top of page

Figure 17: Model of program theory for NT PHC system reform

Figure 17: Model of program theory for NT PHC system reform
[D]

Tables 64 to 68 include the key issues/questions and selected indicators or information sources for monitoring and evaluating the ongoing implementation of the EHSDI and the broader PHC reform agenda, for each of the five building blocks we have considered in this evaluation.41 Each table is organised by short, medium or longer term measures. The list of questions is not complete but is a starting point based on issues identified in this evaluation. Similarly the indicators are included as a starting point. The final selection of indicators should be based on a thorough assessment using criteria related to importance, validity, feasibility, the degree to which they are meaningful, comprehensiveness, and alignment with existing data collections (such as the NT AHKPIs). Furthermore, and as discussed further in Section 7.3.3, the final selection of future questions and indicators for monitoring and evaluation should, in line with principles for developing indicators (Raleigh and Foot 2010), include substantial input from those being monitored.

Table 69 includes measures and indicators for the long-term population health and wellbeing outcomes which sit at the top of the model in Figure 17 and apply across the full spectrum of the health systems building blocks.
Top of page

Table 64: Monitoring and evaluation questions and indicators for service delivery (including core services, providers and quality)
Key issues/questionsIndicators / information source
Short-term (1–3 years)
  • Availability of core PHC services.
  • Equity of access to core PHC services.
  • Utilisation of core PHC services.





  • What are the barriers to and facilitators of the ongoing implementation of regional reform of health services?

  • What are the barriers to and facilitators of the ongoing implementation of the CQI model?
  • Service provision against agreed core PHC service framework.
  • Number of Aboriginal people accessing PHC services, by location, age and sex (including NT AHKPIs 1.1, 1.2, 1.4, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12).
  • Number of Aboriginal people not accessing PHC services, by location, age and sex.
  • Progress with components of regionalisation including merging of ACCHOs, creating new structures, organisational change management.
  • Qualitative research/evaluation.
  • Progress with appointing CQI Facilitators and supporting the model.
  • Service provider engagement in CQI activities (including NT AHKPI 2.16).
  • Qualitative research/evaluation.
Medium term (3–5 years)
  • Coordination of care across the health system (e.g. between primary and specialist services): what has improved what still needs to be addressed?
  • PHC services delivered by regional Aboriginal community controlled health services.
  • Effect of regionalisation, including of different service delivery models, on health service utilisation.
  • To what extent are best practice standards and accreditation processes being implemented, and what factors still need to be addressed?
  • Number of Aboriginal people accessing referred services.
  • Time lag between referral and receiving follow-up secondary or specialist services.
  • Number of regional ACCHOs.
  • Number of DHF-managed remote health centres.
  • Support provided to regional ACCHOs.
  • Utilisation of PHC services in regions that have moved to regional level ACCHOs.
  • Number of regional ACCHOs that are accredited.
  • Qualitative research/evaluation.
Longer term (5–10 years)
  • How has coordination of service delivery improved, especially for families that have a high level of contact with services such as child protection, juvenile justice, corrections, housing, special education and health services?
  • Primary care service delivery is more people-centred.
  • Effect of regionalisation, including of different service delivery models, on health service efficiency.
  • Health service engagement with other sectors.
  • Health service relationship with community.
  • Qualitative research/evaluation.

  • Cost of regionalised services (administrative, corporate, planning, service delivery) against service access and utilisation.
Table 65: Monitoring and evaluation questions and indicators for leadership/governance
Key issues/questionsIndicators/information source
Short term (1-3 years)
  • Greater coherence in leadership and governance.
  • Clear roles, responsibilities and accountabilities.
  • Adequate capacity to support implementation.
  • Clear communications strategy implemented across partner organisations.
  • Clear and consistent messages about regional reform are reaching remote Aboriginal communities.
  • FTEs involved in policy implementation.
  • Qualitative research/evaluation.
Medium term (3-5 years)
  • Level of engagement of Aboriginal communities in health service governance, planning and delivery—what is working well and what factors still need to be addressed?
  • Strengthened ACCHO board governance capacity.
  • Number of RSC and regional ACCHO boards in place (including NT AHKPI 4.18).
  • Qualitative research/evaluation.
Longer term (5-10 years)
  • Ongoing support for community controlled governance and leadership, including competent and capable regional health boards.
  • Existence of system-wide approaches for strategy and policy.
  • Enhanced cultural appropriateness of PHC services.
  • Satisfaction with PHC services.
  • Qualitative research/evaluation.

Table 66: Monitoring and evaluation questions and indicators for workforce
Top of page
Key issues/questionsIndicators/information source
Short term (1–3 years)
  • Assessing the impact of the additional 251 positions created under the EHSDI (e.g. on scope of PHC services delivered and service utilisation).
  • Developing a comprehensive workforce strategy which identifies critical workforce issues, including for Aboriginal health professionals, and approaches to address them.
  • Improvement in systems for recruiting, training and supporting AHWs.
  • Progress in implementing the RAHC, including increase in effectiveness and efficiency over time with more repeat deployments.
  • How the RAHC fits within the context of a comprehensive workforce strategy.
  • Number of operational/active AHWs, by age and sex.
  • Number of AHW trainees, by age and sex.
  • RAHC deployment data.
  • Qualitative research/evaluation.
Medium term (3–5 years)
  • Monitor staff resourcing against agreed workforce strategy.
  • To what extent and how has the implementation of a workforce strategy addressed significant workforce issues?
  • Indicators to be developed within the strategy, but consider NT AHKPIs 2.13, 2.14 and 2.15.
  • Qualitative research/evaluation.
Longer term (5–10 years)
  • What has the impact been on Aboriginal employment and career pathways within the PHC system?
  • What impact has a workforce strategy had on overall workforce capacity, and what are the main barriers that still need to be addressed?
  • Number of additional health professionals recruited and operational.
  • Number of Aboriginal staff in the health workforce, by role, age and sex.
  • Recruitment data (long and short term).
  • Qualitative research/evaluation.
Table 67: Monitoring and evaluation questions and indicators for financing
Key issues/questionsIndicators/information source
Short term (1–3 years)
  • Does the funding model for PHC accurately reflect the costs of providing services to people living in remote NT communities?
  • Monitor financial resources against agreed funding targets.
  • Funding methodology.
  • Financial information.
  • Qualitative research/evaluation.
Medium term (3–5 years)
  • Improved funding equity between regions.
  • To what extent has funding increased as a result of the funding model, and has the system been able to spend additional resources effectively?
  • Do boards of regional ACCHOs have sufficient financial governance capacity to oversee the effective purchase and/or delivery of health services?
  • To what extent is the complexity and multiplicity of funding sources a barrier to effective and efficient regional ACCHOs?
  • Financial information.
  • Qualitative research/evaluation.
Longer term (5–10 years)
  • Assess the efficiency of regional health service models.
  • Is financial information and analysis adequate and timely to effectively support the financial risk management of regional ACCHOs?
  • What impact do ongoing government funding arrangements have on the ability of services to effectively plan for health service delivery?
  • Financial information.
  • Qualitative research/evaluation.

Table 68: Monitoring and evaluation questions and indicators for information
Top of page
Key issues/questionsIndicators/information source
Short term (1–3 years)
  • Are existing PIRS appropriate to support the delivery of coordinated core PHC services in remote communities?
  • Are emerging regional ACCHOs considering appropriate PIRS options, including for training and support?
  • To what extent is NT AHKPI data complete and accurate, and what are the barriers to improving data quality?
  • How is information and data, including NT AHKPI data, collected and used locally, and how effective are CQI Facilitators in supporting this?
  • Analysis of NT AHKPI data and data processes.
  • Qualitative research/evaluation.
  • NT AHKPI 4.19.
Medium term (3–5 years)
  • Is the scope of the NT AHKPIs sufficient, and/or what other measures should be included?
  • How is the health service supporting greater Aboriginal community skills and knowledge through health service reporting and health programs?
  • How do the NT AHKPIs and other information support public/government accountability, and is the data adequate to show what is happening in the NT PHC sector?
  • Qualitative research/evaluation.
  • How effective are the processes for system-wide analysis and reporting of data and information?
  • Qualitative research/evaluation.

Table 69: Monitoring and evaluation questions and indicators for long-term (5–10+ years) population health and wellbeing outcomes
Key issues/questionsIndicators/information source
  • To what extent has the gap in life expectancy between NT Indigenous and non-Indigenous people been reduced?
  • To what extent has the gap in mortality rates between NT Indigenous and non-Indigenous children under five years of age reduced?
  • To what extent has health and wellbeing improved and for whom, how and why; and what else happened?
  • To what extent have social determinants improved and for whom, how and why; and what else happened?
  • Life expectancy (including gap between Indigenous and non-Indigenous)
  • Infant/young child mortality rates (including gap between Indigenous and non-Indigenous).
  • Relevant indicators: NT AHKPI 1.3, NT AHKPI 1.5, NT AHKPI 1.6, avoidable hospitalisations and preventable deaths.
  • Relevant indicators such as housing (overcrowding), education (attendance rates and achievement), employment, access to food and transport.
We have also considered key questions/issues and indicators for the four pillars as outlined in Figure 17: partnership, capacity and communication; improved partnerships/relationships with Aboriginal communities; improved coordination of services; and sustainability. Many of these questions/issues are included in the above tables as they closely relate to a health system building block (e.g. assessing the degree of coherence in the partnership and in communication, and assessing the coordination of service delivery between PHC services and other services). Additional questions/issues include:
  • monitoring consumer satisfaction with health services
  • assessing partnerships and engagement with other sectors and departments
  • assessing the relationship between Aboriginal people and governments, including what factors have improved and what still needs to be developed
  • assessing the engagement of Aboriginal men in health service delivery (both in the health workforce and as consumers)
  • assessing the sustainability of governance models, emerging regional health service delivery models, workforce models and funding.

7.3.3 Further recommendations on future monitoring and evaluation

Evaluation, like other aspects of service delivery, can become something that is ‘done to’ communities rather than a tool in which they are actively engaged and through which they are empowered. Future evaluation approaches need to see the regionalised community controlled organisations as a main reference point and as active participants in the evaluation process. A formative approach, and one in which the participants are active players in all aspects of the process, is recommended.

A major driver of this evaluation has been the quite reasonable desire of the Australian Government to determine whether it is getting value for money from its investment in NT remote PHC services. Future evaluation must to not only meet these central government needs but also give a more predominant role to the questions coming from remote communities.

Our conclusion from this evaluation of the CHCI and the EHSDI is that formative evaluation approaches, 
which are longer term and more interactive, with the learnings internalised by the participants, are likely to 
be most valuable.

41 - It will also be important to develop measures and indicators for the sixth building block—access to medicines, vaccines and technology.


Top of page