An evaluation framework was developed and presented to the Department on 24 April, and approved by the Department on 1 May 2008. This framework conceptualised the evaluation in terms of a hierarchy of outcomes, described in Figure 1 below, and sought to ensure that the project focussed on evaluating the most strategic components of two large and complex national Programs.

Figure 1 – Hierarchy of outcomes model

  1. Ultimate outcomes - impact on overall issue and ultimate goals - progress towards or away from articulated goals
  2. Intermediate outcomes - impacts, outcomes and achievements across strategic and organisational frameworks
  3. Immediate outcomes - impacts, outcomes and achievements in specific program areas
  4. Outputs/activities - models of service delivery and provision, and how these have been implemented
  5. Needs - priority issues that the program must to respond to, the evidence base and conceptual underpinnings for the program.
This model positions the hierarchy of outcomes as the central focus of an evaluation, allowing for analysis of a range of factors that may have contributed to the outcomes at each level. In other words, rather than just having two levels with outcomes flowing directly from outputs, it allows for much more complex mapping of the relationship between program strategies and outcomes.

The development of an evaluation framework was particularly important in the context of a double evaluation, with a large number of research questions provided by the Department regarding both the outcomes and operations of each Program. The framework provides one way of ordering priorities to ensure that the most important research questions of the evaluation are addressed. For this reason, the evaluation framework document actually contained three separate frameworks: one for each Program and a third for the overarching evaluation across the two Programs. The framework was used to frame the topics explored through the consultation process. Table 3 below provides a summary of the overarching hierarchy of outcomes for both Programs.

Table 3 – Hierarchy of outcomes for the joint RCS/UDRH evaluation

Ulimate outcomes

  • Increased workforce capacity
  • Increased training and support
  • Increased rural health research capability and output
  • Integrated rural health training and support programs

Intermediate outcomes

  • Recruitment and retention of health practitioners in rural and remote areas is increased through the provision of a positive rural health education experience
  • RCSs and UDRHs engage with other programs/initiatives within local, State, Territory and Commonwealth governments
  • There is increased and effective collaboration between UDRHs and RCSs, and also with local educational institutions and health service providers

Immediate outcomes

  • Research into rural and remote health issues is taking place
  • Appropriate and effective support is provided to health professionals currently practising in rural and remote settings
  • More rural and remote health practitioners are engaged in education and training opportunities

Activities/Outputs

  • Rural Clinical Schools Program - targeted education, training and support for medical students in rural and remote health, and development of support infrastructure
  • University Departments of Rural Health Program - targeted education, training and support to enhance opportunities for medical, nursing and allied health students in rural and remote health, and development of support infrastructure.