Evaluation of the Medical Specialist Outreach Assistance Program and the Visiting Optometrists Scheme

2.1 Overview of project methodology

Page last updated: 28 February 2012

The evaluation was conducted through four stages, involving 10 discrete steps, which are shown in Figure 3 below. The project commenced in late April 2011 and was completed on 26 September 2011. Throughout the project there were regular meetings of the project team involving Health Policy Analysis and DoHA (see Table 3).

The process of creating this document
Figure 3 – Overview of methodology for the evaluation of MSOAP and VOS


Table 3 – Project stages, timings of meetings of key groups and deliverables
Stage/step Project team meetings Key dates Deliverable Date delivered

Stage 1 – Project governance

  1. Project planning

19/04/2011
03/05/2011

 

  1. Project plan

19/05/2011

Stage 2 – Situation analysis

  1. Literature and document review

17/05/2011

 

 

 

  1. Preliminary data analysis

 

 

  1. First progress report (situation analysis)

23/06/2011

Stage 3 – Consultation and data collection

  1. Key stakeholder consultations (including written submissions from stakeholders)

02/06/2011
15/06/2011

Conducted between June and August 2011

  1. Second progress report

10/06/2011

  1. Survey of service providers

 

Conducted between
18/08/2011 and 02/09/2011

 

 

  1. Community visits

12/07/2011

Conducted between
06/07/2011 and 12/09/2011

  1. Third progress report

08/07/2011

  1. Cost benefit analysis

 

 

 

 

Stage 4 – Data analysis and report preparation

  1. Analysis of programs

 

 

 

 

  1. Draft report and presentation

05/08/2011
23/08/2011

 

  1. Draft report

31/08/2011

  1. Final report

21/09/2011

 

  1. Final report

29/09/2011

Key methods through which evidence was gathered for the evaluation were:
    • review of program documentation and associated literature
    • analysis of program and related data
    • interviews with stakeholder organisations and individuals across Australia
    • review of written submissions provided to the evaluators by stakeholders
    • a survey of clinical service providers who have been supported through MSOAP or VOS
    • eight case studies involving visits to a sample of rural and remote communities where the impact on access for services was assessed along with other social impacts of the programs.
Table 4 sets out the original evaluation questions specified for the review, additional questions developed to explore the issues in more detail and sources of evidence used to address these questions.

Table 4 – Evaluation questions and sources of evidence used to address questions

Evaluation questions

Lit. review

Doc. review

Data analysis

Stakeholders

Surveys

Case studies

Program effectiveness

Evaluation brief:

  • Is MSOAP (including its expansions) effective in providing medical specialist and allied health outreach services is Australian Standard Geographical Classification (ASGC) Remoteness Areas (RAs) 2-5? (question 1)
  • Is VOS effective in providing outreach optometric services in ASGC-RA2-5? (question 2)
  • Determine whether the model of service delivery under each program is effectively meeting the needs of the client groups and other stakeholders (objective 4).
  • Alternatives means of delivering outreach services (1g, 2g)
  • The potential role of team based services within the program (1i)
  • Linkages between VOS supported services and state/territory based spectacle subsidy schemes (2b)

Additional questions:

  • What have been other benefits in addition to facilitating greater access to services? 
  • What issues contribute to service providers not being able to deliver the level of service originally planned?
  • How could the programs be redesigned to minimise this problem?
  • What are the major opportunities to improve the operation and effectiveness of MSOAP and VOS?

x

x

x

x

x

x

Equity

Evaluation brief:

  • Determine whether the model of service delivery under each program is equitably meeting the needs of the client groups and other stakeholders.

Additional questions:

  • Does the allocation of services/funding/expenditure across jurisdictions and remoteness areas reflect relative needs?  
  • What factors have impacted the distribution of spending under the programs?

 

x

x

x

 

 

Program cost effectiveness

Evaluation brief:

  • Determine whether the model of service delivery under each program is efficiently meeting the needs of the client groups and other stakeholders.
  • Costs and benefits (including social benefits to communities) of the program in a sample of communities across the ASGC-RAs (1b )
  • Alternatives to address the barriers to outreach delivery and the budget impact of these alternatives (1h, 2h)

Additional questions:

  • How does the cost of supporting improved access through MSOAP and VOS compare with potential alternative methods of improving access? 
  • What are the opportunities to improve value for money under MSOAP and VOS?

 

 

x

x

 

x

Assessment of need

Evaluation brief:

  • Planning processes for identifying needs and ensuring services are put in place (1d, 2d).
  • Appropriateness of information available to fundholders and advisory fora to make informed decisions (1e)

Additional questions:

  • To what extent are processes undertaken by advisory fora and others effective and appropriate in identifying areas of need?
  • How could these processes be improved? 
  • Are there lessons from some jurisdictions that could be applied elsewhere?
  • What have been the planning processes undertaken for VOS?
  • How could these be improved?

 

x

 

x

 

 

Development, assessment and approval of service proposals: 

Evaluation brief:

  • Look for opportunities to streamline administration of the programs (objective 2).
  • Recruitment and retention of medical practitioners and other service providers (1f, 2f)
  • Appropriateness of information available to fundholders and advisory fora to make informed decisions (1e)

Additional questions:

  • What have been the processes that have been undertaken to develop service proposals and/or invite service providers to submit service proposals?
  • What approaches have been most successful in eliciting proposals for areas of need?
  • Could the criteria and scoring approaches be improved? 
  • What is the most appropriate time period for reassessing and approving existing service proposals? 
  • What advantages and issues would there be in moving from an annual to a multi-year approval cycle?

 

x

x

x

 

 

Administration

Evaluation brief:

  • Efficiency of project administration (1k, 2k)

Additional questions:

  • In what ways could the administrative processes associated with submitting claims and reports be improved and streamlined for service providers, fundholders and DoHA?
  • What are the key cost drivers in administering the programs for fundholders? 
  • What are the key cost drivers in managing the programs for DoHA?
  • In what ways could administrative costs be reduced with redesign of the program?
  • Could reporting under MSOAP be more effective without increasing costs?
  • To what extent is the fundholder model appropriate for MSOAP? 
  • Would there be advantages in having one fundholder for each jurisdiction? 
  • Would there be advantages in having one national fundholder? 
  • Is the fundholder model appropriate for VOS, and how would this best work?

 

x

x

x

x

 

Data management

Evaluation brief:

  • Appropriateness and efficiency of data collection and management for the program (1j, 2j).

Additional questions:

  • How could data reported be enhanced to provide better information on program effectiveness? 
  • How could data flows and reporting be more efficient for service providers, fundholders and DoHA?

 

x

x

x

 

 

Coordination between VOS and MSOAP

Evaluation brief:

  • Identify opportunities for improving the coordination of services between the two outreach programs (objective 1).
  • Continuity of care for patients between VOS and other programs (2c).
  • Patient referral processes (2d).

x

x

x

x

x

x

Future directions

Evaluation brief:

  • Challenges to growth in service delivery (1a; 2a)
  • Recruitment and retention of medical practitioners and other service providers (1f, 2f)
  • Examine the potential impact of the Health Reform Agenda on both programs (objective 3).

x

x

x

x

x

x

Top of page
Provide feedbackIf you would like a response please complete our enquiries form.


Comments will be used to improve web content and will not be responded to.

Real person check

This form uses a CAPTCHA to ensure that it is submitted by a person, instead of a machine or automated software.


Privacy statement