Evaluation of the National External Breast Prostheses Reimbursement Program

3. Evaluation framework for the National External Breast Prostheses Reimbursement Program

Page last updated: 05 November 2010

This section details the program logic for evaluating the National External Breast Prostheses Reimbursement Program.

A program logic model presents a program in a systematic and visual way, highlighting underlying assumptions and providing a theoretical framework. Logic models are tools for service planning and management as well as evaluation. They can be used at any point in the evolution of a program and can lead to better program planning and management. Program logic models describe the sequence of events for bringing about change and relate activities to outcomes. A program logic is a tool for establishing the logical connections in a program (or a number of programs) from the inputs through to the ultimate outcomes. Each outcome level identifies what the program is seeking to achieve at various stages.

In this case, the program logic has been extended to identify a hierarchy of outcomes. This sets out the intended outcomes of a program in layers, from the broad overarching outcomes down to short-term outputs. It is structured as in figure 1.

Urbis uses the hierarchy of outcomes as the central focus of an evaluation, allowing for analysis of a range of factors that may have contributed to the outcomes at each level. The evaluation will assess the extent to which the program is achieving these outcomes, as set out in the evaluation framework which follows. Essentially, the framework breaks down the program into its outcome components and identifies how these outcomes can be measured. The evaluation framework has been structured around the hierarchy of outcomes model, aligning outcome areas with evaluation questions, evidence and data sources.

Figure 1: Hierarchy of outcomes model

Evaluated from the bottom up:
  1. Ultimate outcomes – impact on overall issue and ultimate goals – progress towards or away from articulated goals
  2. Intermediate outcomes – impacts, outcomes and achievements across strategic and organisational frameworks
  3. Immediate outcomes – impacts, outcomes and achievements in specific program areas
  4. Outputs/activities – models of service delivery and provision, and how these have been implemented
  5. Needs – priority issues that the program must address, the evidence base and conceptual underpinnings for the program. Top of page

3.1 Program logic

Ultimate outcome

Improvement to the quality of life of women who have undergone a mastectomy as a result of breast cancer, through a program of financial assistance for external breast prostheses.

Intermediate outcomes

Improved access to and uptake of, the reimbursement program by eligible women.
Demonstrated satisfaction of women with access to the reimbursement program.
Streamlined reimbursement process for eligible women.
Demonstrated sensitivity to the needs of the eligible women accessing the reimbursement.
Appropriate accountability structures in place to monitor and audit the program.
The program is embedded and integrated into Medicare Australia's processes and procedures with smooth operation through the use of appropriate systems.
Women have a choice of breast prostheses suppliers, such that the process of obtaining a prosthesis is consumer driven.

Immediate outcomes

The provision of reimbursement for external breast prostheses.
Increased awareness of the program by eligible women.
National consistency in the reimbursement program.
Ease of access to reimbursement for all eligible women.
Efficiency and appropriateness of the administration of the program.
Levels of knowledge among, promotion for, and support by, healthcare workers.

Activities/outputs

Financial support provided to eligible women by the Commonwealth towards the costs of external breast prostheses.
Development of information sources and dissemination of these.
Development of protocols and policy regarding claims processing.
Implementation of service arrangement to provide one consistent national program.

Needs

Consistent national approach across all jurisdictions.
Appropriate accountability structures in place to monitor and audit the program.
Efficiency of implementation of the program through minimal additional infrastructure costs.
Sensitivity to women's needs.


Outcomes hierarchy

Measures

Information sources

Analytical method

Evaluation questions

Indicators (for example)

Ultimate outcomes

Improvement to the quality of life of women who have undergone a mastectomy as a result of breast cancer, through a program of financial assistance for external breast prostheses.
  • In what ways does the program improve the quality of life of women who have purchased a prostheses as the result of a full of partial mastectomy due to breast cancer?
  • To what extent has the program been successful in achieving its objectives?
  • What improvements could be made to increase the program’s effectiveness, efficiency or appropriateness?
  • Women's satisfaction with the level of support and information offered
  • Satisfaction with levels of financial assistance
  • Extent to which the department is satisfied with the process for reimbursement
  • Women eligible for the program
  • Health care professionals
  • Breast cancer research/ networks/ support organisations
  • The department
  • Survey of women
  • Focus groups with women
  • Qualitative consultations with breast cancer research/ network/ support organisations, and health care professionals

Intermediate outcomes

  • Improved access to and uptake of, the reimbursement program by eligible women.
  • Demonstrated satisfaction of women with access to the reimbursement program.
  • Streamlined reimbursement process for eligible women.
  • Demonstrated sensitivity to the needs of the eligible women accessing the reimbursement.
  • Appropriate accountability structures in place to monitor and audit the program.
  • The program is embedded and integrated into Medicare Australia's processes and procedures with smooth operation through the use of appropriate systems.
  • Women have a choice of breast prostheses suppliers, such that the process of obtaining a prosthesis is consumer driven.
  • How sensitive is the program to the needs of women who have undergone a mastectomy?
  • How sensitive is the program to the needs of women who have undergone a mastectomy in regional/rural areas, of Indigenous or CALD background or who are otherwise disadvantaged?
  • Do women feel they have enough knowledge about all of the program components?
  • How well has the program been integrated into Medicare Australia's processes?
  • How can the program be monitored/ evaluated periodically?
  • How does the program function within the broader policy environment?
  • What choice of breast prostheses suppliers do women currently have?
  • What barriers are there for women in accessing the breast prostheses program?
Administrative
  • Administrative challenges currently faced by Medicare Australia and the department
  • Review of the service arrangement and business rules and analysis of current policy context
Impact
  • Reported levels of awareness and knowledge of the program among women – both generally, and those from regional/rural areas, or of CALD or Indigenous background, or who are otherwise disadvantaged
  • Reported experience and satisfaction with the program to date by women generally, and those from regional/rural areas, or of CALD or Indigenous background, or who are otherwise disadvantaged
  • Systems in place for information sharing and communication between women, breast cancer research/ networks/ support organisations and breast prostheses suppliers; with Medicare and the department
  • Systems in place for ongoing feedback and monitoring of the program
  • Extent to which women feel comfortable with current accessibility
  • Medicare Australia
  • The department
  • Eligible women
  • Health care professionals
  • Breast prostheses suppliers
  • State governments
  • Breast cancer research/ network/ support organisations
  • Survey of women who use breast prostheses as well as those who don't
  • Focus groups with women
  • Qualitative consultations with Medicare, the department, state governments, breast prostheses suppliers and breast cancer research/ network/ support organisations, and health care professionals

Immediate outcomes

  • The provision of reimbursement for external breast prostheses.
  • Increased awareness of the program by eligible women.
  • National consistency in the reimbursement program.
  • Ease of access to reimbursement for all eligible women.
  • Efficiency and appropriateness of the administration of the program.
  • Levels of knowledge among, promotion for, and support by, healthcare workers.
  • What is the uptake of the program in its first 12 months?
  • Do women report consistency in access, awareness and experience of the program nationally?
  • What barriers are there to the efficiency and appropriateness of administering the program?
  • How have claims been processed?
  • How are variations to eligibility assessed?
  • What have been the benefits or limitations of the national program?
  • Are there any gaps in the program?
Administrative
  • Medicare data – number of claims made, number of claims rejected (and reasons why), and demographic characteristics
  • Discussions with Medicare staff regarding systems and processes
Impact
  • Women's experience and satisfaction with the program to date
  • Women reporting number of referrals from clinicians or breast prostheses suppliers
  • Reports from breast cancer research/ network/ support organisations
  • Medicare Australia
  • The department
  • Eligible women
  • State governments
  • Breast cancer research/ network/ support organisations
  • Survey of women who use breast prostheses as well as those who don't
  • Focus groups with women
  • Qualitative consultations with Medicare, the department, state governments and breast cancer research/ network/ support organisations
  • Activities/outputs

  • Financial support provided to eligible women by the Commonwealth towards the costs of external breast prostheses.
  • Development of information sources and dissemination of these.
  • Development of protocols and policy regarding claims processing.
  • Implementation of service arrangement to provide one consistent national program.
  • Does current financial support provided to eligible women sufficiently cover costs of the prostheses?
  • What are women foregoing to pay for prostheses?
  • What are the most efficient aspects of the program (from an administrative viewpoint)?
  • How is program information disseminated? How effective is this dissemination?
  • How well do current protocols operate in providing guidance to eligible women?
Administrative
  • Medicare data – number of claims and demographic characteristics
  • Review of the service arrangement and business rules
Impact
  • Costs of prostheses compared to reimbursement
  • Extent to which women are enabled to make the choice to purchase prostheses with respect to the breast prostheses program
  • Financial challenges experienced by women who have undergone mastectomy
  • Reported information sources
  • Medicare Australia
  • The department
  • Eligible women who have and have not received the payment
  • Review of data
  • Qualitative interviews with Medicare staff, department staff and eligible women