Better health and ageing for all Australians

Evaluation of the NT MOS projects

A: Methodologies

Up to Closing the Gap: Northern Territory

prev pageprev page| TOC |next page

Formative evaluation
Summative evaluation
Cultural safety
Evaluation phases
Evaluation questions

Formative evaluation

Formative Evaluation is usually undertaken during the implementation of a program (such as the establishment and development of the MOS Projects services) to gain further insight and contribute to a learning process. The purpose is to support and improve the management, implementation and development of the program. The objectivity of findings is often not the main concern; more emphasis is put on the direct applicability of results. Operational questions, monitoring of events and early or unintended impacts are addressed.

Our methodology for formative evaluation in this evaluation involved qualitative approaches including site visits to remote areas of the Northern Territory and interviews with key stakeholders, as well as quantitative analysis of the program data as it emerged. Through site visits, the evaluation team engaged 'learning partners' who exchanged information, stories and data, and had an opportunity to express views, raise concerns and build their body of knowledge. Through an early analysis of this data, key issues and potential service barriers or gaps are being identified and fed back to those responsible for the program.

Formative evaluation relies on good relationships between the evaluators and the program providers at all levels. Without the formation of these relationships, an evaluation misses an important understanding of how the program really works. It allows the evaluator to observe, query and identify and to engage with service providers in a participatory action learning approach. The formative evaluator is an insider to the project.

Success Works' approach to formative evaluation is underpinned by Appreciative Inquiry (Cooperrider, 1987). This approach identifies what is working well and the strengths of a program, system or organisation and then engages practitioners in identifying how this good or best practice can be achieved more consistently across the entire initiative or organisation.

Summative evaluation

Summative evaluation determines what has been achieved by the initiative and the overall relevance of those achievements given the research findings. Data sources are similar to those for the formative evaluation (our site visits, interviews with key stakeholders, program data and document analysis), however the questions asked of the data are focussed on what difference the initiative has made and what has been learnt of relevance to other services, programs and policies.
Top of page

Cultural safety

Success Works is committed to cultural safety in our approach to consulting and to reflecting the voices and perspectives of Indigenous communities in our work. Our MOS Projects Evaluation team included experienced Indigenous consultants who lead the initial engagement and local consultation with Indigenous organisations and individuals.

Cultural safety is a worldview that recognises the need to overcome cultural biases in our services and institutions in order to ensure that all individuals and communities have equal rights and equitable access to processes and services which are relevant to them.

Cultural safety starts from the standpoint of Cultural Competence. Cultural competence is about recognising the cultural biases that all of us possess and taking steps to ensure that these biases are clearly acknowledged and that, to the extent possible, they do not get in the way of our practice. Cultural competence allows one to understand that the way in which we experience and understand the world is only one way in which the world can be experienced and understood.

Cultural safety in this evaluation required us to ensure the involvement of relevant cultural perspectives at all stages in the evaluation including:
  • evaluation approach: The action learning/formative evaluation approach used for this evaluation is considered to be best practice in Indigenous contexts 1 (eg Walker, Ballard and Taylor, 2003)

  • evaluation planning: Success Works evaluation team ensured an understanding of cultural imperatives in our planning for the evaluation and in our approach to site visits. This included seeking permission for access to land through the respective Land Councils, and gaining appropriate Ethics approvals. The project also involved the engagement of an OATSIH convened Evaluation Advisory Group which has Indigenous representation. Contact with communities was made in advance, where possible, via the NT MOS Plus staff we accompanied. Engagement and communication is a significant issue in the remote NT communities targeted by the MOS Plus program. We understand remote Aboriginal communities in NT are currently subject to a number of recently introduced outreach services and activities. Our approach was designed to be as least intrusive in communities as possible, by accompanying the MOS Plus service staff already known in community rather than adding to the confusion about 'who is who' in this 'service space' - which can be viewed as one 'welfare mob' (without distinction) by community members.

  • data collection: Our experienced Indigenous consultants acted as the leaders of the consultations at the local level and used 'yarning' or a narrative approach to data collection rather than 'asking questions' which is a westernised model. Yarning requires a high level of skill on the part of the consultant to hear the information needed to answer the evaluation questions within a storytelling approach.

Evaluation phases

Success Works applied a four stage methodology in this evaluation.

Phase 1 – Project establishment

  • Briefing for consultants on current status of MOS Projects

  • Discussion of the National Partnership Agreement on Health Services: Implementation Plan for NT MOS Plus and its impact on the MOS projects and the evaluation

  • Identification of major sources of data and other information

  • Agreement on expectations, boundaries and processes

  • Agreement on final methodology

  • Development of the Project Plan which included:

    • the approach and processes to be undertaken to address the evaluation requirements

    • timelines and deliverables

    • stakeholder analysis processes

    • communication strategy for the evaluation

    • overview of proposed engagement with MOS Projects' clients and stakeholders

    • risk management plan, including identification of back-up personnel
Top of page

Phase 2 – Development of the evaluation framework

  • Review of existing program documentation

  • Development of a literature review

  • Development of a project logic based on documentation available on the MOS Projects, and the literature review, in consultation with both NT Department of Health and Families (NT DHF) and the Department

  • An interactive framework analysis, in consultation with both the NT DHF and the Department

  • Development of the evaluation questions

  • Identification of data sources and management approaches

  • Application to the Northern and Central Land Councils, and Tiwi Islands Council

  • Applications to human research ethics committees:

    • Human Research Ethics Committee (HREC) of the NT Department of Health & Families and Menzies School of Health Research

    • Central Australian Human Research Ethics Committee

    • Remote Health Branch (NT Department of Health and Families)

Phase 3 – Implementation of the evaluation framework

  • Collect and analyse data, both quantitative and qualitative

  • Conduct interviews, focus groups and community site visits

  • Provide regular reports and advice

  • Meet with the Evaluation Advisory Group at key junctures

  • Analyse reports and data collected alongside the information generated from the literature review

  • Identify good practice, gaps and potential improvements

Phase 4 – Analysis and reporting

  • Analyse the data emerging on a regular basis to establish themes, trends and learning's emerging

  • Meet with the Department of Health and Ageing and the Evaluation Advisory Group to test and discuss the key findings

  • Make appropriate and well grounded recommendations in relation to the development of policy directions emerging from this analysis

  • Develop the draft Final Report, including executive summary, key findings and analysis of key findings and issues for future service delivery

  • Circulate the report for comment

  • Incorporate feedback and present the Final Report for submission 30 July 2011
Top of page

Evaluation questions

In relation to consideration of the elements of the MOS service, particularly in the context of service provision in remote communities, the evaluation requirement was to identify the extent to which the MOS Projects:
  • Have met their objectives and outcomes

  • Have reached their target population

  • Have provided an equitable and adequate level of service in relation to need in remote areas of the NT

  • Have attained the required geographic spread of service across remote NT

  • Non case-related services act as an intended pathway to case-related clinical service delivery

  • Case-related clinical services to Aboriginal children are effective, in line with good practice clinical processes, cultural safety and in the context of the legislative framework in the Northern Territory

  • Mobile Outreach Database (MOD) data system effectively records outcome measures for the MOS Projects' clients
The commensurate evaluation questions are - to what extent have:
  • Outreach visits been provided to remote communities?

  • Counselling sessions been provided to children and young people in remote communities?

  • Children, young people and families in remote communities received clinical support to reduce the trauma they are experiencing?

  • Referrals been made to other services?

  • Sexual assault forensic medical examinations been provided to children and young people in remote communities?

  • Access to remotely delivered forensic medical examinations reduced trauma experienced by children and families?

  • Community meetings been provided to families and community members?

  • Community education sessions been provided to families and community members?

  • Practice forums been undertaken for sharing learnings and clinical expertise?

  • Staff participated in external professional development activities?

  • MOD and other reports and documents been produced?

  • MOD and other reports measured outcomes for MOS Plus clients?
Top of pageThe commensurate assumptions to be tested are - is it correct that:
  • MOS Plus services effectively reduce the trauma associated with any form of child abuse and neglect?

  • MOS Plus has expanded in size and scope from MOS?

  • Family focused services are culturally appropriate and reduce trauma in children?

  • Children may experience more than one form of trauma related to abuse?

  • The provision of MOS Plus services to remote communities creates an equitable level of access across remote communities?

  • MOS Plus services have been provided in a timely way to children and young people in remote communities?

  • Referrals to other services are possible and appropriate?

  • Community meetings and community education sessions support and create access to counselling and other support services?

  • The need for MOS Plus services falls within the 0-17 year age group and all children within this age group are receiving service?
In relation to consideration of the MOS service staffing model, and community engagement, the evaluation requirement was to identify the:
  • Key elements of good practice in terms of service provision, clinical governance, staffing profiles, and continuous quality improvement

  • Effectiveness of non case-related services in raising awareness of the issues of child abuse and related trauma in remote communities in the Northern Territory

  • Effectiveness of engagement with, and services to, remote primary health care organisations

  • Extent to which the quality of the relationship of local people with the MOS Projects positively impacts on participation in the services offered and outcomes.
The commensurate evaluation questions are - to what extent:
  • Do children, young people, families and communities know the MOS Projects staff and understand their role?

  • Do families support the participation of children and young people in counselling sessions and forensic examinations?

  • Do families, community members and local agencies develop an understanding of child abuse and related trauma as a result of their participation in community meetings?

  • Do families, community members and local agencies develop skills in dealing with children experiencing trauma as a result of attending community education sessions?

  • Is there engagement with, and services to, remote primary health care organisations?

  • Have MOS Projects staff improved their knowledge of and expertise in the provision of culturally safe trauma related clinical counselling/support and forensic examinations in remote indigenous communities?
Top of pageThe commensurate assumptions to be tested are - is it correct that:
  • MOS Plus counselling services and non-case related services are culturally safe?

  • The staff team has the right mix of skills and expertise to provide all aspects of the MOS services?

  • MOS Plus have appropriate links with NT Closing the Gap initiatives and other outreach services in remote Indigenous communities?

  • Outreach/visiting services are the most culturally appropriate way to provide MOS Plus services to Indigenous children and their families in remote communities, outside the statutory child protection system?

  • A voluntary, therapeutic counselling approach is the most appropriate response for Indigenous children and their families experiencing trauma in remote communities?

  • Communities are wanting access to services outside the child abuse statutory system?

  • Improved access to counselling and forensic examinations in remote communities has a positive impact on the safety of those communities?

  • Local access to MOS Plus services have a positive impact on the safety of those communities?

Footnotes

1 Eg Walker, R, Ballard, J and Taylor C (2003) "Developing paradigms and discourses to establish more appropriate evaluation frameworks and indicators for housing programs" AHURI Final Report No. 29, Western Australia

prev pageprev page| TOC |next page