Home > For Consumers > Aboriginal and Torres Strait Islander Health > Closing the Gap: Northern Territory

Evaluation of the NT MOS projects

Evaluation data collection

Up to Closing the Gap: Northern Territory

prev page| TOC |next page

Evaluation questions and data sources
Evaluation tools
Evaluation protocols: cultural safety

Evaluation questions and data sources

To what extent have outreach visits been provided to remote communities?

• Quantitative data source: MOD (visit summary data)
• Qualitative data source: MOS Plus staff

To what extent have counselling sessions been provided to children and young people in remote communities?

• Quantitative data source: MOD (case related service events, counselling data)
• Qualitative data source: MOS Plus staff

To what extent have sexual assault forensic medical examinations been provided to children and young people in remote communities?

• Quantitative data source: MOD (case related service events, FME data)
• Qualitative data source: none

To what extent have referrals been made to other services?

• Quantitative data source: MOD (case related service events, referrals data)
• Qualitative data source: local community organisations, outreach services, MOS Plus staff, government departments

To what extent have community meetings been provided to families and community members?

• Quantitative data source: MOD (case related service events, community meetings data)
• Qualitative data source: Elders and/or community leaders, local community organisations, MOS Plus staff

Top of pageTo what extent have community education sessions been provided to families and community members?

• Quantitative data source: MOD (case related service events, community education data)
• Qualitative data source: Elders and/or community leaders, local community organisations, MOS Plus staff

To what extent have practice forums been undertaken for sharing learnings and clinical expertise?

• Quantitative data source: MOD (non-case related service events, practice forums data)
• Qualitative data source: MOS Plus staff

To what extent have staff participated in external professional development activities?

• Quantitative data source: MOD (non-case related service events, professional development data)
• Qualitative data source: MOS Plus staff

To what extent have MOD and other reports and documents been produced?

• Quantitative data source: none
• Qualitative data source: MOS Plus staff, government departments

To what extent do MOD and other reports measure outcomes for MOS Plus clients?

• Quantitative data source: MOD (case related service data at case closure)
• Qualitative data source: none

To what extent do children, young people, families and communities know the MOS Projects staff and understand their role?

• Quantitative data source: none
• Qualitative data source: Clients, family, elders and/or community leaders, local community organisations, MOS Plus staff

To what extent have children and young people in remote communities received clinical support to reduce the trauma they are experiencing?

• Quantitative data source: MOD (case related service events, community clinical support data)
• Qualitative data source: Clients, family, outreach services, MOS Plus staff, government departments

To what extent has access to remotely delivered forensic medical examinations reduced trauma experienced by children and families?

• Quantitative data source: MOD (case related service, outcomes data)
• Qualitative data source: Clients, family, elders and/or community leaders, local community organisations, MOS Plus staff, government departments

To what extent do families support the participation of children and young people in counselling sessions and forensic examinations?

• Quantitative data source: MOD (case related service, counselling data)
• Qualitative data source: Clients, family, elders and/or community leaders, MOS Plus staff

To what extent is there engagement with and services to remote primary health agencies?

• Quantitative data source: none
• Qualitative data source: Elders and/or community leaders, local community organisations, outreach services, MOS Plus staff, government departments

Top of pageTo what extent do families, community members and local agencies develop an understanding of child abuse and related trauma as a result of their participation in community meetings?

• Quantitative data source: none
• Qualitative data source: Family, elders and/or community leaders, local community organisations, MOS Plus staff

To what extent do families, community members and local agencies develop skills in dealing with children experiencing trauma as a result of attending community education sessions?

• Quantitative data source: none
• Qualitative data source: Family, elders and/or community leaders, local community organisations, MOS Plus staff, government departments

To what extent have MOS Projects staff improved their knowledge of and expertise in the provision of culturally safe trauma related clinical counselling/support and forensic examinations in remote indigenous communities?

• Quantitative data source: none
• Qualitative data source: Elders and/or community leaders, local community organisations, MOS Plus staff

Is it correct that MOS Projects services effectively reduce the trauma associated with any form of child abuse and neglect?

• Quantitative data source: MOD (case related service events, outcomes data)
• Qualitative data source: Clients, family, elders and/or community leaders, local community organisations, outreach services, MOS Plus staff

Is it correct that MOS Plus has expanded in size and scope from MOS?

• Quantitative data source: MOD (case and non-case related service events)
• Qualitative data source: Clients, family, elders and/or community leaders, local community organisations, outreach services, MOS Plus staff, government departments

Is it correct that family focused services are culturally appropriate and reduce trauma in children?

• Quantitative data source: none
• Qualitative data source: Clients, family, elders and/or community leaders, local community organisations, outreach services, MOS Plus staff

Is it correct that children may experience more than one form of trauma related to abuse?

• Quantitative data source: MOD (case related service events, expanded trauma types data)
• Qualitative data source: Family, elders and/or community leaders, local community organisations, outreach services, MOS Plus staff

Is it correct that the MOS Projects counselling services and non-case-related services are culturally safe?

• Quantitative data source: none
• Qualitative data source: Elders and/or community leaders, local community organisations, MOS Plus staff

Is it correct that the provision of MOS Projects services to remote communities creates an equitable level of access to support across remote communities?

• Quantitative data source: MOD (case and non-case related service events, locations data)
• Qualitative data source: MOS Plus staff, government departments

Top of pageIs it correct that MOS Projects services are provided in a timely way to children and young people in remote communities?

• Quantitative data source: MOD (case related service events, date of referral vs date of visit)
• Qualitative data source: Clients, family, elders and/or community leaders, local community organisations, MOS Plus staff, government departments

Is it correct that referrals to other services are possible and appropriate?

• Quantitative data source: MOD (case related service events, referrals data)
• Qualitative data source: Local community organisations, outreach services, MOS Plus staff, government departments

Is it correct that community meetings and community education sessions support and create access to counselling and other support services?

• Quantitative data source: none
• Qualitative data source: Elders and/or community leaders, local community organisations, outreach services, MOS Plus staff

Is it correct that the staff team has the right mix of skills and expertise to provide all aspects of the MOS Projects?

• Quantitative data source: none
• Qualitative data source: Elders and/or community leaders, local community organisations, outreach services, MOS Plus staff, government departments

Is it correct that MOS Projects have appropriate links with NT Closing the Gap initiatives and other outreach services in remote Indigenous communities?

• Quantitative data source: none
• Qualitative data source: Local community organisations, outreach services, MOS Plus staff, government departments

Is it correct that outreach/visiting services are the most culturally appropriate way to provide MOS Projects services to Indigenous children and their families in remote communities, outside the statutory child protection system?

• Quantitative data source: none
• Qualitative data source: Elders and/or community leaders, local community organisations, outreach services, MOS Plus staff, government departments

Is it correct that a voluntary, therapeutic counselling approach is the most appropriate response for Indigenous children and their families experiencing trauma in remote communities?

• Quantitative data source: none
• Qualitative data source: Elders and/or community leaders, local community organisations, outreach services, MOS Plus staff, government departments

Is it correct communities are wanting access to services outside the child abuse statutory system?

• Quantitative data source: none
• Qualitative data source: Family, elders and/or community leaders, local community organisations, outreach services, MOS Plus staff

Is it correct that the need for MOS Projects services falls within the 0-17 year age group, and all children within this age group are receiving service?

• Quantitative data source: MOD (case related service events, age of clients)
• Qualitative data source: Elders and/or community leaders, local community organisations, outreach services, MOS Plus staff, government departments

Top of pageIs it correct that improved access to counselling and forensic examinations in remote communities has a positive impact on the safety of those communities?

• Quantitative data source: none
• Qualitative data source: Elders and/or community leaders, local community organisations, outreach services, MOS Plus staff, government departments

It is correct that local access to MOS Projects services has a positive impact on the safety of those communities?

• Quantitative data source: MOD (case related service events, outcomes data)
• Qualitative data source: Family, elders and/or community leaders, local community organisations, outreach services, MOS Plus staff, government departments

To incorporate the expanded scope of services in MOS Plus (from MOS), additional variables have been added to the existing Mobile Outreach Database (MOD) data system. An initial assessment of MOD indicates it meets our evaluation output data needs, with the exception of referrals. This information will be obtained through qualitative data collection as indicated. The formative nature of the evaluation enables ongoing review of data sources, and incorporation of consultation with additional stakeholders and interrogation of alternate data sources can occur as appropriate.

Evaluation tools

Some evaluation questions will also require analysis via alternate evaluation and data collection and analysis tools:

• To what extent do families, community members and local agencies develop skills in dealing with children experiencing trauma as a result of attending community education sessions?
  
  
• To what extent have MOS Plus staff improved their knowledge of and expertise in the provision of culturally safe trauma related clinical counselling/support and forensic examinations in remote indigenous communities?
  
  
• Is it correct that the MOS Plus counselling services, forensic examinations and community meeting and education services are culturally safe?

The evaluation data may comprise a qualitative review of documents, such as those relating to the scope and content of community meetings and community education sessions. The data pertaining to cultural safety will also be analysed within a cultural safety evaluation framework.

Evaluation protocols: cultural safety

Success Works is committed to ensuring cultural safety in our evaluation projects. Cultural Safety means that we, as evaluation consultants, understand and respect the different cultural understandings and heritage of the people we are working with and ensure that there is 'no assault, challenge or denial of cultural identity'¹ in our work. Cultural safety requires us to overcome our own cultural biases and ensure, to the extent possible, they do not get in the way of our practice and that the voices and experiences of the people who experience the program being evaluated can directly inform the findings and recommendations.

Cultural safety in evaluation requires us to ensure the involvement of relevant cultural perspectives at all stages in the evaluation including:

• The evaluation approach. The action learning/formative evaluation approach being used for this evaluation is considered to be best practice in Indigenous contexts² (eg Walker, Ballard and Taylor, 2003)
  
  
• The evaluation planning. We need an understanding of cultural imperatives in our planning for the evaluation and in our approach to site visits. This includes seeking permission for access to land through the appropriate source and checking immediately prior to departure that the planned visit is still suitable. Contact with communities will be made in advance, where possible, via the NT MOS Plus staff we will be accompanying. If they consider it necessary, we may also make direct telephone contact with communities through one of our Indigenous team members.
  
  
• Data collection. Our experienced Indigenous consultants act as the leaders of the consultations at the local level and use 'yarning' or a narrative approach to data collection rather than 'asking questions' which is a westernised model. Yarning requires a high level of skill on the part of the consultant to hear the information needed to answer the evaluation questions within a storytelling approach.
  
  
• Data analysis. Indigenous team members are involved in the data analysis phase and we will also consult with other Indigenous informants as we analyse the data to ensure that we have understood the data correctly and that the data is interpreted through an appropriate cultural lens
  
  
• Data reporting. Evaluators should be accountable to the people who have provided the information to inform the evaluation. Our findings should be shared with them and the opportunity provided for community members to offer their own perspectives. In culturally safe evaluation we recognise and accept that our accountability is not only to the client who has commissioned the evaluation but also to the people who inform the evaluation and whose lives will be affected by the evaluation findings.

Footnotes

¹ Robyn Williams (nd) Cultural Safety: What does it mean for our work practice? Available on the University of Tasmania, University department of rural health webpage.
² Eg Walker, r, Ballard, J and Taylor c (2003) "Developing paradigms and discourses to establish more appropriate evaluation frameworks and indicators for housing programs" AHURI Final Report No. 29, Western Australia

Top of page

prev page| TOC |next page