On 21st December 2009, the Office of Aboriginal and Torres Strait Islander Health (OATSIH), Department of Health and Ageing (DoHA) engaged HealthConsult (as prime contractor) working in partnership with Health Policy Analysis to:
- ‘review the current arrangements for the collection, recording, transfer and reporting of national trachoma data’
MethodologyThe project methodology brought together information from an extensive documentation and literature review, and stakeholder consultations to assess the national trachoma surveillance system against the Center for Disease Control (CDC) Guidelines for Evaluating Public Health Surveillance Systems. Key stakeholders from DoHA, the jurisdictions (Northern Territory, Western Australia, South Australia, New South Wales and Queensland) and the National Trachoma Surveillance and Reporting Unit (Melbourne) were consulted. The assessment resulted in the formulation of preliminary findings and ideas for enhancing the ongoing trachoma surveillance activities to meet the needs of the new measure. The findings and ideas formed part of a discussion paper which was used as the basis for a national refinement workshop with key stakeholders. The discussion paper provided feedback to those who had participated in consultations and contained views on the preliminary findings. Input from the workshop was used to refine the preliminary evaluation findings and inform the development of recommendations for enhancements to the current national trachoma surveillance system.Top of page
Overview of current trachoma surveillance arrangementsThe National Trachoma Surveillance and Reporting Unit (NTSRU) was established in November 2006 to improve the quality and consistency of data collection and reporting of active trachoma in Australia. Data are received annually from three jurisdictions in Australia where trachoma is endemic: the Northern Territory (NT), Western Australia (WA) and South Australia (SA). The trachoma surveillance arrangements in each of these jurisdictions were found to be different.
In the NT, trachoma screening is co-ordinated through the Centre for Disease Control, Department of Health and Families (DHF). The majority of the screening for active trachoma takes place as part of the Healthy School Aged Kids (HSAK) program whereas in some areas, like in Katherine West, the Aboriginal Community Controlled Health Service (ACCHS) have assumed responsibility for active trachoma screening. Some screening for active trachoma has also taken place as part of the Australian Government Emergency Initiative. Screening for trichiasis takes place on an ad-hoc basis in the context of Integrated Eye Service clinics conducted by optometrists and or ophthalmologists visiting communities. The data collection process in the NT is outlined in Figure ES.1.
*Only DHF health centres use PCIS the ACCHS use other computer systemsIn WA, trachoma screening is co-ordinated through regional populations health units (PHUs) located in four regions: the Kimberley, Pilbara, Midwest and Goldfields. The PHUs and local primary health care staff screen for active trachoma in a dedicated two week period throughout WA usually in the last week of August and first week of September. Adults are examined for trichiasis as part of the annual influenza vaccination program. The data collection process in the WA is outlined in Figure ES.2.
In SA, trachoma screening is conducted by the Aboriginal Health Council of South Australia (AHCSA) working with six Aboriginal Community Controlled Health Services (ACCHSs). SA currently does not have a widespread screening program. There is only one eye health project officer who coordinates and undertakes the active trachoma screening process with a team of ophthalmologists and optometrists who examine children when they visit selected communities. Trichiasis is not routinely screened for in SA. If ACCHS doctors identify trichiasis they will either be seen by the visiting ophthalmologist or referred for treatment. The data collection process in the SA is outlined in Figure ES.3.
The establishment of the NTSRU has assisted in the standardisation of the data collected about trachoma prevalence in Australia. Data are collected annually from each jurisdiction and are used to
produce an annual surveillance report. To date three reports have been released. Figure ES.4 illustrates the data flow processes that contribute to the national trachoma surveillance system.
Evaluation of national trachoma surveillance system against CDC guidelinesThe CDC Guidelines for Evaluating Public Health Surveillance Systems were used to evaluate the current national trachoma surveillance system. A summary of the evaluation findings against the 10 CDC guidelines evaluation criteria are:
- The level of usefulness: is assessed based on the extent to which the current surveillance system is the right system to support activities aimed at the prevention and control of trachoma and track their effectiveness. This issue was examined separately for the five main system components including:
- Active trachoma screening in children: The evaluation found that active trachoma screening in children is useful and should continue to focus on control and treatment in addition to measuring trachoma prevalence.
- Trachoma control activities: The current system is considered useful if it provides data for assessing the effectiveness of the SAFE (Surgery for trichiasis, Antibiotic therapy, Facial cleanliness; and Environmental improvements) trachoma control activities. The collection of data on environmental conditions, facial cleanliness and interventions is not well catered for in the current system. The evaluation found that the surveillance system data are useful but only partially effective in allowing an assessment of the impact of the SAFE trachoma control activities.
- Trichiasis screening and treatment: Trichiasis screening largely takes place on an ad-hoc basis (excluding WA). This raises issues regarding selection bias as measurement of trichiasis if often based on community outreach clinics which rely on people presenting to the services. There are also gaps in information on access to surgery for trichiasis. The evaluation found that the surveillance system data are useful but only partially effective in monitoring the prevalence of trichiasis.
- Antibiotic resistance: The two main issues identified are associated with the reliability of Indigenous identifiers in laboratory data particularly those outside the NT and whether the current data represents the target population. The evaluation found that the method for monitoring antibiotic resistance is adequate, however, as WA data on antibiotic resistance are largely not included the evaluators consider that the estimates of levels of antibiotic resistance do not reflect the population of interest.
- Community/patient satisfaction data: there is no routine reporting of community/patient satisfaction data. Anecdotally, stakeholders reported that the level of community satisfaction with trachoma screening and control activities is high.
- Simplicity: Considerable variation was found in the arrangements for trachoma screening and data collection across the participating jurisdictions. Furthermore, the use of paper forms and data entry occurring in multiple places adds to the complexity of reporting. The evaluation found that the process of collecting and reporting data at the local level for transfer to the NTSRU is too complex and needs to be streamlined.
- Flexibility: Surveillance data are currently collected at aggregate level; this limits the uses of the data. Flexibility could be improved through the collection of data at the person-level. The evaluation found that the flexibility of the current ‘system’ cannot easily be measured, however the current system is inflexible in terms of obtaining statistics on trachoma prevalence and control activities, other than a limited set of tables.
- Data quality: A number of data quality issues were identified including inadequate systems for checking data quality; variations in data collection skills amongst staff from local communities; delays in data entry and the re-handling of data; and the limited feedback processes particularly at local level. The evaluation found that the current processes for data collection do not support data quality, and do not lend themselves to easily measuring the current level of quality (except for lack of completeness).
- Acceptability: In the current system, acceptability was generally not an issue amongst stakeholders. There were no instances where stakeholders refused to participate in trachoma surveillance activities. The evaluation found that the current system is generally accepted by stakeholders in that all jurisdictions with communities with endemic trachoma participate in surveillance and control activities.
- Sensitivity: The diagnosis of trachoma is not likely to be missed (false negative) or diagnosed as something else (false negative) by practitioners with experience in rural and remote health services. The evaluation found that the current system is highly sensitive at case level (assuming screening staff have received training) i.e. accurately diagnosing children with active trachoma in screened communities; that it allows for the identification of trachoma at the local level in a reasonably timely fashion in screened communities; and that there is no basis for identifying trachoma prevalence in not-screened communities.
- Predictive value positive: It was considered unlikely that appropriately trained screening staff will identify trachoma in a child that does not have trachoma (false positives). The evaluation found that the current system has high PVP, as the risk of false positives is small. Given that the number of false positives in the national trachoma surveillance system is likely to be low (and hence PVP high) and the fact that calculating PVP empirically would be a major exercise, the evaluators believe there is little value in mounting an exercise to make a ‘gold standard’ measurement.
- Representativeness: is assessed by determining whether the current trachoma surveillance system is representative of the actual occurrence of trachoma in Australia. Representativeness was evaluated across the following three areas:
- Population subgroups: The evaluation found that the current system is not likely to be highly representative of the prevalence of trachoma in 1-9 year old children (the WHO recommended target group) neither is it highly representative of the prevalence of trichiasis in older Aboriginals.
- Calculation of rates: The evaluators found that there are currently considerable difficulties in determining appropriate denominators. Difficulties arise due to the historical identification of ‘at risk’ communities, issues associated with determining the number of children in ‘at risk’ communities and the high rate of school absenteeism in Indigenous communities.
- Measurements of risk factor behaviour: Reasonable data are collected on facial cleanliness. However very little information is collected on local environmental conditions and only free text information is collected on the implementation of health promotion and environmental activities. The evaluation found that the current information on environmental conditions is limited and does not facilitate the evaluation of the impact of interventions of trachoma prevalence.
- Timeliness: Timeliness was evaluated against two areas including:
- Timeliness of reporting to the national surveillance system: The evaluation found that there is considerable variation in the timeliness of reporting due to logistical and geographic issues associated with isolated communities as well as resource limitations.
- Timeliness of data for the use in program planning: Most jurisdictions commented that meeting the deadline for data submission was always an issue due to resource levels and the time taken from screening for trachoma through to producing the annual Trachoma Surveillance Report (approximately 12 months). The evaluation found that the time lag from data collection to publication of the annual report is too long for the data to be effectively used for program development and planning.
- Stability: The national trachoma surveillance system is not exposed to issues associated with larger more complex surveillance systems such as unscheduled outages or down times for the system's computer. The evaluation found that volume of data collected annually on trachoma surveillance activities is quite small and therefore easily managed and secured. The biggest risk to the stability of the current system is lack of dedicated resources, particularly in SA where the entire surveillance activity is reliant on one person.top of page
Improvement to the current trachoma surveillance strategyThe evaluators conclude that trachoma surveillance is essential to monitor and evaluate the impact of the Australian Government’s Improving Eye and Ear Health Services for Indigenous Australians for Better Education and Employment Outcomes measure. Although it is important to obtain reasonably precise measures of prevalence, a system designed to estimate prevalence and facilitate the control of trachoma is most appropriate in the Australian situation. The three jurisdictions that have implemented screening have adopted different strategies, but they all focus on trachoma control with surveillance data as a by-product. Standardisation of all aspects of the screening arrangements across jurisdictions would not be sensible. However, it is important to maintain good standards in terms of training of staff that perform screening, standardise the data collected, and standardise processes for assessing the adequacy of screening.
Following the review of current arrangements it is concluded that the current system is not likely to produce highly representative estimates of the prevalence of active trachoma in children. The coverage rates for screening are low and the resulting prevalence rates are unlikely to be fully representative of trachoma in the community. There are also issues with identifying the community coverage rate as it is sometimes difficult to determine the number of children residing in a community at the time of screening. Not all ‘at risk’ communities with endemic trachoma are screened annually and the recent National Indigenous Eye Health Survey has shown that there is trachoma in some communities in Queensland and New South Wales (NSW), jurisdictions in which there is no screening. These problems need to be tackled to ensure that the national trachoma surveillance system can be used to assess the impact of the trachoma surveillance and control component of the Improving Eye and Ear Health Services for Indigenous Australians for Better Education and Employment Outcomes measure announced by the Prime Minister in February 2009.
As the first step, all communities located in remote and very remote locations within NT, WA and SA should be included in the screening program at least once in a five year cycle using a rapid assessment method. Second, active trachoma screening activities should be extended, at least on a once off basis, to remote and very remote communities in NSW and Queensland also using the rapid assessment method. Third, jurisdictions should consider screening ‘at risk’ communities in other regions at least on a once off basis every five years using the rapid assessment method. A decision to continue ongoing annual screening should always depend on the prevalence of active trachoma within the community. In all circumstances, jurisdictions should be able to exercise the discretion to not screen communities in which there is good local evidence that active trachoma is no longer present.
To supplement this more comprehensive approach to trachoma screening the documented screening performance benchmarks must be adopted. First, the Communicable Diseases Network Australia (CDNA) Guidelines which require annual screening in trachoma endemic communities until prevalence is less than 5% for five consecutive years should be incorporated into funding and performance agreements between the Commonwealth and the jurisdictions relating to the allocation of additional funds under the new trachoma eradication initiative. Second, the benchmark coverage rate of 80% for ‘at risk’ communities as required by the National Framework for Delivery of Trachoma Control Programs developed by the Commonwealth and jurisdictions should also be adopted as part of any new funding agreement.
It is important to acknowledge that quality estimates of prevalence require both a correct diagnosis of trachoma in screened children and an accurate measurement of number of children in the target group residing in the community at the time of screening (the ‘denominator’). The evaluators believe that the denominator should be estimated locally at the time of screening, based on a discussion between screening staff and local school and/or health clinic staff, using a list of school enrolees or local resident children within the specified age ranges. Local staff or other informants will need to identify children who are not present in the community at the time of screening or additional children in the target age group who are visiting the community. To support this process, jurisdictions, in collaboration with the NTSRU, should develop guidelines to be followed by screening staff in estimating the denominator. Such guidelines will introduce a degree of consistency into the process and provide an opportunity, to emphasise to screening staff, the importance of obtaining a reliable estimate of the target population to the national trachoma surveillance program.
To round out the enhanced active trachoma surveillance strategy, the evaluators believe that there is value in commissioning a national survey using randomised cluster sampling (the ‘gold standard’ method) once in about a five year period. The nature of the rapid assessment process is that it will not yield the same breadth of information as a survey done using a randomised national sample. Given the costs involved the national survey should address both active trachoma and trichiasis surveillance.
Improvement to the current trichiasis surveillance strategyThe evaluators believe that there is value in continuing with the surveillance of trichiasis. The evaluation has shown that the current system is not likely to produce estimates that are highly representative of the prevalence of trichiasis in older Aboriginal people. It would be appropriate for the NT and SA to adopt a similar strategy to WA, in conducting trichiasis screening as part of the annual adult influenza vaccination program or potentially as an aspect of ‘adult health checks.’ Resources will need to be allocated to ensure training of staff undertaking screening, and to establish mechanisms through which these staff provide the results of screening to regional and/or jurisdictional trachoma units.
Improvement to the current antibiotic resistance surveillance strategyThe evaluators found that the current approach to monitoring antibiotic resistance as part of the national trachoma surveillance system is inadequate. The method used to monitor antibiotic resistance levels in the Aboriginal population in Australia is considered to be appropriate. However the coverage of the pathology service providers that contribute data is thought not to be the most appropriate. The current arrangements result in the Indigenous population of WA not being represented in the data. The evaluators recommended that NTSRU should pursue the collection of data from PathWest. Top of page
Improvement to the current surveillance data setData currently captured for active trachoma following school screening are summarised by staff undertaking screening and then provided to relevant trachoma surveillance units. There are some limitations with the summarised data provided, however, stakeholders considered that the data collected were sufficient to meet the key objectives for the screening program.
Data are also currently captured on provision of antibiotic treatment, but there are limitations with the current national forms. Form 1 does not allow the capture of data on children screened who received treatment outside the two week period. The evaluation found that the enhancement of National Form 1 to allow for reporting of the total number of children screened who received azithromycin should be pursued.Data are captured on health promotion and environmental interventions in communities where active trachoma has been identified on National Form 3. This form allows for a textual description of activities grouped by Surgery, Antibiotics, Facial Cleanliness, Environmental and Other. Reporting of this nature does not appear to be very helpful in understanding exactly what types of interventions have been implemented. There is the potential to improve reporting by further codifying responses. Similarly codes for different types of health promotion/education interventions could also be developed. These codes are best developed through an analysis of programs currently reported in textual form through National Form 3. Form 3 can then be modified so that data are collected in codified form.
The evaluators believe that an important aspect of evaluating trachoma control activities should be the tracking of information on housing and environmental conditions in communities. It is unreasonable to expect staff involved with screening to be able to collect this information themselves. Therefore, systems should be established for collating this information from other available sources. At the national level key housing and environmental indicators are available from the Community Housing and Infrastructure Needs Survey (CHIN). These data are updated every five years, and are considered to be comprehensive and authoritative. OATSIH should obtain approval to access the CHIN data for use in the national trachoma surveillance system and the NTSRU should incorporate the data into the Access database.
The data collection form for trichiasis screening (National Form 4) is adequate to capture relevant information on adults screened. The main issues for trichiasis screening is the establishment of systems for undertaking screening itself. Data on patients receiving surgery for trichiasis is also collected on Form 4. These data are very poorly collected. However, the main issue is the systems for collecting the data rather than the data to be collected.
Improvements to the data collection processThe process of collecting data at the local level for transfer to the national system is complex and can be streamlined, but the arrangements for achieving improvements depend on local circumstances within the jurisdictions. Systems established locally for the collection of screening and other data should facilitate the efficient management of the data and the accurate calculation of summary statistics. We believe the responsibility for improving existing arrangements is best managed by jurisdictional coordinators.
Mechanisms for collecting data on screening for trichiasis need to be improved, particularly in the NT and SA. If arrangements are to be established for screening along with the provision of vaccinations and/or a healthy adult check, then mechanisms need to be developed to capture information on the results of screening. Data collection may be best done by ensuring the information is recorded in the clinical information systems for local primary health care services and arranging for summary data extracts to be obtained from these systems.
A major weakness in current arrangements is the capture of information on surgery for trichiasis. We suggest this issue be addressed through two mechanisms. First local trachoma coordinators should attempt to make arrangements to be able to collect relevant data directly from ophthalmologists providing this surgery for Aboriginal people in the relevant regions. Secondly, at a national level, a specific MBS code could be developed for trichiasis surgery. Combined with the MBS Indigenous identifier, it may be possible to use MBS data to gain a clearer appreciation of the level of access to surgery. This approach could be supplemented by analysis of the national hospital morbidity data, through which data on trichiasis surgery provided on an admitted patient basis can be identified.
The flow of information from the jurisdictional level to the national level occurs once a year with the provision of the Access database. The flow of information could be facilitated through the establishment of a web-based system for data entry. Aside from allowing instant access of the data by the NTSRU, the system needs to provide for the immediate availability of reports that can be generated by local coordinators. The availability of such reports will not only increase the quality of the surveillance and treatment data but also allow greater use of the information at local level to better direct trachoma control activities.Top of page
Improvements to the data analysis and reportingThe national Trachoma Surveillance Report is produced annually, usually around six to nine months after the end of the calendar year to which the report relates. A number of stakeholders indicated that the lag in the release of the report was a problem. One factor impacting the lag is that data provided by jurisdictional and regional co-ordinators are often not finalised until around three months after the end of the calendar year. Establishing a web based reporting arrangement should allow data to be updated progressively through the year, and any data quality issues actioned immediately. The NTSRU could examine opportunities for reducing the time between the receipt of final data and the production of the report.
One issue that requires resolution is the fact that jurisdictional and regional coordinators have not been able to generate reports from the Access database using the data they have entered locally. This issue could easily be addressed within the existing database, or through a new web-based system, by the development of relevant queries and pivot tables that would provide data in a flexible form for users, no matter the level of their computer skills. The reports could provide full detail of local data and appropriate (de-identified) data for benchmarking and performance improvement purposes derived from analyses of the full national data set.
Improvements in trainingThere are two aspects of training that relate to the national trachoma surveillance system; the training to undertake the screening, and the training to collect and enter good quality data. In relation to training of staff collecting the data, the general consensus from stakeholders was that staff in trachoma surveillance and control roles were generally appropriately trained. Some problems with training were encountered with staff from the local communities, where the interest levels and skills varied. It is important to engage local health services workers in screening for trachoma. Effective systems need to be designed and implemented to engage and train local health service workers.
The quality of the data is influenced by many factors including the quality of training and supervision of the person who completes the surveillance forms, and the care exercised in data management. Computer-based forms with cross-checking of information yield more accurate results than paper-based systems with no controls. Data collection and entry are currently paper based, and there are inadequate systems for checking data quality. Computer-based data entry, if well designed, will contribute to resolving this problem.
Implementation and risk managementThe recommendations have been grouped into four categories short term enhancement and expansion (work to commence within six months of this report) and longer term enhancement and expansion (work to commence between six months and two years from delivery of this report). Many of the recommendations have been allocated to the short term enhancement and expansion categories so that the additional and higher quality data that will flow from their early implementation will be available to allow a timely assessment of the impact of the trachoma surveillance and control component of the Australian Government’s Improving Eye and Ear Health Services for Indigenous Australians for Better Education and Employment Outcomes measure. Naturally, it will be important that DoHA and the jurisdictions allocate sufficient resources to allow timely and effective implementation of the recommendations.
The review team believes there are few risks associated with implementing the recommended enhancements to the national trachoma surveillance system so long as the appropriate quantum of resources is allocated. In making the recommendations, the evaluators have considered the need for a balance between resources allocated to trachoma control and treatment, and resources allocated to surveillance. It is considered that the enhancement and expansion program represents an appropriate balance particularly as the total resources being allocated to trachoma surveillance and control by the Australian Government have been increased from $1.12m over three years to $16m over four years (a factor of 10). In such circumstances, the national trachoma surveillance system must be enhanced to deliver the data required to assess the impact of the funding increase. We are confident that implementation of the recommendations of this report will produce the required surveillance data.