Responsibilities for testing have transitioned from NEHTA to DHS. NEHTA still retain “light touch” assurance over releases. The level of quality assurance they perform is based on the assessed risk of the release. NEHTA defines the requirements for sign off to DHS. DHS manages all the processes relating to testing. There are some areas of tension between DHS and NEHTA evident in relation to testing responsibilities and outcomes that appear to be generated by inconsistent expectations across the organisations. The testing processes that could be enhanced include:
- Closer alignment between the DHS testing strategy and test plans and the Clinical Safety Plan developed by NEHTA to confirm that the clinical safety requirements are being met through testing and to assess the impact of any workarounds that may be put in place
- Review of testing timeframes to ensure that NEHTA have adequate time to validate that the system is fit for purpose. DHS provide NEHTA with the release plan, test plan, test cases and outcomes, defects list and defect reports to support their assurance role. Both NEHTA and DHS are required to sign off testing outcomes as a pre-requisite for go live. However NEHTA reported that due to the timing of receipt of the documentation they often have very limited time for assurance activities prior to go/no go meetings for deployment.
Since the commencement of the current CIO at DHS, substantial efforts have been made to resolve all defects and good progress is reported in this area.
A number of limitations around testing of the HI Service in relation to its interaction with the PCEHR system were identified. Under the HI Act IHIs can’t be disclosed for testing or training purposes. This limits the level of testing that can be done increasing the risk of issues occurring in a live environment which could introduce clinical safety risks. There is currently no capacity to undertake production verification testing at a site level, which is necessary given the variability between instances of provider software. This testing, which confirms the configuration of an individual provider’s instance of a piece of software is able to connect to the HI Service, cannot be performed as it would involve retrieving an IHI for the purpose of testing, not for healthcare. DHS do conduct production verification testing for each release to test that services are available.
Although vendor products are subjected to Notice of Connection (NOC) and Compliance, Conformance and Accreditation (CCA) testing, production verification testing is important as there are differences in the way that every organisation implements software that cannot be picked up by the vendor performing NOC and CCA. There are often issues with local configuration or certificates which mean the connection to the HI Service does not work when implemented in production, and this is only discovered when a patient presents.
It is important to healthcare organisations to have certainty that the integration will work on their environment before they start using it in a clinical context. However, there is little understanding among providers of their legal liability if they attempt to perform production verification testing. This could be addressed if individual organisations were able to perform this testing in the test environment, but this would significantly increase the demand on the test infrastructure. Jurisdictions also want to be able to perform User Acceptance Testing prior to implementation, but at this point all User Acceptance Testing is performed by DHS, as requested and contracted by NEHTA. There are similar issues with the availability of environments and data for healthcare organisations to use for training and change management purposes.
Recommendation 17 – Healthcare Identifiers test strategy and environments
It is recommended that a test environment strategy be developed and testing mechanisms and environments be implemented to enable end to end testing of the HI Service and its interactions with other e-Health systems. This should include User Acceptance Testing and Production Verification Testing as well as environments to support training and change management activities.